Joe Caban 844086 Posted November 2, 2007 at 12:31 AM Posted November 2, 2007 at 12:31 AM Look for a few important announcements for ZTL staff and controllers to be posted here very soon. Yes I know you guys are going to say this is stupid so save me the jokes and taunts. Joe Caban vZTL ATM The Future of VATUSA Regards, JX Link to comment Share on other sites More sharing options...
Joe Caban 844086 Posted November 2, 2007 at 12:46 AM Author Posted November 2, 2007 at 12:46 AM Okay item #1: As you guys know the ZTL website is temporarily down. We suspect malicious activities, but have had a backup server for two months now. We are bringing that one online as we speak. In the mean time, I need all staff members to email me from a non ztl email account. With the server down we cannot reach eachother via [email protected] I can be reached at jcaban//at//atpexperience.net or jcaban//at//scandalousmusic.com There will be more announcements and notices as soon as these become available sent to your NON ZTL email. Expect the site to be up again within 72 hours or less. If you have any immediate concerns contact me, if not just wait for the site to be back up. Our Teamspeak server may go down as well if not already. If this happens it will be announced here. Thanks for your patience. Joe Caban vZTL ATM The Future of VATUSA Regards, JX Link to comment Share on other sites More sharing options...
Joe Caban 844086 Posted November 2, 2007 at 12:57 AM Author Posted November 2, 2007 at 12:57 AM ITEM #2 To TRAINEES! If you were looking for training, mentoring, taking exams, and things of that nature we will need you to do it manually for a few days. So, here's what we need: If you had an appointment scheduled you should be able to continue as planned. If you had a test [Mod - Happy Thoughts]igned you will need to contact Mike Stagg our TA for further [Mod - Happy Thoughts]istance. If you are looking to set up a Mentor/Instructor session in the near future please email the Instructor/Mentor directly. If you do not know how to get in contact with one either email Mike Stagg or myself. Once again my email is jcaban//at//atpexperience.net Mike's email is mdstagg//at//verizon.net Joe Caban vZTL ATM The Future of VATUSA Regards, JX Link to comment Share on other sites More sharing options...
Kyle Ramsey 810181 Posted November 2, 2007 at 01:56 AM Posted November 2, 2007 at 01:56 AM Thank you for the leadership at a difficult time, Joe. Kyle Ramsey Link to comment Share on other sites More sharing options...
Luke Kolin Posted November 2, 2007 at 02:33 AM Posted November 2, 2007 at 02:33 AM Look for a few important announcements for ZTL staff and controllers to be posted here very soon. Yes I know you guys are going to say this is stupid so save me the jokes and taunts. Actually, I'm going to echo Kyle and say that you're doing the right thing; taking advantage of a communications media to keep people informed. Good job. I don't know the specifics of the situation, but if this post is any indication, if there's a taunt to be made it's that it might have been done sooner. Let me tell a story - around mid-summer our server had managed a sufficiently high uptime that our CEO started making note of this in public correspondence, and we were advertising it on our home page (we still do; right now we're at around 101 days). The old adage about pride going before a fall came true, and we had a hardware failure. I'm not 100% certain what the issue was (our provider never told us) but based on my postmortem of the box I think the RAM failed. It was both fortunate and unfortunate that the failure occurred around 2 AM ET - not too much traffic at the time, but it didn't get noticed until around 930 AM when I got into work; tried to hit the box, direct SSH and SSH/serial console. Nothing worked, so we called the hosting company and got into their black hole of a support queue. At around 10AM we decided that the problem wouldn't be immediately resolved, so we initiated our disaster recovery procedures. We started downloading our off-site database backups to our backup server, and by lunchtime we had it up and running and were able to send out broadcast messages to our community informing them of the situation. We disabled ACARS and flight logging, but kept the forum open in read-only mode and allowed administrators to post and provide updates. Eventually (around 4PM) our server was restarted, we fscked the drives and restored the DNS and we were back on our way. I mention this because to paraphrase Ron Howard, "Failure is not an option - it is an inevitability" Stuff is going to fail. Servers go down. Software blows up. Network connections are lost. However, when our systems fail we do not need to fail - we can demonstrate a proactive attitude towards communication and recovery that will provide positive feedback from our users. I mention this not as a shot against you or anyone else. We've had two major hardware failures in the last few years. We have used them as a great learning experience by doing the post-mortem in a neutral, analytical fashion, much like an air disaster investigation. And what we've learned from these experiences is that the best two things that you can do are a) proactive communication - even if you have no news, say so; and b) knowing when to cut your losses and switch to the DR site. Those of you reading this thread have either had a major failure, or you are going to. Hopefully my experience and Joe's can give you some tips on what to do. Cheers! Luke[/url] ... I spawn hundreds of children a day. They are daemons because they are easier to kill. The first four remain stubbornly alive despite my (and their) best efforts. ... Normal in my household makes you a member of a visible minority. Link to comment Share on other sites More sharing options...
Joe Caban 844086 Posted November 2, 2007 at 03:10 AM Author Posted November 2, 2007 at 03:10 AM Thanks guys. -------------------------------------------- -------------------------------------------- Can we get a sticky until Monday??? -------------------------------------------- -------------------------------------------- Ok newsflash .. . -- . --- ..... .. -.- .-.. Our site files were deleted. We'll never know why or how. It may or may not have been inadvertent. It may or may not have been a technical problem. Fortunately we have backups and what not. Restoration is in progress. Joe Caban vZTL ATM The Future of VATUSA Regards, JX Link to comment Share on other sites More sharing options...
Elliot Lezam 895373 Posted November 2, 2007 at 05:13 AM Posted November 2, 2007 at 05:13 AM Look for a few important announcements for ZTL staff and controllers to be posted here very soon. Yes I know you guys are going to say this is stupid so save me the jokes and taunts. Actually, I'm going to echo Kyle and say that you're doing the right thing; taking advantage of a communications media to keep people informed. Good job. I don't know the specifics of the situation, but if this post is any indication, if there's a taunt to be made it's that it might have been done sooner. Let me tell a story - around mid-summer our server had managed a sufficiently high uptime that our CEO started making note of this in public correspondence, and we were advertising it on our home page (we still do; right now we're at around 101 days). The old adage about pride going before a fall came true, and we had a hardware failure. I'm not 100% certain what the issue was (our provider never told us) but based on my postmortem of the box I think the RAM failed. It was both fortunate and unfortunate that the failure occurred around 2 AM ET - not too much traffic at the time, but it didn't get noticed until around 930 AM when I got into work; tried to hit the box, direct SSH and SSH/serial console. Nothing worked, so we called the hosting company and got into their black hole of a support queue. At around 10AM we decided that the problem wouldn't be immediately resolved, so we initiated our disaster recovery procedures. We started downloading our off-site database backups to our backup server, and by lunchtime we had it up and running and were able to send out broadcast messages to our community informing them of the situation. We disabled ACARS and flight logging, but kept the forum open in read-only mode and allowed administrators to post and provide updates. Eventually (around 4PM) our server was restarted, we fscked the drives and restored the DNS and we were back on our way. I mention this because to paraphrase Ron Howard, "Failure is not an option - it is an inevitability" Stuff is going to fail. Servers go down. Software blows up. Network connections are lost. However, when our systems fail we do not need to fail - we can demonstrate a proactive attitude towards communication and recovery that will provide positive feedback from our users. I mention this not as a shot against you or anyone else. We've had two major hardware failures in the last few years. We have used them as a great learning experience by doing the post-mortem in a neutral, analytical fashion, much like an air disaster investigation. And what we've learned from these experiences is that the best two things that you can do are a) proactive communication - even if you have no news, say so; and b) knowing when to cut your losses and switch to the DR site. Those of you reading this thread have either had a major failure, or you are going to. Hopefully my experience and Joe's can give you some tips on what to do. Cheers! Luke[/url] Shows how dedicated you guys are. Do you guys run your own servers, because most hosts backup their data periodically to another location. Link to comment Share on other sites More sharing options...
Keith Smith Posted November 2, 2007 at 05:23 AM Posted November 2, 2007 at 05:23 AM Joe, I give you credit for using the VATUSA forum as a temporary place to stay in control at the helm and restore some order, that's good thinking. I can't imagine this is an easy time. Good luck with the restoration. Keith Link to comment Share on other sites More sharing options...
Luke Kolin Posted November 2, 2007 at 03:59 PM Posted November 2, 2007 at 03:59 PM Do you guys run your own servers, because most hosts backup their data periodically to another location. Yes. We pay our hosting company to give us a machine with a bare bones Linux install. We pay them even more to give us the root p[Mod - Happy Thoughts]word and go away. Generally, we operate under the [Mod - Happy Thoughts]umption that all of the sysadmins at our host are either incompetent twits, or about to be replaced by one. I don't really believe this is true, but it never hurts to be pessimistic and make [Mod - Happy Thoughts]umptions based on the worst case. We have a number of cron jobs that automatically dump our database, Subversion repository and application configurations. Our web host gives us 100GB of FTP-accessible storage, and we dump the backups there, as well as to a pair of offsite backups. We have to [Mod - Happy Thoughts]ume that the entire web host will go down or is inaccessible, hence the off site backups. We're responsible for our own DNS, and that too is handled in multiple locations. If there's anything else that I could add, it's to validate your restore procedures even more than your backups. When we did practice restores, we found a few missing items. Even after correcting them, we still had procedural issues when we had to do it for real. Cheers! Luke ... I spawn hundreds of children a day. They are daemons because they are easier to kill. The first four remain stubbornly alive despite my (and their) best efforts. ... Normal in my household makes you a member of a visible minority. Link to comment Share on other sites More sharing options...
Joe Caban 844086 Posted November 2, 2007 at 04:31 PM Author Posted November 2, 2007 at 04:31 PM Good news is our ztlartcc.net site is back up! Bad news is we will be taking it back down next week. We need to switch it to a new server and will be releasing a new website as well. Be on the lookout for that. Joe Caban ZTL ATM The future of VATUSA Regards, JX Link to comment Share on other sites More sharing options...
Joe Caban 844086 Posted November 2, 2007 at 05:35 PM Author Posted November 2, 2007 at 05:35 PM #3 If you received an email giving you log in information to the new server please log in and make sure everything works for you. Also keep the URL confidential for now. I still don't have ztl email capabilities for anyone who is trying to reach me. Also, I would like to thank everyone who helped me out over the past day and a half: Jared Addis JC Rodriguez Brian Sperduto and VATUSA staff Brandon Bartell Mike Stagg Ariel Maisonet Benton Wilmes & Byron Macrae Thanks guys! Joe Regards, JX Link to comment Share on other sites More sharing options...
Brandon Bartell 968788 Posted November 3, 2007 at 09:48 PM Posted November 3, 2007 at 09:48 PM No Problem Joe. I am glad Brian was able to get the current site backup. But soon he will have to take it down again. IRONY. Brandon Bartell Controller Atlanta ARTCC Link to comment Share on other sites More sharing options...
David Reimer 913748 Posted November 5, 2007 at 08:01 PM Posted November 5, 2007 at 08:01 PM (edited) Nvr Mind. Edited November 6, 2007 at 01:39 AM by Guest Link to comment Share on other sites More sharing options...
Joe Caban 844086 Posted November 5, 2007 at 11:22 PM Author Posted November 5, 2007 at 11:22 PM still can't get emails, but you can de sticky i think! No ztlartcc.net emails. We are working on it. Regards, JX Link to comment Share on other sites More sharing options...
David Reimer 913748 Posted November 6, 2007 at 01:40 AM Posted November 6, 2007 at 01:40 AM still can't get emails, but you can de sticky i think! No ztlartcc.net emails. We are working on it. Never mind the de-sticky then Link to comment Share on other sites More sharing options...
Recommended Posts