Jump to content

You're browsing the 2004-2023 VATSIM Forums archive. All content is preserved in a read-only fashion.
For the latest forum posts, please visit https://forum.vatsim.net.

Need to find something? Use the Google search below.

To vZTL Staff and Controllers:


Joe Caban 844086
 Share

Recommended Posts

Joe Caban 844086
Posted
Posted

Look for a few important announcements for ZTL staff and controllers to be posted here very soon. Yes I know you guys are going to say this is stupid so save me the jokes and taunts.

 

Joe Caban

vZTL ATM

The Future of VATUSA

Regards,

JX

Link to comment
Share on other sites

Joe Caban 844086
Posted
Posted

Okay item #1:

 

As you guys know the ZTL website is temporarily down. We suspect malicious activities, but have had a backup server for two months now. We are bringing that one online as we speak.

 

In the mean time, I need all staff members to email me from a non ztl email account. With the server down we cannot reach eachother via [email protected]

 

I can be reached at jcaban//at//atpexperience.net or jcaban//at//scandalousmusic.com

 

There will be more announcements and notices as soon as these become available sent to your NON ZTL email.

 

Expect the site to be up again within 72 hours or less.

 

If you have any immediate concerns contact me, if not just wait for the site to be back up.

 

Our Teamspeak server may go down as well if not already. If this happens it will be announced here.

 

Thanks for your patience.

 

Joe Caban

vZTL ATM

The Future of VATUSA

Regards,

JX

Link to comment
Share on other sites

Joe Caban 844086
Posted
Posted

ITEM #2

 

To TRAINEES!

 

 

If you were looking for training, mentoring, taking exams, and things of that nature we will need you to do it manually for a few days. So, here's what we need:

 

If you had an appointment scheduled you should be able to continue as planned.

 

If you had a test [Mod - Happy Thoughts]igned you will need to contact Mike Stagg our TA for further [Mod - Happy Thoughts]istance.

 

If you are looking to set up a Mentor/Instructor session in the near future please email the Instructor/Mentor directly. If you do not know how to get in contact with one either email Mike Stagg or myself.

 

Once again my email is jcaban//at//atpexperience.net

 

Mike's email is mdstagg//at//verizon.net

 

 

Joe Caban

vZTL ATM

The Future of VATUSA

Regards,

JX

Link to comment
Share on other sites

Kyle Ramsey 810181
Posted
Posted

Thank you for the leadership at a difficult time, Joe.

Kyle Ramsey

 

0

Link to comment
Share on other sites

Luke Kolin
Posted
Posted
Look for a few important announcements for ZTL staff and controllers to be posted here very soon. Yes I know you guys are going to say this is stupid so save me the jokes and taunts.

 

Actually, I'm going to echo Kyle and say that you're doing the right thing; taking advantage of a communications media to keep people informed. Good job. I don't know the specifics of the situation, but if this post is any indication, if there's a taunt to be made it's that it might have been done sooner.

 

Let me tell a story - around mid-summer our server had managed a sufficiently high uptime that our CEO started making note of this in public correspondence, and we were advertising it on our home page (we still do; right now we're at around 101 days). The old adage about pride going before a fall came true, and we had a hardware failure. I'm not 100% certain what the issue was (our provider never told us) but based on my postmortem of the box I think the RAM failed.

 

It was both fortunate and unfortunate that the failure occurred around 2 AM ET - not too much traffic at the time, but it didn't get noticed until around 930 AM when I got into work; tried to hit the box, direct SSH and SSH/serial console. Nothing worked, so we called the hosting company and got into their black hole of a support queue.

 

At around 10AM we decided that the problem wouldn't be immediately resolved, so we initiated our disaster recovery procedures. We started downloading our off-site database backups to our backup server, and by lunchtime we had it up and running and were able to send out broadcast messages to our community informing them of the situation. We disabled ACARS and flight logging, but kept the forum open in read-only mode and allowed administrators to post and provide updates. Eventually (around 4PM) our server was restarted, we fscked the drives and restored the DNS and we were back on our way.

 

I mention this because to paraphrase Ron Howard, "Failure is not an option - it is an inevitability" Stuff is going to fail. Servers go down. Software blows up. Network connections are lost. However, when our systems fail we do not need to fail - we can demonstrate a proactive attitude towards communication and recovery that will provide positive feedback from our users.

 

I mention this not as a shot against you or anyone else. We've had two major hardware failures in the last few years. We have used them as a great learning experience by doing the post-mortem in a neutral, analytical fashion, much like an air disaster investigation. And what we've learned from these experiences is that the best two things that you can do are a) proactive communication - even if you have no news, say so; and b) knowing when to cut your losses and switch to the DR site.

 

Those of you reading this thread have either had a major failure, or you are going to. Hopefully my experience and Joe's can give you some tips on what to do.

 

Cheers!

 

Luke[/url]

... I spawn hundreds of children a day. They are daemons because they are easier to kill. The first four remain stubbornly alive despite my (and their) best efforts.

... Normal in my household makes you a member of a visible minority.

Link to comment
Share on other sites

Joe Caban 844086
Posted
Posted

Thanks guys.

 

--------------------------------------------

--------------------------------------------

Can we get a sticky until Monday???

--------------------------------------------

--------------------------------------------

 

Ok newsflash .. . -- . --- ..... .. -.- .-..

 

Our site files were deleted. We'll never know why or how. It may or may not have been inadvertent. It may or may not have been a technical problem.

 

Fortunately we have backups and what not. Restoration is in progress.

 

Joe Caban

vZTL ATM

The Future of VATUSA

Regards,

JX

Link to comment
Share on other sites

Elliot Lezam 895373
Posted
Posted
Look for a few important announcements for ZTL staff and controllers to be posted here very soon. Yes I know you guys are going to say this is stupid so save me the jokes and taunts.

 

Actually, I'm going to echo Kyle and say that you're doing the right thing; taking advantage of a communications media to keep people informed. Good job. I don't know the specifics of the situation, but if this post is any indication, if there's a taunt to be made it's that it might have been done sooner.

 

Let me tell a story - around mid-summer our server had managed a sufficiently high uptime that our CEO started making note of this in public correspondence, and we were advertising it on our home page (we still do; right now we're at around 101 days). The old adage about pride going before a fall came true, and we had a hardware failure. I'm not 100% certain what the issue was (our provider never told us) but based on my postmortem of the box I think the RAM failed.

 

It was both fortunate and unfortunate that the failure occurred around 2 AM ET - not too much traffic at the time, but it didn't get noticed until around 930 AM when I got into work; tried to hit the box, direct SSH and SSH/serial console. Nothing worked, so we called the hosting company and got into their black hole of a support queue.

 

At around 10AM we decided that the problem wouldn't be immediately resolved, so we initiated our disaster recovery procedures. We started downloading our off-site database backups to our backup server, and by lunchtime we had it up and running and were able to send out broadcast messages to our community informing them of the situation. We disabled ACARS and flight logging, but kept the forum open in read-only mode and allowed administrators to post and provide updates. Eventually (around 4PM) our server was restarted, we fscked the drives and restored the DNS and we were back on our way.

 

I mention this because to paraphrase Ron Howard, "Failure is not an option - it is an inevitability" Stuff is going to fail. Servers go down. Software blows up. Network connections are lost. However, when our systems fail we do not need to fail - we can demonstrate a proactive attitude towards communication and recovery that will provide positive feedback from our users.

 

I mention this not as a shot against you or anyone else. We've had two major hardware failures in the last few years. We have used them as a great learning experience by doing the post-mortem in a neutral, analytical fashion, much like an air disaster investigation. And what we've learned from these experiences is that the best two things that you can do are a) proactive communication - even if you have no news, say so; and b) knowing when to cut your losses and switch to the DR site.

 

Those of you reading this thread have either had a major failure, or you are going to. Hopefully my experience and Joe's can give you some tips on what to do.

 

Cheers!

 

Luke[/url]

 

Shows how dedicated you guys are. Do you guys run your own servers, because most hosts backup their data periodically to another location.

Link to comment
Share on other sites

Keith Smith
Posted
Posted

Joe,

 

I give you credit for using the VATUSA forum as a temporary place to stay in control at the helm and restore some order, that's good thinking.

 

I can't imagine this is an easy time. Good luck with the restoration.

 

Keith

Link to comment
Share on other sites

Luke Kolin
Posted
Posted
Do you guys run your own servers, because most hosts backup their data periodically to another location.

 

Yes. We pay our hosting company to give us a machine with a bare bones Linux install. We pay them even more to give us the root p[Mod - Happy Thoughts]word and go away.

 

Generally, we operate under the [Mod - Happy Thoughts]umption that all of the sysadmins at our host are either incompetent twits, or about to be replaced by one. I don't really believe this is true, but it never hurts to be pessimistic and make [Mod - Happy Thoughts]umptions based on the worst case. We have a number of cron jobs that automatically dump our database, Subversion repository and application configurations. Our web host gives us 100GB of FTP-accessible storage, and we dump the backups there, as well as to a pair of offsite backups. We have to [Mod - Happy Thoughts]ume that the entire web host will go down or is inaccessible, hence the off site backups. We're responsible for our own DNS, and that too is handled in multiple locations.

 

If there's anything else that I could add, it's to validate your restore procedures even more than your backups. When we did practice restores, we found a few missing items. Even after correcting them, we still had procedural issues when we had to do it for real.

 

Cheers!

 

Luke

... I spawn hundreds of children a day. They are daemons because they are easier to kill. The first four remain stubbornly alive despite my (and their) best efforts.

... Normal in my household makes you a member of a visible minority.

Link to comment
Share on other sites

Joe Caban 844086
Posted
Posted

Good news is our ztlartcc.net site is back up! Bad news is we will be taking it back down next week. We need to switch it to a new server and will be releasing a new website as well. Be on the lookout for that.

 

Joe Caban

ZTL ATM

The future of VATUSA

Regards,

JX

Link to comment
Share on other sites

Joe Caban 844086
Posted
Posted

#3

 

If you received an email giving you log in information to the new server please log in and make sure everything works for you. Also keep the URL confidential for now.

 

I still don't have ztl email capabilities for anyone who is trying to reach me.

 

Also, I would like to thank everyone who helped me out over the past day and a half:

 

Jared Addis

JC Rodriguez

Brian Sperduto and VATUSA staff

Brandon Bartell

Mike Stagg

Ariel Maisonet

Benton Wilmes

& Byron Macrae

 

Thanks guys!

 

Joe

Regards,

JX

Link to comment
Share on other sites

Brandon Bartell 968788
Posted
Posted

No Problem Joe. I am glad Brian was able to get the current site backup. But soon he will have to take it down again. IRONY.

Brandon Bartell

Controller

Atlanta ARTCC

Link to comment
Share on other sites

David Reimer 913748
Posted
Posted (edited)

Nvr Mind.

Edited by Guest
signiture.jpg
Link to comment
Share on other sites

Joe Caban 844086
Posted
Posted

still can't get emails, but you can de sticky i think!

 

No ztlartcc.net emails. We are working on it.

Regards,

JX

Link to comment
Share on other sites

David Reimer 913748
Posted
Posted
still can't get emails, but you can de sticky i think!

 

No ztlartcc.net emails. We are working on it.

 

Never mind the de-sticky then

signiture.jpg
Link to comment
Share on other sites

 Share