VATSIM Server problems - was: Maintenance Friday-Saturday

[Kyle Ramsey 810181]

The VATSIM servers will experience maintenance over the next two days. While we will try to not affect pilot and ATC connections, there is no guarantee they won't be affected. Usually switching to an alternative server will alleviate this problem. Some web services may also be affected on specific servers and may not be available for short periods of time.

 

The forums will not be affected and you can find updates posted here if things don't go as planned and we need to communicate any additional downtime or loss of services.

 

Thank you for your patience with these important upgrades and thank you for your support of VATSIM.

[Luca Benelli]

The server is back up, resyncing of all raid arrays is underway and will take a while.

 

We *hope* to have everything back up after the weekend.

[Luca Benelli]

Some new problems... trying to understand what's up.

 

Luca

[Luca Benelli]

So it looks like we have another problem on the RAID system :/ resyncing failed.

 

Now it depends on how fast we get the persons who can look at it on a PC... obviously these things happen on a weekend while people are on holiday, then it will require probably an intervention from somebody at the data center... so there's once again no ETA on this.

 

Sorry.

 

Will update when i have more info.

 

Luca

 

PS: As the vatsim email addresses are down please use alternative emails if you have them to reach whoever you need. You can reach me at [email protected]

 

EDIT : services involved are all websites, dataserver and data files, mailservices @vatism.net and stats. Forums and online services are NOT affected.

[Pan Lalas]

For pilot training related issues I can be reached here: [email protected]

[Luca Benelli]

Latest info is that the server needs a complete rebuild and backup restore.

 

Unfortunately it's the first day back at work for all of us, so there are some priorities to take care of first.

 

I'll try to post another update as soon as i know anything new.

 

Luca

[Norman Blackburn]

Should anybody have any complaints etc and need to contact me directly my alternative email address is [email protected]

[Luca Benelli]

Trying to understand what happened it has been found out that the second disk which has been switched on the box has already errors on it, so we need to have this replaced again. A request for this has been made in the afternoon (European time). I don't know if this activity has already been completed but this means this evening or tomorrow morning we'll start rebuilding the system and possibly restore the backups... hopefully without problems.

[Luca Benelli]

New disk is in and a resync of the empty disks is underway. It's now 22 over here so we decided to run some more checks on the disks overnight instead of waiting and tomorrow morning we will start rebuilding things... first thing is the base system then the DB. Data feed and weather will probably follow. Mail and websites will be queued up.

 

Luca

[Luca Benelli]

Hopes were set a bit too high.

 

The backups are there but it's mostly data, the system re-installation took quite a while.

Some services have already been reactivated (like DNS).

The DBs are being restored from backup while i write, the stats DB is the biggest one and will be last, but the data feeds use it, so these won't be up tonight.

 

We are still trying to get the weather updates back online tonight.

 

Other services will follow in the morning. (Data feeds and mail services first, then CERT web and all the rest to follow).

 

Luca

 

EDIT: Metars should be updating correctly on the network.

[Luca Benelli]

Who is online service "kindly sponsored" by Mike Evans:

 

http://online.vatsim.net

[Luca Benelli]

Data feeds are again being distributed, status.vatsim.net should come up pretty soon and most software should be working again.

 

Please notice we are running this through a workaround on a different server so we have a couple of limitations: no ATIS is displayed and we are still NOT logging connections.

[Luca Benelli]

DB restore is taking even longer than expected, the DB is huge (way over 30 giga).

 

As it is it looks like websites will be restarting tomorrow.. i'm quite confident it will finish during the night. *crosses fingers*

 

We will also try to recover the stats generated by the "temp" data server, but there's no guarantee on this at this time.

 

Luca

 

PS: Mail services are running again.

[Luca Benelli]

Thanks to the work of Christoph Neukirch websites and the Data servers are again working on the main server so we are now again gathering connection info.

 

All services SHOULD be running correctly again, this includes stats (just give me the time to re-enable logins).

PLEASE try not to hammer it doen with stats requests in the next hour. If you have no urgent need for stats try waiting a couple o fhours or even tomorrow before putting your queries

 

If anything doesn't work, please let us know in the network problems thread (general discussion forum - pinned).

 

Thanks for your patience.

 

Luca