Jump to content

VATSIM Tech Blog, Q4 2020


VATSIM

4600 views

Welcome back to the re-ignited Technical Team Posts

This is a long-overdue posting, and we will be aiming to do this more regularly as things go forward into 2021! 

There are many exciting plans for the network. We hope that by writing these posts, we can be more transparent with the network's membership about what has happened, what is currently happening, and where we can say what will be happening! We’ll split these posts out into sections where appropriate; such as Infrastructure and Development, to make things a little clearer.
 

Infrastructure

For a while now VATSIM’s core infrastructure (excluding FSD and AFV) has been in need of a refresh. While reliable, we had a single point of failure and a major OS upgrade due in our future. The current system has been maintained by different members of VATSIM over the years resulting in undocumented configurations in use.

Months leading up to March, the Network Infrastructure team worked on reviewing the current state of application source code, databases, websites, and services running to support FSD. The outcome of this work resulted in a plan to migrate VATSIM to a scalable and redundant infrastructure. 

Kubernetes was chosen as our foundation for providing a scalable and redundant compute environment for our websites and services to run in. Every application and service needed to be moved into a Docker container. Some applications just worked, others needed tweaking to run correctly.

With Kubernetes, applications in most cases run on multiple machines providing a highly available and redundant service. When a compute node fails, it automatically recreates the failed node and starts the applications running on the failed node on healthy ones. In cases where applications aren’t able to run on multiple nodes, they will recover automatically very shortly after the failure automatically.

Our database server needed to be upgraded as well. We were on a MySQL version that had just gone end of life. A week or so leading up to March, rolling upgrades were done using Docker to bring the server up to the latest version of MariaDB.

This early database server upgrade and some data schema changes allowed us to move into a MariaDB Galera cluster. This provides a highly available and scalable database cluster for our developers and users.

On March 2nd for roughly 18 hours, we started moving databases and shifting application workloads to our new compute and database cluster deployments. During this time there was no impact to the network for existing users. Within the next few days (the Network Infrastructure team needed rest), most major issues not caught in testing were resolved. Smaller issues resolved over the following days. 

Since then, we have had only one major outage and several minor outages that have gone unnoticed to the wider network due to the redundancies and fail-safes in place. 

 

Technical Development

vatSys - Jake has been hard at work with vatSys. It continues to progress and we’re seeing more interest in taking up its Eurocat modelled system. 

Euroscope has had several updates on its Beta channel in recent months. Members are encouraged to migrate to this and you can do so at the Euroscope website here and all it requires is to unzip the new version over the top! This removes all aspects of the old voice 
system and adds a variety of other changes which can be viewed in detail here.

The VATSIM Technical Team is in the very early planning stages of upgrading our server architecture. There is no time frame on this and we will share details of this when we absolutely can! 


Something long overdue that we've been getting asked for fairly regularly is a refreshed data feed! If you are using the VATSpy Beta versions, you will already be taking data from this feed!

We'd like to introduce you to Version 3! Moving to production on the 3rd December 0001Z! 

  • Updated more frequently - every 15 seconds!
  • JSON only for easier parsing by developers - no more string delimited lines!
  • Structured format that allows everyone to focus on what matters most to them
  • Additional data not historically present, including pilot ratings and ATIS codes

The structure for this new format can be viewed at https://api.vatsim.dev/#operation/DataServerV3 

The file is retrievable at https://data.vatsim.net/v3/vatsim-data.json 

 

Web Development

Application Programming Interface
The VATSIM Tech Team rolled out a new API to replace the cert auto tools divisions were using. This API is available for use at https://api.vatsim.net/api/ . The new API includes a JSON formatted version of the id_status endpoints, division tools, and more. One of the new endpoints in this new API includes a way to pull user sessions. This means many VATSIM facilities no longer need to store a copy of user sessions that they read from the Data Server, and can simply make a request to the new API. We intend to expand the API in the coming months, to include data such as weather and a wider exposure to other network services. The API will be micro serviced to ensure reliability and speed up development time.  

METAR Update
Many people will have seen, heard or been aware of the recent METAR feed issues whereby they have been jumping back in time. This was due to a change in the feed we were using that meant the file kept jumping back to all hours of the day. We have since moved to a new feed within NOAA that prevents this issue and we have seen a higher reliability on the data since making this change. 

PRAMS
While not focussed overly on the wider membership, anyone who undertakes a pilot rating through an official VATSIM ATO, will find their rating process through PRAMS. This system has been renewed from the ground up to make the ATO’s upgrade and manage members pilot ratings easier and more fluid.

New Statistics Center
In Q3 of 2020, the VATSIM Tech Team began work on a new modern statistics center. The new statistics center will include most of the functionality from the previous site, but will also include broader network statistics. 

myVATSIM
myVATSIM was launched back in early August 2020, in line with the release of Microsoft Flight Simulator, and the new VATSIM Pilot Rating system. myVATSIM is now the house for most self-managed membership functions such as email, division and region change. myVATSIM also houses the new Pilot Learning Center. Members in events teams across the network are now sharing all events to this central location! Be sure to check out what's coming up next! 

Retirement Notice - Txt Data File and JSON v1/v2
These versions of the data will be retired on, or closely following March 1st of 2021, with a potential extension to March 31st if deemed necessary. We highly encourage all users of the old text file to migrate to the new JSON feed which can be found in more detail above. v2 was never advertised as production ready and was purely for an internal project

Retirement Notice - SSO
In April of 2020, we announced VATSIM Connect, the replacement for SSO. VATSIM Connect utilizes new and secure technologies such as OAuth2, allowing users to select which data they would like to share. We highly encourage the various facilities and 3rd party organizations which still use SSO to migrate to Connect before March of 2021. SSO will be retired at the end of March of 2021, and will no longer be available for use. From the 3rd December, no new applications and IPs will be considered and all implementations will be required to migrate for further access. You can find documentation for Connect here and you can begin by creating your organization here.

 

Upcoming Projects

New Website
We are currently working on the new VATSIM.net site. This will be a complete refresh using modernised technologies to showcase the very best of the network. Teasers to come in the future! 

Project Puma
Is all we can say for now! More to be revealed soon!

 

Summary

Development Discord
If you hold a purely technical role, or are responsible for the technical elements, i.e Webmaster, Infrastructure Manager etc, at an FIR, vACC, Division, Region, Affiliated Virtual Airline, Authorised ATO or other VATSIM recognised service, and you have not yet joined our VATSIM Technical Development and Web Discord Server, please send us an email at [email protected] to get a link. Here you can communicate with all technical people across the network, share resources, assist others and get quick access to the VATSIM Tech and Development teams!

 

To finish this off, everyone loves a good stat! 

On average, VATSIM moves between 8 and 10 TB of traffic every month. Roughly equating to ≈ 270GB a day. On Cross the Pond day, we moved 870 GB of traffic. 765 GB of this was through the data server. Levels of traffic this high, over a single 24 hour period have not been seen in VATSIM history! The other 105 GB was traffic routing through SimAware, natTrak and the AFV Map, and other VATSIM services.

1706353557_pastedimage0.thumb.png.4f06000d59c0f63cd611389d7c829b78.png976538069_pastedimage0-2.thumb.png.dbfcbb07809808d8fd4d8a996675d74a.png

  • Like 3
  • Thanks 2

16 Comments


Recommended Comments

very nice update Matt. i enjoyed reading the status of projects and of course there is always the appreciation of triggering interest with upcoming projects.

i look forward to more of these updates and having them hopefully quarterly would be very nice.

Link to comment

Many thanks for the update, always good to hear what's going on behind the scenes. 🙂

Quote

The structure for this new format can be viewed at https://api.vatsim.dev/#operation/DataServerV3 

Unfortunately that link does not work for me - api.vatsim.dev cannot be resolved; is it actually a registered domain or just an internal link for the development team?

Edit: Sorry, haven't seen generic error messages in Chrome for a while and I'm obviously too tired after work. 😅 The link still doesn't work but it's actually just a plain 404, not a DNS issue which also means it should actually not be accessible to anybody else...

Quote

Retirement Notice - Txt Data File and JSON v1
These versions of the data will be retired on, or closely following January 1st of 2021, subject to communication with other solutions. We highly encourage all users of the old text file to migrate to the new JSON feed which can be found in more detail above.

I am a bit surprised by the decision to shut the old service down on such short notice. Unless I missed something the JSON format was just "rumored" not too long ago and it took until April until an official statement was made which by the way said something very different:

Quote

Please note that this does not mean we are deprecating the old file! We will continue to maintain that file for years to come to ensure that all of the current VATSIM tools you know and love continue to function properly.

Not sure why that decision was revised just half a year later; what changed?

Edited by Daniel Neugebauer
Link to comment
3 hours ago, Daniel Neugebauer said:

Unfortunately that link does not work for me - api.vatsim.dev cannot be resolved; is it actually a registered domain or just an internal link for the development team?

Edit: Sorry, haven't seen generic error messages in Chrome for a while and I'm obviously too tired after work. 😅 The link still doesn't work but it's actually just a plain 404, not a DNS issue which also means it should actually not be accessible to anybody else...

The API link appears to be working for me, any chance you can provide more information?

 

Quote

I am a bit surprised by the decision to shut the old service down on such short notice. Unless I missed something the JSON format was just "rumored" not too long ago and it took until April until an official statement was made which by the way said something very different:

The old service does need to go. The JSON feeds provide the same data, and for most users will be easier to implement. The old post does go on to say that we will keep the old feed for a while, but keep in mind that this was made while Aidan was still on the board, as the staff changes, priorities will change.

The biggest changes between the JSON feed, and the text file is formatting, and what data is included. The v1 feed is a mirror of the text file, the v3 version of the JSON feed is focused on removing old properties which are no longer used.

VATSIM has been modernizing a lot of our infrastructure & services (such as getting rid of our old CERT Dashboard for members, and replacing it with myVATSIM). This is a vital step forward. We do understand that old services which are no longer maintained may no longer function as intended, but as some point we do need to take a step forward. This change is inevitable as we continue to modernize the network.  

Edited by Jordan Jolenaar
Link to comment
14 hours ago, Jordan Jolenaar said:

The API link appears to be working for me, any chance you can provide more information?

The API documentation has now become accessible for me as well; there was just nothing served under that URL yesterday.

14 hours ago, Jordan Jolenaar said:

The old service does need to go. The JSON feeds provide the same data, and for most users will be easier to implement. [...]

The biggest changes between the JSON feed, and the text file is formatting, and what data is included. The v1 feed is a mirror of the text file, the v3 version of the JSON feed is focused on removing old properties which are no longer used.

I understand that the old format was inconvenient to parse and did not offer any way for safe extension without breaking all sorts of clients in very weird ways (as seen while the format version number had been incremented from 8 to 9 without any major changes earlier this year). Providing the data in a JSON format, even keeping clients a (limited) choice of the format they want to fetch, clearly is a great improvement and absolutely the right way to go. I am just not sure how that improvement conflicts with providing a legacy format (with reduced/dummy information if fields have gone) "for years to come". That announcement may have actualy led to a slowed adoption of the new format and legacy consumers not getting migrated in time. I have the feeling that there are very many consumers (apart from status tools used by pilots like LittleNavMap or QuteScoop there are also VAs, VACCs and even commercial entities such as FSLabs) of which at least some will not be able to perform the migration on such short notice. A termination of the old legacy format on January 1st will most-likely cause some mid- to long-term service interruptions all across the VATSIM ecosystem for no obvious reason.

Even if the old format's service would need to be terminated due to multiple severe technical issues (e.g. unmaintainable code or incompatibility with upcoming changes to the backend) I don't see why it would not be possible to still provide the legacy format as an additional output generated from the latest JSON format to maintain broad compatibility.

What I would find even more important than just a change of formats is to improve the actual quality of the data encoded in whatever format. Having had a rough look into the v3 feed I immediately noticed that pilots still may have multiple prefilings at a time and it is still not clear if the entry "0000" into a time field (such as departure times) means midnight UTC or "no data". Having had fun with processing the old format for a while I would have wished for such issues to be addressed as well but it's apparently a problem within the backend systems, prefiling forms, pilot clients etc. all not getting cleaned up/sanity-checked on server-side before getting exported to the data feed. Such issues are mostly unrelated to the format being used to transport the data and most of them could even have been fixed in the old legacy format as well.

14 hours ago, Jordan Jolenaar said:

The old post does go on to say that we will keep the old feed for a while, but keep in mind that this was made while Aidan was still on the board, as the staff changes, priorities will change.

Regardless of BoG/staff changes and any internal conflicts that might be involved here I assume that Aidan made the April announcement not as an unqualified individual on his own but in his role of VP Tech (more or less confirmed by Gunnar's thanks in that thread). So at least from my point of view it was an official statement coming from VP Tech, so it was not "Aidan's (personal) decision" but VATSIM's decision. Equally, I don't see the post above as "Matt's (personal) decision" but a change in direction coming from VATSIM/Tech officially.

In case it would in fact be "Matt's personal decision" completely disregarding either "Tech's earlier decision" or just a personal conflict within the team against "Aidan's personal decision" I would highly recommend the BoG to discuss that matter internally (if it has not been done already) as there would be clearly something going wrong in that department. But again: My understanding as a simple member is that both posts are official and have been justified internally at their time. VP positions are equal to high leading positions in a large company, so I would expect decisions and communication to be made as careful and professional as in a company - I have absolutely no doubts in the integrity of both posts. However, the reason for that earlier decision to be revised without apparent technical requirement is just not clear to me.

  • Thanks 1
Link to comment

A change was overdue, but I also vote for a longer transition-period during which the classic datafeed will remain available until 3rd party developers have had a chance to adapt their programs or websites.

Edit: typo

Edited by Andreas Fuchs
  • Like 1
Link to comment
  • Board of Governors

Following internal tech team communications, and a discussion I had with Gunnar yesterday evening, we have extended the EOL date to the 1st March, with the potential to extend to the 31st if deemed absolutely necessary. With staff in the Development Discord, there are several solutions being created by the community that will be shared by VATSIM when they are complete, to aid in the transition and to provide a quick solution for access to the feed.

 

On 12/5/2020 at 11:42 AM, Daniel Neugebauer said:

Regardless of BoG/staff changes and any internal conflicts that might be involved here I assume that Aidan made the April announcement not as an unqualified individual on his own but in his role of VP Tech (more or less confirmed by Gunnar's thanks in that thread). So at least from my point of view it was an official statement coming from VP Tech, so it was not "Aidan's (personal) decision" but VATSIM's decision. Equally, I don't see the post above as "Matt's (personal) decision" but a change in direction coming from VATSIM/Tech officially.

These changes have full approval of the Board of Governors. I, nor anyone, can or should be expected to check or know of every single post ever made historically. The posts made here, and now are the most up-to-date and should be considered as such. There will always be a time of retirement of old solutions, it's the way things inevitably go.

  • Like 6
Link to comment

Thanks for the update! I found that in the new JSON format (v3), latitude and longitude are missing for both ATIS and Stations. Currently, VR Radio Helper (an X-Plane Plugin that helps VR pilots enter frequencies, https://github.com/VerticalLongboard/xplane-vhf-helper) uses the location of a station to display the currently tuned in station, reading it directly from the text datafeed. Since there are duplicates around the planet, sorting by distance is necessary. To not be forced to remove this feature in about two months: Could you please add latitude and longitude to all items in "controllers" and "atis"?

Thanks a lot,

Johannes

PS: There are two other threads with a similar topic: 

 

 

Link to comment
On 12/6/2020 at 6:12 PM, Matt Bozwood-Davies said:

we have extended the EOL date to the 1st March, with the potential to extend to the 31st if deemed absolutely necessary.

Is there an update on the actual date when the old service will be terminated? There's just one week left to March 1st but I think some important tools are still missing JSON compatibility.

Edited by Daniel Neugebauer
Link to comment

Please sign in to comment

You will be able to leave a comment after signing in



Sign In Now
×
×
  • Create New...