Legacy status proxy - providing data feed compatibility to passive clients not migrated to JSON yet

[Daniel Neugebauer]

The data feed providing status information to "passive" clients (i.e. not pilot/ATC clients but external tools) has recently been changed to a completely different new JSON-based data format. The old format has been discontinued as of 7 April 2021 which means old clients have now stopped working.

To get those tools working again they would need to be migrated which, for some tools, may not be easily possible. If you want to restore functionality until a proper migration has been performed, you can use the following unofficial tool:

https://github.com/dneuge/legacy-status-proxy-vatsim

You need Java with a version between 8 and 11 to run the proxy server. If you don't have it installed you can download it for example from AdoptOpenJDK (select OpenJDK 11 with HotSpot JVM).

Please read disclaimer, known limitations and FAQ carefully.

To stress some of the most important points:

• the proxy is intended as a temporary workaround only - if you are a developer, please help migrating the tools currently lacking JSON format support
• the proxy shall only be used to provide access to "passive" clients (i.e. not pilot/ATC clients, those require proper updates to be in line with CoC)
• it is recommended not to host the proxy publicly, ideally you should only run it locally on each machine you need compatibility for
• controller locations have been removed from the JSON data format, so they will be missing on the proxy as well - this may lead to unknown stations/airspaces not being displayed on map applications such as QuteScoop
  • since version 0.90 coordinates are substituted using the VAT-Spy database
  • since version 0.95 online transceivers are also taken into account

 

Edited August 26, 2021 at 01:46 PM by Daniel Neugebauer
coordinate substitution since 0.90/0.95

[Alexandra Robison]

This feels counter-productive.

[Maurizio Cinti]

Do we have to re-run every time we connect this "vatsim.jar" ? This seems to me an incredible and useless amount of "make complicated what was simple".....

Anyway, in Servinfo aircrafts are not visible....

Maurizio

Edited April 7, 2021 at 07:27 PM by Maurizio Cinti

[Daniel Neugebauer]

@Alexandra Robison That's why I waited with publishing the code on GitHub until the announced termination date and only announced it on a larger scale today after the termination had actually happened. The proxy is just a workaround and not a permanent solution. Such a proxy is inconvenient by nature. I don't believe any sane developer will rely on it for any longer than absolutely necessary and I made it very clear that the main goal should remain to migrate the original applications affected by the format change. Users (and maybe even webservice operators) however don't have a choice and may simply want to continue using their status clients today. Due to the service termination we now also notice a lot of services which have not completed migration in time or even were completely unaware of it until their service failed may be in need for a quick interim solution before they can (continue to) properly migrate their applications. That problem has been created by a sudden change in mind from VATSIM: Just last April VATSIM's tech department announced they would "continue to maintain that file for years to come". That promise was broken just 8 months later because after a change in staff apparently the new team was unaware of previous official communications. As you can see by the list of projects which experienced an outage today either the tech blog did not get through to every affected developer or the developers had no time to complete their migrations before the termination date (keep in mind all of us are doing this in our spare time and not everyone may have had enough time to spend on these projects in the past months). I too would have preferred not to have seen the need to write such a proxy but I don't see how it would create any damage. It's simply translating one format to another for interim use and apart from controller coordinates there is basically no loss of information. The data format does not matter, the data does. The projects have now been woken up by the service termination and unless they actively make some decision on how to proceed they will not get back into service.

The library used in the proxy server was readily available through another project of mine so it was possible for me to quickly provide a universal temporary solution. I thought very carefully whether I wanted to invest the time in migrating QuteScoop instead but as a full-time Java developer who only has basic knowledge about C++ I felt that simply wrapping my existing library into a proxy server would be much more helpful in short term while requiring very low effort to write. Because any number of previously unknown affected applications can potentially be covered by that tool the ROI seemed much better than working for weeks or months on a single tool's migration and maybe not finish it in time. The other issue with applications like QuteScoop is that they are often designed for compatibility with other Whazzup-based networks (you know which large one I mean). QuteScoop even seems to have been designed around "that other large network we don't name here". There are a lot more decisions and communication involved in properly migrating such a non-trivial application than just simply slapping a JSON parser on top and changing a URL...

42 minutes ago, Maurizio Cinti said:

Do we have to re-run every time we connect this "vatsim.jar" ? This seems to me an incredible and useless amount of "make complicated what was simple".....

Please note that the proxy is unofficial and not provided by VATSIM. It is a 3rd party tool to help people use a service which would otherwise be completely inaccessible now.

The proxy can also be run in a CLI mode as described on the readme/project page. Using that CLI mode you are also able to run it from a batch file, integrate it with an existing launch tool like SimStarter NG, register it for auto start or as a service etc.

The other alternative you have is to wait until an update is provided for the application you are trying to use, so you would not need that proxy server to translate between the new data format and your legacy application.

Edited April 7, 2021 at 08:05 PM by Daniel Neugebauer

[Nick Harasym]

I don't support this -2

[Daniel Neugebauer]

7 minutes ago, Nick Harasym said:

I don't support this -2

Please clarify if that is just your personal opinion or an official statement from VATSIM staff prohibiting use of that solution.

[Nick Harasym]

10 minutes ago, Daniel Neugebauer said:

Please clarify if that is just your personal opinion or an official statement from VATSIM staff prohibiting use of that solution.

VATSIM isn't prohibiting the use of this however it isn't recommended. Many of the tools that don't support the new JSON format have alternative options.

For some personal things that I didn't have time to port to the new format, I did something similar. A function to map the new data format into the old format.

[Gary Oliver]

VATSIM promised me they would not be removing the old data feeds whilst legacy clients are using them.

An absolute disgrace, and exactly why I shall never be doing any more development for VATSIM.

Too many people playing at their fake job titles, not listening to sense.

This is what happens with change for the sake of change.

Cheers

Gary

Ex-AFV Lead

Edited April 7, 2021 at 11:40 PM by Gary Oliver

[Sean Harrison]

Would be extremely interesting to see how many requests there were on average to the old data, and how many their are now.  I thought ServInfo etc whilst old are stalwarts of the community.  Is there a viable replacement for ServInfo?

Thank you Daniel for stepping in and providing assistance to us users.

[Jason Cochran]

Daniel,

This is awesome. I wasn't aware of this impending change (somehow I missed the change notice), so this could be a lifesaver.

I'm not a hardcore Java guy, so would you be open to help get this running on a server in the cloud some place (for example, Amazon Lambda or Heroku) and making it available through an API?

EDIT: To be clear, I knew this was going to happen at some point, but per the forum Daniel referenced earlier, I thought it was years down the road.

Jason

 

Edited April 8, 2021 at 03:27 AM by Jason Cochran

[Fritz Eisler]

To Daniel: Great job Daniel!

To Staff: I'd still very much like the whazzup.txt also made avble somewhere/somehow please!

Edited April 8, 2021 at 09:39 AM by Fritz Eisler

[Daniel Neugebauer]

9 hours ago, Nick Harasym said:

VATSIM isn't prohibiting the use of this however it isn't recommended.

Thanks, I really wasn't sure how to read your comment earlier. :)

I agree that the proxy is (very obviously) not recommended but it may still currently be the best option for web developers who were unaware of that change until yesterday to quickly restore their service while working on a proper migration. It may also be the only option for users who cannot actively contribute to migration efforts of their favorite desktop applications while they wait for an update.

For a developer, using JSON has some very clear advantages over the old format and is much better to extend. I am very happy with that improvement. I just think the termination of the old format should have been handled in a better way as it does not seem necessary from a technical view point. It's very understandable that VATSIM wants to clean up/remove the old data server - this is totally fine from a developer's perspective. It just may be a bit short-sighted when it comes to previous communications and the user base.

10 hours ago, Nick Harasym said:

Many of the tools that don't support the new JSON format have alternative options.

Well, the only alternative I see is for users is to switch to another service/tool which is something they simply don't want to do. My opinion is that we lost too many good tools over the past years already (not limited to just status trackers etc., for example just think about VATroute). While I agree that it is odd to see someone still using ServInfo 18 years after its last release it also is a strong indication that all available alternatives may be lacking some feature (maybe just the UI?) which is important to the ServInfo user base and still (after almost 2 decades!) has no modern replacement.

Personally, I don't like VatSpy (can't get used to its GUI) but it appears to be the only desktop status monitor that currently has native JSON support. Because my secondary PC while flying online is running on Linux I would have to use some workaround (wine or a VM) to use VatSpy as it is only available for Windows. All the (currently) available web services I've come across so far are lacking something. I am a very happy QuteScoop user, so I want to keep using it (although I always keep an eye on alternatives, just out of curiosity). It would of course be much better if QuteScoop had native support for JSON (which will surely come) but at the moment using the proxy is a working interim solution.

10 hours ago, Gary Oliver said:

An absolute disgrace, and exactly why I shall never be doing any more development for VATSIM.

Too many people playing at their fake job titles, not listening to sense.

I'm sorry to hear that. Thanks again for the great work done so far, AFV is absolutely awesome! :)

However, I'd like to think that even if there are disagreements with current (or former) staff, those disagreements won't last forever as staff is changing on all levels all the time. It's a bit like in politics when the next government may involve other people and have other opinions. (and, indeed, I sometimes feel like VATSIM is loosing itself a bit too much into internal politics)

6 hours ago, Jason Cochran said:

I'm not a hardcore Java guy, so would you be open to help get this running on a server in the cloud some place (for example, Amazon Lambda or Heroku) and making it available through an API?

As laid out in the disclaimer, I strongly advise against openly hosting (any format of) VATSIM data files. Even after conversion the data remains the sole property of VATSIM and is subject to VATSIM policies (incl. possible restrictions in redistribution). Some data fields hold information that might be very sensitive to handle in respect to privacy laws (especially in the European Union since introduction of GDPR). That's one reason why the proxy only allows single IP addresses to be granted access individually. If someone wants to host publicly accessible data files I highly recommend consulting top-level VATSIM staff first and to make sure that is absolutely waterproof for you to provide such information by all applicable regulations (consult a specialized lawyer if unsure). Personally, I think providing that data as a 3rd party is way too much hassle and not worth the time and nerves.

If you only need to provide access to your own services (not redistributing the data files themselves but fetching them for processing by another container/node) that would be possible by simply running the server with all possible IPs granted access through the configuration. I also wanted to add support for configuration through CLI but just couldn't find the time to complete that support before release. It will still be added, however, so if you have dynamically changing IPs you could then restart the proxy with the latest IPs provided as command line arguments.

[Koen Meier]

As a sidenote. I feel the majority of the changes happened after AFV was introduced. Now some of these changes are the result of increased competition. We can globablly classify these as being AFV and the upcomming position frequency updates. Some changes are necessary due to technology being to old and patched together, we can see the FSD as being in here. The main goal would be to have a network with very little hiccups which is also easy to scale and modern.

Now some changes happened due to a new image vatsim wanted to create. I would classify the new website layout, forum and tagline in here and any related changes such as stats. We also changed the pilot training department however not sure how to classify that myself. The flight plan change i am not sure why that happened but it is nice.

But i also think we have to remember nobody likes change. However change also makes us resilient as a person and organisation. If i didnt embrace change on a daily basis i wouldnt be the same person. i wouldnt step out of my comfort zone.

[Andreas Fuchs]

Oh, it is not about change. I guess that almost everyone is welcoming the technical change with open arms. But: someone telling me that there's always replacement for the program that I dearly love to use just shows that they might not really know the program that we are talking about. At present, there's no similar alternative to Qutescoop. If you love ServInfo, switching to VATSpy might be the easiest solution. But Qutescoop? There's nothing like this outstanding program.

There's no harm done if some users are running a proxy tool on their own, private computers in order to feed these programs. Our staff should rather be supportive and happy that a member of the VATSIM community has once again created an application that helps solving issues. All for free, uncomplicated, from the base for the base.

Edited April 8, 2021 at 07:36 PM by Andreas Fuchs
EDIT: for wording, typing errors

[Fraser Cooper]

So many devs use the old data-feed still. Turning it off is unreal. Many devs will now have to spend many hours to make their product compatible. Whereas, VATSIM could push the data in the old format as well as the new surely?!

 

[Alexandra Robison]

3 minutes ago, Fraser Cooper said:

So many devs use the old data-feed still. Turning it off is unreal. Many devs will now have to spend many hours to make their product compatible. Whereas, VATSIM could push the data in the old format as well as the new surely?!

 

This change was announced 4 months ago. I was able to convert my applications to the v3 feed in about 15 minutes. JSON is far, far easier to use than a colon-delimited text file.

[Luke Kolin]

What's worse is that the "new" data feed (even from v2 to v3) removes certain important fields that breaks applications, including our own. Some of us have been warning VATSIM about this for several weeks and months without any official response only to be told to email someone (which says something about the importance VATSIM gives to these forums ... and by implication our feedback).

We need a structured text feed like JSON ... what we do NOT need is something that removes critical data fields and complete radio silence from VATSIM when we point this out in advance.

Luke

[Maurizio Cinti]

This discussion is getting more and more an exercise of surrealism....

I'm not a Java or whatever expert , but only a final user, more than 20 years enjoying flying online on Vatsim (and in Satco before it...), so I'm not interested in dispute around technical aspects of software. I'm only saying that this decision to switch to another kind of online graphical imaging of Vatsim traffic is incomprehensible to many users.
No one spent his time explaining us why and how.

Maurizio

Edited April 8, 2021 at 07:14 PM by Maurizio Cinti

[Sean Harrison]

This may sound stupid, but what is recommended ‘now’ to work out what atc is online?  I’m a long time servInfo user, and found it extremely hard to visualise who I should be contacting [atc wise] over last week.

[Alexandra Robison]

3 minutes ago, Sean Harrison said:

This may sound stupid, but what is recommended ‘now’ to work out what atc is online?  I’m a long time servInfo user, and found it extremely hard to visualise who I should be contacting [atc wise] over last week.

https://vattastic.com is a good in-browser solution

Otherwise if you'd like a desktop client, VATSpy is a good choice. https://vatspy.rosscarlson.dev/

 

Note: this is my personal opinion.

[Andreas Fuchs]

I think we all understand the reasons for change. Don't we all agree that the new server software (fast position updates), the new VATSIM Voice (thanks Gary!), the new website, the new ICAO flightplans etc. are all great stuff? I will not be able to express my gratitude sufficiently to those who have invested countless hours into this, for free. But I am profoundly thankful to all Devs!

Yes, there has to be some cutoff at some point, I get that. So, to help users with "legacy" programs, a nice member of our community decided to create this small Java program, great! Why would anyone in VATSIM's staff strongly disagree with this? I do not understand this at all. Rather be happy that someone has done the work and provided it to everyone. And take him onboard, use him as a resource, do not alienate him or write off his work. We are all here for the same thing here, let's work together!

[Andreas Fuchs]

6 minutes ago, Sean Harrison said:

This may sound stupid, but what is recommended ‘now’ to work out what atc is online?  I’m a long time servInfo user, and found it extremely hard to visualise who I should be contacting [atc wise] over last week.

I suggest you try VATspy, that's the closest you get to what you are used to from ServInfo. It already has native support for the "new" JSON format.

Edited April 8, 2021 at 07:44 PM by Andreas Fuchs

[Andreas Fuchs]

 

1 hour ago, Alexandra Robison said:

This change was announced 4 months ago. I was able to convert my applications to the v3 feed in about 15 minutes. JSON is far, far easier to use than a colon-delimited text file.

That's great that you were able to do this! Hats off to you! But there also older programs that have been written in older languages by several Devs that are a bit more delicate to change. If anyone could help us out on Qutescoop, that would be cool.

[Sean Harrison]

13 minutes ago, Alexandra Robison said:

https://vattastic.com is a good in-browser solution

Otherwise if you'd like a desktop client, VATSpy is a good choice. https://vatspy.rosscarlson.dev/

 

Note: this is my personal opinion.

Thanks for that.  I often have problems accessing vattastic.  

is Volanata affected by the changes? And is it approved/recommended by VATSIM?   It seems to promote VATSIM strongly.

Edited April 8, 2021 at 07:56 PM by Sean Harrison

[Koen Meier]

1 minute ago, Sean Harrison said:

Thanks for that.

is Volanata affected by the changes? And is it approved/recommended by VATSIM?   It seems to promote VATSIM strongly.

I think volanta and simtoolkitpro for that matter rely just like maps on the feed. So if they were created with the old feed they would have to be updated.