Legacy status proxy - providing data feed compatibility to passive clients not migrated to JSON yet

[Luke Kolin]

4 minutes ago, Sean Harrison said:

Thanks for that.  I often have problems accessing vattastic.  

They've got a (poorly configured) wildcard cert. Try www.vattastic.com.

Cheers

Luke

[Gunnar Lindahl]

Gosh, so much interest! I think the first thing I'll say is that it's great that we this much passion and interest from so many people - without it VATSIM wouldn't exist.

Thanks to Daniel for creating the tool. I bow to others in this thread who have infintely more technical knowledge than I, but it looks to me like you have created a great temporary workaround for those who are using applications which haven't yet updated to the new format or are abandonware, so thank you for your service to the community.

Those who know me know that I am not a technical expert in this or any field, so I just want to make a few short points. Matt's team are reviewing some of the correspondence and will put out a more useful technical summary of why we've done things the way we've done them this weekend. 

For those who might be reading this thread and wondering what it's all about for the first time, you can read the technical blogs posted last year which announced the transition to this new data feed. We plugged this change several times in our blogs, over social media and also via correspondence with users of the datafeed. The vast majority were switched well ahead of the 31st March date (the feed was actually switched off on the 7th April, a week after) - to answer a couple of the questions, apps like Volanta have already been updated and are functioning well.

If there is anyone who maintains a tool which runs off the VATSIM datafeed and are unaware of the update or struggling to update, please do point them in our direction. There is also an extemely lively Dev Discord server where all of this is, and much more, is discussed, with some great collaboration not just amogst VATSIM members but our close partners also. You can find details of that in the technical blog if interested in taking part.

As I say, more detail to come, this message is to acknowledge that there are some queries that we will endeavour to give context to.

 

[Christiaan Hoek]

I'm just an end user, but I am amazed at the ignorance of breaking every second client and leaving people wondering what the hell is going on.  What happened to "if its not broken - don't fix it!".

My map viewer of choice is https://afv-map.vatsim.net/  (is it just me, or is that an official Vatsim url?)  Guess it should be working then?  Guess again...

Ok, lets try VatMap on my iPhone.  Dead.

VATSpy on PC?  Dead.

Googled, got this.  Still frustrated.

[Alexandra Robison]

47 minutes ago, Christiaan Hoek said:

I'm just an end user, but I am amazed at the ignorance of breaking every second client and leaving people wondering what the hell is going on.  What happened to "if its not broken - don't fix it!".

My map viewer of choice is https://afv-map.vatsim.net/  (is it just me, or is that an official Vatsim url?)  Guess it should be working then?  Guess again...

Ok, lets try VatMap on my iPhone.  Dead.

VATSpy on PC?  Dead.

Googled, got this.  Still frustrated.

VATSpy is working. You might just need to download the latest update.

[Mike Sweeney]

If possible, re-consider changes to facilitate newer technology, so that legacy applications [SERVINFO] will also still continue to work.

[EDIT]
Thank you Daniel for providing a workaround to enable SERVINFO to continue working, greatly appreciated!
Thanks Andreas for providing a link for the instructions , and with updated explanation for VATBOOK (formerly Eurobook) option within Servinfo.

Edited May 2, 2021 at 03:30 PM by Mike Sweeney

[Eric Fisher]

For anyone wanting a web based solution,  try this https://simaware.ca/ as other than Volanta, there is none better.

What irks me is the only halfway decent Android app, Vatscan, is now dead thanks to this. Currently there are no Android apps for Vatsim. Sucks...

 

[Robert Shearman Jr]

15 hours ago, Eric Fisher said:

What irks me is the only halfway decent Android app, Vatscan, is now dead thanks to this. Currently there are no Android apps for Vatsim. Sucks...

 

[Luke Kolin]

On 4/8/2021 at 4:17 PM, Gunnar Lindahl said:

Those who know me know that I am not a technical expert in this or any field, so I just want to make a few short points. Matt's team are reviewing some of the correspondence and will put out a more useful technical summary of why we've done things the way we've done them this weekend. 

I hope I am jumping the gun a bit here but given that there's been complete silence both on the forums and to my e-mails regarding access I'd like to throw a few things out there for you to consider.

I have been very critical of VATSIM's technology development ecosystem over the years. I've been significantly less active over the years for two reasons - one, my life has moved in different directions and online flight with ATC isn't a priority for me, but most importantly you have been doing things way, way better than before for the past year or two. I don't want to understate it - there seems to be at last a recognition that VATSIM technology needs to move forward, it is moving forward and that is fantastic to see. I have no issue that you've deprecated the old feed. I've probably said in these forums repeatedly that we need something that is structured text and handles proper character encoding and escaping, and JSON makes perfect sense.

What I have big issues with is the way the rollout (and more importantly, the fallout) has been handled. I and others have posted repeatedly in the developer forums about our concerns with the v3 feed, specifically that it omits several critical fields that were present in the v2 feed. The only responses have been "it's hard" (even though the v2 feed has it without issues) or to email Matt. If you're going to have a developer forum here, then can you please make it something more than a "write-only" forum? I have /dev/null and an old OCZ SSD if I want my posts to vanish. 🙂

If Discord is where all of the cool kids are at these days, then the developer forum shouldn't exist. Put in a sticky telling everyone where to go and close it down. But Discord and the development team need to be responsive to access requests. I've sent in requests for access last week (since apparently the "old" Discord that I was part of way back no longer exists) and have heard nothing. I understand that the tech team may be busy, but at least getting people connected should be a major priority. Right now I don't know what's going on - maybe there is all sorts of info being posted there that I'm not seeing or able to influence, and I'll be presented with another fait accompli? I don't know.

What this has demonstrated is that VATSIM is an ecosystem and has stakeholders that far exceed VATSIM staff and ATC - as you make good technical process, then next step you will need to do as part of this transformation is upping the responsiveness of the tech team to feedback, or even their availability. Again, my concern is not that stuff (or even my stuff) broke, it's that feedback was given and seemingly ignored, and there's been radio silence just when basic messaging (beyond yours, thank you!) is needed.

Cheers

Luke

[Cor de Bruin]

Good afternoon,

Yesterday I was notified of the problems with our Intergraded PirepSystem not working anymore for VATSIM. As the developer of that great piece of software now fulltime working in Hamburg with Airbus I have a little problem as I am not a programmer. I can do small things with php but  this is not so simple.

So I hope somebody can give me some advice or better the solution as more Virtual Airlines will face this problem. For now I sadly can say only to the members, use a Acars program, pirep manually or change network.

Thank you in advange,

 

Regards,

 

Cor

[Daniel Neugebauer]

If using the proxy is an acceptable interim solution until you find a new developer to properly migrate your software, it is possible to run the proxy on a server as well (the proxy's GUI is optional, it works from CLI). After starting the proxy the URL in your existing PIREP software needs to be changed from http://status.vatsim.net/ to (by default) http://localhost:8080/ and of course you need to test if everything works. This should be a very quick change, maybe the previous developer can have a look at it. 

Please make absolutely sure that access to the proxy server is blocked by a packet filter aka "firewall" and the port is not used for anything else. In particular please make sure there is no accidental publication of the proxy's converted data feed through any web server/proxy sitting in front on or of the same server. For easier configuration of the data proxy (if you need to make any changes) I would recommend to use the proxy's GUI on a local machine, then copy the properties file over to the server as the options cannot be configured from CLI yet.

Note that ATC coordinates are currently still missing (they are no longer provided by VATSIM) but you probably don't need those for PIREPs? I already have an update in work to locate ATC by their callsign from VAT-Spy data but it is not ready for release yet. I haven't checked it so far but if pilots logged in as observers (i.e. the "copilot" in a shared cockpit flight) are seen as ATC observers by VATSIM then they are probably affected by removed coordinates as well (can be seen as online but may always be located at a 0°/0° placeholder).

Edited April 12, 2021 at 02:19 PM by Daniel Neugebauer

[Cor de Bruin]

4 hours ago, Daniel Neugebauer said:

If using the proxy is an acceptable interim solution until you find a new developer to properly migrate your software, it is possible to run the proxy on a server as well (the proxy's GUI is optional, it works from CLI). After starting the proxy the URL in your existing PIREP software needs to be changed from http://status.vatsim.net/ to (by default) http://localhost:8080/ and of course you need to test if everything works. This should be a very quick change, maybe the previous developer can have a look at it. 

Please make absolutely sure that access to the proxy server is blocked by a packet filter aka "firewall" and the port is not used for anything else. In particular please make sure there is no accidental publication of the proxy's converted data feed through any web server/proxy sitting in front on or of the same server. For easier configuration of the data proxy (if you need to make any changes) I would recommend to use the proxy's GUI on a local machine, then copy the properties file over to the server as the options cannot be configured from CLI yet.

Note that ATC coordinates are currently still missing (they are no longer provided by VATSIM) but you probably don't need those for PIREPs? I already have an update in work to locate ATC by their callsign from VAT-Spy data but it is not ready for release yet. I haven't checked it so far but if pilots logged in as observers (i.e. the "copilot" in a shared cockpit flight) are seen as ATC observers by VATSIM then they are probably affected by removed coordinates as well (can be seen as online but may always be located at a 0°/0° placeholder).

Hi Daniel,

I will pass the information through to our ICT manager to see if he can make it work on our server. Thank you and I understand that things are moving forward but sometimes people forget that within the Flightsim community al lot of elderly participating and they wanna hold on to the programs they are used to and know how they work. Personally I find it great to read that somebody still using servinfo. When I started Flightsim 16 years ago was a program you wanted to use.😀

Regards,

Cor

[Alistair Thomson]

1 hour ago, Cor de Bruin said:

sometimes people forget that within the Flightsim community al lot of elderly participating and they wanna hold on to the programs they are used to and know how they work

Whoa! Hold on there, sunshine! :) As someone who has quite a few years behind him, I do know that some of us older types can get a bit set in their ways, but I dislike the assumption you seem to be making, that "they" (ie. folks "of an age") globally cling desperately to their past. That is not a characterisation of older folks: it's a caricature.

It could equally be postulated, as a caricature, that all young folks today think only of themselves and care naught for others, so that when they present the Next New Thing, they are so fixated on its Greatness that they forget the basics of continuity and how important that is to the community. Like Apple did with an upgrade to Final Cut, for which they were rightly pilloried: and like Vatsim did with Json 3, the "fallout" (to use Luke's word) from which is still falling.

That seems to be the basis of Luke's disappointment, and I feel he is justified. Comments from some of the tech department staff have reinforced that justification, denying the appropriateness of temporary fixes which were offered in an attempt to ameliorate their mis-step.

And when the lead developer of the much-acclaimed AFV turns his back on his involvement with any future Vatsim development, we should all pause and think what is happening here.

[Sean Harrison]

Alistair, That isn’t how I took Cors’ post.  There are people of all ages who love change and embrace it with open arms, there are others [of all ages]  whom hate change.  As Tswizzle says it may be love, hate or indifference.....  It happens, and whatever the reason for it, you can always expect to have some fall by the wayside as a result.  

[Alistair Thomson]

10 minutes ago, Sean Harrison said:

Alistair, That isn’t how I took Cors’ post.

Please don't misunderstand. I'm totally behind Cor in his unhappiness about this situation, but maybe I'm sensitised to the age thing, so I thought it would be helpful to point out that the age thing works both ways. That was the only point of my post.

I've seen some pretty derogatory remarks about Vatsim being an "old people network" and I just wanted to say that for me, and possibly for others, the sun doesn't shine out of every youngster, no matter how technically adept they may be. Andreas Kohl welcomes the move to Json because it makes sense, and it truly does. But being a good developer in Vatsim requires more than simply being a good technologist. I feel that the BoG is losing track of that when it allows this nonsense which we have seen reported  in the forums.

[Cor de Bruin]

Hi Allistar,

I apologize if I have offended you. That was by no means the intention, because not every elderly person has problems with change. I do not know in which category I fall because I don't think I am young or old, while my grandchildren consider me old (54).
But from the practice of running a VA for 15 years and where a large number of the 170+ members are close to retirement age (65) or far above,  it is sometimes very difficult for these people to change. We have members who are still flying with FS9 and really don't want to change because it works so well.

And certainly IPS, which enables members to fly on the networks without having to think about it and not have to worry about an extra program on their computer or forgetting to turn it on.
Personally, it doesn't matter to me because I still use an ACARS program, but I do have responsibility towards the members and to say that it no longer works is annoying especially because 
there is no replacement for IPS (at the moment).

Again I apologize if I offended you and have a nice day.

Regards,

Cor

Edited April 16, 2021 at 11:59 AM by Cor de Bruin

[Alistair Thomson]

2 hours ago, Cor de Bruin said:

Hi Allistar,

I apologize if I have offended you.

Actually, I'm not at all offended. You are absolutely correct in the assumption that some folks who are as old as you and I :) are not as adventurous as they once were in their youth, but I also know that some folks who are still afflicted with youth hold the belief that everyone over their age plus ten is past it, and that the experience built up over many (many, many) years is irrelevant in the light of the white heat of their immaturity. Your post gave me the opportunity to say some things about that.

[Andreas Fuchs]

For those using ServInfo: Daniel's app works like a charm with it! Just make sure you download the latest version of his proxy tool (0.80.2 or newer) and configure ServInfo according to the instructions on his page:

Quote

ServInfo

The status proxy needs to be configured as a network, do not configure anything in Proxy settings.

as a custom network

Open Options/Connection Parameters. In section Custom Network Settings select a free slot such as Custom Network 1. Choose a Network name (for example VATSIM Legacy) and enter the proxy address as Locator path url (default: http://localhost:8080/). Confirm the settings with OK.

Use the icon with two networked computers to retrieve data through the proxy; do not click the VATSIM logo.

as "VATSIM"

Close ServInfo and navigate to the application's folder. Open servinfo.ini with a text editor. Find the VATSIM Servers Locator Url setting and change the URL to your server address (default: http://localhost:8080/). Save the file, close the editor and start ServInfo.

Click the VATSIM logo to retrieve data through the proxy.

Probably, option 2 (editing servinfo.ini) will be the best way.

 

By the way, if you would like to download and display VATBOOK data (ATC reservation system for VATSIM), follow these instructions:

 

[Daniel Neugebauer]

1 hour ago, Andreas Fuchs said:

By the way, if you would like to download and display VATBOOK data (ATC reservation system for VATSIM)

Just as an addendum if you use Qutescoop: I missed that it disables the "ATC bookings" option when you switch from VATSIM to "user defined network". If you are missing bookings please make sure to re-enable that option. The setup instructions on GitHub have already been amended.

[Mike Lehkamp]

So as I understand this, and please correct me if I'm wrong, us FS9'ers using Squawkbox, can no longer participate with VATSIM? I have over 1100 hundred hours of flying online with VATSIM. I am not the only one. And please, before anyone suggests SWIFT, understand that it works a 1/3rd of time with many issues. These issues were addressed over a year ago with the actual authors of Swift - Not Andreas. Can anyone recommend a interface to VATSIM for users of FS9 going forward?

Respectfully,

Michael

[Mats Edvin Aaro]

4 hours ago, Mike Lehkamp said:

So as I understand this, and please correct me if I'm wrong, us FS9'ers using Squawkbox, can no longer participate with VATSIM? I have over 1100 hundred hours of flying online with VATSIM. I am not the only one. And please, before anyone suggests SWIFT, understand that it works a 1/3rd of time with many issues. These issues were addressed over a year ago with the actual authors of Swift - Not Andreas. Can anyone recommend a interface to VATSIM for users of FS9 going forward?

Respectfully,

Michael

That is correct, Swift is per now the only way you can use FS9 on the network.

https://www.vatsim.net/pilots/download-required-software

[Daniel Neugebauer]

11 hours ago, Mike Lehkamp said:

So as I understand this, and please correct me if I'm wrong, us FS9'ers using Squawkbox, can no longer participate with VATSIM?

The upcoming changes to the network protocol actually really require a deprecation of all unmaintained clients. This was communicated back in March:

I'd like to point out that the deprecation of legacy pilot & ATC clients has absolutely nothing to do with the data feed discussed in this thread:

The data feed we are talking about here is just a regularly server-side generated dump of the current network state and can be accessed without any login or pilot client via a web URL, it does not allow you to actively connect and participate with the network, it is just used to display global maps of all client activity etc. To be able to distinguish such applications from actual pilot/ATC clients I chose to call those status applications "passive" clients.

It might still be necessary for pilot/ATC clients to read the server information from the data feed to know where to connect to. But, especially without having any insight into the NDA'ed actual client protocol, I don't see any other reason why a pilot client should have to fetch the data feed. The disclaimer of my proxy advises against using it in combination with an "active" client (i.e. pilot or ATC client) just in case there would be some way (unknown to me) by which you could theoretically use the proxy to make a connection with an otherwise prohibited client or otherwise influence how an "active" client participates with the network which would effectively turn a permitted client into prohibited software.

Edited April 30, 2021 at 07:17 PM by Daniel Neugebauer

[Andreas Fuchs]

13 hours ago, Mike Lehkamp said:

And please, before anyone suggests SWIFT, understand that it works a 1/3rd of time with many issues.

You have a very special setup -most users of FS9 have a single computer that is running FS9 and for them it does work reliably. Same for me, although I am using X-Plane 11 99% of the time. FS9 is sooooooo old and I only use it for trouble shooting. I am sorry to say, but you are probably in the position to change something in your setup. Yes, change is not easy in a elaborate home cockpit as yours and I do not envy the task that may lay ahead of you. I am far from being a computer person, even I managed to change to another simulator platform 😉

[Daniel Neugebauer]

Release 0.90 of the legacy proxy is now available:

Quote

New Features

• missing controller coordinates are now looked up from a VAT-Spy database ("Station Locator" feature)
  •  a copy of the VAT-Spy Client Data Update Project is included internally with the proxy for easy use
  • a warning will appear at startup if the internal database is older than 270 days (check can be disabled in configuration)
  • alternatively, the proxy can be configured to read the database from an external directory

Improvements

• base URL is now guarded by a checkbox to prevent accidental modification
• warning if JAR is started from a directory/class path which cannot be read properly by Java
• added disclaimer to data file output
• added option to log errors when parsing upstream data

Fixed Bugs

• proxy restarts in quick succession were previously blocked on Linux and should now work

Searching coordinates using VAT-Spy works completely offline and without having to install VAT-Spy (the database is included within the proxy application itself). Having controller coordinates means that QuteScoop is now able to display labels for center sectors again. The configuration dialog provides several options to fine-tune the search. Those options are explained in the user manual which was split from the readme file for better readability.

The Station Locator feature was supposed to offer an option to also use the list of online transceivers which is currently being propagated by VATSIM as the official method of computing the otherwise missing controller coordinates. However, I already had the VAT-Spy-based solution implemented before that method was made official and for me it has worked remarkably well so far. I just wanted to get the one nicely working feature released before so that the release would not get delayed even longer than it already had been.

The roadmap for a future "version 1.0" (whenever that will be) currently includes online transceivers, general bug fixes, better usability (minimize on start, minimize to system tray) and (hopefully) more CLI options if that's of any actual use to somebody. I cannot commit on any of those features yet nor can I provide any time estimates - I mainly worked on the proxy during my vacation and public holidays with bad weather; as the weather gets better (and other projects are calling as well 😉 ) it may take a while until I can continue with those features.

Nevertheless, I plan to provide monthly updates to include the latest updates from VAT-Spy database from now on. The proxy does not "phone home" which means there is no automated update notification. Please check regularly for new releases. As AIRAC/VAT-Spy updates are frequent and predictable I will not post to this thread when I release a new version that only updates the database. If I release any further changes (features or fixed bugs) I will continue to post here as I release such updates.

[Daniel Neugebauer]

Release 0.95 is now available.

If coordinates cannot be substituted from VAT-Spy data, the proxy will now by default fall back to the online list of Audio for VATSIM transceivers. The options allow alternate strategies to be selected for the Station Locator: You can also revert back to the old behaviour of just using the VAT-Spy database or change to only use online transceivers to substitute missing coordinates. For optimization, location data will be cached on several levels and thus may not by synchronous/consistent to every request (which cannot be expected from the original data sources anyway). If you change any Station Locator options you may have to restart both the proxy as well as your status client (QuteScoop, ServInfo, ...) to immediately see the data according to your newly selected options.

The Station Locator and its options are also described in full detail in the user manual.

As previously announced, I publish updates for the included VAT-Spy database regularly which I may not announce in this thread. You can always find the latest release on the release overview.

[Daniel Neugebauer]

As already reported on IT media today, a widely used Java library (Log4j, CVE-2021-44228) had a critical security vulnerability which can be abused for remote code execution. That library is also used by the Legacy Proxy and could, in worst case, lead to malware downloads (viruses, trojan horses etc.). I was able to reproduce the general issue but cannot tell if an attack is realistic (depends on too many variables including unknown specifics of the VATSIM ecosystem). An attack vector seems at least plausible, so full caution is advised.

All versions before the just released 0.95.5 are vulnerable. Please update as soon as possible to 0.95.5 and delete all copies you might still have lying around to avoid running an affected version by accident.

As a precaution I deleted all JAR files of older releases from GitHub. Unfortunately I did not implement any update check due to privacy concerns, so this is the only way I can inform about the need for an update. In case you are immediately aware of users who cannot be reached by this message please inform them about the need for an immediate update.

Full details are available in the release notes.

Edited December 11, 2021 at 12:22 AM by Daniel Neugebauer