Aidan Stevens Posted April 12, 2020 at 07:10 PM Posted April 12, 2020 at 07:10 PM We've heard your requests, and we understand that the current, colon-delimited datafile has been a pain to deal with and acts as a barrier to entrance for many new developers. As such, we have rewritten the dataserver which comes with a brand new JSON file that includes a 1:1 field mapping with the colon-delimited file. Documentation for this new file can be found here. This is officially v1.0 and is now considered in production. We are also doing our best to embrace open-source and community involvement with our technology, so you can find the code for the new dataserver on our GitHub, here. Please feel free to create issues and open pull requests. Please note that this does not mean we are deprecating the old file! We will continue to maintain that file for years to come to ensure that all of the current VATSIM tools you know and love continue to function properly. Please see the latest tech blog post for an update on the legacy data feed. Thanks for reading, and I hope you enjoy! Regards, 7 4 Link to comment Share on other sites More sharing options...
Board of Governors Gunnar Lindahl Posted April 12, 2020 at 08:49 PM Board of Governors Posted April 12, 2020 at 08:49 PM Great work team - thank you! GUNNAR LINDAHL [email protected] Link to comment Share on other sites More sharing options...
Luke Kolin Posted April 13, 2020 at 09:48 PM Posted April 13, 2020 at 09:48 PM (edited) Is there any metadata in the feed as to when it was generated, or is it real-time? Cheers! Luke PS: Do you support TLS on the data server? Edited April 13, 2020 at 10:10 PM by Luke Kolin ... I spawn hundreds of children a day. They are daemons because they are easier to kill. The first four remain stubbornly alive despite my (and their) best efforts. ... Normal in my household makes you a member of a visible minority. Link to comment Share on other sites More sharing options...
Mateusz Zymla Posted April 13, 2020 at 11:49 PM Posted April 13, 2020 at 11:49 PM Would love to know it as well, if it's live, or at least close-to-real (as previous data txt distributed is basically too old for some real-time managing apps), it's a game-changer. Mateusz Zymla - 1131338 VATSIMer since 2009, IRL pilot rated. Link to comment Share on other sites More sharing options...
Nick Harasym Posted April 14, 2020 at 03:35 AM Posted April 14, 2020 at 03:35 AM 4 hours ago, Mateusz Zymla said: Would love to know it as well, if it's live, or at least close-to-real (as previous data txt distributed is basically too old for some real-time managing apps), it's a game-changer. Maybe Kafka will be a thing, more info shortly??? 1 Nick Harasym VATSIM Senior Developer Team Lead, Infrastructure [email protected] Link to comment Share on other sites More sharing options...
Mateusz Zymla Posted April 14, 2020 at 12:49 PM Posted April 14, 2020 at 12:49 PM 9 hours ago, Nick Harasym said: Maybe Kafka will be a thing, more info shortly??? any further reading for now 😉 ? Mateusz Zymla - 1131338 VATSIMer since 2009, IRL pilot rated. Link to comment Share on other sites More sharing options...
Nick Harasym Posted April 14, 2020 at 01:06 PM Posted April 14, 2020 at 01:06 PM 15 minutes ago, Mateusz Zymla said: any further reading for now 😉 ? Basically, write a consumer client for a Kafka server and you will be able to consume a real time feed of data shortly. Most languages already have support for Kafka https://cwiki.apache.org/confluence/display/KAFKA/Clients Nick Harasym VATSIM Senior Developer Team Lead, Infrastructure [email protected] Link to comment Share on other sites More sharing options...
Andrew Ogden Posted April 14, 2020 at 01:20 PM Posted April 14, 2020 at 01:20 PM (edited) Here's an open source one I've purpose written if you're interested for now: https://github.com/andrewogden1678/VATMap_Kafka Edited April 14, 2020 at 01:21 PM by Andrew Ogden Andrew Ogden Gander Oceanic OCA Chief Vancouver FIR Senior Instructor Visit us: https://ganderoceanic.ca Contact: [email protected] Link to comment Share on other sites More sharing options...
Nick Harasym Posted April 14, 2020 at 01:34 PM Posted April 14, 2020 at 01:34 PM The above example works however it was written for our "closed beta" of the Kafka feed. Building up the consumer will be the same however the data will change slightly. Nick Harasym VATSIM Senior Developer Team Lead, Infrastructure [email protected] Link to comment Share on other sites More sharing options...
Mateusz Zymla Posted April 14, 2020 at 01:34 PM Posted April 14, 2020 at 01:34 PM (edited) Yeah, we are thinking about introducing an A-CDM plugin/software/website (not decided yet), and for now on there are no publicly available any complete soft for that, so we would be forced to extract data from EuroScope. We don't know how we'd like it to look like, but extracting data directly from VATSIM data servers would make our life easier for sure 😉 Edited April 14, 2020 at 01:35 PM by Mateusz Zymla Mateusz Zymla - 1131338 VATSIMer since 2009, IRL pilot rated. Link to comment Share on other sites More sharing options...
Luke Kolin Posted April 14, 2020 at 04:47 PM Posted April 14, 2020 at 04:47 PM 3 hours ago, Nick Harasym said: The above example works however it was written for our "closed beta" of the Kafka feed. Building up the consumer will be the same however the data will change slightly. Is it too late to get access to this? Cheers! Luke ... I spawn hundreds of children a day. They are daemons because they are easier to kill. The first four remain stubbornly alive despite my (and their) best efforts. ... Normal in my household makes you a member of a visible minority. Link to comment Share on other sites More sharing options...
Luis Henrique Posted April 14, 2020 at 11:09 PM Posted April 14, 2020 at 11:09 PM Thank you so much! Just a random simmer... Link to comment Share on other sites More sharing options...
Nick Harasym Posted April 15, 2020 at 03:31 PM Posted April 15, 2020 at 03:31 PM 22 hours ago, Luke Kolin said: Is it too late to get access to this? Cheers! Luke For the current Kafka offer that is floating out around there, yes we won't be giving credentials for it till we have the production one ready. Nick Harasym VATSIM Senior Developer Team Lead, Infrastructure [email protected] Link to comment Share on other sites More sharing options...
Luke Kolin Posted April 18, 2020 at 11:04 PM Posted April 18, 2020 at 11:04 PM Thanks. Any chance we can get an 'effective date' field in the data server JSON feed? Cheers! ... I spawn hundreds of children a day. They are daemons because they are easier to kill. The first four remain stubbornly alive despite my (and their) best efforts. ... Normal in my household makes you a member of a visible minority. Link to comment Share on other sites More sharing options...
Mario Trunz Posted April 26, 2020 at 07:14 PM Posted April 26, 2020 at 07:14 PM Is there a way we can get the last 10-15 flights based on the VATSIM User ID? meteoNautics We are a community that provides tools for Global Flight Simulation Operationsmeteonautics.com Link to comment Share on other sites More sharing options...
Jason Cochran Posted May 4, 2020 at 12:04 PM Posted May 4, 2020 at 12:04 PM +1 for Kafka. Very nice, guys. Though we have struggled with what seems to be higher than normal CPU consumption of the brokers, we are currently using Kafka at scale in a very large enterprise... and it's super dependable and mostly runs itself. Link to comment Share on other sites More sharing options...
Martijn Rammeloo Posted May 22, 2020 at 07:55 AM Posted May 22, 2020 at 07:55 AM Is there something going wrong with the Charset?Image Link to comment Share on other sites More sharing options...
Matt Bozwood-Davies Posted May 24, 2020 at 12:32 AM Posted May 24, 2020 at 12:32 AM On 4/19/2020 at 12:04 AM, Luke Kolin said: Any chance we can get an 'effective date' field in the data server JSON feed? This was pushed up in a new version the other day with a non-breaking change. There is a new top-level bject "general" which contains the following properties version - incremental changes reload - not sure what this was used for in the txt file but we left it there for similarity update - same format as the txt timestamp update_timestamp - UTC timestamp iso format connected_clients unique_users "general": { "version": 1, "reload": 1, "update": "20200524002549", "update_timestamp": "2020-05-24T00:25:49.9636563Z", "connected_clients": 1168, "unique_users": 1105 } --- On 4/26/2020 at 8:14 PM, Mario Trunz said: Is there a way we can get the last 10-15 flights based on the VATSIM User ID? Please see the API for this: Link to comment Share on other sites More sharing options...
Luke Kolin Posted May 25, 2020 at 08:28 PM Posted May 25, 2020 at 08:28 PM Fantastic - thank you! Cheers Luke ... I spawn hundreds of children a day. They are daemons because they are easier to kill. The first four remain stubbornly alive despite my (and their) best efforts. ... Normal in my household makes you a member of a visible minority. Link to comment Share on other sites More sharing options...
Martijn Rammeloo Posted June 3, 2020 at 06:34 PM Posted June 3, 2020 at 06:34 PM (edited) @Aidan StevensThe datafeed (JSON) seems to be serving outdated data. Screenshot was taken seconds ago, 1833Z Edited June 3, 2020 at 06:34 PM by Martijn Rammeloo Link to comment Share on other sites More sharing options...
Joshua Channin Posted July 19, 2020 at 05:00 PM Posted July 19, 2020 at 05:00 PM (edited) Is it intended that the resource is blocked by CORS for client side javascript requests as an example? "Access to fetch at 'http://cluster.data.vatsim.net/vatsim-data.json' from origin 'http://localhost:8080' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled." @Aidan Stevens Edited July 31, 2020 at 11:59 PM by Joshua Channin Link to comment Share on other sites More sharing options...
Joshua Channin Posted August 23, 2020 at 03:37 PM Posted August 23, 2020 at 03:37 PM @Aidan Stevens bump Link to comment Share on other sites More sharing options...
Nestor Perez Posted August 23, 2020 at 03:44 PM Posted August 23, 2020 at 03:44 PM Hey Joshua, what would be the usecase for us to allow CORS requests? I think we’d rather have all requests proxied (and cached) through each application’s server; thus reducing the final load and cost of VATSIM’s free-to-use services. Me. Link to comment Share on other sites More sharing options...
Joshua Channin Posted August 26, 2020 at 09:50 PM Posted August 26, 2020 at 09:50 PM (edited) That makes sense Nestor. Mostly wanted to confirm the behavior was intended. I do think from a developer advocacy level, this restricts working with VATSIM to those developers able and willing to support their frontend client with an application server caching the data rather than a frontend client only and the caching occurs at the VATSIM level. But all things considered with VATSIM's free-to-use services as you mentioned, I concur with your thoughts. Edited August 26, 2020 at 09:51 PM by Joshua Channin Link to comment Share on other sites More sharing options...
Nestor Perez Posted August 26, 2020 at 10:16 PM Posted August 26, 2020 at 10:16 PM Forgot to add: If there is a legitimate usecase for which it is the only option available, I'm sure everyone in the team is looking forward to learn more about it 🙂 Me. Link to comment Share on other sites More sharing options...
Recommended Posts