VATEUD API

[Svilen Vassilev]

Just for clarity, now that all planned endpoints are in place: here's a breakdown of the current caching times on the API:

 

Backend

 

No caching on the backend! Data is pulled from the DB on each request.

 

Frontend

 

• All EUD members: html responses: no caching; json, xml and csv responses: 3 hours
  
• vACC members: html responses: no caching; json, xml and csv responses: 4 hours
  
• Members by country: html responses: no caching; json, xml and csv responses: 4 hours
  
• Members by rating: html responses: no caching; json, xml and csv responses: 4 hours
  
• Single member details: all formats: 2 hours
  
• Member validation endpoint: no caching
  
• vACC profiles: all formats: 2 hours
  
• Staff member listings: all formats: 5 minutes
  
• Who's online data: all formats: 5 minutes
  
• Notams: all formats: 12 hours
  
• Charts: all formats: 4 hours
  
• ATC frequencies: all formats: 2 hours
  
• Events calendar: all formats: 10 minutes
  
• Airport profiles: all formats: 10 minutes

[Svilen Vassilev]

Update based on feedback:

 

• Creating a weekly event will now only create 26 instances of the event (6 months ahead), down from 52 instances
  
• Editing a weekly event instance will now cause the changes made to be propagated to all other instances of that weekly event if the "weekly" flag is set to true (checkmark selected)
  
• If the weekly flag is not ticked when editing an instance of an event, the changes made will only affect the edited instance and will not propagate to other instances.
  
• The HTML listing of events on the frontend now only displays future (upcoming events). The json, xml, csv and ics/ical formats still give the full data
  
• Events frontend sorting: the nearest upcoming events are now on top, with later ones below based on starting dates/times. All columns are sortable in both directions (click on column header)
  
• Fixed pagination bugs for events and staff members on the HTML frontend

[Todor Atanasov 878664]

Do you sleep???

 

P.S. On the Freq html page, some of the longer descriptions are being cut http://api.vateud.net/frequencies/BULG

[Svilen Vassilev]

Do you sleep???

 

I do

 

 

On the Freq html page, some of the longer descriptions are being cut http://api.vateud.net/frequencies/BULG

 

Yup, unfortunately the DB field that stores the names is limited to 50 chars and we can't change it right now for legacy/compatibility reasons. I've opened a task on your behalf for Miguel (VATEUD8) to trim the longer names a little bit as a workaround: http://tasks.vateud.net/tasks/440

[Miguel Frias]

Bulgaria issues have been corrected.

[Svilen Vassilev]

Thank you, Miguel!

[Todor Atanasov 878664]

Bulgaria issues have been corrected.

Thanks guys

[Svilen Vassilev]

Update based on feedback:

 

Custom airport charts

 

Some vACCs prefer to override the real world charts with their own renditions for various reasons: VATSIM-specific procedures, scenery compatibility, advanced customization, etc.

 

The API can handle those overrides in the following manner:

 

• vACCs supply a custom charts index as a CSV file in the legacy VATEUD charts format, as seen for example here
  
• CSV format description: the field separator is "|". The only relevant data used by the API to p[Mod - Happy Thoughts] the file is: ICAO code at position `0`, chart name at position `3` and chart url at position `10`. The rest is irrelevant and can be left blank when creating the CSV file.
  
• vACC staff members with API backend access create a "Custom Charts Source" entry on the backend (vACC Staff Zone) defining the URL for the custom CSV listing (pictured below)
  
• That's it from user perspective, the rest is handled automatically by the API application

 

 

 

Application logic

 

• The API pulls the vACC Custom Chart Source listings once per day, p[Mod - Happy Thoughts]s and injects them in a local DB of custom charts
  
• Whenever an API user sends a request for charts, the API first looks into the custom charts DB to find any matches based on the requested icao code. If matches are found, it sends out the custom listings, if matches are not found it proceeds to query and pull the RW charts for that airport
  
• There are no differences in the request and response syntaxes between custom and RW charts: the same calls and the same response logic and formats apply, i.e. you get json, xml, csv or html responses in both cases

 

Examples:

 

vACC Germany prefers to use custom charts and below you can see the listings let's say for Berlin Tegel and Frankfurt airports:

 

http://api.vateud.net/charts/eddt

http://api.vateud.net/charts/eddf

[Even Rognlien]

Looks like the API website is down? Any idea when it will be up and running again? The VATEUD API is really great!

[Adam Trzcinski]

It should be back up

[Even Rognlien]

Great!

[Svilen Vassilev]

Updated, based on feedback

 

The single member details endpoint (section J, requests of the type: http://api.vateud.net/members/id/[vatsim-cid]) has been augmented to fallback to obtaining the public cert record (idstatus.php) for the requested CID and display the available data from there if the supplied vatsim CID does not match any internal VATEUD records. This allows API users to individually query members not belonging to VATEUD (or not [Mod - Happy Thoughts]igned to a division).

 

Also some bugfixes and performance improvements.

 

Also, I'm trying to eradicate all 500 code returns, no matter how ludicrous the request. The plan is to have the API always return 20x codes for successful calls and 40x codes for client-side errors and in no circomestances under normal operation return a 50x code. Thus all 50x returns shall be treated as bugs: currently those are either legitimate bugs, or internal application exceptions that are not caught and rescued gracefully. I've traced and removed some of those instances, but there are more and if you notice a 500 return anywhere, please send me request URL so that I can look at it and make amendments.

[Svilen Vassilev]

And another requested update:

 

vACCs now have the ability, apart from blanket-replacing all charts for a given airport with custom versions, to selectively add single (individual) custom airport charts, that will be used to augment the RW listings obtained form our upstream sources.

 

This is done via the backend interface, the relevant menu entry is vACC Staff Zone > Individual custom charts and it looks like this:

 

 

Example: vACCBiH maintains a custom unofficial parking chart and guide for Sarajevo Airport (LQSA), but otherwise the vACC does not maintain custom charts. It's now possible to add this single extra custom chart to the RW chart listings already provided by the API by creating an Individual custom chart record via the backend.

 

To summarize the charts logic available in the API:

• We have automatic Real World chart listings, obtained from upstream sources, matching the respective country's official AIP publications. Automatically updated, no maintenance required
  
• We have the ability to selectively override misspelled, misleading or incorrect chart names (from the RW listings) via the administrative backend interface (in the docs that's section H, paragraph "Overriding chart titles")
  
• We have the ability to blanket replace all charts for a given airport with a custom stack, created and maintained by the respective vACC, and fed to the API through a CSV file hosted by that vACC (in the docs, that's section H, paragraph "Custom airport charts")
  
• We have the ability to augment the RW chart listings for any airport with individual custom charts, created and maintained by the vACCs, and fed to the API via records, editable from the admin backend interface (in the docs that's section H, paragraph "Individual Custom charts")

 

Enojy

[Svilen Vassilev]

Update:

 

Q. ATC Bookings and m[Mod - Happy Thoughts] bookings

 

Individual ATC bookings

 

Registered backend members now have the ability to create, edit and delete ATC bookings that are synchronized with the VATBOOK service (vroute, etc). Look for the "ATC Bookings" entry in the "Events and bookings" menu.

 

Users can also update or delete existing bookings, but only if they were initially created by themselves.

 

M[Mod - Happy Thoughts] bookings

 

When needed, backend users can also create m[Mod - Happy Thoughts] bookings (multiple bookings at once) by uploading a CSV file with the desired bookings details. Look for the "M[Mod - Happy Thoughts] Bookings" entry in the "Events and bookings" menu.

 

The following rules apply:

 

• The CSV file should be UTF encoded (Unicode). Legacy latin, etc encodings will work as well granted that non-latin characters are avoided
  
• The CSV file should have 4 entries per line, separated with commas `,`. Those entries in their respective order are: `Controller name and surname`, `ATC position`, `Starting date and time of the ATC session`, `Ending date and time of the ATC session`. All times are zulu (UTC).
  
• The m[Mod - Happy Thoughts] created bookings can later be individually edited or deleted if needed, but only by the user who originally created the m[Mod - Happy Thoughts] booking
  
• Both date/time entries in the CSV file should have the following syntax: `YYYY-MM-DD HH:MM:SS`

 

Example CSV file syntax:

 

    Svilen V[Mod - Happy Thoughts]ilev,LQTZ_APP,2013-12-19 13:00:00,2013-12-19 15:00:00
   Mirza Ibrahimovic,LQBK_APP,2013-12-19 14:00:00,2013-12-19 16:00:00
   Nikola Tutoric,LQMO_APP,2013-12-19 15:00:00,2013-12-19 17:00:00
   Adam Stanojevic,LQSA_APP,2013-12-19 18:00:00,2013-12-19 21:00:00

 

Initial post has been updated accordingly. Enjoy

[Even Rognlien]

Hello again! In the XML docomeent all the cached charts have the same URL, for example the ENGM charts: https://charts.aero/airport/ENGM instead of a direct link to the actual docomeent like it was before. Any chance these will be back?

[Svilen Vassilev]

Hello again! In the XML docomeent all the cached charts have the same URL, for example the ENGM charts: https://charts.aero/airport/ENGM instead of a direct link to the actual docomeent like it was before. Any chance these will be back?

 

Hello! Just to clarify: all formats provide the same data. Removing the direct links to the cached files was an intentional change on our side, due to the inconsistency of charts.aero caching: recently they seem to randomly cache some charts, skip others, even though it's the same airport, or include dead links.

 

Based on that, I'm more inclined to deprecate the cached URLs, as they don't really provide additional user value.

[Even Rognlien]

Thanks for your reply!

 

You see, I have had some problems with my application around 22:00z-00:00z. It cannot find the charts. I have done some troubleshooting and it looks like the URLs coming from Eurocontrol in the XML-docomeent expire around this time. However, on your website everything is working fine. It looks like the XML-docs isn't 100% synced with your website.. (?)

 

Even

[Svilen Vassilev]

All right. The temporary discrepancies you see between the XML data and the HTML rendered data are due to server-side caching on our end. Even though , as mentioned in my previous post, the data is exactly the same among all formats (those are merely a presentation layer), we do apply server side action caching for nearly all endpoints, as docomeented here. The charts data in particular had a rather large expiration time of 4 hours.

 

So what must have been happening is that some chart links have expired in between the server-side cache refreshes. It's impossible to predict which chart links for which airports will change at what times. The only way to make sure you'll have fresh and valid data on each request is to remove the charts caching altogether, but that's a "no go".

 

What I did was reducing the cache expiration time for charts in half: from 4 to 2 hours. That should reduce the probability of having broken links by 50%. Let me know how it goes.

 

Also added table of contents to the docomeentation at http://api.vateud.net/ as it was getting too long to navigate, even for me

[Even Rognlien]

Thanks Svilen! It still seems to happen - at the same time every night. As I'm writing this post (around 0030z) almost all European airports have broken links... However, the waiting time for the links to get updated is probably shorter now

 

EDIT: Charts back at 0200z +- 5 minutes

[Svilen Vassilev]

Thanks for the feedback, Even. When this happens next time, let's try to check if charts.aero has working URLs on their site for the affected airports. If yes, then it's a matter of timing/tweaking the sync and cache on our end, which I can do. If at the same time however charts.aero links are also broken, due to recycling/refreshing, then there's nothing we can do on our end. I'll monitor the API the next few nights to try and catch this period.

[Daniel Hawton]

As of now, Italy's chart links are outdated compared to the current links provided by charts.aero.

[Svilen Vassilev]

As of now, Italy's chart links are outdated compared to the current links provided by charts.aero.

 

That's a bit general statement Please be as precise as you can when making reports and give me as many details as you can. Does outdated mean expired, unreachable URLs or different chart versions or same chart versions on different URLs? I just checked at random LIRF, LIML, LIMJ and LIPZ: there's 100% match between the API and charts.aero. Of course, I'm responding 1 hour after your post, so if it was a caching issue, the cache may have been rebuilt since.

 

I was also monitoring the API tonight to investigate Even's report: indeed at 0000z the chart links for several countries expired. Not all countries were affected and I didn't have the time to manually check each country, but from those I did check about half were still working and half were missing (including UK). A parallel trace to charts.aero showed the urls have expired there as well, so I presume that's the time they run their resync and rebuild their URLs. The links were rebuilt at charts.aero at various times for different countries, but all the missing ones were back at around 0100z, after approx 1 hour. There's nothing much I can do to offset charts.aero maintenance windows, so I'll just force an API charts cache cleanup at 0100z every day to make sure that no expired links are served on our side by the time the new URLs are available and will further reduce the charts cache expiration time to 1 hour (down from 2 hours). Let's see how this plays.

[Daniel Hawton]

As of now, Italy's chart links are outdated compared to the current links provided by charts.aero.

 

That's a bit general statement Please be as precise as you can when making reports and give me as many details as you can. Does outdated mean expired, unreachable URLs or different chart versions or same chart versions on different URLs? I just checked at random LIRF, LIML, LIMJ and LIPZ: there's 100% match between the API and charts.aero. Of course, I'm responding 1 hour after your post, so if it was a caching issue, the cache may have been rebuilt since.

 

The links served were invalid and matched the same from the previous date (outdated). Other than that, I don't have enough information to tell you anymore additional. It may have just been an out of sync Italy section (I checked 3 different, LIRF LIML and LIPZ and ran into the same issue on all, a 404 error from EuroControl with an invalid charts.aero cache URL [pointed to the airport itself on their site]).

[Svilen Vassilev]

Update: addressing charts sync window

 

As described above, the maintenance window for charts.aero rebuilding their links database is every day approx between 0000z and 0100z. The duration and exact times of this window vary from country to country. During that time the old links for some countries are no longer working and there aren't any new ones yet. To mitigate the impact of this data-starvation on the EUD API, the following changes were made:

 

• File-level caching: Cached chart URLs are once again active. Not all charts get cached, but for those that do, the cached url's will provide a fallback plan. Charts that don't have a cached version will have a nil value for the cached url field.
  
• Action-level caching: The response caching time for the charts endpoints has been further reduced to 1 hour (down from 2) to minimize the chance of stale data being served after the re-sync window period
  
• Cache sweeping: Local action cache is now also force-purged daily at 0105z to ensure fresh data is served for charts shortly after the re-sync window closes.

 

Enjoy!

[Even Rognlien]

Great Svilen, thanks!