VATSIM Stats API

[Benjamin Gustafson 1212890]

Since it seems that getting a decent API to replace our datafeed is not going to happen anytime soon, is there any potential of an API being set up for us to pull data from the VATSIM Stats database? Specifically looking to compute controller hours, as well as which position they're logged in as. Right now we have to rely on the datafeed which gives very inaccurate numbers (since we can't tell exact time of disconnect). It appears the VATSIM stats database actually has accurate, to the minute data. It would be great if our websites could pull this.

 

Thanks.

[Colin Schoen]

It has been asked before. The response has historically been that VATSIM has no plans to release a public API for the statistics center.

Edited August 7, 2015 at 05:48 PM by Guest

[Miguel Frias]

I've been asking for that practically since day 1 of my VATSIM programming era. It has always been denied.

 

This was because historically, the stats were extremely slow in the past. They have been modified recently (latest year) and I believe a step forward could be made into getting an API. The API would allow for very precise queries that could be made to avoid getting endless rows of data.

 

It is a matter for Kieran to decide upon and eventually implement. The main issue nowadays (I think) is actually finding the time to implement an API.

[Kieran Hardern]

Broadly speaking there is no current plan to implement such an API. There are plans to provide more data and in better formats, though this is dependent upon a lot of work getting done before that can start. As Miguel says, available time is key - though to clarify, this isn't that I'm not available (or not bothered), it means that I have to balance a lot of day to day maintenance/jobs with dev work.

 

Directly relating to your request, in my eyes you are essentially saying 'I want to download VATSIM stats into my own database'. (hours and list of callsigns is pretty much everything relating to a member). Such an extensive API is unlikely ever to be available and it defeats the purpose of having the stats centre altogether. There are many factors at play there relating to the old data, the abuse of such a system and of course why this would be useful in the first place.

 

Perhaps you could suggest what you're trying to achieve here, so that other developers can pitch in and discuss what they're considering doing (or having trouble doing). I expect there are many ways to achieve the same goal and I presume that a lot of what people are trying to achieve can be better done through the stats centre itself - not an API.

[Miguel Frias]

Benjamin,

 

Now that you have an answer from Kieran, what most people do is get the data from the data servers and interpret it. As you said => yes this renders a not-so-accurate amount of information. But you do have the logoff time, just depends on how you program it. For instance, I have something programmed that does this:

- check the data feed every 2 minutes. If the HTML header is the same, ignore (no change yet); otherwise download the newest data feed

- check data with my own database that saves atc comeulative logged in time:

pseudo-code below

    FOREACH data_feed_atc AS 'a' DO:
      IF (last feed contains 'a') [or alternatively] IF (SQL "NOW()-last_seen" < 3) THEN
        // user was online just now so add two minutes to total logged on time
        SQL "update users set total_time=total_time + 2 where CID='a';"
      ELSE
        // add new connection but do not add 2 minutes
        SQL "insert into [OR] update users set last_seen = NOW() where CID='a';"
      ENDIF
   ENDFOR

 

If the ATC went offline, there is no code to add the minutes so at most he will have logged off in the last 1-2 minutes. Not that accurate -- not really a problem.

Of course this is all very nice when the network is working probably. If your site goes down or if the data servers go bezerk, you will missing data. But that's life.

[Benjamin Gustafson 1212890]

Kieran,

 

Most ARTCCs track how much their controllers have controlled on a month by month basis and display this. It's usually broken down by position as well.

 

And yes, I too poll the stats server and if they disappear from the last poll, consider that the log off time. I poll this every five minutes (since I believe that's the frequency the data feed as published), and yes, it can provide a 4-5 minute error. If a controller logs thirty sessions in a month, that is a potential error of over two hours.

 

Many ARTCCs like to recognize those who control the most, and a +/- of that scope is unacceptable, requiring our senior staff to manually tabulate each controllers time from the VATSIM stats page which is rather time consuming.

[Miguel Frias]

Benjamin, it's around 2 to 3 minutes. If you do it like I suggested you'll get the maximum update possible:

 

POLL every 60 seconds but only get the HEAD of the HTML to compare. if equal then don't donwload further. Otherwise, new data is ready for download.

 

And... very important... go round-robin or random throughout the servers. Don't stick to one so that the load is balanced.

[Kieran Hardern]

Kieran,

 

Most ARTCCs track how much their controllers have controlled on a month by month basis and display this. It's usually broken down by position as well.

 

I'm aware, though the majority that I've seen do this in a "top 5/top 10" way, often keeping those numbers regularly up to date. I can't see how repeated downloads over the month of all of your member's connections and then filtering by callsign is a better way than tracking the network callsigns and adding up the totals. The only advantage that would give you would be a few seconds accuracy at the cost of that huge amount of processing.

[Benjamin Gustafson 1212890]

Kieran,

 

Most ARTCCs track how much their controllers have controlled on a month by month basis and display this. It's usually broken down by position as well.

 

I'm aware, though the majority that I've seen do this in a "top 5/top 10" way, often keeping those numbers regularly up to date. I can't see how repeated downloads over the month of all of your member's connections and then filtering by callsign is a better way than tracking the network callsigns and adding up the totals. The only advantage that would give you would be a few seconds accuracy at the cost of that huge amount of processing.

 

Except now we are having to poll every two to five minutes, p[Mod - Happy Thoughts] through the entire stat feed, find our positions and record them (losing accuracy per connection) versus say polling nightly and getting up to date values once a day. If we had access to the stats we wouldn't need to poll so often, we are doing that because that's the only way to get the information at all.

 

If we don't want to provide the information, that's fine, but its a whole wealth of accurate information that's stored on the stats historian and only revealed via a web front end, which makes the data not easily as useful and much slower to tabulate.

 

If we had web controls where ARTCC staff could build views, or view activity for all members of the ARTCC, etc and could view all of the data for the ARTCC via one web interface, instead of going CID by CID and grabbing the data and adding up just the positions for our ARTCC (since controllers visit elsewhere and we don't count those hours), that would be great too.

 

It just seems silly to have this wealth of information available and to be doing very little with it.

[Arne Hunsdiek]

Benjamin,

 

if you just want to count the ATC hours of your members (without the used stations) how about getting the hours with the idstatus tool?

 

Run it every 1st at 12am and make your own calcualtions.

 

Just a thought

[Daniel Hawton]

Benjamin,

 

if you just want to count the ATC hours of your members (without the used stations) how about getting the hours with the idstatus tool?

 

Run it every 1st at 12am and make your own calcualtions.

 

Just a thought

 

He's trying to get activity information. atctime through idstatus wouldn't display information to be able to determine if a controller met activity requirements within Benjamin's ARTCC as there is no way to differentiate between them controlling in Denver vs controlling in London (for example).