METAR Service for Scripts

[Mike Evans]

Good Day All...

 

I'm looking at the logs, and seeing lots of different creative ways to retrieve the METAR data from VATSIM..

 

If you are using the old www.vatsim.net/data/search_metar.php?id=ICAO feed, please move to metar.vatsim.net/ICAO immediately.. the old search_metar.php will be going away very shortly...

 

Additionally, if you need ALL the metars, or a large number of them, please retrieve www.vatsim.net/data/metar.txt

 

This file contains all metars currently active in the network.

 

Please ONLY retrieve this file once every 15 minutes.. if you retrieve any sooner than that, your site WILL be banned..

 

Is there any other functionality in this area that would help you?

[Gerry Hattendorf 935415]

Hey Mike,

 

Just one thing, I have a WX page that's called from a link on our home page. There is no cron, however I can't throttle users selecting this page as it's a link on our homepage. I hate to think that I need to create a table and cron to store this data locally, but I don't want our site banned because 10 users all wanted to see the metars in a short time.

 

Other ideas perhaps?

[Anthony Lawrence]

Hey Mike,

 

Just one thing, I have a WX page that's called from a link on our home page. There is no cron, however I can't throttle users selecting this page as it's a link on our homepage. I hate to think that I need to create a table and cron to store this data locally, but I don't want our site banned because 10 users all wanted to see the metars in a short time.

 

Other ideas perhaps?

 

Download the result and store it in a file locally? Requires very little change to your code that way - other than the check to see if it's x minutes out of date

 

Regards,

A...

[David Klain 874106]

That's exactly how I process VATSIM data as well as weax. Download file and save it locally every 15 min. Every time a person makes a data request on the site, we pull and process from the local copy instead of downloading a new one. Provides much faster response as well since the file is on the server...just ensure you point to its local address and not an http: address!

 

Dave

[Gerry Hattendorf 935415]

Thanks guys, I'm working with some VERY OLD code, and the page we have is just a stupid anchor to the vatsim site. I guess I'll busy for the next day to update my database, and my scripts.

 

BTW Dave, what is "weax"?

[Thomas Mathieu 998318]

So we would need to setup a cron job to take down the file locally and present it on the local server every 15 minutes, correct? I think I understand.

[Anthony Lawrence]

Thanks guys, I'm working with some VERY OLD code, and the page we have is just a stupid anchor to the vatsim site. I guess I'll busy for the next day to update my database, and my scripts.

 

Nobody said you have to update the database So if you're wanting to cut corners you can do as suggested and save the file locally

 

So we would need to setup a cron job to take down the file locally and present it on the local server every 15 minutes, correct? I think I understand.

 

Nope - if you're existing method is to request the metar when a user views your site, all you need to do is modify your script to download the file and store it locally. Then, read the metar from the file. If the file is older than 15 minutes, it should then re-download it.

 

This way, you don't waste resources when nobody is looking at the metars (for example, late at night)

 

Regards,

A...

[Thomas Mathieu 998318]

Ah got it. Thanks!

[Brian Beach 915973]

If you are using the old http://www.vatsim.net/data/search_metar.php?id=ICAO feed, please move to metar.vatsim.net/ICAO immediately.. the old search_metar.php will be going away very shortly...

Will http://metar.vatsim.net/data/metar.php still remain? This is what VatsimPHP uses, and it gets that from the status.txt file itself.

 

Please ONLY retrieve this file once every 15 minutes.. if you retrieve any sooner than that, your site WILL be banned..

As a note, VatsimPHP retrieves this date every THIRTY minutes, for those of you worried if you are using VatsimPHP for your information.

[David Klain 874106]

weax is a common abbreviation for weather in the us military (we also sometimes write wx as shorthand). When spoken, a person would say "WEE AXE"...

 

Dave

[Luca Benelli]

Nope - if you're existing method is to request the metar when a user views your site, all you need to do is modify your script to download the file and store it locally. Then, read the metar from the file. If the file is older than 15 minutes, it should then re-download it.

 

If you have only few requests a day it would be better remaining with the old system.. or even better, fetch the data you need and store only this with a timestamp, if somebody else requests them within 15 minutes you already have them if not download them.

 

It doesn't make sense to download the compete WX list every 15 minutes if you have 20 METAR requests a day.

 

Luca

[Jeff Thomas]

You guys are just talking about the entire metars.txt file download correct?

 

If I just want Ohare and Midway I can just use the simple service and hit that as many times as I want correct?

[Mike Evans]

Well, ideally you'll only hit it once every 15 minutes, and cache it locally... the data will not change any sooner than that..

[Norman Blackburn]

Would it not be better to only get the data when required? By this I mean if the data is requested a check should be made locally. If no current data exists (ie is older than 15 mins) then it should download and keep this data in the even somebody else requires the info.

 

If no request is made no data should be needed rather than a cron pulling data every 15 mins even if its not being used.

 

Thoughts?

[Gerry Hattendorf 935415]

Please ONLY retrieve this file once every 15 minutes.. if you retrieve any sooner than that, your site WILL be banned..

 

Is there a grace period for this so I can test my new scripts?

 

Thanks!

[Matthew Cianfarani]

Would it not be better to only get the data when required? By this I mean if the data is requested a check should be made locally. If no current data exists (ie is older than 15 mins) then it should download and keep this data in the even somebody else requires the info.

 

If no request is made no data should be needed rather than a cron pulling data every 15 mins even if its not being used.

 

Thoughts?

 

Makes sense, Ideally what I think I will due is not display the METAR on the homepage but have an area for a person to check, and only download the data when they need it.

[Gerry Hattendorf 935415]

I've finished updating my script, and due to file permission errors and "racing" with local text files, I decided to create a table in my database. A simple code snippet;

 

function GetMetar($icao)
{
  // Sanitize the user input to prevent SQL Injection
  $icao = mysql_real_escape_string($icao);
  $icao = strtoupper($icao);
  if (strlen ($icao) == 0)
     return "";

  $string = "";
  $curtime = time();

  // Lets look in the database to see if we have a METAR for this airport and it's within 20 minutes current use it.
  $query = mysql_query("SELECT * FROM currentIcoMetar where airportNameID = '$icao'");
  if ($metar = mysql_fetch_object($query))
  {
     $string = $metar->theMetarStuff;
     if ($metar->lastUpdateTime + 1200 < $curtime)
     {
        if (!$string = file_get_contents("http://metar.vatsim.net/$icao"))
           $string = "CURRRENT METAR UNAVAILABLE";
        else
           mysql_query("update currentIcoMetar set lastUpdateTime = $curtime, theMetarStuff = '$string' where airportNameID = '$icao'");
     }
  }
  else
  {
     if (!$string = file_get_contents("http://metar.vatsim.net/$icao"))
        $string = "CURRRENT METAR UNAVAILABLE";
     else
        mysql_query("insert into currentIcoMetar airportNameID , unixtime, theMetarStuff values ('$icao', $curtime, '$string')");
  }
  $rowString = "<tr><td cl[Mod - Happy Thoughts]=\"metarleft\">$icao</td><td nowrap cl[Mod - Happy Thoughts]=\"metarright\">$string<td></tr>";
  return $rowString;
}

 

 

As I'm caching METAR data for 20 minutes, this would save bandwidth, but I'm still plagued with a user wanting to check METARS several airports we don't normally keep.

 

So my question is, if we have a flood of pilots checking ICAO codes we don't store, is my site going to be banned because of unexpected visitors exceeding the 15 minute limit?

 

Cheers!

[Jeff Thomas]

Ok, maybe I wasn't clear. If I put this snippet:

 

$OHAREMETARLOC = "http://metar.vatsim.net/kord";

.

.

.

$ordMetar = file_get_contents($OHAREMETARLOC);

echo $ordMetar ;

 

onto my homepage so anyone who comes to the site can see the current Ohare VATSIM metar , it will be hit EVERY time someone hits the page or refreshes the page.

 

I don't see the need to store the file off anywhere and just hit the service in real time.

[Luca Benelli]

If your page should ever get thousand hits a minute, for whatever reason, that would be a thousand hits on the metar page.

 

For quite some time i used something like this (in php) :

 

$metartime=filemtime("mymetarfile.txt"); 
$now=time(); 
$diff=$now-$metartime; 
if ($diff > 900){

  download the KORD metar to mymetarfile.txt 

  }

echo the content of mymetarfile.txt

 

In this way you don't have to donwload the complete metar file, you donwload it at most once every 15 minutes and you don't download it if it is not needed.

 

This is good also for those who need more than one metar... if you get lots of requests you'd better donwload the complete file with a cronjob but for low traffic sites this should be enough.

[Jeff Thomas]

Ok, thanks Luca. If I ever get 1000 hits in a minute, I'll have a whole lot of bigger problems to deal with but no sense in p[Mod - Happy Thoughts]ing on the problem.

 

Here's what I came up with (pretty much the same as Gerry's above but with files instead of a database):

 

public static function getMetar($airportCode) {
	$delay=900; //15 minutes
	$metarFile=$airportCode."_metar.txt";

	//* Update metar file if needed
	if(!file_exists($metarFile) || time()-filemtime($metarFile) > $delay) {
		copy("http://metar.vatsim.net/".$airportCode, $metarFile);
	}

	return $file_get_contents($metarFile);

}

 

I only call it for Ohare right now, but this gives me the ability to call it with other codes if I ever want to expand. Of course, if I expand to too many airports, I'll switch and download the entire file and p[Mod - Happy Thoughts] it.

[Mike Evans]

Jeff.. what's the IP address of your web server, I can tell you pretty quickly if you'll need to download the whole file or not based on our logs.. If you have any doubts, download the whole file.. a .txt file is far less "expensive" for our server versus a php script, no matter how small it is..

[Sean Fisher 1042348]

The chance of getting 100+ hits in a minute?? I wish I had a 1/4 of that! Well unless your Delta VA of course

[Luca Benelli]

Anybody ever heard of DDOS?

[Sean Fisher 1042348]

Well thats why I also push for entire page caching for when people aren't logged in. And the object caching for when users are. Saves a bunch of CPU juice!

[Florian Harms]

Just check the publishing times in the METAR, you will find out, that just in a very rare numbers the metar will be published more often then every 30 minutes. Most stations publish them hourly, others less frequently.

Flo