Jump to content

You're browsing the 2004-2023 VATSIM Forums archive. All content is preserved in a read-only fashion.
For the latest forum posts, please visit https://forum.vatsim.net.

Need to find something? Use the Google search below.
PLEASE READ - Webmaster Support Forum
This forum will be retired in the near future. Please direct all queries to our dedicated GitHub support page https://github.com/vatsimnetwork/developer-info/discussions 
Here you can find documentation on our services and we are continuing to migrate pertinent information into the Wiki pages https://github.com/vatsimnetwork/developer-info/wiki

METAR Service for Scripts


Mike Evans
 Share

Recommended Posts

Mike Evans
Posted
Posted

Good Day All...

 

I'm looking at the logs, and seeing lots of different creative ways to retrieve the METAR data from VATSIM..

 

If you are using the old www.vatsim.net/data/search_metar.php?id=ICAO feed, please move to metar.vatsim.net/ICAO immediately.. the old search_metar.php will be going away very shortly...

 

Additionally, if you need ALL the metars, or a large number of them, please retrieve www.vatsim.net/data/metar.txt

 

This file contains all metars currently active in the network.

 

Please ONLY retrieve this file once every 15 minutes.. if you retrieve any sooner than that, your site WILL be banned..

 

Is there any other functionality in this area that would help you?

Mike Evans

Link to comment
Share on other sites

Gerry Hattendorf 935415
Posted
Posted

Hey Mike,

 

Just one thing, I have a WX page that's called from a link on our home page. There is no cron, however I can't throttle users selecting this page as it's a link on our homepage. I hate to think that I need to create a table and cron to store this data locally, but I don't want our site banned because 10 users all wanted to see the metars in a short time.

 

Other ideas perhaps?

Gerry Hattendorf

ZLA Webmaster

VATSIM Supervisor

Link to comment
Share on other sites

Anthony Lawrence
Posted
Posted
Hey Mike,

 

Just one thing, I have a WX page that's called from a link on our home page. There is no cron, however I can't throttle users selecting this page as it's a link on our homepage. I hate to think that I need to create a table and cron to store this data locally, but I don't want our site banned because 10 users all wanted to see the metars in a short time.

 

Other ideas perhaps?

 

Download the result and store it in a file locally? Requires very little change to your code that way - other than the check to see if it's x minutes out of date

 

Regards,

A...

0
Link to comment
Share on other sites

David Klain 874106
Posted
Posted

That's exactly how I process VATSIM data as well as weax. Download file and save it locally every 15 min. Every time a person makes a data request on the site, we pull and process from the local copy instead of downloading a new one. Provides much faster response as well since the file is on the server...just ensure you point to its local address and not an http: address!

 

Dave

Link to comment
Share on other sites

Gerry Hattendorf 935415
Posted
Posted

Thanks guys, I'm working with some VERY OLD code, and the page we have is just a stupid anchor to the vatsim site. I guess I'll busy for the next day to update my database, and my scripts.

 

BTW Dave, what is "weax"?

Gerry Hattendorf

ZLA Webmaster

VATSIM Supervisor

Link to comment
Share on other sites

Thomas Mathieu 998318
Posted
Posted

So we would need to setup a cron job to take down the file locally and present it on the local server every 15 minutes, correct? I think I understand.

Best Regards,

Thomas Mathieu

VATAME1 Region Director

VATSIM Africa Middle East

http://www.vatame.net

[email protected]

TMSIGVATAME.png

Link to comment
Share on other sites

Anthony Lawrence
Posted
Posted
Thanks guys, I'm working with some VERY OLD code, and the page we have is just a stupid anchor to the vatsim site. I guess I'll busy for the next day to update my database, and my scripts.

 

Nobody said you have to update the database So if you're wanting to cut corners you can do as suggested and save the file locally

 

So we would need to setup a cron job to take down the file locally and present it on the local server every 15 minutes, correct? I think I understand.

 

Nope - if you're existing method is to request the metar when a user views your site, all you need to do is modify your script to download the file and store it locally. Then, read the metar from the file. If the file is older than 15 minutes, it should then re-download it.

 

This way, you don't waste resources when nobody is looking at the metars (for example, late at night)

 

Regards,

A...

0
Link to comment
Share on other sites

Thomas Mathieu 998318
Posted
Posted

Ah got it. Thanks!

Best Regards,

Thomas Mathieu

VATAME1 Region Director

VATSIM Africa Middle East

http://www.vatame.net

[email protected]

TMSIGVATAME.png

Link to comment
Share on other sites

Brian Beach 915973
Posted
Posted
If you are using the old http://www.vatsim.net/data/search_metar.php?id=ICAO feed, please move to metar.vatsim.net/ICAO immediately.. the old search_metar.php will be going away very shortly...

Will http://metar.vatsim.net/data/metar.php still remain? This is what VatsimPHP uses, and it gets that from the status.txt file itself.

 

Please ONLY retrieve this file once every 15 minutes.. if you retrieve any sooner than that, your site WILL be banned..

As a note, VatsimPHP retrieves this date every THIRTY minutes, for those of you worried if you are using VatsimPHP for your information.

Brian Beach

VatsimPHP Developer: http://www.bbflights.com/VatsimPHP/

AFA Detroit Hub Director: http://www.flyafa.com/

userbar603862kx9.gif

Link to comment
Share on other sites

David Klain 874106
Posted
Posted

weax is a common abbreviation for weather in the us military (we also sometimes write wx as shorthand). When spoken, a person would say "WEE AXE"...

 

Dave

Link to comment
Share on other sites

Luca Benelli
Posted
Posted

Nope - if you're existing method is to request the metar when a user views your site, all you need to do is modify your script to download the file and store it locally. Then, read the metar from the file. If the file is older than 15 minutes, it should then re-download it.

 

If you have only few requests a day it would be better remaining with the old system.. or even better, fetch the data you need and store only this with a timestamp, if somebody else requests them within 15 minutes you already have them if not download them.

 

It doesn't make sense to download the compete WX list every 15 minutes if you have 20 METAR requests a day.

 

Luca

Luca Benelli - C3 - P2

Link to comment
Share on other sites

Jeff Thomas
Posted
Posted

You guys are just talking about the entire metars.txt file download correct?

 

If I just want Ohare and Midway I can just use the simple service and hit that as many times as I want correct?

Jeff Thomas

VP-IT

https://joinava.org

Link to comment
Share on other sites

Mike Evans
Posted
Posted

Well, ideally you'll only hit it once every 15 minutes, and cache it locally... the data will not change any sooner than that..

Mike Evans

Link to comment
Share on other sites

Norman Blackburn
Posted
Posted

Would it not be better to only get the data when required? By this I mean if the data is requested a check should be made locally. If no current data exists (ie is older than 15 mins) then it should download and keep this data in the even somebody else requires the info.

 

If no request is made no data should be needed rather than a cron pulling data every 15 mins even if its not being used.

 

Thoughts?

Norman

sig_FSLBetaTester.jpg

Link to comment
Share on other sites

Gerry Hattendorf 935415
Posted
Posted
Please ONLY retrieve this file once every 15 minutes.. if you retrieve any sooner than that, your site WILL be banned..

 

Is there a grace period for this so I can test my new scripts?

 

Thanks!

Gerry Hattendorf

ZLA Webmaster

VATSIM Supervisor

Link to comment
Share on other sites

  • Board of Governors
Matthew Cianfarani
Posted
Posted
Would it not be better to only get the data when required? By this I mean if the data is requested a check should be made locally. If no current data exists (ie is older than 15 mins) then it should download and keep this data in the even somebody else requires the info.

 

If no request is made no data should be needed rather than a cron pulling data every 15 mins even if its not being used.

 

Thoughts?

 

Makes sense, Ideally what I think I will due is not display the METAR on the homepage but have an area for a person to check, and only download the data when they need it.

Matthew Cianfarani
Vice President , Technology 
VATSIM Board of Governors

Link to comment
Share on other sites

Gerry Hattendorf 935415
Posted
Posted

I've finished updating my script, and due to file permission errors and "racing" with local text files, I decided to create a table in my database. A simple code snippet;

 

function GetMetar($icao)
{
  // Sanitize the user input to prevent SQL Injection
  $icao = mysql_real_escape_string($icao);
  $icao = strtoupper($icao);
  if (strlen ($icao) == 0)
     return "";

  $string = "";
  $curtime = time();

  // Lets look in the database to see if we have a METAR for this airport and it's within 20 minutes current use it.
  $query = mysql_query("SELECT * FROM currentIcoMetar where airportNameID = '$icao'");
  if ($metar = mysql_fetch_object($query))
  {
     $string = $metar->theMetarStuff;
     if ($metar->lastUpdateTime + 1200 < $curtime)
     {
        if (!$string = file_get_contents("http://metar.vatsim.net/$icao"))
           $string = "CURRRENT METAR UNAVAILABLE";
        else
           mysql_query("update currentIcoMetar set lastUpdateTime = $curtime, theMetarStuff = '$string' where airportNameID = '$icao'");
     }
  }
  else
  {
     if (!$string = file_get_contents("http://metar.vatsim.net/$icao"))
        $string = "CURRRENT METAR UNAVAILABLE";
     else
        mysql_query("insert into currentIcoMetar airportNameID , unixtime, theMetarStuff values ('$icao', $curtime, '$string')");
  }
  $rowString = "<tr><td cl[Mod - Happy Thoughts]=\"metarleft\">$icao</td><td nowrap cl[Mod - Happy Thoughts]=\"metarright\">$string<td></tr>";
  return $rowString;
}

 

 

As I'm caching METAR data for 20 minutes, this would save bandwidth, but I'm still plagued with a user wanting to check METARS several airports we don't normally keep.

 

So my question is, if we have a flood of pilots checking ICAO codes we don't store, is my site going to be banned because of unexpected visitors exceeding the 15 minute limit?

 

Cheers!

Gerry Hattendorf

ZLA Webmaster

VATSIM Supervisor

Link to comment
Share on other sites

Jeff Thomas
Posted
Posted

Ok, maybe I wasn't clear. If I put this snippet:

 

$OHAREMETARLOC = "http://metar.vatsim.net/kord";

.

.

.

$ordMetar = file_get_contents($OHAREMETARLOC);

echo $ordMetar ;

 

onto my homepage so anyone who comes to the site can see the current Ohare VATSIM metar , it will be hit EVERY time someone hits the page or refreshes the page.

 

I don't see the need to store the file off anywhere and just hit the service in real time.

Jeff Thomas

VP-IT

https://joinava.org

Link to comment
Share on other sites

Luca Benelli
Posted
Posted

If your page should ever get thousand hits a minute, for whatever reason, that would be a thousand hits on the metar page.

 

For quite some time i used something like this (in php) :

 

$metartime=filemtime("mymetarfile.txt"); 
$now=time(); 
$diff=$now-$metartime; 
if ($diff > 900){

  download the KORD metar to mymetarfile.txt 

  }

echo the content of mymetarfile.txt

 

In this way you don't have to donwload the complete metar file, you donwload it at most once every 15 minutes and you don't download it if it is not needed.

 

This is good also for those who need more than one metar... if you get lots of requests you'd better donwload the complete file with a cronjob but for low traffic sites this should be enough.

Luca Benelli - C3 - P2

Link to comment
Share on other sites

Jeff Thomas
Posted
Posted

Ok, thanks Luca. If I ever get 1000 hits in a minute, I'll have a whole lot of bigger problems to deal with but no sense in p[Mod - Happy Thoughts]ing on the problem.

 

Here's what I came up with (pretty much the same as Gerry's above but with files instead of a database):

 

public static function getMetar($airportCode) {
	$delay=900; //15 minutes
	$metarFile=$airportCode."_metar.txt";

	//* Update metar file if needed
	if(!file_exists($metarFile) || time()-filemtime($metarFile) > $delay) {
		copy("http://metar.vatsim.net/".$airportCode, $metarFile);
	}

	return $file_get_contents($metarFile);

}

 

I only call it for Ohare right now, but this gives me the ability to call it with other codes if I ever want to expand. Of course, if I expand to too many airports, I'll switch and download the entire file and p[Mod - Happy Thoughts] it.

Jeff Thomas

VP-IT

https://joinava.org

Link to comment
Share on other sites

Mike Evans
Posted
Posted

Jeff.. what's the IP address of your web server, I can tell you pretty quickly if you'll need to download the whole file or not based on our logs.. If you have any doubts, download the whole file.. a .txt file is far less "expensive" for our server versus a php script, no matter how small it is..

Mike Evans

Link to comment
Share on other sites

Sean Fisher 1042348
Posted
Posted

The chance of getting 100+ hits in a minute?? I wish I had a 1/4 of that! Well unless your Delta VA of course

Regards,

 

5

[email protected]

I am Pushing Forward. Are You?

Link to comment
Share on other sites

Luca Benelli
Posted
Posted

Anybody ever heard of DDOS?

Luca Benelli - C3 - P2

Link to comment
Share on other sites

Sean Fisher 1042348
Posted
Posted

Well thats why I also push for entire page caching for when people aren't logged in. And the object caching for when users are. Saves a bunch of CPU juice!

Regards,

 

5

[email protected]

I am Pushing Forward. Are You?

Link to comment
Share on other sites

  • 2 weeks later...
Florian Harms
Posted
Posted

Just check the publishing times in the METAR, you will find out, that just in a very rare numbers the metar will be published more often then every 30 minutes. Most stations publish them hourly, others less frequently.

Flo

Florian Harms

VATSIM Europe Division / DCRM

Supervisor

Link to comment
Share on other sites

 Share