servers banned from downloading data files

[Luca Benelli]

During a log check on the pcflyer.net server (due to the provider throttling the bandwidth) we found some IPs downloading the data file more than 5000 times a day, considering that following the 2 minutes rule and the random server you shouldn't get more than 200 times a day per server something is very wrong.

 

Anyway, the following IPs have been banned from that server.

 

176.28.16.243

62.146.152.18

210.186.59.60

67.68.212.43

213.239.201.245

84.111.8.160

213.163.64.32

93.35.67.35

[Colin Schoen]

Strange that: 213.239.201.245 appears to be IVAOs forum server.

[Brian Pryor 810138]

And 213.163.64.32 returns Hostname: srv32.logistics.ivao.aero

[Luca Benelli]

ouch.

i used the complete log file instead of limitign it to the data file downloads. list will change slightly... some were vatsim indicator downloads.

 

In fact only the first IP changes, and it's probably some crawler going through some forum...

 

210.186.59.60

67.68.212.43

213.239.201.245

62.146.152.18

84.111.8.160

213.163.64.32

93.35.67.35

[Adam Trzcinski]

176.28.16.243 - vatsim.slt-media.de

That's a rotating banner service that we are using in Germany and other vACCs.

 

It currently looks, that this server has been banned from loading the vatsim indicators.

Can you verify?

 

http://vatsim.slt-media.de/rotban/?id=vatoic&vid=1125672

Should load:

http://pcflyer.net/indicators/1125672.png

 

Or has the URL for the indicators changed completely?

[Steven Cullen]

Hi Adam,

 

The URL has not changed. I will email you.

 

Steven

[Luca Benelli]

another bunch of blocked IPs.

 

109.201.152.238

85.72.42.69

154.20.126.147

213.239.201.245 - IVAO once again, don't these guys have somebody who KNOWS what he's actually doing?!

62.146.152.18

173.165.172.33

213.163.64.32

89.65.219.91

216.155.139.3

[Miguel Frias]

A bit more in-depth:

 

another bunch of blocked IPs.

109.201.152.238 (Netherlands)

85.72.42.69 (Greece)

154.20.126.147 (Canada dialup?)

213.239.201.245 - IVAO once again, don't these guys have somebody who KNOWS what he's actually doing?!

62.146.152.18 (Switzerland)

173.165.172.33 (Atlanta/US)

213.163.64.32 (IVAO)

89.65.219.91 (Poland)

216.155.139.3 (simba.aircharts.org)

 

Quite international!

[Larry Wimble]

Wow, I've got some heavy hitters on data.vattastic.com:

 

203.97.109.198: requests per hour: 729

93.35.71.142: requests per hour: 331

87.230.42.143: requests per hour: 135

84.62.71.177: requests per hour: 729

93.39.11.167: requests per hour: 486

 

These are just the ones who make more than 100 requests per hour. I've got many others who are making between 60 and 100 as well. I've started delivering a "you are banned" file to those who make more than 60 requests per hour.

 

Larry

[Luca Benelli]

You are too good considering that they shouldn't be making more than 7.5/h on average...

[Bradley Grafelman]

I've started delivering a "you are banned" file to those who make more than 60 requests per hour.

Aww, that's no fun... if you've got an iptables-like firewall, it's more fun to add DROP rules for such hosts. That way, you save your internal/egress resources and you punish them by making them wait for a timeout.

[Daniel Hawton]

You are too good considering that they shouldn't be making more than 7.5/h on average...

 

You're still [Mod - Happy Thoughts]uming 1 ip = 1 site.

[Colin Schoen]

If you are hosting multiple sites on one machine and IP then why wouldn't you just fetch the data once and have your sites pull it locally? Now if people are using shared hosts, who dump multiple sites on the same IP, that would have to mean that 13 (or more) VATSIM related sites were stuck on the same box, fetching data 7.5 times per hour from one feed provider to cross the 100 threshold. That is not likely.

[Daniel Hawton]

If you are hosting multiple sites on one machine and IP then why wouldn't you just fetch the data once and have your sites pull it locally? Now if people are using shared hosts, who dump multiple sites on the same IP, that would have to mean that 13 (or more) VATSIM related sites were stuck on the same box, fetching data 7.5 times per hour from one feed provider to cross the 100 threshold. That is not likely.

 

I don't own every site that runs on my server, that's why. I am now a source for the data files, so I'm working on moving the queries to fetch locally, but since I don't run all the sites my server hosts I cannot make them share from each other. Even then my server got listed with 2-3 sites pinging the data stream every 2 minutes.

[Svilen Vassilev]

Extracting upstream communication & data exchange logic code into libraries helps.

 

For example: the VATEUD servers have multiple sites (divisional, API, several vACC ones, and several event sites) all of them using the VATSIM data. Each of these apps uses the vatsim_online library as a data source. But that library is designed to work on a server level, so even when installed and accessed from multiple apps on the same machine, it caches the data locally in the OS specified shared temp folder and only polls for changes when those cached local files go stale (checks based on file timestamp). There can be hundreds of requests per minute going from the individual websites to the library. Regardless the library will only poll VATSIM in no less than 4 hour intervals for the status file and no less than 5 minutes for the data file from random server.

 

Of course, this is for Ruby/Rails. For PHP, I'm pretty sure Jelle Vink's VatsimPHP library follows similar patterns.

[Rahul Parkar]

Svilen,

 

While I agree that extracting into libraries is the way forward. When you don't own the sites on your box and are just a server host. You can only really recommend things for people to do, based on the consequences of them not doing so.

 

Cheers

RP

[Greg Barber]

Can someone please provide a definitive list of data feed servers. I'm currently sharing the load between 3 but I understand it could be more.

[Colin Schoen]

http://status.vatsim.net/status.txt

 

url0=http://www.pcflyer.net/DataFeed/vatsim-data.txt

url0=http://fsproshop.com/servinfo/vatsim-data.txt

url0=http://info.vroute.net/vatsim-data.txt

url0=http://data.vattastic.com/vatsim-data.txt

[Daniel Hawton]

And soon there will be a 5th source for data

[Luca Benelli]

yep, my fault for not adding it yet...

http://status.vatsim.net reports the available sources.

[Jonas Kuster]

62.146.152.18

 

That's mine. Really sorry about.

I developped a small tool for doing specific statistics, especially for events of vACC Switzerland (like the VA-Meeting last weekend). I had some difficulties at the beginning using the different servers by random. In the meantime, everthing works fine. Request should not exceed the given limits. Very appreciate to remove this IP from the ban list. You can contact me for further explanations any time.

 

Is there a docomeent stating the requirements for becoming a additional data provider of the data file? Is there a need for it?

[Luca Benelli]

We just added a 5th server, we shouldn't need more.

 

Luca