Corrupt client data feed -- STILL HAPPENING!

[Nick Johnston]

Hi there,

 

In recent weeks there have been a handful of entries in the client data feed that appear to be invalid, or at least are missing basic information such as the PID/CID.

 

There is such a situation occurring in the feed as I write: the entry for an ATC client with the callsign ZMO_63_CTR is currently being reported as:

 

ZMO_63_CTR:::ATC:135.200:23.00000:-72.00000:0:::0::::USA-C::5::6:450::::::::::::::::::20180816151259::::

 

It looks like it is missing its:

• CID
  
• Controller Name
  
• Protocol Version

 

For reference, here's what a normal ATC client looks like (taken from the same file):

 

ANC_20_CTR:846001:Bob Scott:ATC:132.300:61.16787:-149.96014:0:::0::::USA-W:100:7::6:600::::::::::::::::$ rw.liveatc.net/ZAN_20^<A7>Anchorage Center^<A7>PANC Information SIERRA Winds calm Vis 10SM Sky FEW015 BKN055 OVC090 Alt 2998^<A7>Landing rwys 07L & 07R. Departing rwy 33.:20180816153938:20180816150908::::

 

I was going to attach the data feed file but don't have permission for adding attachments. Any feed from the last half hour will have the same entry in it, but for the record the extracts above came from a download with the header:

 

; Created at 16/08/2018 15:43:20 UTC by Data Server V4.0

 

I'm not sure if this is an issue with the feed, the servers, or perhaps with the ATC client being used, but I would have thought that not including the CID/PID is a non-normal state?

 

 

Nick

Edited September 4, 2018 at 03:11 PM by Guest

[Bahaeddine El-Zarif]

Which server are you using?

 

The reason why I ask is, VATSIM announced that USA-C is being retired and re-directed to USA-S.

 

Source:

[Nick Johnston]

Well, the feed file was saying that he was using USA-C. In terms of which server the data feed file came from, I've got the same result from at least three different servers over the period that the entry was incorrect, which was ~15:14 - 15:50 UTC.

 

According to the Stats Center site - https://stats.vatsim.net/atc_details.php?conn=54164385 - the user in question was online for a couple of hours without issue and then (was) disconnected at ~15:06, just prior to the data in the feed changing.

[Matthew Cianfarani]

Hi Nick,

 

I will look into this tonight. We have been (and are in the middle of) some significant infrastructure changes. I am unsure if this is contributing to the issue... but we will look into it regardless.

[Nick Johnston]

Thanks, Matt. As is always the way, the issue disappeared almost literally the moment I hit submit on my original message.

 

If you need them I have copies of the data feed output during the period (although all of the salient bits are in my original post).

 

As Bahaeddine pointed out, the timing of the issue did coincide with the post on the Facebook page re the decommissioning of the USA-C server. I just went back through the data for the last errored feed and noticed a couple of other clients that also had no CID/Real Name/Protocol Version in the same file, and they were also showing as being on the USA-C server:

 

AFR334:::::49.00156:2.55890:406:0::0::::USA-C::1:1200::::::::::::::::::::20180816151249:10:29.913:1012:
KLM1017:::::-9.81427:25.46865:32339:470::0::::USA-C::1:2000::::::::::::::::::::20180816151245:315:30.27:1025:

 

Stats Center shows that AFR334 was connected until 15:06 (same as ZMO_63_CTR): https://stats.vatsim.net/callsign_details.php?id=1224743&callsign=AFR334

Similarly KLM107 was connected until 15:06 as well: https://stats.vatsim.net/callsign_details.php?id=1432164&callsign=KLM1017

 

The broken entries for these clients first appeared in the datafeed generated at 15:13 and last appeared in the feed at 15:47.

 

In fact the 15:13 datafeed contains all three of the broken client entries that are in the 15:47 one, plus several others. All of them are listed as being connected via USA-C, so it does look as if the issue was specific to that server at that time.

 

 

Nick

[Calum Towers]

Seeing a few issues with the data feed that is likely related to the above.

 

There are instances of "ghost" connections (http://prntscr.com/kk4wbu) but then everything will disappear and there will be no connections at all.

 

Seems to sort itself within 30-60 seconds.

[Nick Johnston]

Yeah, I'm getting paged again at the moment for more entries in the data feed that are missing CID/PID, Real Name, and Protocol Version.

 

; Created at 18/08/2018 14:37:07 UTC by Data Server V4.0

EGPH_GND:::ATC:121.750:55.95250:-3.36139:0:::0::::UK1::2::3:10::::::::::::::::$ uk.voice.vatsim.net/egph_gnd^<A7>"Edinburgh Ground"^<A7>We love feedback! Submit some at vats.im/atcfb:20180818143146:20180818141423::::

HECC_CTR:::ATC:127.700:32.33222:30.27000:0:::0::::::5::6:500::::::::::::::::::20180818141423::::

HKG_E_CTR:::ATC:121.300:22.30892:113.91460:0:::0::::USA-E::10::6:360::::::::::::::::$ rw1.vatpac.org/hkg_e_ctr^<A7>Hong Kong Radar^<A7>Realistic correlation in use. hkvacc.org^<A7>Covering VHHH, VMMC, VHHX:20180818143241:20180818141423::::

JC_OBS:::ATC:199.998:22.31049:113.92134:0:::0::::::4::0:147::::::::::::::::Observer:20180818143258:20180818141423::::

LOWW_N_APP:::ATC:118.770:48.11772:16.56716:0:::0::::GERMANY2::4::5:150::::::::::::::::$ voice.vacc-austria.org/loww_n_app^<A7>Wien Radar^<A7>For charts visit www.vacc-austria.org:20180818143324:20180818141422::::

LROP_ATIS:::ATC:118.500:44.57111:26.08500:0:::0::::GERMANY2::5::4:0::::::::::::::::$ uk.voice.vatsim.net/lrop_atis^<A7>THIS IS BUCHAREST-OTOPENI INFORMATION V1400 arrival runway 08R^<A7>departure runway 08L TRANSITION LEVEL 50 WIND RUNWAY 08R^<A7>TOUCHDOWN ZONE 080 DEGREES 10 KNOTS VISIBILITY RUNWAY 08R^<A7>TOUCHDOWN ZONE 10 KILOMETERS OR MORE sky conditions clear^<A7>TEMPERATURE 33 DEWPOINT 15 QNH 1015 HECTOPASCALS QFE 1004^<A7>HECTOPASCALS WARNING FOR BIRD ACTIVITY RUNWAY 08R THIS WAS^<A7>BUCHAREST-OTOPENI INFORMATION V:20180818143339:20180818141423::::

LRUB_CTR:::ATC:129.750:46.50000:25.00000:0:::0::::GERMANY2::5::6:400::::::::::::::::$ uk.voice.vatsim.net/lrub_ctr^<A7>Black Sea West^<A7>FULL ATC for LBSR LRBB LUUU:20180818143339:20180818141423::::

LSGG_ATIS:::ATC:135.570:46.23833:6.10944:0:::0::::UK1::2::4:0::::::::::::::::$ voice.vacc.ch/lsgg_atis^<A7>This is Geneva information O runway 05 in use ILS approach^<A7>transition level 80 MET report Geneva at time 1350 UTC wind 040^<A7>degrees 9 knots visibility 10 kilometers Clouds few 4500 feet^<A7>temperature 26 dew point 15 QNH 1021 NOSIG^<A7>advise on initial contact you have information O Report^<A7>aircraft type on initial contact with Geneva Arrival:20180818143339:20180818141423::::

LSGG_TWR:::ATC:118.700:46.23833:6.10944:0:::0::::UK1::2::4:50::::::::::::::::$ voice.vacc.ch/lsgg_twr^<A7>Geneva Tower, Bonjour^<A7>ATIS 135.57^<A7>Comply with the procedures and read our charts *charts.vacc.ch*, feedback *fb.vacc.ch*:20180818143340:20180818141423::::

OMAE_CTR:::ATC:119.300:24.43306:54.65111:0:::0::::::8::6:200::::::::::::::::$ voice1.vatsim-germany.org/omae_ctr^<A7>"U.A.E Radar" - twitch.tv/chrissKLO^<A7>Report callsign with heading and altitude on first contact^<A7>Charts/procedures @ :20180818143406:20180818141423::::

OMDB_ATIS:::ATC:131.700:25.25278:55.36445:0:::0::::SINGAPORE::8::4:0::::::::::::::::$ voice1.vatsim-germany.org/omdb_atis^<A7>This is Dubai international airport information F at time 1400^<A7>expect ILS approach arrival runway 30L departure runway 30R^<A7>wind 310 degrees 8 knots variable between 280 and 340 degrees^<A7>visibility 6 kilometers no significant clouds temperature 36^<A7>dew point 27 QNH 995 NOSIG^<A7>advice ATC  that you copied information F:20180818143406:20180818141423::::

PB_SUP:::ATC:199.998:51.47750:-0.46139:0:::0::::GERMANY2::11::0:300:::::::::::::::::20180818143420:20180818141423::::

ZGKL_ATIS:::ATC:126.450:25.21667:110.03833:0:::0::::SINGAPORE::3::4:0::::::::::::::::$ voice.vatprc.net/zgkl_atis^<A7>ZGKL info D 1400 UTC runway 01 expect ILS approach runway 01^<A7>wind variable at 1 m/s visibility 10 kilometers few 1290^<A7>meters temperature 29 deg C dewpoint 22 deg C QNH 1003 hPa^<A7>advise on initial contact you have info D^<A7>and confirm you will implement RNAV procedures:20180818142156:20180818141422::::

ZGKL_CTR:::ATC:133.600:25.21667:110.03833:0:::0::::::5::6:600::::::::::::::::::20180818141422::::

ZGKL_TWR:::ATC:118.000:25.21667:110.03833:0:::0::::SINGAPORE::3::4:50::::::::::::::::$ voice.vatprc.net/zgkl_twr^<A7>Callsign Guilin Tower^<A7>China RVSM in use^<A7>For charts and more info please visit www.vatprc.net:20180818142157:20180818141422::::

ZPKM_CTR:::ATC:125.750:30.58000:103.94833:0:::0::::GERMANY2::5::6:600::::::::::::::::$ voice.vatprc.net/zpkm_ctr^<A7>Callsign Kun Ming Control^<A7>China RVSM in use^<A7>For charts and more info please visit www.vatprc.net:20180818142156:20180818141423::::

 

So that's 16 ATC from a range of different servers. So it does look like this issue is something beyond just the USA-C server.

 

No aircraft this time though.

 

The first reported p[Mod - Happy Thoughts] fail was with the 20180818140312 data feed and every one that's been loaded here since then (one every two minutes) has failed.

 

 

Nick

[Nick Johnston]

Actually, it looks like the data p[Mod - Happy Thoughts]d correctly at 14:12 and 14:14 UTC before failing again:

 

[Nick Johnston]

Looks like the problem persisted for about four hours before coming right just after 1800 UTC (the times in the graphs are NZST):

 

[Nick Johnston]

Any progress on resolving this issue? Looks like it happened again for about four and a half hours on Sunday afternoon/evening UTC.

 

 

Specifically: the data feed files with the missing values were from 14:48 -> 19:23 UTC on September 2.

 

Is someone investigating this? I have the data feed files in question saved if anyone needs them.

[Matthew Cianfarani]

In Short -

 

We know - We Know -

No, we cannot fix it quickly.

 

I'm in meetings with the day job for the rest of the day and when I can, I'll write a full formal summary of *Why* we are having these issues.

 

But the best way to summarize it is, We re rolling out a LOT of new tech, and the Dataserver is probably one of the oldest systems we have in place. We broke it, and instead of patching it with a huge band-aid are taking a new approach. I hope to have something to report within the next week.

 

I really do appreciate the patience