Pilots not correctly changing frequency

[Michael Kuhne]

Since a few days it looks like many pilots are unable to change correctly to the next frequency and still transmit on the old one after having been sent to the next sector.

Is this an issue with AfV, the heavy traffic on the network in general or just way too many pilots forgetting to push the frequency switch button? It didn't seem to be that bad some weeks ago.

[Ryan Geckler]

It's an issue with AFV, the developers are aware and working on it.

[Michael Kuhne]

Thanks for the information. Are the pilots alreeady transmitting on the next frequency and it is only a kind of lag thing that I can hear them or did AfV not permit them to switch correctly?

[Ryan Geckler]

It was a lag issue. With CTP yesterday setting record numbers, the vox servers just couldn't keep up. Should be fine now.

[Simon Kelsey]

Just to fill in a bit of background on this:

 

When the AFV ranging servers become saturated, pending frequency requests build up in a queue on the server. They will sooner or later be actioned - so if you notice this starting to happen the absolute worst thing you can do is start mashing the frequency swap button and trying to go back and forth between frequencies. This simply adds more and more frequency change requests to the queue which gets longer and longer and everything grinds to a halt.

 

So if you do happen to experience this -- the best thing you can do is wait and don't touch anything. If everybody does this, the servers will probably catch up (they can process millions of these calculations every second) but the more you keep mashing the frequency change switch the worse it gets.

 

Techy background: we have a number of servers which work to calculate, essentially, who is in range of whom. This involves knowing which frequency you are on and then calculating the distance between you and everybody else on the same frequency worldwide, every few seconds or so.

 

This 'ranging' is obviously a pretty mathematically-intensive (and therefore server CPU intensive) operation and it is one of the bottlenecks in the AFV system. The good news is that we are able to monitor the system and identify when a problem is occurring -- the solution is to simply add some extra ranging servers to the cluster as needed, which generally only takes a few minutes and doesn't affect the rest of the service. (As an aside, the fact we are able to do this is thanks to one of the many server-side enhancements and optimisations which Mark has been adding quietly in the background over the last six months or so that has enabled us to scale the system up from supporting about 1000 users at release to >3000 now!)