Loss of inbound sound

[Wade Williams 877539]

I've recently started experiencing an extremely frustrating problem.

 

I use Windows XP, which is kept up-to-date. My VRC is the latest version.

 

Recently, I've started losing my incoming sound at seemingly random times. For example, last night, I had a pilot come on that wanted to shoot practice ILS approaches. He tried to call me, but I could not hear him. So, I reset my comm channel and waited for him to call me back. After 5 minutes or so, he tried to call me back. Again, I couldn't hear him. He indicated he could hear me fine. My instructor cleared him for takeoff while I restarted VRC. Once I got back on, I was able to hear him.

 

We went through a full approach with everything working fine. Then he landed - on climbout, when he tried to call me, I could not hear him. However, my instructor told me he called, so I replied "radar contact, turn left heading 270" and then I heard his readback. Again, things were fine throughout the approach. Then he landed. Again, I did not hear him as he tried to call me on climbout - again, because my instructor told me he tried to contact me, I issued radar contact + turn instruction and then heard his readback and things were fine throughout the approach, until he landed. After I gave him his taxi instructions, he taxi'd to the gate and said something right before signing off, which I could not hear.

 

Now, it sounds almost as if my problem is altitude based, but in addition to that making zero sense, I know I've experienced the problem before with aircraft approaching my airspace.

 

The odd thing about this problem is that it's very sporadic. I did a FNO last Friday and never experienced a single problem all night. Additionally, the entire time these problems are occurring, I'm talking on teamspeak (cursing actually) with my other controllers, so it's definitely not a headset problem or loss of all sound on the system problem.

 

I have noticed the following:

 

1) When I cannot hear the pilot, my RX light does not turn orange

 

2) Un[Mod - Happy Thoughts]igning the comm channel in VRC by clicking off the primary, rx and tx checkboxes and then clicking them back on again usually fixes it.

 

3) Based on the story above, I'm suspicious now that my transmitting is somehow "waking up" my voice channel - i.e. I cannot hear a pilot initially, but if I transmit to him then I can hear him on readback.

 

Has anyone ever experienced anything similar?

 

Ross, got any thoughts?

 

Thanks,

 

Wade

[Norman Blackburn]

Wade,

 

You are suffering from UDP timeout.

 

Basically your router has no idea where to deliver the sound packets until you open the door (by transmitting).

 

Unfortunately that same door closes after a little while. You need to check your router settings to either increase the timeout and / or forward the port (3290).

[Ross Carlson]

3) Based on the story above, I'm suspicious now that my transmitting is somehow "waking up" my voice channel - i.e. I cannot hear a pilot initially, but if I transmit to him then I can hear him on readback.

 

Has anyone ever experienced anything similar?

 

Ross, got any thoughts?

 

This makes it sound like a NAT timeout problem in your router. When you first check the HDST/SPKR box in the comms panel to connect to a voice channel, you are opening a socket through your router to the voice server. This action effectively creates a temporary hole in the router's firewall, and allows data to flow in and out on that socket. Many routers have a timeout feature whereby if no data flows in or out of this socket, it will close the hole and thus block any more data from flowing in from the Big Bad World. You can still transmit because the router implicitly trusts data from inside your network. When you transmit you reopen the hole and reset the timeout timer, and you can then hear pilots again. As long as you or a pilot transmits often enough to keep the hole open, all is well, but as soon as there is enough silence on the frequency, the hole closes and you have to transmit to reopen it.

 

What kind of router do you have? Does it allow you to set NAT or port forwarding timeouts?

[Wade Williams 877539]

Thanks guys. I'm familiar with UDP timeouts - the thing perplexing me is that it started happening without any router change on my part. However, rather than beating my head against the wall trying to figure out why, I'll just increase the timeout and see what happens.

[Norman Blackburn]

Wade,

 

Did the IP address of your VRC change recently? Perhaps you had things set for an IP address that is no longer being [Mod - Happy Thoughts]igned.

[Wade Williams 877539]

Nope...and thus the confusion.

 

I am somewhat perplexed by the whole situation. Initially, I did not setup a port forwarding through my router, and just depended on the dynamic udp translation. In short, I tried the more secure configuration first, and it worked.

 

However, in theory, that shouldn't work at all. Unless you setup a static port translation, the only way a pilot should ever be able to contact you is if you contact him first (or, if you just got done talking to someone and the timeout hasn't expired).

 

Without a port translation, what would you set for a UDP timeout? My router is 2 minutes by default, but what value would work? 10 minutes? On VATSIM one can easily go for hours without a transmission on channel and who really wants to set their global UDP timeout to 4 hours?

 

Anyhow, I've re-setup my port forwarding for 3290 and will report how it goes. If I still experience the problem, I'll start doing some logging on the router, and if necessary, I'll start increasing the global UDP timeout.

[Ross Carlson]

However, in theory, that shouldn't work at all. Unless you setup a static port translation, the only way a pilot should ever be able to contact you is if you contact him first (or, if you just got done talking to someone and the timeout hasn't expired).

 

The reason it works is because the pilot doesn't contact you directly ... everything goes through the voice server. So, you initiate the connection from your end to the voice server, and the voice server takes care of sending pilot tranmissions back to you through the same socket that you originally opened from your end to the server.

 

In other words, Air-to-Ground comms are not peer-to-peer ... they are client/server. Ground-to-Ground comms ARE peer-to-peer, however, which is why you need to forward UDP 3290 to your VRC machine in order to receive overrides and intercom calls, or to be monitored by another controller.

[Wade Williams 877539]

Thanks Ross, I had not looked into the actual implementation, so that clears it up.

[Harold Rutila 974112]

This recently began happening to me again. I'm glad this topic came up. Did that solution work for you?

[Wade Williams 877539]

Ross,

 

After some more testing, I can definitely say something more insidious is going on here.

 

Since there are not widespread reports, it may be something unique to Harold's and my machine. However, read on for why it may be voice-server specific.

 

However, here's what I found out:

 

1) While tuned to LA_CTR last night in my comm panel, the problem was occurring approximately every minute

 

2) The problem continued to occur when I byp[Mod - Happy Thoughts]ed all routers and firewalls, and plugged the machine directly into my cable modem

 

3) A packet capture showed that voice packets continued to be received when there was no inbound audio - they just were not being played and the Rx light did not illuminate

 

4) Depressing transmit forced the UDP packets to be [Mod - Happy Thoughts]igned a new port, and that always resulted in resumption of voice. However, voice would quickly drop out again. As previously mentioned, voice packets continued to to be received when there was no sound and this was verified both behind the firewall/router and with the firewall/router byp[Mod - Happy Thoughts]ed.

 

5) The majority of my testing was with channels on rw.liveatc.net - however, I was able to try it briefly with a channel on voice2.vacc-sag.org and had the same result. However, later in the evening, I was able to listen on rw2.vatpac.org for more than 30 minutes without issue.

 

One data point which may or may not be significant - on rw.vatpac.org, pressing transmit does not result in a new UDP port being [Mod - Happy Thoughts]igned.

 

Next step:

 

1) More testing on voice servers other than rw.liveatc.net and more testing on rw.liveatc.net

 

2) If #1 produces no clues, uninstall and reinstall VRC (doubtful it would have a positive impact)

 

3) Prayer

 

Let me know if you have any thoughts.

 

Wade

[Wade Williams 877539]

Ross,

 

After a lot more investigation, I believe the problem is restricted to the rw.liveatc.net server. I believe my previous report of seeing the problem on a different server was incorrect.

 

The problem is that VRC is not sending UDP 57-byte what I [Mod - Happy Thoughts]ume are "keepalive" packets to the server. Watching UDP only, all that is ever received is 60-byte UDP packets from the server, and of course the 142-byte UDP voice packets. Upon pressing the transmit key, the UDP-port changes and playing of the sound resumes until presumably, the VRC client times out because it hasn't seen a response to the keepalives it never sent.

 

On working servers, I see the keepalives in both directions, and pressing the transmit key does NOT cause the voice port to change.

 

In short, I see different behavior when connected to rw.liveatc.net than when connected to other voice servers.

 

Please let me know what additional testing I can perform to help sort out the problem. I'll be glad to provide a complete packet trace from rw.liveatc.net and a working server if you wish.

 

Wade

[Wade Williams 877539]

Well, one of the good things about being older is you've had plenty of opportunity to admit when you were wrong. And indeed, I was wrong.

 

I use VRC through VMWare, and eventually found a udp timeout configuration variable buried deep within a VMWare config file. After changing that to never timeout, rw.liveatc.net began working properly.

 

That doesn't explain why it only happened with one server while others work correctly or why it used to work correctly. I still think the difference in packet behavior I saw had something to do with triggering the problem, but clearly it's not worth spending the time to troubleshoot it now.