VPilot loses Comms

[Greg Hateley 1243958]

Hey Guys,

 

On the night before worldflight, our sim has decided to kill itself. Everything has been working fine up until now. We are running P3D V3, with VPilot. When we connect to a channel, we can hear everything they are saying, they can hear us, everything is fine. However after about 7 minutes, we drop off the channel, and cannot connect onto any channels again without restarting the whole computer. We have found the P3d V2.5 works fine with v pilot, no issues, however if possible we would like to get it working with V3

[Ernesto Alvarez 818262]

double check its not a port issue. port forward UDP 3290

[Ned Hamilton 1215492]

I've had similar issue recently. Everything is working fine and then (without any changes) suddenly one channel doesn't work. Disconnecting and then connecting sometimes fixes it.

 

Doesn't seem to me that a port issue would suddenly drop channels. A port issue isn't likely to be intermittent is it?

[Don Desfosse]

It can appear intermittent. I'm not a network engineer, don't play one on TV, and didn't sleep in a Holiday Inn Express last night, so anyone smarter (especially if I say anything wrong) please jump in, but without port forwarding or a data exchange before the port timeout value is reached, the port will close.

 

What I don't know and haven't reasearched/tested is whether or not you can send a command to the router to reopen the port or reboot the router as opposed to having to reboot the computer.

 

The fact that it worked fine with 2.5 has me scratching my head, as that would indicate maybe it's not a forwarding/timeout issue, though the other symptoms sound like a forwarding/timeout issue.

[Ned Hamilton 1215492]

Don, I'm not a network engineer either but I know a little bit--maybe enough to be dangerous.

 

Why would a router have a port-timeout setting? I thought the clients (vPilot in this case) establish communication with the VATSIM server over certain ports for the UDP or TCP voice connections. These connections can of course time-out but isn't it the responsibility of vPilot to re-establish the connection? I guess I'm confused about how it could be a router problem.

 

I've often wondered if my ISP is mucking around with my VATSIM connections!!

[Don Desfosse]

I don't know from a network/design standpoint why it would be advantageous for the timeout and/or what drives the need for the forwarding. All VATSIM software that I've used, controller and pilot clients alike, not just vPilot, rely on the port to be forwarded, so I would have to imagine that it's certainly not a vPilot limitation (i.e., it is NOT vPilot's responsibility to reestablish a connection). Hopefully Ross or someone well-versed in this area can respond; I look forward to learning more about it myself.

[Ernesto Alvarez 818262]

none of us being network engineers the simplest way is to think of it like a revolving door. door opens and closes letting packets in and out. occasionally someone gets their foot stuck in the door and poof, doors stuck in either direction until someone removes the jam which is what port forwarding essentially does in kinda simple terms, pushes it through

 

none of the clients to my knowledge are designed to force those doors to open when they get stuck. other programs may (pretty sure some do). but none of the VATSIM clients that i know of will do that. why or how difficult that is to add is beyond me, im no programmer lol

 

if you want a quick way to test if its your router or not, if you are able to byp[Mod - Happy Thoughts] the router (if its a stand alone router and not modem and router combo), connect to your modem directly. if it stops happening, youll know its the router. never seen a modem cause it on its own

[Luke Kolin]

the simplest way is to think of it like a revolving door. door opens and closes letting packets in and out. occasionally someone gets their foot stuck in the door and poof, doors stuck in either direction until someone removes the jam which is what port forwarding essentially does in kinda simple terms, pushes it through

 

That's not what port forwarding does, at all. The problem is that the VVL was written for an Internet that doesn't exist anymore. Bear with me, it's a long story.

 

In the old days when the Internet was populated by crazy dudes with awesome facial hair, everything had its own, unique IP address and was more or less reachable from everywhere. Life was simple, and good. When machines wanted to talk to each other, it was straightforward. If a machine wanted to talk to another, it would open a random, local port and connect to a known destination port on the other machine. When the destination machine did an accept() it would keep the known destination port open ([Mod - Happy Thoughts]uming you wanted multiple connections). The point us that while you have a known port, it's really only needed on the destination. The source ports should be random.

 

Pretty soon, people discovered that having machines wide-open to everyone made as much sense as not putting locks on your doors, so we made firewalls and restricted what machines can talk on what ports, but that didn't change matters very much. Either you could talk to each other or not.

 

Where the Internet completely changed was when we ran out of IPv4 addresses. There's room for a bit under 4 billion, but that [Mod - Happy Thoughts]umes proper utilization; which doesn't happen. For a while I was employed by a subsidiary of GE so my desktop IP address started with 3.4..... but was entirely non-routable to the outside world. That's 16 million addresses, wasted. Combined with the world getting access to the Internet instead of bearded academics, we had an explosion of devices connected to the Internet.

 

In my household, I have two servers, five desktops and a laptop. There are three phones, five tablets and two music players. There are two TVs, two set-top boxes and one DVD player with Internet connections. Then there are three thermostats a Wifi-enabled light switch and probably by the end of the year we'll have a pair of smoke detectors and a webcam for doing time-lapses out of the back deck. Add to that my router and WiFi access point each have an IP address. That's a lot of addresses - and while we're somewhat tech-savvy and ahead of the curve, the reality is that most households with consumer electronics and teens are going to be using around a dozen or two addresses. Home automation (thermostats, smoke detectors, switches, timers) is going to explode because with Wifi or Bluetooth you no longer need to retrofit a house with low voltage wiring.

 

This is where Network Address Translation (or NAT) comes in. It's awesome, but really annoying.

 

There's a bunch of addresses that are defined in the RFC1918 spec as "non-routable". The one you're most familiar with is the 192.168.0.0/16 block, but there's 10.0.0.0/24 and 172.16.0.0/20. That's why in a lot of cases your home router gives you an IP address starting with 192.168.0 - millions of you can have that same address without conflicts. What happens is that NAT works on the router to make all of your traffic look like it's coming from the router. When there's only one of you behind it, that's not much; when there is more than one of you that's where the magic happens.

 

Let's say there's two computers connecting to the same server on port 3290. The router creates two connections using two local ports to the destination port on the server. It keeps track of which connection and port is for which computer in a small in-memory table and as it gets back packets from the server it sends them to the right machine on the non-routable address. It seemed like magic the first time I saw it; most people today just expect that's how the Internet is supposed to work. It's not - it's a giant hack.

 

So long as you're using TCP, life is good. Because the protocol is connection-oriented, it deals with state and disconnections well. The router's TCP/IP stack gets notified on disconnects and it can easily keep its little table updated. There's a limit, but most consumer grade routers can easily handle a dozen or two connections for a dozen or two machines without much trouble. No port forwarding or anything required.

 

NAT gives you some neat security advantages. Because those machines' addresses are "non-routable", you can't connect to them via the Internet. The only way those machines can talk to anyone is if they initiate the connection. So long as you don't have malware on your computer, you get basic firewalling for free. Now there are cases where you want to run something behind a NAT gateway, and this is where port forwarding comes in - you can tell the gateway to forward all packets on port X to a given private IP address. But you only need port forwarding if you're running a server, and even then it's a blunt instrument. You are limited to 1 IP address/port combination. (This is important later).

 

Where our tidy little world goes to you know where in a handbasket is when UDP crashes the party. Unlike TCP, UDP doesn't handle connections or state. It doesn't guarantee in order transmission. This tempts people a great deal when writing network software because UDP is technically a little faster than TCP because it doesn't have the same overhead and for voice, the guarantees TCP provides can be somewhat unhelpful.

 

Because there's no connection support, our poor little NAT router has to make a lot of guesses. When it sees a UDP packet come from an internal machine, it adds another entry to its little table so that when the response comes back from the server it knows who to send it back to. But since UDP has no concept of a "connection", our router has to make guesses. If a packet isn't sent from the internal machine in a while, it [Mod - Happy Thoughts]umes the link is gone and drops the entry from the table. And if the entry isn't in the table, then when a packet comes from the server it gets discarded, because our router doesn't know which internal address to send it to.

 

This is why VVL stops working after a while if you don't transmit. If you don't transmit, your router doesn't see any packets from you, times out the forwarding rule and then you can't hear anything because the UDP packets aren't being forwarded.

 

(On a side note, UDP has another big problem with NAT that I'm not sure if VATSIM has ever run into - if your server has multiple IP addresses, you absolutely need to keep track server-side of what IP address the incoming packets came in on, and send the response back on the same address, otherwise the NAT router doesn't know who to send them to. TCP does this automatically for you; not so UDP.)

 

This is where people advise others to either increase their router timeouts or port forward 3290. Both are bad ideas, one worse than the other. The first one isn't terrible, but the point is that you shouldn't have to tweak NAT to make things work. It should work out of the box. Period.

 

The second is a bad idea because what happens when more than one person behind a NAT wants to use VVL? Port forwarding as we mentioned before is limited to 1 address/port combination, and the world where each user has their own IP address fundamentally breaks down with NAT - you can have dozens, maybe hundreds of people coming from behind a NAT gateway. (Read up on carrier-grade NAT.) It's no longer a 1 IP to 1 computer, or even 1 IP to one residence or site.

 

What's more disturbing is that the problem is so darn easy to fix. All VVL needs to do is send a ping every few seconds. A nice small UDP packet whose entire purpose is to remind any NAT gateways in between that "I'm still here". Compared to the bandwidth utilized by voice traffic, it's pretty trivial - and it's similar to TCP keepalives. (This, by the way, is why I tell anyone wanting to use UDP that unless they really know what they're doing - I've been doing this for over a decade and I don't qualify - all they will end up doing is reimplementing TCP. Badly.)

 

So port forwarding with UDP isn't a matter of stuck packets or blockages of any kind. It's actually due to an absence of data, and an implementation that is woefully out of touch with how the Internet exists today.

 

I won't get into IPv6 support - each day hundreds of new users are added to the Internet that have no IPv4 connectivity or are going through NAT. If VATSIM cannot support IPv6 and NAT properly they are excluding far more users than the deaf or blind ones we seem to worry endlessly about at times.

 

Cheers!

 

Luke

[Ernesto Alvarez 818262]

see, hes an engineer

[Don Desfosse]

Luke, thanks so much for a layman's description! Now I really get it!

[Ross Carlson]

I had a conversation with the VVL developers years ago, and I recall that they did implement a ping to prevent UDP NAT timeouts. I don't recall, however, why it was removed, if it was even removed at all. It seems like it didn't work in all situations and so they just fell back to recommending port forwarding, but I could be remembering wrong.

 

Luke, some devs are looking at replacing VVL ... would you suggest we just use TCP? Is the TCP overhead no longer a factor with modern 'net connections?

[Luke Kolin]

I had a conversation with the VVL developers years ago, and I recall that they did implement a ping to prevent UDP NAT timeouts. I don't recall, however, why it was removed, if it was even removed at all. It seems like it didn't work in all situations and so they just fell back to recommending port forwarding, but I could be remembering wrong.

 

Interesting. It certainly doesn't behave like it's working, and I'm not aware of anything within NAT that would prevent it from working unless the ping interval was too low.

 

Luke, some devs are looking at replacing VVL ... would you suggest we just use TCP? Is the TCP overhead no longer a factor with modern 'net connections?

 

I am not an engineer, nor do I play one on TV. Certainly the overhead of TCP is a rounding error, but UDP has a few other interesting attributes, most importantly that it doesn't guarantee in-order delivery and has actual packets rather than being stream-based. I think this is what voice developers like most about it - you're better off dropping a packet than delaying the entire stream waiting for a piece of data to go to through.

 

If I were writing a voice infrastructure from scratch, I'd probably just say go with TCP unless (or until) you have problems. But given that VVL has done most of the work necessary and works in a significant number of cases, the best course of action might just be to ensure it works properly with NAT, IPv6 and multi-homed servers. That might be less work than a significant rewrite.

 

Again, I've done it enough to get over the Dunning-Kruger effect, little more. I am not an expert.

 

Cheers!

 

Luke

[Dave Seminchuk 1266919]

Hi

 

Hope its ok to revive an old thread as the previous notes maybe relevant.

 

Ive been looking forward to trying out vatsim but keep running into problems. The latest being voice comms being lost. Meaning the connection is still there but can only see text. If I tune to another frequency and go back, I will hear it for another 10sec or so and then its gone.

 

According to these old posts, it has to do with port forwarding of 3290. Its cleared in my security software firewall, but my router is completely IPv6 which doesnt have any sort of port forwarding, apparently because its not needed.

Not sure what else to do about this. Any suggestions?

[Ross Carlson]

my router is completely IPv6

 

I'm not a network engineer, so I may be missing something here, but as far as I can tell, all the VATSIM servers are IPv4. So if you can connect to VATSIM, then your router supports IPv4. What leads you to think it is IPv6 only? What router is it?

[Dave Seminchuk 1266919]

my router is completely IPv6

 

I'm not a network engineer, so I may be missing something here, but as far as I can tell, all the VATSIM servers are IPv4. So if you can connect to VATSIM, then your router supports IPv4. What leads you to think it is IPv6 only? What router is it?

 

Good question. Im sure it has backwards compatibility for IPV4 connections built in under the hood, but since it has IPV6 as its only connectionm port forwarding was apparently removed from a firmware update. The manual had port forwarding before and now its removed. Only IP port filtering is still there. Tried to use that but to no avail.

 

Its a modem router combo. Manual is here

https://www.upc.ch/dam/www-upc-cablecom-ch/business/pdf/Support1/b2b-ConnectBox_Manual_EN.PDF

[Ross Carlson]

So you're saying the port forwarding stuff mentioned on page 40 is not actually in the router UI anymore?

 

If that's the case, I'd say it's time for a new router, if you can't do a firmware downgrade to get the full functionality back.

[Dave Seminchuk 1266919]

Thats correct. No longer there. And port filtering specifically says IPV6 source IP. Seems very odd to me. But thats what they supplied. Guess I'll have to wait to hear from them.

Thanks for your time.