Voice Codecs, NDAs and Open Source

[Ross Carlson]

Ahh, finally Luke has joined the thread, restoring balance to The Force.

 

I suspect that Ross only wrote VPilot out of desperation or necessity;

 

I had already written the vPilot core for a commercial project, so the hard part was done. The main reason I ported it to VATSIM was that I wanted better model matching.

 

I don't know if things have gotten better (I think Ross was doing some work making C# libraries) but when you start writing code for VATSIM you'll discover that there are all sorts of impediments beyond the NDA. You'll end up writing a lot of boiler plate code to handle the basics of the protocol and unless you are one of the special breed of masochists who insist on writing C++, you have a ton of P/Invoke in your future dealing with VVL.

 

Things have gotten *slightly* better ... I put out C# libs for networking and for VVL, so no P/Invoke needed. Since then we've had another C# client released based on those libs. (vATIS)

 

Nothing has really changed in terms of good docomeentation, and there still isn't any development discussion happening in any accessible way. Mostly just emails between developers.

 

Luke, setting aside the problems with VATSIM's developer ecosystem and focusing strictly on the question of open-sourcing all of VATSIM's tech, I'd love to hear your thoughts on my concern about protecting the network from poorly-written clients. I'm really on the fence about this ... I don't know if it would actually be a problem. I just cringe at the idea of any rookie developer with Visual Studio Express being able to write a half-baked client and connect it to the network. (Not sure if you read the whole thread, but I gave an example earlier of how this could be disruptive.) Am I being overly paranoid? Or is it a legit concern and we need a way of vetting client code and controlling which clients can connect? And if so, how do we accomplish that and still be compatible with open source?

[Mathew Sutcliffe]

Hi Craig,

 

We at swift are pleased to see this subject getting more attention. It was Wade Williams (then VP Development) who first suggested and provided a lot of the political will to move VATSIM towards open source in 2012, but real world commitments forced him to step down before he could really achieve anything. Those forum threads have since been lost when the forums got hacked.

 

We have been working on swift for three years, dedicating a huge amount of our spare time to it. Our team leader has signed the NDA, the other three team members including myself have not, 99% of the code is not covered by the NDA. I can't speak for the others, but I would not be working on it if it were not going to be open source. So we have a strong interest in ensuring the future of open source in VATSIM, as that would be effectively three years of my life wasted if it didn't pan out.

 

Also - if any of the developers of Swift are reading this: you say your client is open source. Where's the source? Your project website makes lots of references to repositories, but doesn't actually link to them.

Our license is the GPL. When we release a binary, we'll be obliged to release the source. We haven't got to that stage yet. This is a cop-out answer, but it's technically the state of affiars as it stands. Yes, it's open source; no, we haven't released the source yet because we haven't released anything yet. Even Linux started life like this. ESR talks about it in The Cathedral & the Bazaar.

 

Ideally, we would like to release the source sooner than when the GPL legally obliges us to. A precondition for doing this is a stable and scalable platform for receiving bug reports and pull requests, and a project management platform that is generally better than what we're currently using. Github etc. don't really satisfy our needs. None of us is a web guru, and any time we spend setting this up is time we could be spending working on the code of swift. We've been asking for help with this, so if you're interested, drop us a line.

 

Every platform, whether it is open or closed, has its share of hacks. Just look at the Apache webserver, OpenSSH, Firefox, Google Chrome, Android and the Linux kernel. In fact, open source projects could be argued to be more secure, as there are far, far more eyes available to pour over the codebase than any software company can muster.

Even banks use open protocols for secure communications. There is no security in obscurity. These arguments have been made before. There is a slow movement on the political side of VATSIM towards accepting open source, and swift is something of a test case. This is another reason why releasing the code in an unfinished state might be a tactical mistake.

 

What if someone writes a client that makes it easy to disrupt others by deliberately hovering over aircraft and spamming text frequencies or someone implements the VVL protocol creating a client which facilitates the use of sound cards to play music over frequency.

This can already be done. The NDA and closed source does nothing to stop this.

 

It is nearly impossible to 100% enforce connections are coming from nonmalicous clients if the protocol is open sourced.

Absolutely and demonstrably untrue. You're using open protocols when you log into your online banking service, when you provide your credit card details to Amazon or Paypal, basically whenever you use the Internet in any way.

 

These are nontrivial problems that have serious and irreversible consequences. It isn't as simple as just "open sourcing" the protocols.

Please remember that you are talking to people who do this in their day jobs or who hold university degrees in it. We understand there are problems to overcome, nobody is suggesting it is simple. Management has an important job to do, but when an airline captain says he needs to land at the nearest suitable airport, it's time for management to get out of the way and let the pilots fly the plane.

 

nor is its development rapid.

Noone is more aware of this than us. Well played BTW Luke.

Edited January 28, 2016 at 08:40 PM by Guest

[Luke Kolin]

I had already written the vPilot core for a commercial project, so the hard part was done. The main reason I ported it to VATSIM was that I wanted better model matching.

 

Thanks. If you don't mind my asking, how much overlap is there between your commercial and hobbyist software projects? I certainly did a lot more sim-related development while there was still some significant overlap.

 

Nothing has really changed in terms of good docomeentation, and there still isn't any development discussion happening in any accessible way. Mostly just emails between developers.

 

Which is a huge shame. There's a ton of institutional knowledge getting lost in people's email accounts, and it's also a lot more difficult for people to come up to speed when they don't have the benefit of all those past conversations.

 

I didn't mention it earlier, but one of the best things that's ever happened to software development was Google, and then Stack Overflow - mostly because they facilitate and encourage the sharing of information. I can't think of how many times I've had to overcome some minor issue that was a major issue simply because I hadn't done it before. The modern software ecosystem is all about sharing, and anyone who's come up in that ecosystem can't imagine anything but. (It also helps deal with that initial development hump when you don't understand the platform well enough.)

 

In my mind, the next (if ever) VATGOV5 needs to open the platform, consolidate the docomeentation and ensure continuity of development via collaboration and code sharing. A brilliant developer that does not share is of limited to no use - what separates homo sapiens from other social animals that can communicate is our ability to record knowledge from generation to generation. No wolf, lion, dog or honeybee can do that.

 

I just cringe at the idea of any rookie developer with Visual Studio Express being able to write a half-baked client and connect it to the network.

 

Ross, when you go to bed tonight, get on your knees and pray that "tomorrow, 20 new developers will fire up Visual Studio Express and want to write a VATSIM client". Is it a badly behaved client a legitimate concern? Maybe. But it's so far down your list of concerns that it's really not worth worrying about. It's like building a house that can withstand an metorite strike but doesn't have heat or hot water.

 

Here's the reality - you already have half-baked clients connecting to the network each and every day. They're called FSINN and SB4 (and whatever generates those NaNs that are going out). Having that debug [Mod - Happy Thoughts]ertion or completely messed up weather above FL240 does far more damage to the VATSIM community's experience (and the reputation of this network) than a pilot client that updates at 2Hz instead of 0.2Hz. The network has spent all this time guarding against a problem, and its cure is worse than the disease.

 

(As an aside, FSD needs to do some sanity checking of position payloads, and if you're worried about flooding things add a rate limiter on them. And VATSIM needs to either STF about bandwidth or GZIP the TCP streams.)

 

(If it makes you sleep better at night, just give each client a private access key and have it salt the hash of the CID with said key. It prevents trivial attacks or new clients, but makes no claims to being real security. You don't need real security. You need developers and clients.)

 

Again, the lack of new development and ability to fix existing broken clients is the security greatest threat VATSIM currently faces. That needs to be fixed before tactical edge cases.

 

Cheers!

 

Luke

[Colin Schoen]

 

What if someone writes a client that makes it easy to disrupt others by deliberately hovering over aircraft and spamming text frequencies or someone implements the VVL protocol creating a client which facilitates the use of sound cards to play music over frequency.

This can already be done. The NDA and closed source does nothing to stop this.

 

 

You are right it just makes it a hell of a lot harder. If we had an infinite amount of resources we would fix VVL so that it had some authentication layer. If you open source it as is there will literally be nothing that we can do to stop voice abusers.

 

It is nearly impossible to 100% enforce connections are coming from nonmalicous clients if the protocol is open sourced.

Absolutely and demonstrably untrue. You're using open protocols when you log into your online banking service, when you provide your credit card details to Amazon or Paypal, basically whenever you use the Internet in any way.

 

I feel like we are going in circles here. See the previous discussion about how many of these protocols are not comparable to FSD.

 

These are nontrivial problems that have serious and irreversible consequences. It isn't as simple as just "open sourcing" the protocols.

Please remember that you are talking to people who do this in their day jobs or who hold university degrees in it. We understand there are problems to overcome, nobody is suggesting it is simple. Management has an important job to do, but when an airline captain says he needs to land at the nearest suitable airport, it's time for management to get out of the way and let the pilots fly the plane.

 

I'm not making any [Mod - Happy Thoughts]umptions about the backgrounds of those involved in the conversation. I too have expressed in many previous forum discussions and including personal talks with VATGOV1 on multiple occasions that I am in favor of open sourcing the code base with some limitations.

 

I am not going to go into detail, but as I mentioned previously there are some tools that help keep the trouble makers away that just would not be as effective if the internals were public. Yes, it is security by obscurity, but for the majority of members whose sole intention of joining the network is to cause chaos, the tools have proved rather powerful.

 

Perhaps this is just a risk we have to take in order to step forward...

[Ross Carlson]

If you don't mind my asking, how much overlap is there between your commercial and hobbyist software projects?

 

In terms of code? It's hard for me to quantify, but I can say that probably 70% of the code for vSTARS and vERAM is used in my commercial projects, and probably 50% of the vPilot code. Those chunks represent maybe 20% of the code base for my commercial projects, if that. Not sure if that answers your question ... why do you ask?

 

I certainly did a lot more sim-related development while there was still some significant overlap.

 

What do you mean by that?

 

I just cringe at the idea of any rookie developer with Visual Studio Express being able to write a half-baked client and connect it to the network.

 

Ross, when you go to bed tonight, get on your knees and pray that "tomorrow, 20 new developers will fire up Visual Studio Express and want to write a VATSIM client".

 

That comes across as a bit of a straw man ... I'm not saying I don't want new developers ... I'm saying I don't want rookie hack developers being able to connect garbage clients to the network fully unchecked.

 

Again, I don't know if it would actually be a problem ... I have no idea. I suppose we could always tighten things back up if it does actually become an issue.

 

Here's the reality - you already have half-baked clients connecting to the network each and every day. They're called FSINN and SB4 (and whatever generates those NaNs that are going out).

 

I have to disagree there ... those are not half-baked clients in the way I'm saying. Those clients all behave properly on the network as far as sending the right packets at the right time.

 

I'm actually not concerned with poorly-behaved pilot clients ... the network protocol surface area that they deal with is very small and easy to validate on the server. It's the ATC clients that are the concern for me ... the protocols for things like handoffs, track state management and the like are where a poorly-behaved client could cause disruption, and it could happen in subtle ways that are hard to track down.

 

(As an aside, FSD needs to do some sanity checking of position payloads, and if you're worried about flooding things add a rate limiter on them. And VATSIM needs to either STF about bandwidth or GZIP the TCP streams.)

 

I'm with you there. Have another look at my easy-vs-complex example ... I mentioned the position update rate as a simple thing to validate on the server. The more complex inter-ATC client interactions are the concern. And as I said in that same reply, really the whole protocol needs a ground-up rewrite so that handoffs, track ownership, and track state management are all done on the server. That would make this issue moot.

 

(If it makes you sleep better at night, just give each client a private access key and have it salt the hash of the CID with said key. It prevents trivial attacks or new clients, but makes no claims to being real security. You don't need real security. You need developers and clients.)

 

How would you do that and still have the client be open source?

 

Again, the lack of new development and ability to fix existing broken clients is the security greatest threat VATSIM currently faces. That needs to be fixed before tactical edge cases.

 

I absolutely agree ... we need to get things straight in the development house first, but I see no harm in discussing concerns about open sourcing everything. Don't let the fact that I'm raising this concern lead you to think that I see it as the #1 priority.

[James Horgan]

I must raise my hand, it was me who originally got Craig's attention with my post over at the VATSIM UK website. I wasn't bitter about not p[Mod - Happy Thoughts]ing my C1 - far from it. People put a large amount of effort into my mentoring, and I made mistakes. But the problems I caused/mishandled were exacerbated tremendously by the voice codec and the lag especially that I had to fight against to either be heard, or understood.

 

I like this idea of a bunch of cretins with no programming knowledge firing up Visual Studio and writing a client. I am one of those cretins. I am currently teaching myself C++ (I have no life) so I can write PFD/ND/ECAM etc software for my A330 home simulator project. One of my aims is to eventually integrate the upgraded TCAS into the MCDU, which displays callsigns and relative direction, but also be able to connect to VATSIM though the MCDU directly, and handle Mode-S messages through Hoppie's client or through VATSIM when it is instigated.

 

It just seems that things move at an incredibly slow pace. Yes people are volunteers, I understand that, but at the same time, it would be nice to have timeframes and publicised goals for moving forward at least, so people can see we are moving in the right direction.

[Luke Kolin]

If we had an infinite amount of resources we would fix VVL so that it had some authentication layer. If you open source it as is there will literally be nothing that we can do to stop voice abusers.

 

The people with the ability to comprehend source code and devise an attack already have sufficient technical ability to launch a successful attack against a poorly design service or protocol.

 

I didn't need the FSD source code or the VVL source code (neither of which I have ever seen, by the way) to understand the proper way to attack FSD, just a decent understanding of network protocols and software engineering. If I did not have either, the source code would not be of any value to me.

 

I feel like we are going in circles here. See the previous discussion about how many of these protocols are not comparable to FSD.

 

You're only going in circles because you don't understand. The protocols are the same - the only difference is that they're properly written. FSD is a toy, and a poorly written one at that.

 

I am not going to go into detail, but as I mentioned previously there are some tools that help keep the trouble makers away that just would not be as effective if the internals were public. Yes, it is security by obscurity, but for the majority of members whose sole intention of joining the network is to cause chaos, the tools have proved rather powerful.

 

See my first statement. There are enough public details about the nature of the software design and architecture that a anyone competent enough to understand the source code could already cause significant damage to the system without that being available.

 

Cheers!

 

Luke

[Luke Kolin]

Not sure if that answers your question ... why do you ask?

 

Mostly personal curiosity - you've lasted longer than I in the active FS development space; I did so because a lot of what I was writing for flight simulation could be reapplied commercially which keep the interest going. I was curious if there was a parallel.

 

You and I collaborated on software in 2004 - how many folks are around and active from then?

 

That comes across as a bit of a straw man ... I'm not saying I don't want new developers ... I'm saying I don't want rookie hack developers being able to connect garbage clients to the network fully unchecked.

 

Yes, there's a little dramatic license - but the point remains. Writing clients should be easy and the core components (network, voice) should be pre-made. I understand that you want to restrict clients, so keep the notion of a simple access key that prevents anyone from hooking up to production FSD at will. But that doesn't require an NDA.

 

The point is that we've gone so far towards protecting the network that our defense mechanisms actively "protect" against new developers.

 

I have to disagree there ... those are not half-baked clients in the way I'm saying. Those clients all behave properly on the network as far as sending the right packets at the right time.

 

Right, but that's just a small part of the whole experience. Where they are half-baked (and that is a little generous) has caused far more network and user disruption than a single rogue client. I fear this worry about a rogue client distracts us from the real problems and disruptions our members face every day.

 

By all means have a client kill switch. It just needs to be as simple as possible.

 

It's the ATC clients that are the concern for me ... the protocols for things like handoffs, track state management and the like are where a poorly-behaved client could cause disruption, and it could happen in subtle ways that are hard to track down.

 

You already know this, but if you're depending on your clients to be well-behaved, that's most of the problem. I agree that the server should have enough intelligence to sanity check certain messages and manage state. Never trust the client.

 

How would you do that and still have the client be open source?

 

As far as I am concerned, the key isn't part of the source code and is just a compile time definition. (I've not read up on the exact GPL requirements, but so long as no other GPL code is in the client the author needs only to waive their right to exercise GPL rights on the secret.) Worst case, you just say that users are liable for poorly behaved clients and keep a "safe harbor" list.

 

Again, at the end of the day you really can't trust the client very much. The server needs to have basic protections built into it.

 

Cheers!

 

Luke

[Craig Watson 1344682]

Wow ... thanks to everyone for getting involved - lots of great discussions

 

No, the security mechanisms there are to secure against something more likely and dangerous than a cyber attack - a fork.

 

Finally, a reason for an NDA that I can (kind of) understand - although it is still flawed. There is one world-famous community that is currently in a fair amount of uncertainty due to the danger of a fork - and that is Bitcoin.

 

That said, the Bitcoin fork is primarily being debated due to splits in the leadership. In actual fact, the danger of forking could be a concrete case for open source - bear with me!

 

Say, for example, that the VATSIM network gets into a state where a good number of people are unhappy at the way the network is run - be it technological, or governmental - sound familar? That unhappy portion of the membership would then have the power, and the means, to fork the core VATSIM code and create their own network.

 

However, that risk would be enough to keep the leadership from letting the network get into that state in the first place. The leadership of any community is always accountable to their members, and this would put the membership firmly in the driving seat.

 

I need to stress that the possibility of a fork should not be a scythe to be held over the heads of the leadership, however it does help focus the leadership team on the things that should already matter the most - keeping the network happy and moving things forward, instead of getting complacent and introverted.

 

Our license is the GPL. When we release a binary, we'll be obliged to release the source. We haven't got to that stage yet. This is a cop-out answer, but it's technically the state of affiars as it stands. Yes, it's open source; no, we haven't released the source yet because we haven't released anything yet. Even Linux started life like this. ESR talks about it in The Cathedral & the Bazaar.

 

Thanks Matthew for responding on behalf of Swift - though I'm going to nit-pick for a moment. Specifically on the use of the word "obliged". Publishing the source isn't (IMO) shouldn't be an obligation that is done through gritted teeth because you're bound by a license to do so.

 

Despite that, the decision to keep the code under wraps isn't what I'd necessarily agree with as it elongates the entire process and makes it even less likely that developers would actively contact you and offer their [Mod - Happy Thoughts]istance, and also actually reinforces the (false) view that Swift is actually not open source at all.

 

If active development is done "in the open", then progress would be clearly visible and encouraging to those waiting for the client, and pull requests could come in from literally anyone - you could essentially crowd-source the entire application.

 

I appreciate that Linux was closed until the initial release, but with all due respect, Swift is not the Linux kernel - I really do think that opening up the codebase could be the catalyst to getting things moving.

 

A precondition for doing this is a stable and scalable platform for receiving bug reports and pull requests, and a project management platform that is generally better than what we're currently using.

 

Atl[Mod - Happy Thoughts]ian's JIRA is free for open source projects. If you don't want anything that "heavy", there's always Trello. As for bug reports and pull requests - GitHub, or if you really want to keep it in the Atl[Mod - Happy Thoughts]ian family, BitBucket.

 

Personally, I'd recommend the Trello+GitHub approach - you can make use of GitHub's wikis for docomeentation and pages for the static website. No muss, no fuss.

 

If you open source it as is there will literally be nothing that we can do to stop voice abusers.

 

This continutes to puzzle me. Why would an open source client be any more likely to be used for "voice abuse" than an already existing closed client? I also refer you to my question from a few posts ago:

 

What's stopping me plugging my iPhone into my microphone or line-in port and holding down my PTT key - with the current protocol and client software?

 

Finally:

 

Perhaps this is just a risk we have to take in order to step forward...

 

This.

[Ross Carlson]

Mostly personal curiosity - you've lasted longer than I in the active FS development space; I did so because a lot of what I was writing for flight simulation could be reapplied commercially which keep the interest going. I was curious if there was a parallel.

 

Definitely a parallel ... I'm not sure I would still be doing VATSIM dev if it weren't for the fact that the dev work I've done for VATSIM in the past led to commercial opportunities that I'm still involved in today.

 

You and I collaborated on software in 2004 - how many folks are around and active from then?

 

I think I'm the only one.

 

Yes, there's a little dramatic license - but the point remains. Writing clients should be easy and the core components (network, voice) should be pre-made. I understand that you want to restrict clients, so keep the notion of a simple access key that prevents anyone from hooking up to production FSD at will. But that doesn't require an NDA.

 

The point is that we've gone so far towards protecting the network that our defense mechanisms actively "protect" against new developers.

 

I [Mod - Happy Thoughts]ume you keep repeating that for the benefit of others reading this thread ... I'm fully aware of how diseased the dev ecosystem is here.

 

I fear this worry about a rogue client distracts us from the real problems and disruptions our members face every day.

 

I agree, IF the rogue client concern is not a real issue. If it is a real issue, then it's not a distraction ... it's just another real issue that we would have to deal with if we opened things up. I realize you don't see it as a valid concern ... I'm still not convinced either way. The fact that I've been so deeply involved with the ATC client dev side of things, where you have not (right?) may explain that disparity.

 

Also, a way to control which clients can connect will go a long way towards getting the VATSIM staff on board with open source.

 

By all means have a client kill switch. It just needs to be as simple as possible.

 

Definitely ... a new access key would need to be given out to any dev wanting to experiment ... perhaps initially the key would only work on a sandbox server, and then be approved for use on the live network after some code review happens. Just thinking out loud here ... what do you think?

 

Or would you go even simpler and allow devs to connect unproven clients to the live network right away, and then shut them down if they are found to be causing issues?

 

You already know this, but if you're depending on your clients to be well-behaved, that's most of the problem. I agree that the server should have enough intelligence to sanity check certain messages and manage state. Never trust the client.

 

I'm with you 100% ... and I think a full server and protocol rewrite will be necessary to get us there. That would be a wonderful thing.

 

How would you do that and still have the client be open source?

 

As far as I am concerned, the key isn't part of the source code and is just a compile time definition. (I've not read up on the exact GPL requirements, but so long as no other GPL code is in the client the author needs only to waive their right to exercise GPL rights on the secret.) Worst case, you just say that users are liable for poorly behaved clients and keep a "safe harbor" list.

 

I had long discussions about this with other devs years ago, and I thought we concluded that you couldn't have private keys compiled into any app that used GPL code, since that app couldn't be GPL unless you distributed everything that was required to compile it. However, now that I think about it, it wasn't the secret key that was the issue ... it was the code that made use of that key and implemented the protection protocol. So as long as our validation scheme was open source, we could still keep the individual client keys private, and still be GPL compatible, right?

[Bradley Johnson 1218984]

I've read the entire thread to this point, but before I post my thoughts or opinions on the subject, I do have one question that only the Founders and the BoG can answer:

 

Can the Founders and the BoG put aside a fear of a competing network being started with Open Source software if VATSIM's infrastructure were to be redeveloped using an open methodology?

 

Put all security, code standards, QA vetting, rogue clients concerns aside to just a moment when answering that question. it is an important question, and one that needs to be thoughtfully answered to ensure that alignment is even possible. Because, putting all technical concerns aside, it is the one thing that would make any move away from the current model impossible.

[Mathew Sutcliffe]

Put all security, code standards, QA vetting, rogue clients concerns aside to just a moment when answering that question. it is an important question, and one that needs to be thoughtfully answered to ensure that alignment is even possible. Because, putting all technical concerns aside, it is the one thing that would make any move away from the current model impossible.

And ironically, sticking with the current model is one thing that can make a competing network built from scratch more likely in the long term. OTOH, there are already competing networks in existence, and VATSIM is still here. Whatever our beliefs about software, we're all here because we love VATSIM for its community.

 

Thanks Matthew for responding on behalf of Swift - though I'm going to nit-pick for a moment. Specifically on the use of the word "obliged". Publishing the source isn't (IMO) shouldn't be an obligation that is done through gritted teeth because you're bound by a license to do so.

I wholeheartedly agree with you and I'm sorry if I didn't make that point strongly enough. And I'm not responding "on behalf of" swift, I'm just one member and like most teams we don't always agree about everything, so I didn't want to come on too strong with some of my opinions which aren't necessarily in line with the rest of the team.

 

Atl[Mod - Happy Thoughts]ian's JIRA is free for open source projects. If you don't want anything that "heavy", there's always Trello. As for bug reports and pull requests - GitHub, or if you really want to keep it in the Atl[Mod - Happy Thoughts]ian family, BitBucket. Personally, I'd recommend the Trello+GitHub approach - you can make use of GitHub's wikis for docomeentation and pages for the static website. No muss, no fuss.

I edited my post because I forgot to mention Github wouldn't really fulfull what we want. We've looked at the options and so far we're gravitating towards Phabricator. We want to have control over the server that everything runs on. We need continuous integration for our three target operating systems, we're currently using Jenkins which is great, but we need a dedicated person to sysadmin it all for us.

 

I had long discussions about this with other devs years ago, and I thought we concluded that you couldn't have private keys compiled into any app that used GPL code, since that app couldn't be GPL unless you distributed everything that was required to compile it. However, now that I think about it, it wasn't the secret key that was the issue ... it was the code that made use of that key and implemented the protection protocol. So as long as our validation scheme was open source, we could still keep the individual client keys private, and still be GPL compatible, right?

swift is GPL with a "further restriction" (as defined by §7 of GPLv3) which permits linking with our closed source vatlib shim.

 

The point is that we've gone so far towards protecting the network that our defense mechanisms actively "protect" against new developers.

 

I fear this worry about a rogue client distracts us from the real problems and disruptions our members face every day.

This is basically the whole point in a nutshell. It's like an immune system rejecting a donor kidney, even though that kidney could save its life.

[Luke Kolin]

The fact that I've been so deeply involved with the ATC client dev side of things, where you have not (right?) may explain that disparity.

 

It's possible, but look at my earlier posts here. You have two potential problems - either a malicious attack or an honest software bug. The former isn't going to be caused by someone registering via your development program. The latter is always a risk, and has been so long as we've had pre-release software being tested on the network.

 

Definitely ... a new access key would need to be given out to any dev wanting to experiment ... perhaps initially the key would only work on a sandbox server, and then be approved for use on the live network after some code review happens. Just thinking out loud here ... what do you think?

 

I don't have a problem putting in some basic restrictions as to what clients (and versions) can connect to the production network, to allow deprecation of clients and basic security measures.

 

(If you want to test, providing a sandbox isn't enough or the best way. VATSIM probably should release some EBS images so that people can launch FSD on a t2.nano in AWS. They're $4.75/month to run 24/7.)

 

I'm with you 100% ... and I think a full server and protocol rewrite will be necessary to get us there. That would be a wonderful thing.

 

Absolutely. The best decision I ever made with the ACARS protocol was to design it from the ground up to have multiple concurrent protocol versions in place. There's a few rough edges still, but the protocol allows for revision escalation and compression on the fly without forcing a big-bang change. VATSIM needs to start the prep work on what FSD 2.0 would look like, running in parallel with FSD v1.0 (protocol-wise).

 

However, now that I think about it, it wasn't the secret key that was the issue ... it was the code that made use of that key and implemented the protection protocol. So as long as our validation scheme was open source, we could still keep the individual client keys private, and still be GPL compatible, right?

 

Yes. The app will compile and operate if the secret key is #defined as 1234; it just won't connect to the VATSIM servers. The binaries wouldn't be identical, but then again they're not identical if you compile with -O2 instead of -O1 or -O3. I think providing the #define without its actual value would be compliant with the spirit of the GPL.

 

 

Looking back in retrospect, the challenge that I have is that I'm a software developer that believes in collaboration, garbage collected languages and making a really kick-[Mod - Happy Thoughts] experience for pilots. (I find ATC interesting, but there's enough people fascinating with its esoterica like yourself that it will be OK). What was so much fun about what we did a decade ago was that it was exciting and providing new data that at the time we were only starting to think of what we could use it for.

 

What makes me sad and disappointed is what's happened over the past decade. In 2002 I was flying and getting two online friends to teach me online flying by manning multiple positions in two different ARTCCs throughout our flight. I still remember when this was driven by the WOW factor of flying commercial airliner similators, seeing your friends right beside you and being able to fly and control all over. Now there just seems to be a great deal of fear - fear of unqualified participants, poorly behaved software or hack attacks.

 

That's a bit of an oversimplification, but to me the wonder and joy seemed to leach out of this ecosystem. Which is a shame, because flying and controlling planes together with people from all over is a wonderful, joyous idea. Writing really good software to facilitate this has been equally wonderful. It's just a shame that VATSIM needed to not be involved for it to be that way.

 

It's not too late to fix it.

 

Cheers!

 

Luke

[Mike Evans]

The Servinfo feed is equally pitiful. It's again that same unecoded colon-delimited data delivery. The data update interval hasn't changed and at 2-3 minutes it's so slow to be almost useless. It shouldn't be hard to crank the update interval down to seconds and put a network of caching proxies in front of the generator. Don't even get me started on the server list - it boggles the mind that we still have people typing in IP addresses in 2016.

 

Cranking the interval down is trivial, the Data Server already generates a realtime feed that isn't published, but if its changing every second, what exactly would a caching proxy provide? The cache is immediately out of date.

[Luke Kolin]

Cranking the interval down is trivial, the Data Server already generates a realtime feed that isn't published, but if its changing every second, what exactly would a caching proxy provide? The cache is immediately out of date.

 

Sorry, I think you misunderstood - the refresh time should be reduced to seconds, not necessarily a single second. Even 30 or 45 seconds would provide a lot more (and better) information than it does now.

 

As an aside, you'd be amazed what even a 1s TTL can do on a high volume service.

 

Cheers!

 

Luke

[Richard Jenkins]

I haven't read all the comments but I am sure they are very eloquent and well thought out.

 

One way or another OS is coming to VATSIM. The founders will have to accept it and many already do! Over the last 3 or 4 years the cost of running VATSIM has decreased dramatically. Costs are down and bandwidth is up! The protections that were put in place to protect certain aspects of running the network are no longer needed.

 

I see OS happening as soon as we can develop a hardened server that can deal with differing clients. Who is going to do that? Got me!

 

RJ

[Kirk Christie]

I don't have any thing to contribute to the topic of discussion, however I will say this.

 

This is a hobby that I am p[Mod - Happy Thoughts]ionate about, it takes a lot of my time, so im always reading information to see whats going on and how the future of my hobby will be maintained, one of those things I read are the EC and BOG meeting minuets, this things tell me that there is actually stuff going on at the top end to improve and enhance the network.

 

To the OP and the person with the codec problems, maybe you should consider reading these docomeents, you would have already known that the ball was rolling on improving the voice codecs as mentioned at item 6 of the Q3 BOG meeting you would have saved your self a 3 page argument. After that you need to remind your self that this is a volunteer organisation, and that things take time.

[Craig Watson 1344682]

To the OP and the person with the codec problems, maybe you should consider reading these docomeents, you would have already known that the ball was rolling on improving the voice codecs as mentioned at item 6 of the Q3 BOG meeting you would have saved your self a 3 page argument. After that you need to remind your self that this is a volunteer organisation, and that things take time.

 

Thanks for taking the time to reply. My entire reason for bringing up the NDA is my view that it is holding the network back and contributing to the glacially slow response to a problem that should not exist.

 

I am not, under any circomestances, bashing anyone for a lack of intent - it's clear that there are people in the community who genuinely do want to further things. I also write, contribute to, maintain and publish several open source projects, and am actively involved in a community of open source development - please don't think me ignorant of what actually goes into writing and publishing software, and please don't think me ungrateful of those efforts.

 

However, actions speak louder than words. The BoG can sit in meetings and discuss things for as long as they want, but it still won't stop issues like James' happening. Simply - the only progress that actually means anything (to me) is code being committed, and releases being done to production. Anything else is management waffle.

[Christopher Collins]

I haven't read all the comments but I am sure they are very eloquent and well thought out.

 

One way or another OS is coming to VATSIM. The founders will have to accept it and many already do! Over the last 3 or 4 years the cost of running VATSIM has decreased dramatically. Costs are down and bandwidth is up! The protections that were put in place to protect certain aspects of running the network are no longer needed.

 

I see OS happening as soon as we can develop a hardened server that can deal with differing clients. Who is going to do that? Got me!

 

I guess I can stick my neck out here in the long run if I can get an agreement to relinquish the NDA over new protocol/platform development or any from-scratch server development. (I'm happy to work within the NDA for code maintenance, but I'll be damned if I'm rewriting something that big to be kept behind sealed doors).

[Michael Filhol 1269327]

Hello,

Coming across this thread while looking up the network protocol, I must say I felt some warmth in my veins reading. First because I now fully understand that my unwillingness to tie myself to NDAs and my wish to contribute to the sim world with free software only, are shared feelings. Moreover, I realised for once that the discussion does not date back years but rather shows an actual, present and public debate. So thank you.

 

I am the author of ATC-pie, a recent ATC client which has already approached VATSIM in a few ways, and incidentally been used by several VATSIM ATCs too, including long-time top-rated mentors. So I am one of those candidates for "new client connection", and I promise I try never to half-bake my Pie.

 

Some Swift people know me because their closed lib is a hanging option for bridging VATSIM and the free world, but it is apparently not ready yet so I have to wait. Or, I sign the NDA and do my own closed lib, but will need to maintain compiled versions for different platforms, two of the major ones I do not even (want to) own...

 

I claim nothing great to say here but for what it is worth, I vividly testify that the piece of secrecy being discussed in this thread is indeed the tiny and last piece that is preventing me from proposing a VATSIM connection.

[Ernesto Alvarez 818262]

I claim nothing great to say here but for what it is worth, I vividly testify that the piece of secrecy being discussed in this thread is indeed the tiny and last piece that is preventing me from proposing a VATSIM connection.

 

not totally the last. your software would also have to go through a code review to be approved last i heard. its not just gaining access to the keys and off you go, have to actually get it approved as well

[David Zhong]

And the latter requirement would likely remain even if the protocol were made public.

[Mathew Sutcliffe]

The existence of an approval process doesn't prevent Michael from proposing a new client.

[Michael Filhol 1269327]

I was indeed going to answer that sooner if I were not to board a plane

I did mean last before proposing, and moreover do appreciate and condone the existence of a reviewing and approval process.

[Halldor Bui Jonsson]

Not sure if i'm performing a cardinal sin here by reviving such an old thread, but I guess I´ll take my chances.

 

Have there been any new updates regarding improved voice codecs? The ancient roger-wilco like voice quality, especially the lag keeps me from getting back behind a scope. I just can't deal with it now that I work on the airwaves from day to day.

 

Is there any light at the end of this tunnel?