New Forum?

[Andreas Fuchs]

Perfect, thanks for the reply, Richard!

[Don Desfosse]

Hi everyone,

A few facts and/or thoughts for folks to consider.

At the time of this writing, our most recent CID issued is 1700488.  With the exception of a couple dozen Founders CIDs and CIDs used as part of our infrastructure, member CID issuance started at 810001, meaning we've had 890,487 people initiate a registration.  There are many of those that never confirm their registration, some that come and go, etc.  Another measure is "active registrations".  We currently show just shy of 140,000 "active" users, meaning they have registered and/or connected to the network at least once in the last 18 months.  If you shorten that to 3 months, there are roughly 30,000 active members.  

We are currently operating at a cost of roughly $12K US/year.  That is a significantly degraded service, as we've been cutting and slicing and chopping services to save money over the last year or so.  We are very new to this fundraising stuff -- not even a month in -- so are learning more each day.  That said, in order to restore some of the capability we had to cut, and thinking of slight growth/improvement, our initial projections are needing somewhere in the range of $18K/year.  You are almost correct that roughly 1% currently contribute, though, sadly, it's even less than that.  Currently only 0.75% (yes, that's only three quarters of a percent) have donated.  And very few of those, less than 20%, signed up for recurring donations.  For a more exact number, only 0.14% (or about an eighth of a percent) of our active users have set up a recurring donation to keep VATSIM running in perpetuity.  I personally find this to be an astonishingly low number for the amazing benefit the members of our community receive, but that's what the current data shows.  

If every one of our active members donated an equal share to meet our initial budget, the contribution would be sixty cents (US) per year.  However, at the current less-than-1% response rate, our current contributors, if those are the only folks/numbers we could count on donating, they would need to contribute an average of $100 per year.  We are nowhere near this.  Nor, frankly, should we be asking that much from any one person.  The alternative, of course, is to greatly increase the number of people who donate to the community.  

Currently, donations made to date account for roughly 30% of our 2023 need, most of which were made within the first five days of our call for donations, and dropped to near-zero since.  If you consider sustainability, and look at the recurring donations, we are closer to 20% of our need in order to sustain (and maybe even slightly grow/improve) VATSIM.

So, yes, we need to increase our donation rate, both in terms of size, commitment to recurring donations, and most importantly number/percentage of people in the community donating.  

Some members in our community truly can't afford to donate anything at all.  They are barely able to run a sim or a radar screen on their computer, but have a love -- many have a true passion -- for aviation.  This community is absolute life and a stepping stone toward achieving a passionate dream for those people.  Some members in our community could likely afford to donate dozens, if not hundreds, of dollars per year toward their own enjoyment as well as advancing a community filled with aviation passion and stepping stones to others' potential careers and/or passions.

We don't charge a fee; we ask for the grace and kindness and compassion, in time, talents and treasures, from those that can afford to contribute to our community and our community members' dreams.

So back to the question at hand, yes, the cost vs. value of the forums is very much in question.  I am personally a huge proponent of keeping the forums, at least in some way, shape or form, for all of the reasons already stated.  The decision is not yet a done deal, though if our situation doesn't improve quickly, it may be, and not in a way I'm in favor of.  

In terms of fundraising reminders, we're 3 weeks into our new community funding opportunity/journey.  We are trying to find the right balance between too frequent outreach (which some have already accused us of), and not asking enough (which others have suggested) which could cause the organization to fail financially.  And with so many folks in the community, that balance is a little different individually.  But we're trying to find the right overall balance for the community.  We've never done this before within this community/environment.  Which isn't to say that we're completely ignorant -- we're not -- but we're trying to find the best balance with respect to community outreach for fundraising, publishing financials to give folks transparency, etc.  It will never be perfect for most folks individually, but we're working to make it at least reasonable -- if not as close to perfect as possible -- for the overall community.  And we appreciate everyone's patience as we tweak variables to help get us there.

I hope this helps add some data and some perspective.

[Andreas Fuchs]

Hi Don,

thanks for sharing all these details! In short: I think that these figures and the need for further donations should be published in the places that are visited by more members: VATSIM's Facebook page, VATSIM's Discord and it should be propagated down to all Regions and Divisions for distribution in their local communication channels.

[Don Desfosse]

I appreciate and agree.  I'll speak to my colleagues in Marketing and Communications.  Thanks for the suggestions!

[Callum McLoughlin]

Has any consideration been given to the overall rationalisation of services and costs? There are so so many local websites and forums running, that have to now (and will continue to be?) paid and financed separately. Is there any economy of scale where everybody wins? I think that would also yield improvement in some areas such as time duplicating building and running websites, when only one is needed, and so on. I think it would also lead to better sharing of capabilities (be it web systems or people) that some places have but others do not.

I think 'the other network' has everything in one place and it works for them. Seeing VATSIM not only survive but prosper is in everybody's interest!

Edited February 8, 2023 at 10:02 AM by Callum McLoughlin

[Don Desfosse]

It's a great point, and one that we'll look into.  One challenge that we've had in the past is local facilities' interest in total control.  But I think it's been a while since it's been seriously looked into.  Thanks for the suggestion!

[Richard Jenkins]

13 hours ago, Callum McLoughlin said:

Has any consideration been given to the overall rationalisation of services and costs? There are so so many local websites and forums running, that have to now (and will continue to be?) paid and financed separately. Is there any economy of scale where everybody wins? I think that would also yield improvement in some areas such as time duplicating building and running websites, when only one is needed, and so on. I think it would also lead to better sharing of capabilities (be it web systems or people) that some places have but others do not.

I think 'the other network' has everything in one place and it works for them. Seeing VATSIM not only survive but prosper is in everybody's interest!

Hello -

 I have been looking at this for several months now and what I have found is the level of web content varies greatly between facilities. Some are existing on very cheap DreamHost type accounts for a few dollars per month; while others have built and maintain extremely complex and resource intensive systems. This is the nature of VATSIM and feel it is this individuality that pushes VATSIM forward. Yes, you are correct, the "Other Place" does provide hosting for their facilities and  uses a boilerplate type system to achieve this and they have some incredibly miserable sites as a result. Just compare our ZLA with their ZLA as an example. In short - Some of the systems built by facilities we just can't afford. Others we could probably help with and we have in the past. 

Economy of scale, in my opinion, is great for the bottom line but does take a lot of the flexibility away from the local staff. I guess it depends how much independence facilities would be willing to give up to have the resource funded. Which at this point is kind of a pointless discussion because we are just trying to keep the VATSIM core services up and running. We have things that need to be looked at immediately and in the very near term:

1. Kubernetes cluster needs additional nodes and/or upgrades to existing nodes.

2. AFV ranging queue serves have extended wait times during high utilization of the FSD network. Code review and/or additional compute resources needed.

3. Increased demand for FSD network. Recently deployed additional server to increase cap limit to 2,400 connections. Review use of multiple ATIS policy and determine impact on FSD network. 

4. Potential migration of voice ATIS service from existing location to core VATSIM location. Required resources and associated costs.

5. Growth of online stats database. Now contains in excess of 20+ years of information and causing performance issues. Feasibility of archiving/pruning/architecture of data to decrease resource demand and improve performance.

6. Resource planning for additional compute for upcoming ATC clients.

7. Forums - What to do? Continue with existing and pay for a new license (non-profit pricing query) and the semi-annual update fees or move to a new free version with data migration issues to be resolved?

8. Review of existing development requests and their associated costs and ROI.

9. Review of email services. We are currently sending in excess of 31,000 emails per month from core services. Not a cost issue but more so a demand issue.

[Callum McLoughlin]

Thanks Richard, that is a really good summary. The transparency on this issue, and the level of communication is very good and extremely refreshing.

[Sean Harrison]

If members currently donate to their Region/Division how does that help out the USA?  I donate money; and previously time and efforts; locally.  Do we expect members to also donate to VATSIM?  If I stop donating locally and give it to VATSIM inc, will there be any benefit to local regions/divisions?  Hopefully some thought has gone into the holistic Vatsim structure and the support thereof.

[Trevor Hannant]

I agree with Callum regarding the transparency here.

From an "economy of scale" perspective, is there something that can be done on licensing of software platforms and hosting that would still allow the individuality of Regions/Divisions?   For example,  software licenses that could be combined under a single purchase account with multiple license purchases leading to bulk discounts?

[Don Desfosse]

50 minutes ago, Sean Harrison said:

If members currently donate to their Region/Division how does that help out the USA? 

Hi Sean, I'm not sure I understand this first question.

 

50 minutes ago, Sean Harrison said:

I donate money; and previously time and efforts; locally.  Do we expect members to also donate to VATSIM?  

I would say although there is not any specific expectation, if we want VATSIM to continue, it is certainly necessary.  As I mentioned above, we know that not every member of the community can afford to donate, but the hope certainly is that those that can, will.

 

50 minutes ago, Sean Harrison said:

If I stop donating locally and give it to VATSIM inc, will there be any benefit to local regions/divisions?  Hopefully some thought has gone into the holistic Vatsim structure and the support thereof.

There will always be a need at the global and at the local level.  For many folks, it won't be an all or nothing thing in terms of where their time, talents and treasures are distributed.

Thought has certainly gone into the holistic structure.  However, as was mentioned above, right now there isn't enough funding to keep the global VATSIM network alive through the end of the year.  If VATSIM dies, there will be no local regions or divisions to worry about.

The Marketing and Comms folks are probably going to be upset about how blunt I'm being, but this is truly transparency.... 😉

[Richard Jenkins]

1 hour ago, Sean Harrison said:

If members currently donate to their Region/Division how does that help out the USA?  I donate money; and previously time and efforts; locally.  Do we expect members to also donate to VATSIM?  If I stop donating locally and give it to VATSIM inc, will there be any benefit to local regions/divisions?  Hopefully some thought has gone into the holistic Vatsim structure and the support thereof.

Hi Sean -

I would suggest that if you are contributing to a local facility (ACC/Division/Region)  you continue to do that and not contribute to VATSIM. It would not be appropriate to drain resources from one need to another need in this scenario. The amount of people donating to local facilities is extremely small in comparison to the entire active VATSIM population and we should be able to support VATSIM core services from that population.

[Sean Harrison]

As an observation,  twenty years ago when member numbers were smaller, we managed to exist with donations of servers and time and effort.  We grow and have substantially more members, servers are reduced and we have a massive staff list.  I understand that the cost of infrastructure increases but not dramatically against inflation.  (Members earn more now, and infrastructure costs more).  We appear to have rationalised infrastructure, so numerically we have less hardware.  Members develop and offer software.  From an outsiders view it would appear we are spending more than we have(at top level).  Why is this so?  Should our strategic plan include devolution and revert back to regions which provide servers etc, do away with the huge upper structure, and simply make each Regional director a board member.  Without a detailed expenditure and income sheet what else should we consider.  It is hard to fathom that with reduced servers etc we still don’t have sufficient donations with an increased membership.

[Sean Harrison]

Further on this, driving membership involvement can only assist.  At the moment I don’t believe there is sufficient demonstrated reason to donate.  This should be done by open advice to members.  I used to commit myself to 10% of my online time was ATC so that I gave time back.  That stopped because of the “restrictive” policies etc that were put in place, so I gave up trying to offer ATC services to other members.  It’s not something I enjoyed, but I did it to help others.

 

So with sufficient openness on the reason to donate, maybe we could frame a suggestive costing; “ten hours on line by a member effectively costs $1” please consider supporting our community.

[Don Desfosse]

Lots of good questions -- I can answer at least some of them.

Decentralized (think donated) servers and contributions to infrastructure have caused significant issues in the past, both in terms of lack of / delay in standardization, breakage, downtime, response time, etc.  They have also caused a certain amount of risk to the network.  A decision was made to bring everything in house, as the community was growing beyond the historical patience with such issues.  This caused our costs to roughly triple.

For over two decades, we've had a small number of people that were footing the bill for the whole network.  As costs increased, and frankly as some of those folks have quite literally died off, the remaining folks have found it untenable to continue footing the bill themselves.  To ensure the sustainability of the network, we needed a renewed funding and sustainment plan.

Each region lead is already a VP on the Board of Governors.

I hear you, but we've been hesitant to put a "suggested contribution" out there.  Some of that discussion has included that there are plenty of people of all ages and all walks of life that might not be able to afford even a meager donation.  So we've been reticent to publish a "recommended" number out of respect to those folks.  We really want to keep VATSIM free to use for all that can use it.  Peoples' generous donations will allow us to do that.  

But if you keep the numbers in context, out of over 140,000 active members, our current contribution rate is 0.14% of members that have made any kind of contribution.  Amazingly, that tremendously small number of people have committed to contributing roughly 20% of what we need to sustain the network for the next year.  That's only one-seventh of a percent of people that have found it in their heart and means to donate, but they were very, very generous.  That is a tremendously generous donation made by precious few people.  If we could only get 100, or 200, or even 300 more folks to commit to donating $5/month USD (or their local equivalent), and a thousand folks to contribute $5-10 USD a year, we would be in wonderful shape to not only sustain, but to modestly grow/improve, the network.  I have to imagine out of 140,000 active members, at least a thousand folks those could forego the equivalent of one coffee/drink at Starbucks (or their preferred establishment) per year to support this phenomenal network.

Thanks for all the great ideas!  Keep them coming!

[Callum McLoughlin]

If VATSIM does not achieve the level of donations required / sought, are plans being considered for a subscription level membership which carries perks? I am not sure what these could be without causing a whole manner of grief, but I am just wondering how far ahead the contingency planning has got.

[Sean Harrison]

I think you are correct Callum.  As an organisation we are exceeding our funding.  Option 1 raise funds, Option 2 reduce quality, Option 3 do nothing and fold.

The Wikipedia model of asking for donations, I don’t believe stops people using it freely!  It may be a discomfort, but hey nothing is free.

[Nicholas Watkins]

3 hours ago, Don Desfosse said:

Thanks for all the great ideas!  Keep them coming!

I'm flabbergasted that I have yet to receive an email on the incorporation, as a 501-C3 in the US, of VATSIM. When I signed up for the network, I agreed to receive communication from the network in a manner consistent with updates to the network and/or the organization itself. I received an email when Audio for VATSIM was launched, I received an email when the privacy policy was updated, I even received an email when VATSIM was looking for a VATGOV2 back in 2015. While I understand there are GDPR concerns, every VATSIM member agreed to receive email when we signed up for the network. Wouldn't the creation of the 501, the change in accepting monetary donations, and a whole host of other things that have happened since the last official email [NOTAM] I received from VATSIM (Audio for VATSIM launch) qualify for sending a single email to the members of the network?

Going a little further, there is yet to be a donate button on the website itself. Moving to social media; When I look at Facebook I have to expand the post and read the whole post, which mind you most will not do as it's a TLDR, to even see the donate.vatsim.net link. The word simply isn't out there to the masses that VATSIM is now a 501 and is now accepting and asking for donations. I think the messaging hasn't been great.

As for ideas, well the donate button on the website, even if at the very bottom next to the social links or wedged between the footer and the join now link, will do wonders. Add to that the email that really needs to be sent out to the membership and you'll see the numbers rise. I would caution however, to what many have said here already. A well thought out plan needs to be in place, beyond 'hey we just need to pay for servers' and a serious look at finances and all possible avenues of discounting existing services needs to be taken into account. For example, when I ran a non-profit just a few years ago Microsoft was willing to donate upwards of $50,000/yr in Microsoft Services (think Azure) through the TechSoup partnership that we paid only $100/yr for. Email & Office365 was provided in 1,000+ user licenses at no cost to the organization. I know there have been strides to lower the costs, but I think even more can be done to achieve that while continuing to cut costs. Add to that the well thought out plan for not only maintaining the existing infrastructure, but expanding and improving on the services the organization is already offering will move the needle enough for members to consider contributing.

To be completely frank, right now it feels as if the organization is just pleading for us to help make rent and the look, in my humble opinion, isn't great.

[Nick Harasym]

  

5 hours ago, Sean Harrison said:

As an observation, twenty years ago when member numbers were smaller, we managed to exist with donations of servers and time and effort.  We grow and have substantially more members, servers are reduced and we have a massive staff list.  I understand that the cost of infrastructure increases but not dramatically against inflation.  (Members earn more now, and infrastructure costs more).  We appear to have rationalised infrastructure, so numerically we have less hardware.  Members develop and offer software.  From an outsiders view it would appear we are spending more than we have(at top level).  Why is this so?  Should our strategic plan include devolution and revert back to regions which provide servers etc, do away with the huge upper structure, and simply make each Regional director a board member.  Without a detailed expenditure and income sheet what else should we consider.  It is hard to fathom that with reduced servers etc we still don’t have sufficient donations with an increased membership.

I'd like to talk about the servers part of your comments since well I do servers mostly on VATSIM.

Servers these days for VATSIM are very different even going back a few years. While users of VATSIM only see websites, FSD and AFV, there are other servers supporting the network. Not directly for the simulation side but the operational side which then supports the simulation side. These servers that people don't see support software listed below (not a complete list).

• Grafana and Prometheus - provides us with a dashboard system where we can make graphs to have better understanding how the network is performing and running. Alerting of problems gets driven by this data aswell. Collecting, processing and storing this data does come with a cost but without it we have a much harder time with Software observability - Wikipedia. We have solved real problems impacting the network due to this data. Without it would have been very hard to prove so that we can bring it up with our infrastructure providers.
• Salt - provides us with a toolset to configure servers automatically when they come online and work across multiple servers at once.
• FreeIPA - Until late 2019, authentication for VATSIM servers at the operational level was handled manually. Administrators would have to manually update keys to grant or remove access. This is now all centralized into one place. Users that have access to servers to do administrative tasks automatically get access if they should get access. This makes onboarding new tech team members much easier. FreeIPA isn't a slim application, its multiple applications and that does require a decently sized server to run.
• dnshaiku - It does DNS things for VATSIM. More information to come on this in the following weeks.
• MariaDB + Galera + ProxySQL - With VATSIM being bigger than ever we have a large set of databases to maintain that require a high amount of read queries. These operate in a cluster which has protected the network during server outages where the database kept working since we have replicas of the data. If we hadn't been in a cluster we would have major network impacts.
• Task queues + workers - A lot of VATSIM processes have been moved to task queues and is no longer processed by cron job type scripts. This allows for greater flexibility.

What I am trying to get at here is we don't have less servers, we have more than we have ever had. We are running more websites, more traffic through FSD (with more capacity) and now have AFV. The tech team is also following better practices regarding how servers are operated which in some cases adds small costs.

[Jacob Boyles]

21 minutes ago, Nicholas Watkins said:

 I know there have been strides to lower the costs, but I think even more can be done to achieve that while continuing to cut costs.

Respectfully, what do you have to back up this statement? We are at the limit of what us as the tech team can reliably manage while keeping costs low. If we cut more we have to pick between reliability or cost. We are doing an absolutely incredible amount with the current infrastructure, everything new we try to do is painful because we simply don't have the resources.

 

21 minutes ago, Nicholas Watkins said:

For example, when I ran a non-profit just a few years ago Microsoft was willing to donate upwards of $50,000/yr in Microsoft Services (think Azure) through the TechSoup partnership that we paid only $100/yr for. Email & Office365 was provided in 1,000+ user licenses at no cost to the organization.

Microsoft will give $5,000 a year in Azure, which yes the plan is to take advantage of, but the impact will not be as large as you'd expect. All of our web services must stay in the same provider, all under the same networking backbone to take advantage of private networking between our services and the databases. $5,000 is not enough to move all of our web services and the database to Azure, and the extra cost on top of the $5,000 we would get is on par with what we pay currently due to Azure being so much more expensive. So we would have to carve out services that can act on their own. We also have to keep in mind that doing that work requires people, and it is not a small job to do.

We also already have google workspaces for free through google, but the savings from that were minimal as our previous email solution was very cheap.

 

Reference Nick's post above for more information about some other services we need to run for us to manage the infrastructure effectively.

[Nicholas Watkins]

4 minutes ago, Jacob Boyles said:

Respectfully, what do you have to back up this statement?

Have you reached out to the reps at Digital Ocean or through TechSoup? You can't just go by what is published on TechSoup.

5 minutes ago, Jacob Boyles said:

Microsoft will give $5,000 a year in Azure, which yes the plan is to take advantage of, but the impact will not be as large as you'd expect.

Actually they will give more, you just have to speak with the Microsoft rep that is assigned to the organization. Ask and you shall receive. Again, you can't just go off the numbers published on TechSoup. Add to that what Microsoft has already offered the organization in terms of a partnership and potentially a monetary donation, the leverage is there to request and receive more. The networking and relationship building just needs to exist between the two entities in order to make it happen.

[Jacob Boyles]

23 minutes ago, Nicholas Watkins said:

Have you reached out to the reps at Digital Ocean or through TechSoup? You can't just go by what is published on TechSoup.

Actually they will give more, you just have to speak with the Microsoft rep that is assigned to the organization. Ask and you shall receive. Again, you can't just go off the numbers published on TechSoup. Add to that what Microsoft has already offered the organization in terms of a partnership and potentially a monetary donation, the leverage is there to request and receive more. The networking and relationship building just needs to exist between the two entities in order to make it happen.

We have reached out to DO, they would not give us any credit.

As for Microsoft I thought we had reached out but I will check on that, either way, it is not something that can be done overnight. Lifting and shifting the entire network (or a subsection) to a new provider would likely be the biggest infrastructure project we have ever done.

It is not as simple as a provider gives us money and all our problems are solved, it would be a massive undertaking by a team who already gets pulled in every direction but everyone else in every department.

I am also not aware of a Microsoft partnership which included money, I'd love for that to be sent our way.

[Nicholas Watkins]

19 minutes ago, Jacob Boyles said:

We have reached out to DO, they would not give us any credit.

As for Microsoft I thought we had reached out but I will check on that, either way, it is not something that can be done overnight. Lifting and shifting the entire network (or a subsection) to a new provider would likely be the biggest infrastructure project we have ever done.

It is not as simple as a provider gives us money and all our problems are solved, it would be a massive undertaking by a team who already gets pulled in every direction but everyone else in every department.

I am also not aware of a Microsoft partnership which included money, I'd love for that to be sent our way.

All of that relates directly to the well thought out plan that I mentioned earlier, which based on your reply I can only assume hasn't happened. Of course a lift and shift can't be done overnight, show me where I said it could be done? This isn't my first rodeo Jacob. If your department is being pulled in every direction by every department then I suggest that the VP of Tech needs to prioritize and lay down a process for bug fixes, enhancements, and other things that come along naturally through the course of a technical development project or set of projects. For anyone or any department to be pulled in multiple directions is a indicator to me that there appears to be a lack of leadership within. Perhaps I am wrong and I truly hope that I am, for the sake of the network.

[Matthew Cianfarani]

3 minutes ago, Nicholas Watkins said:

All of that relates directly to the well thought out plan that I mentioned earlier, which based on your reply I can only assume hasn't happened. Of course a lift and shift can't be done overnight, show me where I said it could be done? This isn't my first rodeo Jacob. If your department is being pulled in every direction by every department then I suggest that the VP of Tech needs to prioritize and lay down a process for bug fixes, enhancements, and other things that come along naturally through the course of a technical development project or set of projects. For anyone or any department to be pulled in multiple directions is a indicator to me that there appears to be a lack of leadership within. Perhaps I am wrong and I truly hope that I am, for the sake of the network.

Hi Nicholas,

With regards to Azure, we have already evaluated it in the past, and all options remain open for consideration in the future should we determine that a change in our infrastructure direction is necessary. I am confident in the progress my team has made in recent years, as we have successfully rolled out numerous new features, improvements, and stability fixes. We have a well-established development pipeline and roadmap in place.

Our technology teams are led by experienced and capable individuals, and my team leads and I communicate daily to ensure that our strategy and vision are aligned with our goals. With over 60 years of combined experience in running infrastructure among my senior team and I, we are well-equipped to handle the growth demands of VATSIM. Our team members come from diverse backgrounds, including developers and sysadmins, giving us a well-rounded and comprehensive group to manage our infrastructure.

If you would like to discuss this further, please feel free to reach out to me via email.

Best regards,

Matt

[Nicholas Watkins]

1 hour ago, Matthew Cianfarani said:

If you would like to discuss this further, please feel free to reach out to me via email.

Sure Matt. Thanks for the conversation earlier. I appreciate it. It is heartening to hear that the messaging will be coming in due time and there are protections in place to prevent donations and other funds from being misused. As you know I've spoken with Tim on the matters of concern and will be following up with an email on the tech topics with you as well as operational topics with Tim and RJ. I appreciate your transparency in these matters and correcting the picture that was painted to me earlier. All of the work you have been putting into the technical side of the network, hasn't gone unnoticed. Thanks again.