- Fri Sep 09, 2016 5:51 pm
Dhruv Kalra 878508 wrote:So, if we were to take a crack at building a test regimen, what sorts of empirical data points would you be after, and how can we get the ball rolling?
I suppose step 1 would be to list the popular weather engines. Obviously there's the various flavors of Active Sky. I'm not sure what others are out there. (I've only used Active Sky and now ASN myself.)
Then we need to know if those engines all pull real world precipitation weather radar return data in order to determine placement of clouds in the sim. If they don't, then they're just generating clouds randomly based on the precip info in the closest METAR, and we can stop there.
One thing I'm not sure of here is what happens if Active Sky is configured to pull VATSIM weather. (If that's even still a feature.) As I understand it, when that feature is enabled, AS pulls METARs from VATSIM for the departure and destination fields. When that happens, and there is precipitation noted in the METAR, does AS randomly place the appropriate type of clouds somewhere around the field, and does this override any precipitation radar data that it pulls from a real world source? If it does override real world precip data, then I suppose we can just advise users to NOT enable that feature. Since VATSIM METARs come from the real world anyway, that feature seems kind of pointless. (Maybe it's useful in parts of the world where VATSIM's METAR sources are delayed or unreliable, causing a mismatch with real-world weather?)
If we can get by all that, and we know that the popular weather engines are pulling real world precip radar data, then the next step is to determine how accurately the engine can place cloud formations in the sim. This is where things get fuzzy. I vaguely recall reading somewhere that the sim doesn't give developers very good control over the placement of clouds. This is where we would need to do some testing. We'd have to try each weather engine in the same location on several different computers each and see what we get. Probably the low-hanging fruit here is to start with the most popular engine (ASN) and check the consistency of cloud placement across many different computers all running the latest version of ASN, all with their aircraft placed near an area of known precipitation. (Something strong enough to show returns on SkyVector, for comparison.) If it's consistent within ASN, then we can move on and test to see if it is consistent across other weather engines. If it's not even consistent within ASN, then we're done.
Obviously, the definition of "consistent" here is somewhat subjective, but I don't think it's reasonable to try to come up with actual tolerance values. We'll need to make a judgement call based on what we see in the sims.