VATUSA TESTING

[Graeme Florance 906409]

NOTAM: This thread is 3 years old.

 

I suspect it's been resolved by now.

 

Eh not sure if you remark is tounge in cheek but the thread was started a few hours ago in response to very recent events.

[Jen Galbraith 1084907]

Gentlemen.

 

I worked in the Nuclear Industry for 20 years and was involved in writing test questions at one point, as well as evaluating test questions and answers through test analysis. The nuclear industry is very strict about how test questions are written, and if they are poorly worded, or incorrect, then they are thrown out. If a student takes the test with those poor questions, and gets them wrong, prior to the test analysis, then he has the opportunity to challenge the questions. If his challenge is successful (and from what I've read, the 2 questions Graeme had concerns about was successfully challenged due to poor wording or whatever) then the student's final grade is given without considering the poor questions. For instance, if it was a 100-question exam, and 1 question was deemed to have been a poor question or even wrong, then that question gets thrown out and the student is graded on a 99-question exam. Maybe this might be taken into consideration either for Graeme right now or for future students when exam questions are founded to be poor or bad questions.

 

That seems like a very fair and reasonable policy that maybe just hasn't been implemented here because it hasn't been brought up before (pure speculation on my part). Great suggestion!

[Graeme Florance 906409]

So using the logic expressed by management, I should of answered the other four questions correctly and this issue would not have arisen (two of which were under protest as well but that has been lost along the way ) Well i am glad i missed those questions because if i hadn't i would of p[Mod - Happy Thoughts]ed this exam and the two wrong questions would not have come to light and the next student may well of been in the same predicament I find myself in today. It is suggested that the tests are open book and the information is readily available. Let me say that I spent four hours looking for such answers last Monday. I didnt just use the resources available in print. I asked fellow controllers senior controllers instuctors and a TA, for pointers on a couple. So I hear you say should be pretty easy, I too would say the same looking outside in. I would expect to see the majority of people taking these tests at various levels achieving very high scores.

 

Well lets look at the latest VATUSA Kudos board to see if that is indeed the case

 

Test Score

 

ZBW (Boston) - Delivery/Ground 97

VATUSA Basic Controller Exam 92

VATUSA Basic Controller Exam 92

ZSE (Seattle) - Tower 90

ZHU (Houston) - SOP 90

VATUSA Basic Controller Exam 80

ZOB (Cleveland) - DTW ATCT 80

VATUSA Basic Controller Exam 80

 

names removed to protect privacy

 

That works out to a 87% average if you combine all the scores together hmmmm an averarge score below 90%. I think with such a wealth of knowlege available to us that 90% would be a very achievable score with a legitmate test and with legitamate questions on an open book test. If the information is as clear, consise, and available as I am being told it is lets make the test past rate 100%.

 

I speculate that alot of people in management and members of the community are educated to a high standard. Along the way they may have recieved some sort of certification or even diploma that meant they had to take a test to succeed. I can gaurantee if it had come to the final question on the final test after four years of college, high school, whatever. They would be fighting just as hard for the right thing to be done as I am here.

 

YOUR TESTS ARE FLAWED and not just one or two questions, I am led to believe you have a fourm dedicated to this very topic where questions are continually discussed. Stop making this an issue about Graeme's Test, because that is secondary. I frankly dont care if i ever achieve my C1 Rating under this regime. Be big enough to hold your hand up and say you are wrong and take steps to make sure that it dosent happen again...

[John Cierpial 1008209]

Graeme and everyone else here, I think you are missing the big picture which Gary and Alex have constantly been trying to reiterate to you. The two questions you protested were indeed incorrect and are reworded for future exams. However, you submitted this exam and then protested only after you failed. If you had p[Mod - Happy Thoughts]ed, would you have protested those questions, especially if it marked you right even though you knew it was wrong? Like the VATUSA policy says, if you fail, you must wait 14 days to retake the test. Simple as that. That gives you 13 days from today (I believe don't quote my math ) to continue studying, learning more about the enroute position from your ARTCC's docomeents and the .65. Don't look at this as a negative experience because a positive came out of it (you fixed two questions for every other C1 candidate). Remember, this is NOT the end of the world and you will now be able to take the test a second time and positively get a 100%

 

-Joe

[Graeme Florance 906409]

Graeme and everyone else here, I think you are missing the big picture which Gary and Alex have constantly been trying to reiterate to you. The two questions you protested were indeed incorrect and are reworded for future exams.

 

And you know this based on what? Do you even know what the questions were. No you dont so to make a statement as to you know what was being conveyed is speculative IMO. I firmly believe the answers I gave to the ASKED questions were the best option available. Hence why they were deemed badly worded as the answers they wish to be were not indeed the correct answers to the questions INTENDED. Do you want me to post the actuall questions ?

 

I took an incomplete test if two questons are to be thrown out, is it not fair that i get to complete all 25 questions? And then be judged on the merits of a valid 25 questions as required to p[Mod - Happy Thoughts] the examination. If this can be done with wrong answers, whos to say they cant at some point say those correct answers you had were poorly worded we are just going to take those out and make you take the test again....

 

Thank you for your response, all are appreciated and you are the first without a title under your name to disagree with my opinion and stance on the issue.

[Gary Millsaps 830104]

To all,

 

Every attempt has been made to handle this situation in an equitable and fair manner. Mr. Florance not having received the results he deems acceptable through the available methods, has chosen to bring it into the court of public opinion. I have outlined in the various correspondences I've had with Mr. Florance and others the factors and reasoning upon which my decision lies. Opinions on their value do and will vary wildly...which is as expected. I have no intention of further feeding the furor.

 

The facts remain thus:

 

Mr. Florance erred on answering 6 questions - two of which have been challenged and appropriately modified to improve their readability and clarity. This adds to the improvement of the validity of the question base for this and other VATUSA examinations. Be that as it may, Mr. Florance erred on an additional four questions which are not challenged and have no reason to be so. They were simply answered wrong.

 

Mr. Florance will have every opportunity to re-take the full examination just as any other member has in the past and will do so in the future as defined in the VATUSA policies.

 

In short gentlemen, my decision stands.

[Graeme Florance 906409]

I think Mr Bailey and Mr Vodnansky are in recipt of an initial email that sates i challenged 4 out of the six, again this why I suggested to you Gary that perhaps we could discuss the situation. You failed to respond to such invitation. I can post that one too if you would like

[Gary Millsaps 830104]

Graeme,

 

I'll gladly discuss any aspect of this issue with you. You know where to find me...

[Brian Swintal 867195]

In the real world and among the millions of tests I have taken I have seen questions challenged by students. They think the question is worded wrong or does not clearly express what is needed in the answer. In those instances the question was thrown out and just not counted and reworded for a future exam. So instead of 50 questions we are now down to 49. This was done because it makes no sense to punish the student because the instructor wrote an ambiguous question. Using the argument "every other cl[Mod - Happy Thoughts] got it right" is not a valid excuse either. The way the material was presented in past cl[Mod - Happy Thoughts]es might have been different and something said in those cl[Mod - Happy Thoughts]es might have related to the question. If the instructor doesn't think the question is ambiguous and thinks its fine worded how it is then you move on BUT when an instructor admits fault, you don't continue to punish the student.

 

Punishing Graeme because of a poorly worded question is just plain wrong and mean spirited.

[Luke Kolin]

This is very similar to what we do with our pilot examinations at Delta Virtual. In the past, our questions were poorly worded and we had a number of issues with them turned out to be quite valid.

 

If one or more questions were deemed unsuitable, we didn't count them and determined if the pilot received a p[Mod - Happy Thoughts]ing score. If so, then the questions were marked correct so that the exam would p[Mod - Happy Thoughts]. If not, we took a second step - if all of the successfully challenged questions were answered correctly, would the pilot have p[Mod - Happy Thoughts]ed? In such a circomestance we would usually punt and delete the examination, making the pilot eligible for an immediate re-test.

 

One shouldn't punish a pilot for a flaw in the testing mechanism, and what's more important the challenge process needs to encomp[Mod - Happy Thoughts] procedures that retain pilot faith in the process. It seems like here, the procedure is more important than the people. The process has been successfully defended, at the cost of pilot goodwill. I don't see how VATUSA wins in such a scenario.

 

Cheers!

 

Luke

[Jim Davey]

When someone fails an open-book exam, it speaks of not taking one’s time to review the information, thinking through the question and applying what one reads to selecting the best answer.

 

This comment here is what does it for me, and why I quit controlling several years ago. I was trying to get on with the DC ARTCC (while already a C1, with 85 hours of controlling to that point, in addition to time mentoring other controllers), and their "open book" tests included some information that wasn't to be found ANYWHERE on the VATUSA or DC ARTCC web sites. When I challenged it, I was ignored. This current event leads me to believe the "sweep it under the rug" mentality of ignoring what is blatantly wrong if it involves change or doing "the right thing" is more wide-spread across at least the VATUSA leadership than I thought. I support your cause whole-heartedly, Mr. Florance.

Mr. Millsaps, the "He got 4 other questions wrong" statement holds no water. Isn't 4 wrong still a p[Mod - Happy Thoughts]ing grade? I understand up to 5 wrong is still p[Mod - Happy Thoughts]ing. You should be ashamed of the example you're providing Mr. Millsaps.

[Alex Bailey 969331]

If we're going to bring the word "ashamed" into it, then everyone should be ashamed to not realize what is actually happening here. This is the typical 'jump on the bandwagon' approach taken by many people in the court of public opinion, who think that the person bringing the complaint is telling the truth. Or maybe it's because he has posted on his ARTCC website for people to come in here and help his cause... You tell me.

 

The facts are straight. The questions were valid, they had a correct answer, and out of the kindness of my heart I agreed to REVISE the questions to make them even easier (cut and paste). I will not entertain any more discussion regarding the validity of the test, since it is quite evident that most people posting here do not have any idea as to the facts here.

 

If you feel like we're horrible people who are ruining your VATSIM experience, then I invite you to file a complaint with conflict resolution and see how much mud you can really throw at us. Until then, it's business as usual. I tend to get atleast one email per day from a student who failed his/her exam who wants my [Mod - Happy Thoughts]istance locating the correct answer so they can p[Mod - Happy Thoughts] the re-take. Students like that deserve my attention.

 

The funny thing is, Graeme... I went to bat for you on more than one occasion previously which I've yet to receive thanks for, and I don't even want it. I do apologize for going out of my way to help your case (it kept some things off your CERT record), and I promise it will not happen again.

[Brian Swintal 867195]

This is far from everybody jumping on the bandwagon and I'm sorry you feel that way. This is a case of people making sure that a student isn't being punished because of a mistake on the training department. I can't even believe that this type of discussion is being handled in the forum. The fact that it had to make it this far shows how badly the process must have been handled behind the scenes. Alex, the fact that you are now angry with Gareme shows just how badly the whole situation has been handled and the process should be looked at very closely.

 

No where was it ever stated that the challanged questions were right so let me ask here and now and see the facts. Graeme got 6 questions wrong that made him fail the test. Of those 6 wrong 2 were worded poorly and "changed for future tests". So if those 2 questions were thrown out or given to Graeme with an explaination of the correct answer he would have gotten 4 wrong and that would have allowed him to p[Mod - Happy Thoughts]? If a question is marked wrong, then later decided it was poorly worded and changed, IT SHOULD BE THROWN OUT! Simple as that. Your current procedure, if there even was one, only works to punish the student and leave them feeling angry.

 

What is the current procedure for a challanged question?

 

Alex with an attitude like that I don't think you are going to be getting much thanks. It would have been nicer to keep the conversation civil but I wouldn't know how to respond to you last statement. You poorly chose your words and comments of a personal nature should be left to an email not aired on a public forum.

[Brian Swintal 867195]

If you feel like we're horrible people who are ruining your VATSIM experience, then I invite you to file a complaint with conflict resolution and see how much mud you can really throw at us. Until then, it's business as usual. I tend to get atleast one email per day from a student who failed his/her exam who wants my [Mod - Happy Thoughts]istance locating the correct answer so they can p[Mod - Happy Thoughts] the re-take. Students like that deserve my attention.

 

Everybody should be getting the same amount of attention from you regardless of your personal feelings towards them. You have to be able to take the good with the bad and if you can't then maybe you aren't the right person from the job. This is a perfect example. Somebody challanged the current procedure and felt they were being treated unfairly and they were met with no response or no adequate response to suit their needs. Just saying "that's the way it is so deal with it" is not an adequate response. The full outlining of why they decision was made, and some background, or previous examples should have been given to support the reason for the decision. If you give a reason with no explaination is going to get a response like you have seen here.

 

"What we have here is a failure to communicate!"

[Alex Bailey 969331]

Brian,

 

Yet again, somebody has posted with no knowledge of what has happened. There were many emails exchanged through the proper channels for days, and I actually thank Graeme's ATM for his professional handling of the situation. Graeme didn't like the resolution he got from Gary and decided to make it public.

 

I'll say it again. The questions were valid and were correct.

 

All I did was revise the questions to make them even more clear. It was a service to the membership to change a question that one person takes issue with out of hundreds who have taken the test.

Edited December 23, 2008 at 05:13 PM by Guest

[Keith Smith]

It's disheartening to see how so many people are so quick to admonish the people that volunteer their time in management positions on this network. This situation seems so simple...a person failed a test. Two of the questions have since been reworded for clarity (this is a far cry from the questions being 'wrong' or 'thrown out'). While it is FEASIBLE to SUGGEST that the person might be able to take the test again immediately, I'm guessing this was considered, but ultimately rejected for reasons that Gary mentioned in his email reply. Not satisfied with the detailed, civil replies, though, the original poster takes the entire issue to a public forum, includes private email exchanges, and so, wioth pitchforks at the ready, and an incomplete picture of all of the facts surrounding the issue, the uninformed m[Mod - Happy Thoughts]es start chiming in.

 

Let's [Mod - Happy Thoughts]ume, for argument's sake, that there WAS a grave injustice...was this REALLY the right way to handle it? Absolutely not...it's a waste of many peoples' time. If this was a truly valid issue, it could be escalated higher up the chain, through the appropriate channels.

 

Really, though, this shouldn't be escalated. For the controller to post that he probably won't be able to control again after this is so far off base, it's comical. If HE doesn't want to control again, that's his choice, but saying such a thing seems nothing more than an attempt to increase the drama and gravity of the situation.

 

Sure, it's ok to be a little frustrated that the test couldn't be taken again sooner, but a decision needed to be made, one way or another. To keep things simple, and not open a can of worms for the future, the decision was to handle it as any other test failure. It's completely unnecessary that it has come to this, and the decision was NOT unreasonable.

 

Please give the people who volunteer their time in management positions a break, and a little bit of credit. That, and I don't see why Alex and Gary are expected to have a _bottomless_ supply of patience for people who don't come to the table with a sense of reason or perspective.

[Brian Swintal 867195]

Brian,

 

Yet again, somebody has posted with no knowledge of what has happened. There were many emails exchanged through the proper channels for days, and I actually thank Graeme's ATM for his professional handling of the situation. Graeme didn't like the resolution he got from Gary and decided to make it public.

 

I'll say it again. The questions were valid and were correct.

 

All I did was revise the questions to make them even more clear. It was a service to the membership to change a question that one person takes issue with out of hundreds who have taken the test.

 

Obviously Graeme felt like he didn't get an adequate explaination of the ruling otherwise i'm positive it wouldn't have made it this far. It is one thing to just tell somebody that they are wrong and get over it but it is a completely different to give them a full explaination as to why and what made you come to that conclusion. Also any question that is reworded, on any test, for any reason, makes that question invalid and should not be counted wrong or right. Changing a question is an admission that they question was faulty. If it wasn't faulty then there would have been no reason to change it, right?

 

Graeme is an excellent controller. I have flown in and out of ORD several times while he was controlling and he is one of the best controllers that I have seen in ORD in quite some time. Hopefully the poor handling of this situation doesn't leave him with too bad of a distaste for controlling on the VATSIM network.

[Jason Baker 1028693]

Everybody should be getting the same amount of attention from you regardless of your personal feelings towards them. You have to be able to take the good with the bad and if you can't then maybe you aren't the right person for the job. This is a perfect example. Somebody challanged the current procedure and felt they were being treated unfairly and they were met with no response or no adequate response to suit their needs. Just saying "that's the way it is so deal with it" is not an adequate response. The full outlining of why they decision was made, and some background, or previous examples should have been given to support the reason for the decision. If you give a reason with no explaination is going to get a response like you have seen here.

 

"What we have here is a failure to communicate!"

 

What this more closely resembles is an attempt to wear down leading figures the audience chooses to be in disagreement with. It's sandbox politcs - nothing else.

Gary gave a perfect example and provided enough insight to the OP, as to state that he, for the sake of credibiliy would not adjust the grade from fail to p[Mod - Happy Thoughts]. The fact remains that people who fail in a certain system are always feeling treated unfairly, or wronged. Yet, this test, in it's prior wording must have been p[Mod - Happy Thoughts]ed before. Everybody understands the concept of a misunderstood question, and sometimes such questions can be worded a bit twisted, but if this particular test has ever been p[Mod - Happy Thoughts]ed by anyone it is nothing but a personal perception. This perception lead to change, in a positive way. OP has the opportunity to retake and p[Mod - Happy Thoughts] the test. In fact, I challenge you to take any (!) FAA written examination, fail it, and then complain about the wording and see what happens.

You will be taking this test again, if you are lucky the question will experience change within 5 years.

 

The whole matter seems blown out of proportion, like many things that have to do with VATSIM and VATUSA & it's membership. It's these kinds of forum posts that keep people like me from pursuing ATC, and it most certainly helps to realize that whatever position within this community one seeks, it will turn out to leave a very sour taste, besides being entirely thankless. I have yet to contact any of the accused gents in here without receipt of a prompt and optimistic response, but that may be caused by the fact that I can ask a question, get an answer and accept it as it is.

Posting stuff like this to publicly demoralize and accuse people we are in disagreement with is a sign of lack of taste - at best.

 

 

Merry Christmas!

Edited December 23, 2008 at 05:27 PM by Guest

[Alex Bailey 969331]

Brian,

 

Thanks for your concern and I'm glad you are enjoying your flights there. If you will read the previous posts, Graeme has posted Gary's response which I'm sure you will agree is adequate and addresses the issue with due diligence. There were other email exchanged that follow the same line of thinking, and no decision was issued to Graeme (or Jason) without reasoning. I'm going to respect the nature of private conversation from all parties and not post the remaining emails.

[Keith Smith]

Jason,

 

If you are interested in pursuing ATC on VATSIM, you should do it, regardless of what you see here on the forums. The forums are very much removed from the scopes, I [Mod - Happy Thoughts]ure you. I can come here and be completely baffled by some of what I read. I get back to the scopes, and the p[Mod - Happy Thoughts]ion and energy comes right back, in waves....nearly every time. If I am remembering correctly, you also have a presence on other r/w flying forums, so I take it you're a r/w pilot. I can tell you that providing ATC on VATSIM will almost certainly make you a more aware pilot.

[Brian Swintal 867195]

Brian,

 

Thanks for your concern and I'm glad you are enjoying your flights there. If you will read the previous posts, Graeme has posted Gary's response which I'm sure you will agree is adequate and addresses the issue with due diligence. There were other email exchanged that follow the same line of thinking, and no decision was issued to Graeme (or Jason) without reasoning. I'm going to respect the nature of private conversation from all parties and not post the remaining emails.

 

I love flying in all of VATUSA's airspace, except ZLA, as there are some wonderful controllers out there and it obviously speaks to the training they recieved. In no way am I or was I attempting to bash VATUSA or the people that run it. I think the crux of the disagreement between al is what to do with the questions. On one side there are those that say the question is/was valid and will be counted, even through Graeme got them right. On the other side you have those that say any question that is deemed poorly worded and ambiguous should be removed from that particular exam and the exam should be rescored. That is what it boils down to on the basic level. I believe the test should be rescored based on good faith. Not because a stink was through or aired on a public forum were things got heated, but because that is the right thing to do. I've delt with situations like this when I was a TA and back when I was going through flight training. The FAA Exam is a bad reference because all you need to do with those tests is memorize the answers because all of the test questions are available before hand. The questions were word-for-word so there is no reason for a challenge except when a computer mistake may have happened. The problem arises when you start putting a different word in to change the question up a little and that can lead to a misinterpretation on the part of the taker.

 

Your right though. We don't know what was said in email between all parties involved. What we have to go on is what is posted here. There are always 2 sides to every story and then there is the truth. I'm not saying anybody is lying but it is human nature to play up your cause to make you look like the victim. Obviously Graeme didn't agree with the decision that was made. He brought it here for our thoughts on the subject matter. We've all weighed in on the matter as it was presented and voiced our opinions on the subject. Obviously many of us are siding on the test takers side as we all believe that any question that is reworded on future tests to avoid confusion invalidates those questions on the current test and the test should be rescored.

[Alex Bailey 969331]

Fair enough Brian, but if a re-score were to take place we could only really throw out one of his two challenges. One of the questions was a simple exchange of one word, which I would never throw out. The second was the removal of a word to make the question much more clear, and IF I were to throw that out then his score is 19/24 and is still a failure.

 

Hope that gives you some insight into the decision that was made and I think it is time to put this issue to rest.

[Jason Baker 1028693]

you also have a presence on other r/w flying forums, so I take it you're a r/w pilot.

 

Keith,

 

if I am not entirely mistaken we have had some PM conversations on this "other" forum. If you are the same guy - drop me a line, I'd like that. Cheers & Merry Christmas!

[Dan Leavitt 1081600]

Fair enough Brian, but if a re-score were to take place we could only really throw out one of his two challenges. One of the questions was a simple exchange of one word, which I would never throw out. The second was the removal of a word to make the question much more clear, and IF I were to throw that out then his score is 19/24 and is still a failure.

 

Alex,

 

2 questions had to be changed, which means there were 2 problems with the test. It doesn't matter how small YOU think the error is. That 1 word may have caused someone(Graeme OR another person taking the same test), to misinterpret the question.

 

The other question you are disputing had "All of the above" rather than "Any of the above" as the answer choice. Now come on... Any time you take an exam you know that all of the above means that each individual choice above is correct. Again, I made a revision to make it much more clear

 

The word in question is the difference between ANY or ALL. Now on a test, ANY means that it can be ANY of the answers, but not simultaneously. ALL refers to all of the answers being correct simultaneously.

 

A test is supposed to be subjective, not left up to opinion and interpretation. If the difference of that 1 word causes there to be a misinterpretation of the question then that question has to be thrown out immediately, because as soon as the question becomes ambiguous, it is open to the test takers opinion, which an opinion can never be marked right or wrong.

 

So, by my math, a 19/23 is a p[Mod - Happy Thoughts]ing grade. That's all there is to it.

 

Dan Leavitt (1081600)

[Brian Swintal 867195]

Ok time out. Before I open my mouth again how many questions are on the exam and what is a p[Mod - Happy Thoughts]ing grade. From the sounds of it there are 25 questions and p[Mod - Happy Thoughts]ing is an 80%