Academic Testing

[Ten oz]

My employer was extended and opportunity to have someone attend a senior leadership school taught by one of the Uniform Armed Services. I was selected to attend. One of our first assingments was to take a pre-test. We were specifically directed to not answer any questions we were not 100% certain about. Though multiple choice we were all instructed not to guess. To simply leave blank answers to question we were not absolutely positive about. The purpose of the test is to guage the effectiveness of the course. They are looking for a baseline knowledge of the subject matter upon entry into the class. The test doesn't count toward class completion and does not negatively effect ones final grade.

 

After the test was complete, during a break, I chatted with the services members and was surprised to find to to a person every had answered every question on the test despite universal acknowledgement from everyone I spoke with that they were not familiar with the information. I got strange looks when stating that I had left at least 3/4 or more of the test blank. I reminded the people I spoke to of our instruction but most stated they felt that they were able to eliminate some of the answer option and basically had 50/50 odds when answering. The explaination is a bit confusing. 50/50 odds is not equal to knowing the answer. So conditioned to test strategies many of the students seemed unable to provide an honest account of their knowledge base.

 

So my question is; should there be a built in penalty for guessing on multiple choice tests? A standard where questions left blank are not counted but questions answered wrong are subtract from questioned answer correctly? When I school I was always taught that there was no harm in guessing. To always be sure to answer every questions. Perhaps that is a bad philosophy. Students should be held accountable for what they actually know and should be able to evaluate the difference between good/lucky test taking and acquired knowledge. On a 100 question test were a student knows 50 answers for sure and are able to reduce the rest to 50/50 guesess a passing grade is well within reach despite only actually know half the material. If wrong answers subracted from the score guessing would become very risky. Students would be force to provide more honest evaluations of there know and educators would have more reliable feedbak.

 

What are this forums thoughts on this?

 

[fiveworlds]

Negative marking is common in many schools. There is however the simple problem of people thinking they know the right answer and being wrong.

[swansont]

Points off for wrong answers, as fiveworlds said. A lot of standardized test mark off a fraction of a point for wrong answers, so the raw score can be below zero, but it becomes statistically advantageous to guess if you eliminate more than 1 wrong answer. But if you change the value of the negative marking, you can make it so even 50-50 guessing isn't break-even.

 

 

After the test was complete, during a break, I chatted with the services members and was surprised to find to to a person every had answered every question on the test despite universal acknowledgement from everyone I spoke with that they were not familiar with the information. I got strange looks when stating that I had left at least 3/4 or more of the test blank. I reminded the people I spoke to of our instruction but most stated they felt that they were able to eliminate some of the answer option and basically had 50/50 odds when answering. The explaination is a bit confusing. 50/50 odds is not equal to knowing the answer. So conditioned to test strategies many of the students seemed unable to provide an honest account of their knowledge base.

 

I would have been tempted to cast this as disobeying a direct order, just to make them uncomfortable with their admission.

[MigL]

I don't mean that it applies in any way to your case Ten oz, since you were clearly instructed that you were being tested only on what you know.

 

But in many tests your reasoning is clearly on test. If you can use deductive reasoning to eliminate answers which are more wrong, then the one which is least wrong must be the correct one. Why should it be an issue, then ?

[StringJunky]

Isn.t the primary problem that multiple-choice exams introduce a statistical improvement in getting the right answer by sheer chance?

Edited September 5, 2015 by StringJunky

[Endy0816]

I would always use those to help me know what to study later, though I never had an instructor tell us not to guess.

 

You know the practical is going to be in the same format with similar or identical questions. For every military test I can think of, guessing will net you a fraction of a point on average per question you guess on(and yes this is pretty archaic). There is something to be said though about sustaining winning habits.

 

Realistically you know the back office folks who figure out the statistics can factor in people guessing as well in any case.

 

I have taken nonmilitary academic tests where there are penalties for wrong answers. SAT was like that and it was just a matter of subtracting the penalty from how many fractions of a point on average you can expect from the process of elimination and guessing. Then you know whether to guess or not.

[Ten oz]

Points off for wrong answers, as fiveworlds said. A lot of standardized test mark off a fraction of a point for wrong answers, so the raw score can be below zero, but it becomes statistically advantageous to guess if you eliminate more than 1 wrong answer. But if you change the value of the negative marking, you can make it so even 50-50 guessing isn't break-even.

 

 

I would have been tempted to cast this as disobeying a direct order, just to make them uncomfortable with their admission.

I was thinking a full point off and not just fractions of a point. I suppose it could be taken a step further where correct answers count and 1 but incorrect answers count as -2 or greater. Then a person would truly have to be sure of every answer.

 

I don't believe they saw any issue with answering all questions including the ones they were unsure of. As people have noted in this thread being able to deduce an answer or superior guess is often viewed as on par with knowing the answer out right.

 

I don't mean that it applies in any way to your case Ten oz, since you were clearly instructed that you were being tested only on what you know.

 

But in many tests your reasoning is clearly on test. If you can use deductive reasoning to eliminate answers which are more wrong, then the one which is least wrong must be the correct one. Why should it be an issue, then ?

On most multiple choice tests with A-D options, test for which you know little regarding the subject, an answer or two can usually be eliminated. The majority of questions provide 1 in 3 or 1 in 2 chance of guessing. Given those chances it would not be miraculous to guess 7 of 10 questions right. Of course it could go the other way as well. It make evuating the persons true knowledge nebulous. I am sure we all have seen or heard of people being up for a few hours or days in Las Vegas before eventually losing all gains. How large of a sample size do you need to truly rate a gamblers skill? Or is a gamblers skill a fallacy and in truth every who plays long enough loses unless they cheat?

 

I would always use those to help me know what to study later, though I never had an instructor tell us not to guess.

 

You know the practical is going to be in the same format with similar or identical questions. For every military test I can think of, guessing will net you a fraction of a point on average per question you guess on(and yes this is pretty archaic). There is something to be said though about sustaining winning habits.

 

Realistically you know the back office folks who figure out the statistics can factor in people guessing as well in any case.

 

I have taken nonmilitary academic tests where there are penalties for wrong answers. SAT was like that and it was just a matter of subtracting the penalty from how many fractions of a point on average you can expect from the process of elimination and guessing. Then you know whether to guess or not.

In the military the back office folks have had a hard time over the last few years:

 

http://www.nytimes.com/2014/03/28/us/air-force-fires-9-officers-accused-in-cheating-scandal.html?_r=0

 

http://news.usni.org/2014/08/20/navy-expels-34-sailors-nuclear-cheating-scandal

 

http://archive.armytimes.com/article/20120705/NEWS/207050329/Record-number-OCS-candidates-booted-amid-cheating-allegations

 

I realize that for any test there will be cheats. However the above stories are all officers; leaders with their organizations. And these test were job proficiency tests. It isn't comparable to the SAT. No one is expected to know the answer to all the questions on the SAT. Job proficiency tests are meant to ensure a minimum knowledge base amongst the ranks. When leadership is caught cheating on that type of test you know the issue must be much larger. What I witnessed was a room full of senoir leadership unable to provide an honest accessment of their knowledge base. While there may be something to be said for sustained winning habits I think in the end there is more to be said for just knowing information straight up. I know the 2 + 2 = 4 and do not need to apply any sort of test taking strategy to arrive at that answer.

 

Perhaps with in any section of a test there should be questions that a person must answer correctly to pass. A question so fundamental to the subject matter that failure to answer it can not be excused? The test may be 100 questions but there are 10 questions that must be answered correctly on the test and failure to answer those 10 is an automatic failure regardless of how the other 90 questions are answered.

[imatfaal]

At Med School our multiple choices were full negative marking. +1 for correct answer, zero for a blank, -1 for a wrong answer. Get 50 right and 50 wrong and end up with zero marks

[John Cuthber]

At Med School our multiple choices were full negative marking. +1 for correct answer, zero for a blank, -1 for a wrong answer. Get 50 right and 50 wrong and end up with zero marks

Interesting.

When I was a student the medic students told me that the mark scheme was simple. 1 mark for the right answer

zero marks for no reply and

- 5 for a wrong answer.

This was said to reflect the fact that , in medicine,if you don't know, and you guess, you can do a lot of damage. On the other hand, if you accept that you don't know, you can go an ask someone.

 

Sounds reasonable to me.

[fiveworlds]

quote

Perhaps with in any section of a test there should be questions that a person must answer correctly to pass. A question so fundamental to the subject matter that failure to answer it can not be excused? The test may be 100 questions but there are 10 questions that must be answered correctly on the test and failure to answer those 10 is an automatic failure regardless of how the other 90 questions are answered.

/quote

 

Yeah I had a few of those. Mostly they were part of practical work. For example you could be asked to make an omelette. But you forgot to check if the ingredients were stale.

[Endy0816]

I don't believe they saw any issue with answering all questions including the ones they were unsure of. As people have noted in this thread being able to deduce an answer or superior guess is often viewed as on par with knowing the answer out right.

 

On most multiple choice tests with A-D options, test for which you know little regarding the subject, an answer or two can usually be eliminated. The majority of questions provide 1 in 3 or 1 in 2 chance of guessing. Given those chances it would not be miraculous to guess 7 of 10 questions right. Of course it could go the other way as well. It make evuating the persons true knowledge nebulous. I am sure we all have seen or heard of people being up for a few hours or days in Las Vegas before eventually losing all gains. How large of a sample size do you need to truly rate a gamblers skill? Or is a gamblers skill a fallacy and in truth every who plays long enough loses unless they cheat?

 

In the military the back office folks have had a hard time over the last few years:

 

http://www.nytimes.com/2014/03/28/us/air-force-fires-9-officers-accused-in-cheating-scandal.html?_r=0

 

http://news.usni.org/2014/08/20/navy-expels-34-sailors-nuclear-cheating-scandal

 

http://archive.armytimes.com/article/20120705/NEWS/207050329/Record-number-OCS-candidates-booted-amid-cheating-allegations

 

I realize that for any test there will be cheats. However the above stories are all officers; leaders with their organizations. And these test were job proficiency tests. It isn't comparable to the SAT. No one is expected to know the answer to all the questions on the SAT. Job proficiency tests are meant to ensure a minimum knowledge base amongst the ranks. When leadership is caught cheating on that type of test you know the issue must be much larger. What I witnessed was a room full of senoir leadership unable to provide an honest accessment of their knowledge base. While there may be something to be said for sustained winning habits I think in the end there is more to be said for just knowing information straight up. I know the 2 + 2 = 4 and do not need to apply any sort of test taking strategy to arrive at that answer.

 

Perhaps with in any section of a test there should be questions that a person must answer correctly to pass. A question so fundamental to the subject matter that failure to answer it can not be excused? The test may be 100 questions but there are 10 questions that must be answered correctly on the test and failure to answer those 10 is an automatic failure regardless of how the other 90 questions are answered.

 

There has been a general decline in ethics over the years. That much is true. Not really sure whether it stems from the population at large or modern military culture. Lot of money involved these days.

 

I was mostly thinking of the people who compile all the data points though. The data goes up to them and they make changes from on-high. Eventually filters back down and courses are adjusted as a result.

 

Yeah I had a few of those. Mostly they were part of practical work. For example you could be asked to make an omelette. But you forgot to check if the ingredients were stale.

 

It is like that for qualifying for your actual rating/mos(Enlisted).

 

Practical is honestly most important part. Everything else is in a procedure or manual somewhere.

 

 

I know the 2 + 2 = 4 and do not need to apply any sort of test taking strategy to arrive at that answer.

 

Not always that cut and dry. Mainly thinking about Rating exams where you can be lumped in with people who share a rating but wildly different job responsibilities. It would be like a clock maker and Swansont being asked questions from each others field of expertise.

 

In theory yes, they could study to the point that they could answer any question thrown at them, but realistically they are best off studying what they can and then studying the test itself to gain the best advantage(even if only what not to do in the case of negative points for wrong answers).

 

 

I was thinking a full point off and not just fractions of a point. I suppose it could be taken a step further where correct answers count and 1 but incorrect answers count as -2 or greater. Then a person would truly have to be sure of every answer.

 

You basically need to take off 1/2 a point per question minimum to remove any incentive for guessing(assuming correct answers are worth 1 point). You can take off more it just doesn't gain you anything more besides the psychological effect.

 

You eliminate 1 of 4 answers and you have 1/3 a chance to guess correctly.

You eliminate 2 of 4 answers and you have 1/2 a chance to guess correctly.

You eliminate 3 of 4 answers and you know the answer.

 

 

Main thing is that either way it was a zero point value test. In that case even "negative" points for wrong answers might not have any impact.

[fiveworlds]

Quote

It is like that for qualifying for your actual rating/mos(Enlisted).

/Quote

 

I know I tried to get into the military here a few years ago. Of course I am a small guy and we were told of our group of fifty guys they would take the best two. If I was to try again I would need to train seriously hard beforehand. I honestly didn't realise going down exactly how fit I needed to be.

[overtone]

I am sure we all have seen or heard of people being up for a few hours or days in Las Vegas before eventually losing all gains. How large of a sample size do you need to truly rate a gamblers skill? Or is a gamblers skill a fallacy and in truth every who plays long enough loses unless they cheat?

At the poker table, real skill can win in the long run - because there is skill involved, and you aren't playing the house.

 

But as a result of this, poker skill takes a very long time to verify - I ran across a calculation once that seemed to show (the math seemed competent) that being ten thousand dollars ahead after three months of daily poker at a typical Las Vegas casino meant almost nothing - you were about as likely to be a below average player (and in the process of losing all your money, probably) as an above average one.

 

There are a couple of casino games in which the odds are technically in your favor given perfect play and sufficient mental resources, such as blackjack, but you are playing the house: the house will likely win anyway because it cannot be driven from the game and forced to accept its losses. The gambler's skill there includes quitting - for good - while the right amount ahead.

 

At all other games not even the odds are in your favor, and no skill can beat bad odds in the long run. The skill there is to not play unless forced, if forced make the shortest run possible; know the highest expected payoff (smallest expected loss) single bet and make it with all your money. Then quit, win or lose.

 

So skill in gambling largely rests in knowing when not to gamble, which is almost all the time, but still being willing and competent to gamble in that rare situation when it's a good idea. And that is real skill, so there is skill in gambling. But it's difficult to test for.

 

The guy who wrote "Thinking, Fast and Slow" got his early career start evaluating military recruits for officer potential - and failing, but being unable to discard the evaluation procedure for interesting reasons. That's a really good book for anyone interested in evaluation via testing.

 

Most tests don't work, I think, except that if you can screen for cheats that's often valuable information (good for jobs cheats do well, bad for jobs cheats will screw up).

Edited September 7, 2015 by overtone