A Perspective on 2012 VR paper
Moderators: Section Moderators, Forum Moderators
A Perspective on 2012 VR paper
Along with many parents, I have been puzzled by the seemingly lower scores on the 2012 verbal reasoning paper. This post represents some statistical analysis and my best effort to shed some light on the results.
The boring statistical part
I have analysed the scores posted in this forum for the 11 + exam for the 2009,2010,2011 and 2012. The result is as follows:
Mean and standard deviation scores are for VR paper only
2009: mean 70.3%, st dev 13.3%, mean overall standardised score 348 (sample size 13)
2010: mean 78.1%, st dev 9.1%, mean overall standardised score 350 (sample size 19)
2011: mean 79.6%, st dev 10.5%, mean overall standardised score 344 (sample size 49)
2012: mean 74.8%, st dev 10.0%, mean overall standardised score 341 (sample size 73)
A breakdown analysis of the VR scores can be done by looking at what percentage of scores achieved a certain score
............................................ 2011 2012
Score in VR paper of 95% and above 2% 0%
Score in VR paper of 90% and above 19% 4%
Score in VR paper of 85% and above 31% 16%
Score in VR paper of 80% and above 50% 30%
Note 1: one standard deviation represents the amount of deviation or variance in the scores. 68% scores lie within one standard deviation of the mean.
Note 2: the sample size in 2009 and 2010 is rather small and the confidence that these results are as representative iin 2011 and 2012 is similarly limited
Note 3: these scores are not representative of all the scores from pupils taking the 11+. As can be seen from the very high mean overall standardised score, the scores posted on this 11+ forum represents the higher scores. For example, from the 2011 results, over half of all scores on the forum would have been definitely offered a place (344 was the score of the last person to receive an offer from KEGS, which has the highest cut-off mark of all CSSE schools), and over 70% would (could?) have got an offer at one of the CSSE scores
Other 'observations'
-Immediate feedback from pupils was that the VR paper was not particularly hard
-Not many pupils commented on time trouble
-On receiving the actual scores, many parents have expressed their surprise at how low their child's VR scores were. Some parents remarked that their child's score was 90+% in practice but the scores in the exam were 20% lower
-I am not aware of any changes to the format of the VR paper - still 12 blocks out of the 21 possible question types, 80 questions in 50 minutes
-anecdotal information is that there were several blocks of questions involving tricky vocabulary where the meanings of the words were similar
My Interpretation
1. There has been a drop in the average (mean) score in the 2012 VR paper from around 79.6% in 2011 to 74.8% in 2012.
2. Even the top students found the 2012 VR paper harder than pupils in 2011 as reflected in the scores eg in 2011 19% of pupils achieved 90% or higher, but only 4% managed this in 2012. Not a single score in 2012 here was 76 or higher out of 80 meaning that every one of the 76 pupils, bright though they are, all managed at least five questions wrong.
3. There is no reason to consider that the sample size is a factor when comparing 2011 to 2012 results (the sample size is sufficiently large, and the representation of candidates is of similar overall scores). The variance or standard deviation in 2012 is similar to that of 2011.
Possible hypothesis
It is possible that there may have been the odd one or two trickier than normal questions involving moving a letter, letter series, related numbers, etc, but this should account for only a few marks and certainly not five. The best logical explanation is that the 2012 VR paper contained at least two, probably three sections, which on the surface may have appeared easy to the child, but actually were difficult questions to answer correctly. For example, the following types of questions do not consume much time, but are extremely hard to get correct, and very bad odds of being able to guess..
Two odd ones out (http://www.elevenplusexams.co.uk/forum/ ... =3&t=24046" onclick="window.open(this.href);return false;)
Need to find two words that are different from the other three
eg split, cut, break, parting, sever
Opposite or same meanings (http://www.elevenplusexams.co.uk/forum/ ... =3&t=23628" onclick="window.open(this.href);return false;)
Can you find the two words with the most opposite in meanings?
1. (fiend, ferocious, demon) (mild, gentle, vicious)
2. (fluid, gritty, dense) (solid, soft, firm)
Related words (http://www.elevenplusexams.co.uk/forum/ ... =3&t=21459" onclick="window.open(this.href);return false;)
Stitch is to (Sew, needle, cloth) as stroke is to (pat,paint, cat)
Unfortunately we do not have access to the VR paper so we may never know...
The boring statistical part
I have analysed the scores posted in this forum for the 11 + exam for the 2009,2010,2011 and 2012. The result is as follows:
Mean and standard deviation scores are for VR paper only
2009: mean 70.3%, st dev 13.3%, mean overall standardised score 348 (sample size 13)
2010: mean 78.1%, st dev 9.1%, mean overall standardised score 350 (sample size 19)
2011: mean 79.6%, st dev 10.5%, mean overall standardised score 344 (sample size 49)
2012: mean 74.8%, st dev 10.0%, mean overall standardised score 341 (sample size 73)
A breakdown analysis of the VR scores can be done by looking at what percentage of scores achieved a certain score
............................................ 2011 2012
Score in VR paper of 95% and above 2% 0%
Score in VR paper of 90% and above 19% 4%
Score in VR paper of 85% and above 31% 16%
Score in VR paper of 80% and above 50% 30%
Note 1: one standard deviation represents the amount of deviation or variance in the scores. 68% scores lie within one standard deviation of the mean.
Note 2: the sample size in 2009 and 2010 is rather small and the confidence that these results are as representative iin 2011 and 2012 is similarly limited
Note 3: these scores are not representative of all the scores from pupils taking the 11+. As can be seen from the very high mean overall standardised score, the scores posted on this 11+ forum represents the higher scores. For example, from the 2011 results, over half of all scores on the forum would have been definitely offered a place (344 was the score of the last person to receive an offer from KEGS, which has the highest cut-off mark of all CSSE schools), and over 70% would (could?) have got an offer at one of the CSSE scores
Other 'observations'
-Immediate feedback from pupils was that the VR paper was not particularly hard
-Not many pupils commented on time trouble
-On receiving the actual scores, many parents have expressed their surprise at how low their child's VR scores were. Some parents remarked that their child's score was 90+% in practice but the scores in the exam were 20% lower
-I am not aware of any changes to the format of the VR paper - still 12 blocks out of the 21 possible question types, 80 questions in 50 minutes
-anecdotal information is that there were several blocks of questions involving tricky vocabulary where the meanings of the words were similar
My Interpretation
1. There has been a drop in the average (mean) score in the 2012 VR paper from around 79.6% in 2011 to 74.8% in 2012.
2. Even the top students found the 2012 VR paper harder than pupils in 2011 as reflected in the scores eg in 2011 19% of pupils achieved 90% or higher, but only 4% managed this in 2012. Not a single score in 2012 here was 76 or higher out of 80 meaning that every one of the 76 pupils, bright though they are, all managed at least five questions wrong.
3. There is no reason to consider that the sample size is a factor when comparing 2011 to 2012 results (the sample size is sufficiently large, and the representation of candidates is of similar overall scores). The variance or standard deviation in 2012 is similar to that of 2011.
Possible hypothesis
It is possible that there may have been the odd one or two trickier than normal questions involving moving a letter, letter series, related numbers, etc, but this should account for only a few marks and certainly not five. The best logical explanation is that the 2012 VR paper contained at least two, probably three sections, which on the surface may have appeared easy to the child, but actually were difficult questions to answer correctly. For example, the following types of questions do not consume much time, but are extremely hard to get correct, and very bad odds of being able to guess..
Two odd ones out (http://www.elevenplusexams.co.uk/forum/ ... =3&t=24046" onclick="window.open(this.href);return false;)
Need to find two words that are different from the other three
eg split, cut, break, parting, sever
Opposite or same meanings (http://www.elevenplusexams.co.uk/forum/ ... =3&t=23628" onclick="window.open(this.href);return false;)
Can you find the two words with the most opposite in meanings?
1. (fiend, ferocious, demon) (mild, gentle, vicious)
2. (fluid, gritty, dense) (solid, soft, firm)
Related words (http://www.elevenplusexams.co.uk/forum/ ... =3&t=21459" onclick="window.open(this.href);return false;)
Stitch is to (Sew, needle, cloth) as stroke is to (pat,paint, cat)
Unfortunately we do not have access to the VR paper so we may never know...
-
First-timer
- Posts: 698
- Joined: Wed Sep 23, 2009 1:47 pm
- Location: Essex
Re: A Perspective on 2012 VR paper
Wow! You've gone to a lot of trouble! Not sure what can be drawn from the skewed sample scores that get posted on here but supposing this year's VR was a bit harder than last year - does it make a difference? I thought that the papers were standardised after the event to take account of differences in difficulty. Everyone sat the same paper. Last year the Maths was easy and the English was supposed to be hard. It's just the way it falls. We've seen that they can change format without notice (as with English in recent years) and we just need to roll with it. At least the VR seems to still be following the same 21 types. It's obviously something that's needling you, but as you say, we'll never know. I don't think it will have made a difference as to who got in. Those who were strongest at VR should still be the strongest with a trickier paper.
Re: A Perspective on 2012 VR paper
I totally agree that it is about the relative performance and that in theory it is the "same" for all. I have no personal interest in this year's paper as my child already sat the 11+ last year. I feel sorry for those children and parents who were disappointed with their VR scores this year especially after all their hard work and effort. The frustating part is not knowing why...
Re: A Perspective on 2012 VR paper
Interesting anng, from the postings I guesstimated 
it was about 6% (nearer 5% it would appear) more difficult than last year.
Do we know for sure only 21 question types used?
it was about 6% (nearer 5% it would appear) more difficult than last year.Do we know for sure only 21 question types used?
Re: A Perspective on 2012 VR paper
Aang, I am totally impressed and did not find the statistical part boring at all. Thank you so much for taking the time to analyse this.
I had been concerned that maybe there was some type of computer glitch as so many parents had expressed surprise but your analysis helps put the context around it and shows that although a tough year, it wasn't a complete aberration.
Thank you
I had been concerned that maybe there was some type of computer glitch as so many parents had expressed surprise but your analysis helps put the context around it and shows that although a tough year, it wasn't a complete aberration.
Thank you
Re: A Perspective on 2012 VR paper
Aang, U are just mind blowing. I believe that many future 11+ parents will be definitely benefited from your analysis. Excellent work. Keep it up.
Re: A Perspective on 2012 VR paper
Thank you aang for taking the time to do this.
'What we have learned is like a handful of earth; What we have yet to learn is like the whole world.' Auvaiyaar.
Re: A Perspective on 2012 VR paper
Thank you very much, aang!
Re: A Perspective on 2012 VR paper
Thanks for your research aang - very interesting! These were they types of questions that my DS would have found more challenging, so if there were more of these types than usual then that may account for his low score in VR. However, as you say, without access to the paper, we will never know!!!
Re: A Perspective on 2012 VR paper
aang just a wild thought, the sort of questions you gave as examples would require a fair grasp of English. I know someone who didn't do as well in the VR as they should have and without doubt English was their weakest subject.
With all the data you have, can you statistically show a link between English and the VR this year compared to others? Again it's a pure guess but I think the English paper overall was a little easier than last year, for last years VR the DC mentioned codes created a bit of a problem as it gobbled up the time.
With all the data you have, can you statistically show a link between English and the VR this year compared to others? Again it's a pure guess but I think the English paper overall was a little easier than last year, for last years VR the DC mentioned codes created a bit of a problem as it gobbled up the time.