A Perspective on 2012 VR paper
Posted: Wed Mar 07, 2012 4:41 pm
Along with many parents, I have been puzzled by the seemingly lower scores on the 2012 verbal reasoning paper. This post represents some statistical analysis and my best effort to shed some light on the results.
The boring statistical part
I have analysed the scores posted in this forum for the 11 + exam for the 2009,2010,2011 and 2012. The result is as follows:
Mean and standard deviation scores are for VR paper only
2009: mean 70.3%, st dev 13.3%, mean overall standardised score 348 (sample size 13)
2010: mean 78.1%, st dev 9.1%, mean overall standardised score 350 (sample size 19)
2011: mean 79.6%, st dev 10.5%, mean overall standardised score 344 (sample size 49)
2012: mean 74.8%, st dev 10.0%, mean overall standardised score 341 (sample size 73)
A breakdown analysis of the VR scores can be done by looking at what percentage of scores achieved a certain score
............................................ 2011 2012
Score in VR paper of 95% and above 2% 0%
Score in VR paper of 90% and above 19% 4%
Score in VR paper of 85% and above 31% 16%
Score in VR paper of 80% and above 50% 30%
Note 1: one standard deviation represents the amount of deviation or variance in the scores. 68% scores lie within one standard deviation of the mean.
Note 2: the sample size in 2009 and 2010 is rather small and the confidence that these results are as representative iin 2011 and 2012 is similarly limited
Note 3: these scores are not representative of all the scores from pupils taking the 11+. As can be seen from the very high mean overall standardised score, the scores posted on this 11+ forum represents the higher scores. For example, from the 2011 results, over half of all scores on the forum would have been definitely offered a place (344 was the score of the last person to receive an offer from KEGS, which has the highest cut-off mark of all CSSE schools), and over 70% would (could?) have got an offer at one of the CSSE scores
Other 'observations'
-Immediate feedback from pupils was that the VR paper was not particularly hard
-Not many pupils commented on time trouble
-On receiving the actual scores, many parents have expressed their surprise at how low their child's VR scores were. Some parents remarked that their child's score was 90+% in practice but the scores in the exam were 20% lower
-I am not aware of any changes to the format of the VR paper - still 12 blocks out of the 21 possible question types, 80 questions in 50 minutes
-anecdotal information is that there were several blocks of questions involving tricky vocabulary where the meanings of the words were similar
My Interpretation
1. There has been a drop in the average (mean) score in the 2012 VR paper from around 79.6% in 2011 to 74.8% in 2012.
2. Even the top students found the 2012 VR paper harder than pupils in 2011 as reflected in the scores eg in 2011 19% of pupils achieved 90% or higher, but only 4% managed this in 2012. Not a single score in 2012 here was 76 or higher out of 80 meaning that every one of the 76 pupils, bright though they are, all managed at least five questions wrong.
3. There is no reason to consider that the sample size is a factor when comparing 2011 to 2012 results (the sample size is sufficiently large, and the representation of candidates is of similar overall scores). The variance or standard deviation in 2012 is similar to that of 2011.
Possible hypothesis
It is possible that there may have been the odd one or two trickier than normal questions involving moving a letter, letter series, related numbers, etc, but this should account for only a few marks and certainly not five. The best logical explanation is that the 2012 VR paper contained at least two, probably three sections, which on the surface may have appeared easy to the child, but actually were difficult questions to answer correctly. For example, the following types of questions do not consume much time, but are extremely hard to get correct, and very bad odds of being able to guess..
Two odd ones out (http://www.elevenplusexams.co.uk/forum/ ... =3&t=24046" onclick="window.open(this.href);return false;)
Need to find two words that are different from the other three
eg split, cut, break, parting, sever
Opposite or same meanings (http://www.elevenplusexams.co.uk/forum/ ... =3&t=23628" onclick="window.open(this.href);return false;)
Can you find the two words with the most opposite in meanings?
1. (fiend, ferocious, demon) (mild, gentle, vicious)
2. (fluid, gritty, dense) (solid, soft, firm)
Related words (http://www.elevenplusexams.co.uk/forum/ ... =3&t=21459" onclick="window.open(this.href);return false;)
Stitch is to (Sew, needle, cloth) as stroke is to (pat,paint, cat)
Unfortunately we do not have access to the VR paper so we may never know...
The boring statistical part
I have analysed the scores posted in this forum for the 11 + exam for the 2009,2010,2011 and 2012. The result is as follows:
Mean and standard deviation scores are for VR paper only
2009: mean 70.3%, st dev 13.3%, mean overall standardised score 348 (sample size 13)
2010: mean 78.1%, st dev 9.1%, mean overall standardised score 350 (sample size 19)
2011: mean 79.6%, st dev 10.5%, mean overall standardised score 344 (sample size 49)
2012: mean 74.8%, st dev 10.0%, mean overall standardised score 341 (sample size 73)
A breakdown analysis of the VR scores can be done by looking at what percentage of scores achieved a certain score
............................................ 2011 2012
Score in VR paper of 95% and above 2% 0%
Score in VR paper of 90% and above 19% 4%
Score in VR paper of 85% and above 31% 16%
Score in VR paper of 80% and above 50% 30%
Note 1: one standard deviation represents the amount of deviation or variance in the scores. 68% scores lie within one standard deviation of the mean.
Note 2: the sample size in 2009 and 2010 is rather small and the confidence that these results are as representative iin 2011 and 2012 is similarly limited
Note 3: these scores are not representative of all the scores from pupils taking the 11+. As can be seen from the very high mean overall standardised score, the scores posted on this 11+ forum represents the higher scores. For example, from the 2011 results, over half of all scores on the forum would have been definitely offered a place (344 was the score of the last person to receive an offer from KEGS, which has the highest cut-off mark of all CSSE schools), and over 70% would (could?) have got an offer at one of the CSSE scores
Other 'observations'
-Immediate feedback from pupils was that the VR paper was not particularly hard
-Not many pupils commented on time trouble
-On receiving the actual scores, many parents have expressed their surprise at how low their child's VR scores were. Some parents remarked that their child's score was 90+% in practice but the scores in the exam were 20% lower
-I am not aware of any changes to the format of the VR paper - still 12 blocks out of the 21 possible question types, 80 questions in 50 minutes
-anecdotal information is that there were several blocks of questions involving tricky vocabulary where the meanings of the words were similar
My Interpretation
1. There has been a drop in the average (mean) score in the 2012 VR paper from around 79.6% in 2011 to 74.8% in 2012.
2. Even the top students found the 2012 VR paper harder than pupils in 2011 as reflected in the scores eg in 2011 19% of pupils achieved 90% or higher, but only 4% managed this in 2012. Not a single score in 2012 here was 76 or higher out of 80 meaning that every one of the 76 pupils, bright though they are, all managed at least five questions wrong.
3. There is no reason to consider that the sample size is a factor when comparing 2011 to 2012 results (the sample size is sufficiently large, and the representation of candidates is of similar overall scores). The variance or standard deviation in 2012 is similar to that of 2011.
Possible hypothesis
It is possible that there may have been the odd one or two trickier than normal questions involving moving a letter, letter series, related numbers, etc, but this should account for only a few marks and certainly not five. The best logical explanation is that the 2012 VR paper contained at least two, probably three sections, which on the surface may have appeared easy to the child, but actually were difficult questions to answer correctly. For example, the following types of questions do not consume much time, but are extremely hard to get correct, and very bad odds of being able to guess..
Two odd ones out (http://www.elevenplusexams.co.uk/forum/ ... =3&t=24046" onclick="window.open(this.href);return false;)
Need to find two words that are different from the other three
eg split, cut, break, parting, sever
Opposite or same meanings (http://www.elevenplusexams.co.uk/forum/ ... =3&t=23628" onclick="window.open(this.href);return false;)
Can you find the two words with the most opposite in meanings?
1. (fiend, ferocious, demon) (mild, gentle, vicious)
2. (fluid, gritty, dense) (solid, soft, firm)
Related words (http://www.elevenplusexams.co.uk/forum/ ... =3&t=21459" onclick="window.open(this.href);return false;)
Stitch is to (Sew, needle, cloth) as stroke is to (pat,paint, cat)
Unfortunately we do not have access to the VR paper so we may never know...