Dear Parents and Carers,
Following my last letter on Friday 20 September, I am writing with further details about the Secondary Transfer Test and the solution we have agreed with The Buckinghamshire Grammar Schools, following an independent review and verification.
The issue
In the Secondary Transfer Test on Thursday 12 September, there were issues with two questions and a practice question in the Verbal Skills paper:
• For question 10 of the Verbal Reasoning section of the paper, the correct answer could be found on the answer sheet, however three of the other answer options did not match the options on the question paper; • For question 13 in the same section, the answer could not be found on the answer sheet; • A practice question in the English section could also be answered, however one of the five responses on the answer sheet did not match the answers on the question paper.
We have undertaken a thorough investigation into how the error occurred. While a correct answer sheet had been produced and verified by TBGS, a failure in our final quality assurance process meant that a previous answer sheet was used in its place. This resulted in the errors outlined above.
To address this moving forwards, we will conduct a systematic review of all our internal processes. In addition, we will also appoint an external, independent reviewer to provide a final quality assurance check of our future admissions test papers.
The key concerns
Our overriding priority has been to ensure all children get a fair result. Our statistical analysis has therefore focused on two core areas:
1. The reliability of the test to determine whether or not a child is suited to a grammar school in Buckinghamshire; and 2. The impact the errors may have caused on the ability of the children to complete the Verbal Reasoning section of the paper, given the different conditions under which the children sat the paper (for example, the children who were advised not to attempt the questions, children who were told part-way through the paper, and children who were not informed).
GL Assessment’s statisticians have reviewed the test performance in detail and passed their findings to an independent statistician, who has been approved by The Buckinghamshire Grammar Schools (TBGS). Our key findings, which have been independently verified, are outlined below.
The findings
In relation to the reliability of the test to determine whether or not a child is suited to a grammar school, our analysis found that the overall reliability of the test has not been compromised and remains very good. However, we recognise the need to consider the impact on each individual child given the different instructions they may or may not have been given.
We considered two specific possibilities: that the time spent by children on attempting to resolve the problem might mean they were less likely to complete the test; and that children might have been unsettled by being unable to find an answer to the two questions, so that their performance on later ones was affected. We also compared and analysed completion rates between the 2018 and 2019 papers in order to understand how children performed against a similar paper in similar circumstances.
Our analysis found that, while the test reliability and completion rates as a whole remain statistically sound, there was evidence that completion rates started to drop during the last six questions of the Verbal Reasoning section of the paper. This evidence is consistent with some of the feedback we have received from parents. Our solution has therefore been based on this evidence.
The solution
The solution we have agreed with The Buckinghamshire Grammar Schools is:
1. All children will be awarded a mark for each of the two erroneous questions, thereby ensuring no advantage or disadvantage from these two questions;
2. In order to ensure no individual child is penalised for not being able to complete the test, the last six questions of the Verbal Reasoning section will not count towards the final mark.
It is important to reiterate that the independent statistician has verified that the outcome of the test, without those questions, is still fair for all children, highly reliable and above the accepted conventions for admissions tests.
What this means for you and your child
The test as a whole is very reliable. This reliability means that the test results will still be sent to you, as planned, on 18 October ahead of the deadline for school applications on 31 October.
Further information
Please be reminded that Primary headteachers and test centres do not have any further information to support this letter at this time and so we would ask you not to contact them directly in relation to this matter. The mailbox for further enquiries is:
bucks@gl-assessment.co.uk.
A set of Frequently Asked Questions will also be published on the TBGS website later this week:
www.thebucksgrammarschools.org/faqs. These questions will aim to provide you with information prior to any review or appeal.
I would like to apologise once again for any distress this situation has caused your child and your family, as well as thank you for your patience as we have conducted our analysis and reached this solution.