News from GL regarding errors on test.

[greengekho]

Dear Parents and Carers,

Following my last letter on Friday 20 September, I am writing with further details about the Secondary Transfer Test and the solution we have agreed with The Buckinghamshire Grammar Schools, following an independent review and verification.

The issue

In the Secondary Transfer Test on Thursday 12 September, there were issues with two questions and a practice question in the Verbal Skills paper:

• For question 10 of the Verbal Reasoning section of the paper, the correct answer could be found on the answer sheet, however three of the other answer options did not match the options on the question paper; • For question 13 in the same section, the answer could not be found on the answer sheet; • A practice question in the English section could also be answered, however one of the five responses on the answer sheet did not match the answers on the question paper.
We have undertaken a thorough investigation into how the error occurred. While a correct answer sheet had been produced and verified by TBGS, a failure in our final quality assurance process meant that a previous answer sheet was used in its place. This resulted in the errors outlined above.

To address this moving forwards, we will conduct a systematic review of all our internal processes. In addition, we will also appoint an external, independent reviewer to provide a final quality assurance check of our future admissions test papers.

The key concerns

Our overriding priority has been to ensure all children get a fair result. Our statistical analysis has therefore focused on two core areas:

1. The reliability of the test to determine whether or not a child is suited to a grammar school in Buckinghamshire; and 2. The impact the errors may have caused on the ability of the children to complete the Verbal Reasoning section of the paper, given the different conditions under which the children sat the paper (for example, the children who were advised not to attempt the questions, children who were told part-way through the paper, and children who were not informed).
GL Assessment’s statisticians have reviewed the test performance in detail and passed their findings to an independent statistician, who has been approved by The Buckinghamshire Grammar Schools (TBGS). Our key findings, which have been independently verified, are outlined below.







The findings

In relation to the reliability of the test to determine whether or not a child is suited to a grammar school, our analysis found that the overall reliability of the test has not been compromised and remains very good. However, we recognise the need to consider the impact on each individual child given the different instructions they may or may not have been given.

We considered two specific possibilities: that the time spent by children on attempting to resolve the problem might mean they were less likely to complete the test; and that children might have been unsettled by being unable to find an answer to the two questions, so that their performance on later ones was affected. We also compared and analysed completion rates between the 2018 and 2019 papers in order to understand how children performed against a similar paper in similar circumstances.

Our analysis found that, while the test reliability and completion rates as a whole remain statistically sound, there was evidence that completion rates started to drop during the last six questions of the Verbal Reasoning section of the paper. This evidence is consistent with some of the feedback we have received from parents. Our solution has therefore been based on this evidence.

The solution

The solution we have agreed with The Buckinghamshire Grammar Schools is:

1. All children will be awarded a mark for each of the two erroneous questions, thereby ensuring no advantage or disadvantage from these two questions;

2. In order to ensure no individual child is penalised for not being able to complete the test, the last six questions of the Verbal Reasoning section will not count towards the final mark.
It is important to reiterate that the independent statistician has verified that the outcome of the test, without those questions, is still fair for all children, highly reliable and above the accepted conventions for admissions tests.

What this means for you and your child

The test as a whole is very reliable. This reliability means that the test results will still be sent to you, as planned, on 18 October ahead of the deadline for school applications on 31 October.

Further information

Please be reminded that Primary headteachers and test centres do not have any further information to support this letter at this time and so we would ask you not to contact them directly in relation to this matter. The mailbox for further enquiries is: bucks@gl-assessment.co.uk.







A set of Frequently Asked Questions will also be published on the TBGS website later this week: http://www.thebucksgrammarschools.org/faqs" onclick="window.open(this.href);return false;. These questions will aim to provide you with information prior to any review or appeal.

I would like to apologise once again for any distress this situation has caused your child and your family, as well as thank you for your patience as we have conducted our analysis and reached this solution.

[sookipixeldust]

Thanks greengekho

If the solutions is this: "In order to ensure no individual child is penalised for not being able to complete the test, the last six questions of the Verbal Reasoning section will not count towards the final mark."

Will this unfairly penalise kids with stronger VR skills, or not? Personally I was depending on my child getting more points in VR (I know they finished this section) to help balance out their slightly weaker maths skills. Or do you think the weighting will fix this issue (so that one question would be worth more than it would have been before this crisis, so getting one wrong will also be worse)?

Please correct me if I'm wrong, but my interpretation is as follows:

The last 6 questions are now not counted (questions 14-19?), and everyone will get 2 points for those incorrect ones (questions 10 & 13), so basically, if there were 19 questions to start with, they’ll have to judge the VR skills of the child on 11 questions instead of 19. Thus, there is far less chance of the child being able to distinguish themselves in the VR section. Also, as the questions usually get harder towards the end of each section, it’s likely that most kids will get many of the first 11 questions right. The points that make the difference - that make the children's VR ability obvious - are gained in the later ones.

In other words, a child will now have had to have done better in all the other sections in order to score above average overall. If they were best at VR skills, then it seems to me that this would penalise them.

Also, with the smaller pool of marks, I wonder if the differences between 120 and 121 for instance, will probably now be down to only 1 or 2 marks, instead of maybe 3 or 4.

I'd be interested in what other people think.

I agree that there wasn't an obvious better or fairer solution to this problem.

[1lurker]

My very simple understanding is... each of the three areas are standardised separately, so overall still only 30% or so of children will score 121 or above in the VR / English section. The standardised score is then divided in two to give 50% of the total marks ( added to 25% from Maths and 25% from NVR both of which are standardised separately so only 30% or so children will score over 121 in each) in theory those strong in VR / English will still perform better than those who are not.

[mm23292]

But surely a child who was stronger on the VR component of the paper, compared with the English component, will be adversely affected.

[fabs]

I completely disagree with the approach taken:

1) From my understanding, there were a number of sittings where the children were told before the start of the exam, to ignore questions 10 and 13 - why should they be awarded the additional two points?

2) Some children will naturally not work through the questions as quickly as others. Surely the decision not to mark the last six questions will favour the children that work at a slower speed. In the same way, other children that sped through earlier questions (with less care and attention) in order to complete the paper in time, will be disadvantaged vs. children that took more time on the earlier questions?

Surely a better approach would have been to have separate proposals based on whether the children were;

a) told to ignore the questions before the start of the exam
b) told during the exam
c) not told anything

Thoughts??

[Morty999]

Yes.
They don’t say when they did the statistical comparison of 20-8 vs 20-9 whether it was all papers or just those from centres not told in advance.
It’s all a farce!
They should ignore the 2 wrong w: and not just give everyone 2 marks surely?
And as for discounting the last 6 that will surely skew the results for those kids who got those last 6 correct.

[1lurker]

I don’t think they could adjust the scores in any way and please everyone. They are clearly relying on the review process scooping up any children who slip through the net.

Unfortunately the borderline children who don’t have supportive parents, will just accept the result and grow up thinking they weren’t good enough.

[Deb70]

https://schoolsweek.co.uk/bucks-11-plus ... l-appeals/