I'd be wary of saying "not until year 6." So much of "it" depends on the primary school and the child, if the child is allowed to coast while a low expectation is sought class-wide then you need to be ready to supplement this and boost the child's aptitude for on-going learning (education not just being a 9-3 thing or a dirty word).
This came home to me recently when on holiday. A woman, a teacher, was surprised that our Y6 boy was allowed out of school to go away. This was last week.
I said that there had been no probs with him, only with our Y8 boy (bear with me, there IS relevance!!) I asked her why she was surprised about the Y6 boy, thinking that in Y6 they tend to coast along and not do much (at least at our gaff).
"Because of SATS," she said.
DS2 was on 5s when he was in Y5, and he's secured his grammar place so maybe I'm slacking off big time but I murmured vaguely that that was all OK and she said,
"oh, well, maybe it's just a problem if they're borderline 3/4."
I was shocked (easily done) at the paucity of expectation. Maybe I'm naive, OK, I'm naive, but our school aims for 5s (gets 3s in nigh on 100% of the year in Y2 so need to get 5s to demonstrate the necessary uplift) and here, already, the assumption is that 3s or 4s might be on the cards. Talk about aiming low. (And here, am not denigrating any child who is not academic or who works hard and is rightly proud to get their 4, just that if the ability is there to get higher, the support should be too.)
So, if your primary is a vibrant lively one, promoting an interest in learning, keeping the brain buzzing in all sorts of different ways, then it's one thing. Keep them alert and interested, as yoyo suggests. You can ask the HT for their VRQ scores. These aren't told to you unless you ask but provide a marker against which you can assess their relative placings (which is what it all comes down to) and how much above the national average they are as plotted from, I think, Y1. Our head had great charts of it all, and would tell you (no names) what children who achieved grammars in years ahead had got at the earlier stages to give you an idea of cross reference/comparison.