each year group had its swings and roundabouts
Quite right and some people are more suited to different year groups. I have taught every year group at primary, including nursery - which I was awful at. It takes a very special person to teach nursery classes.
My experience is that primary can be more exhausting than secondary
I think the key words are 'My experience'. This is my experience too, but no one is saying marking A level work isn't exhausting and time consuming too.
When I taught foundation and KS1 the planning took ages and I was exhausted at the end of the day - young pupils are very demanding of your attention, but of course the marking didn't take long and I used to mark with the pupils. KS2 pupils are more independent, but create more marking and for the school SATs in Y6 are pretty high pressure. Tiring in a different way. I moved to secondary because I prefer teaching older pupils and as a result, in my experience, I find it less stressful and I'm definitely doing less hours at home even though I have increased my hours at work.
Anyone considering teaching as a career should get themselves into as many different schools as they can across a wide range of ages.