My case is an odd one but I'll share anyway.
For various reasons I had to start school a year earlier than my peers.
Despite being younger I was clearly ahead of the class, but when my class went up I was kept down to be with my correct age-group. I got rather upset about this since I was being treated as the classroom helper - (which I wouldn't doubt is what is happening to your son, and is what happened to both of my son's too). I became so frustrated that I began to hate school and hover outside the higher classes, being facinated by their work and subjects.
Eventually they moved me up a year, which was a great relief and once again I was the youngest in that class, but doing really well.
When I reached Year 6 my friends went merrily off to secondary school and the local authorities decided that I could not go to secondary school until I was the correct age. I had to do year 6 again, with a 'baby' class. Mentally and physically, I was ready to move off and I can't tell you how frustrating it was to remain in primary school and once again be 'the helper.' I had to complete the entire year all over again, same topics, same subjects and nothing at all was done to ease the blow. To make matters even worse, salt was rubbed into my wounds by my now secondary school friends wanting nothing to do with a 'little primary school kid.'
If your son can't go to secondary school a year earlier, I would suggest that he remain with his peers and just enjoy being the class brain-box.
I refused offers to move my kids up a year for this very reason, but insisted that they be stretched intellectually. Thankfully we have this London gifted and Talented thing going on which is great at giving more able kids greater opportunities. In addition to this my eldest son was allowed to quietly study Key Stage 3 science in primary school. I also tapped into their needs a lot at home by getting them good books to read, finding good internet sites, watching documentaries with them and so on....
My youngest child is in Nursery now and she can already read - I actually don't think this is a particularly great achievement as I think most kids probably could at age 4 if they had the right stimulation and encouragement...but that's another issue. Her teacher has suggested she might like to go up to Reception and spend time in there for a while and I have firmly resisted.
Finally - my eldest son's first teacher was/is married to a Psychologist and after I refused to allow my son to be moved up a year she told me that she had discussed this with her husband and he felt that it would potentially do more harm than good for a child to be elevated only to be demoted again in Year 6.
I believe each parent instinctively knows what is best for their child, and am sure you will make the right decision regarding him. Just thought I'd share my personal experience.