I don't think you would be silly to turn down a Grammar place - IF you thought another school was better suited to your son.
I think you are doing the right thing by looking at all the options and not just going for the easy one!
A lot may depend on you DS's personality. TWGSB is a big school. I know many on this forum are very happy with it for their sons but I have always felt my DS has been lost in a big crowd because he is 'middle of the road' and causes no problems! The teaching has always been good though.
I don't know much about the grammar streams/school to which you refer, other than I know children at Skinners Kent Academy who are thriving.
Its such a hard decision and interesting that he is keen on a particular school.
Would subjects taken at GCSE be deal breaker? Its worth looking into that also. Good luck.