Though the average age in aikido class is a lot higher than it was in karate class (my guess is 40 versus 25), I'd say most of the student under 50 are quite fit (thanks to aikido training I'd say). Past 55 ageing seems to start affecting joints of some, sometimes from old injuries.

I don't see that an average age of 40 makes it an old man's art, in the sense that everyone is so fragile from old age that training has to be extra slow and careful (in fact I think the 16 year olds seem more fragile than the 40 year olds. Actually, also in karate class, I think the 35 to 50 year olds were the toughest)

For me there is one major disadvantage of starting late: because I started at 41, I may not be able to continue training for as many years as I hope to.