I think this is really true... people seem to divide up different kinds of physical activity in their minds as being 'for kids' or 'for adults' and often gendered as well (e.g. strength training = adult and male, aerobics = adult and female, yoga = adult and female, gymnastics = little girls, jogging = adults of both sexes, skiing = adults and kids of both sexes, etc) Not necessarily that people will give you a hard time if you don't fit the expected age and gender, but they'll be a little surprised.

Traditional martial arts, with the gis and foreign language, seem to be seen as mainstream IF it's for kids and much more niche if it's for adults. Something about that whole mantra about 'martial arts teach discipline and build character' that we often hear seems to translate as 'it's good for kids' to many people and make it seem vaguely incongruous or eccentric for adults. Not to the point of weird or anything, but it's an 'interesting' activity rather than an obvious expected one (like 'going to the gym' or jogging would be).

I also notice that it's more common to see kids with very regular predictable attendance than adults... Adults will make sure their kids show up at their scheduled activities regularly more strictly than they'll do the same themselves.