well, in the US going topless is considered immodest for women, but not so socially taboo for men.
In my dojo all people can wear Hakama from the get go, but people gradually start to adopt them whenever they feel 'ready'.
Some people do so earlier than others. It doesn't change much except for the initial adjustment time.
It's nice to see some uniformity in the room, but it's a purely aesthetic consideration I think. That's not to downlplay the aesthetic aspect either.