I have heard this argument before. I read an excerpt from one of Saotome Sensei's books (sorry I don't reccollect which one) where he says that this is (and this in my words not Saotome Sensei's) rediculous. If Aikido is about everything it is supposed to be about then even suggesting that someone would be having improper thoughts about the women in class because they are not wearing hakama defeats the purpose.
Not as well put as Saotome Sensei put it.
My understanding is that the hakama was initially required as it was part of formal wear. Those who didn't have one could not practice. Therefore one had to beg, borrow, or buy a hakama to attend practice. Then afte WWII, fabric being so scarce, it was allowed to practice without.