For me, on the few occasions when I trained without a hakama, after being very used to one, I felt like a dork. With the hakama, I feel like I'm working with a wide, solid foundation, and without it, I feel like there are spindly little sticks holding me up. I don't really have skinny legs, either. It's just a weird psychological thing that's built up. Although it doesn't bother me when I'm doing non-Aikido stuff in a gi.
From a practical viewpioint, we'd be better off without hakama, or even obi. I have practiced an interactive, relaxation game that is like a dynamic, wide-open version of pushing hands (developed by Peter Ralston). Since there were no uniform rules I just wore a karate gi top with the strings tied on the sides - no belt. The automatic increase in relaxation and freedom of movement was very surprising.