I'm in the same position as Sean (yoshinkan) and I also feel there are issues with safety and with training.
Ukemi is difficult enough to learn properly without a hakama. I don't personally see the point in beginners risking their bodies just to wear something that has only symbolic meaning. I've even heard some address the hakama as part of 'role playing'...while I don't go that far myself, I just can't see the value of risking injury to newbies. Too many broken toes, and worse from trying to push ukemi skills in a pleated set of baggy pants.
The training issue is that at least in the yoshinkan, instructors want to see your whole posture, hips, legs, foot positions, everything. I think that is part of the reason they don't use hakama til third dan. In the rougher styles of aikido (yoseikan, shodokan in particular), the hakama also impeads 'mixing it up'.
These are style specific things though...when in Rome...