First, a note on safety: Have none of you ever caught your foot in the hem of someone's hakama while taking a breakfall? I mean, it's not the end of the world, but I and others I know have had some seriously pulled groin muscles. Maybe it's just our local ukemi style, but I doubt it.
I do not yet wear hakama and have found it to be a bit of a problem when visiting other (especially non USAF) dojos. The problem is really in difference of promotion speeds and rank signifiers. Where I train, it takes a looooong time to get to sho dan, and thereby to wearing hakama. When I visit other dojos, the immediate assumption, based on the state of my uniform, is that I've been training for maybe a year or two, when in fact it's getting close to 8 years. One of the reasons I'm looking forward to getting my sho dan is that I think it will make visiting other dojos less awkward in terms of rank and experience confusion (though I'm sure there's a flip side to this).
Besides, just about everyone looks better wearing hakama.