I have only worn one in iaido as I am not allowed to in aikido. I find that it makes me more aware of my body and my surroundings. In a lot of the kata, I am doing them from seiza and sometimes go into standing after it started. You have to be aware of your feet and your hakama so you don't trip while getting up.
There's also the aspect of preserving tradition. I certainly don't train in martial arts to help defend myself (although I would like to think that I may be able to if needed). I train to gain skill and knowledge, and thus help to preserve the best of what I can. That should include the clothing and the etiquette. If each generation chips away little things that are no longer relevant, then what they're training in soon begins to change.