I agree that rank should not be equated with status. I should add that I've mostly worked with people from my own organization, so we all know approximately what the rank symbolizes.
However, I was at a seminar in January that drew people from many different styles. We simply talked about what we were comfortable with before working on a technique. This was encouraged by the instructor.
So I guess that at their best, symbols of rank can be a rough indicator of ability and people of different abilities get the chance to work together and learn from one another. The dark side is the egoism and divisivness you've noticed in some places.
As for the hakama, I'm comfortable with it being worn at shodan and above -- and I'm not there yet.