This was also one of my personal moments. At the Shodokan Honbu dojo there is a system of coloured belts which really does serve a purpose since some people train once a week and others train multiple sessions every day and it really is hard to remember all the faces. It is a good rough indicator of background skill.
The university clubs don't have coloured belts but they all enter the club together, train loads together and know each other well.
I still remember how cocky I was as a newly minted green belt and how shocked I was playing shihan to a mere white belt. One of these "Kinki girls" (Kinki Daigaku) shorter and lighter than I was broke my grip and tossed me with a perfect gedan-ate - I never even saw it coming. I was more careful after that.
Coloured belts have no more meaning than a tool to organize teaching. My general suggestion is that when you visit another club, even in the same style, you wear your white belt.
I agree with you. I have visited another dojo while my shihan was away on vacation. I elected to wear my white belt. I received funny stares and could hear the rumblings from the main students for they felt I might have known something. Irregardless, I still maintained respect for their school and their teaching from their sensei. Bottom line, you represent your school and teacher when you walk and/or train somewhere else. I ever want to hear is that another instructor mentions to mine, I can see he was taught by you and you should be proud.