For me, it's cultural as I'm Japanese and have that culture integrated into my life.
My current teacher is Japanese. I call him "sensei" when I speak to him directly and also when I speak about him to others. This occurs both off and on the mat. (Eg "Sensei, did you want to catch a bite to eat after class?" Now, if I weren't involved in aikido at all and he were just your average run-of-the-mill guy, I'd probably still call him by his last name and "san" as that's natural and polite in the Japanese language/culture (as he's older than I am and, hence, "higher" on the hierarchical societal structure).
As it stands, I pretty much call all Japanese aikido teachers "sensei" both on and off the mat, even if I'd never trained with that person in my life. It's just natural to me. I'd feel like I were slighting the person by not using "sensei." It just feels, well, incorrect almost at a biological level or something if I started referring to them as "Yamada," "Chiba," and such.
Usually in Japanese culture, only people who are at about the same hierarchical level call others by their last name without "san" or such, and even that after some acquaintance. When I was doing research in Japan during college, folks in the undergratuate section called me "Akiyama san" while the doctoral candidates and such called me "Akiyama kun." If, say, I had been classmates with one of them for a while, they might have called me "Akiyama."
Now, non-Japanese aikido teachers I'll usually address by their first name but use "sensei" on the mat (eg "Mary sensei," "John sensei") but just by their first name off the mat. Even our 6th dan senior student here I'll just call by his first name off the mat. Some folks, though, I'll say have earned my respect and the whole Japanese cultural hierarchical thing kicks in; I do call some Western aikido teachers (most with thirty to forty years of experience) "sensei" too...
But, when I'm speaking of teachers in the third person to folks who may not know them personally, especially to folks who may hold the teacher in high regard, I'll usually use "sensei" in that context to be polite.
Hope that didn't muddy up the waters too much.