I think the term implies a type of relationship with one's instructor. I think Ellis Amdur coined the phrase "at once intimate and distant". It certainly makes sense in a Japanese cultural context and I would certainly use the term for Japanese instructors, but even this is flexible.
For example, I would refer to my Shakuhachi teacher as "xxx Sensei" when addressing him in public (in relation to shakuhachi activities), but I always addressed as "xxx San" outside of that context because it "felt" right. I always referred to my martial Arts instructor as "xxx Sensei".