He was talking more, it would seem, about the four classes in Japan, which became especially clear during the Tokugawa Shogunate. The Four classes (in order of the most powerful and respected) were Samurai, farmers, artisans, and merchants. Farmers were not necessarily well thought of, but they provided the rice crop. The merchants, or eta, were looked down upon and lived in their own dirty villages, because it was believed it was wrong to make a living off someone else's work.
Anyways, back on topic-
before becoming an Aikidoka, I did Shito-Ryu karate. My instructor was in his 50's, and while he was in good shape, you wouldn't be able to tell he was an instructor if he wore street clothes. But watching his kata, and especially doing bunkai, you could tell you didn't wanna mess with him.
In the eyes of the warrior, everyone is as dangerous as their inner spirit, not their outer appearance.