Why? He didn't teach religion. If a Shinto priest was teaching mathematics, do you think these same people would have a problem learning from him?
Probably not, mathematics isn't something you can colour with religion or philosophy in the way Aikido
can be and mathematics wasn't invented by a Shinto priest, as far as we know.
Take for example the Christian martial arts groups, think they'd be comfortable studying the creation of a Shinto priest? Some of them might be out of the door faster than you can say, "Pagan idol worship."
Switching things around, if I turned up at a new martial art and the creator was, lets make this fun, the Pope and the Popes new martial art was supposedly deeply philosophical I'd want serious reassurances that they weren't in some way trying to teach me Christianity and I'd be highly suspicious if they told me that the Christian clerics philosophy wasn't essentially Christianity.