Though it is possible—in fact highly likely—that horseriders wore hakama, there is no obvious connection between hakama and horse-riding.
The word is usually written as 袴, (Nelson No. 5448, Radical 145). The character has been written in three different ways, with different radicals on the left side of the character. The radical above is koromo-hen (R.120), and means 'clothes', the other two radicals being ito-hen, meaning 'thread' or 'yarn', and kawa-hen (R.177), meaning 'leather'.
The fact that the word appears with a radical meaning 'leather' suggests that trousers were actually made of leather and thus might have been worn by horseriders, but the latter two ways of writing the word have dropped out of use. Nevertheless, all three ways can be found in Morohashi's "Dai Kanwa Jiten", where the meaning of the word is also quite clear: basically hakama are trousers, and the wide, narrow, flowing, pleated varieties all appeared much later than the first use of the word, given in the "Kokugo Dai Jiten" (in 732).
I should perhaps add that there is a whole raft of theories concerning horseriders coming to Japan (usually from Mongolia), and even that these brought to Japan Mongolian martial arts which have been preserved. These theories originate in a thesis proposed by Egami Namio in 1948 and subsequently refined and published as a book entitled "Kiba minzoku kokka" (Tokyo, Chuokoronsha, 1967). The theory turns upon the differences in archaeological remains thought to go back to the Kofun period, when the Yamato state was established.
I should also add that the ice on which Egami skates is awfully thin.
Best regards to all,