Why are they separate, and are they necessarily so?
When I think about martial arts globally, I really wonder about a few things.
Why is this internal stuff limited so strongly and for so long to one culture? Why so few, why so supposedly secret, why not well supported by science? Why didn't someone in Wales or The Northwest Territories or Brazil or Mali find and develop the same thing?
They are separate because they use a completely different methodology to manipulate the body and create power, that are not compatible with each other; doing things one way pretty much negates doing the other.
AFAIK, "internal stuff" was possibly far more widespread long ago than it is now. If I recall correctly some of the historic research, what we would recognize as "internal" exercises used for meditative and health purposes traveled east with monks out of India, and was not initially part of any martial art. Over centuries, such groups and individuals may have had the opportunity to experiment with their bodies and create a body of knowledge that became integral to their particular sects. In the countries where those sojourners set up camp, warrior clans or individuals within them may have picked up on the practical applications after exposure to esoteric religious or spiritual practices.
Once it got into MAs, I don't think it's surprising that individuals who possessed the skills would suppress their dissemination, to protect their martial advantage over their enemies. Religious sects and spiritual groups may still maintain the core practices, for all we know. Being esoteric often means being invisible to the world at large.