George S. Ledyard
A person who trains seriously in almost anything will handle someone who doesn't train.
I don't buy it. The evidence I've experienced tells me this is simply not true. If you are for example training complex lock flows for 10 years, and never spar, then come across a tough guy who spazes out and swings wildly you are going to get it. If you are not used to getting hit ( the case for most martial artists sadly) it now comes down to what kind of person are you. Do you crumple or continue. Most people crumple. This is why tough guys can win fights without training and why guys can 'cave man' as you said above.
Even as a serious student of combat sports, it is hard to fight a strong aggressive attacker who is untrained, does unexpected things, and wants to hurt you. I have over a decade of martial arts training. The first time I got punched in the face (should that stil l happen after a decade of training? :-p ) I turned my head away, was hit again, started moving straight back, was hit again, and finally was overwhelmed. Why? Because in all my years of martial arts, I was never really punched in the face. It hurt, I didn't know what to do, I didn't know how to stop him. I got confused, lost my distance, and it was over. I had multiple point sparring trophies and a black belt in TKD, over a year in aikido, over a year in Krav Maga previously which was (solely focused on self defense), and was a at the time a serious student of judo and bjj. This was a MMA ruled sparring event where I could of used any of that knowledge, yet I was unable to against just a guy throwing simple untrained swings.
I was unprepared because nobody ever rang my bell in years and years of martial arts training. They assumed my repetitive forms and partnered katas would just let me block/catch the strike and counter with some kind of complicated response.
And it is not true. You talk to anyone who is getting into the same situation and they have the same basic story. They were unprepared. Unprepared after 5, 10, 15 years of arts that didn't spar. Even something as simple as adding slapping to a bjj sparring session can crumple a skilled practitioner if he is not used to getting his regularly. What's the saying, "Punching a bjj black belt in the face turns him into a blue belt".
To me, an agressive, untrained, thug is more dangerous then a guy with years of martial arts training. He is strong, fast, angry, doesn't care about life, and has probably had more real fighting experience then you (as in the martial arts student he is attacking).
This is why I advocate real sparring with grappling, striking, clinching, etc for anyone who seriously wants to use anything they are learning against an attacker. I don't do it enough, and it is obvious to me every time I do.