Well, first I wouldn't call it "negative baggage."
Kinda like eye and ear pro, no?
Its how you look at it. As no training fully replicates reality, concessions to safety are simply a key to being able to do repeat repetitions and not so damage your students (or expose them to unacceptable risks) that they are not able to actually perform what you are training them for.
The issue with force on force is the level of protection. A decent helmet simply has not been made that allows for full perception and full protection against both rounds and full on strikes. Even these helmets are less than ideal protection because sim rounds to get through.
The helmets add a certain unreality, but it is a tradeoff - what is more important? Training without a "martian" head and yet severely curtailing the confrontational dynamics, or training sans protection and severely curtailing the realistic dynamics - which many martial arts and defensive tactics systems do.
Realistic dynamics trumps the minor issues that protective equipment cause.
Even body armor is an issue. I prefer guys to train sim without it for the pain penalty - more important than training in your gear. You make up for that by training other drills in full gear to offset not doing so in sims training.
The same drill can be done without sim, (blue or red guns), contact can be limited (in other words, non-force on force training or limited force on force training), or certain techniques can be limited - and made up in other training.
When you view training as an integrated whole, as a system of training versus individual, unrelated aspects, you can put the varied pieces together to make up what individual drills lack, and address individual skill sets by either dialing down to the micro level or back out to the macro level.