Personally, I think that violence isn't a physical thing, its spiritual. In other words, whether or not a thing becomes violent depends on our attitude, our intent, not necessarily on what we actually do. (Caveat: up to a certain point, that is.)
For istance: The all-time-greatest-sport-on-Earth is, of course, Hockey. Hockey is said to be a very violent game. To my mind, it is not - it's rough, certainly; crashing into a guy with a closing speed of 30 mph, sending him and his equipment all over the ice is a punishing experience, but that doesn't make it violent. Why? Because its not done maliciously - you check someone for tactical reasons; driving him off the puck or the puck handler, disrupting his game. It's hard, it hurts, (it's wonderfully cathartic...
)but its a part of the game; not only do players accept it, they'd be insulted if you didn't check them for fear of getting 'too rough'.
It can become violent of course, it wouldn't be Hockey unless there were fights, but the game itself isn't.
Same thing for martial arts, I think. We use MA for self defence - to protect ourselves and others from a violent attack. We're not promoting or using violence, we're stopping it; when we use it properly. Things change when our attitude changes - if we decide to use our skills beyond what is required to defend ourselves or others, i.e. if we opened shihonage out to break the attackers arm rather than just dropping him to the ground - THEN it becomes a violent act. That's my opinion on the matter.
Hope I explained my own position clearly; it's difficult isn't it?