It took a year of strikes and grabs to wonder about it, huh? ;-)
As Chuck pointed out, intent is a major factor here. I think the problem is to some degree semantic: What do we mean by "violent"? Aikido IS a martial art; when de-escalation and simple evasion are not enough, it involves direct physical action. I would probably say that in this context, by "violent" we mean "aggressive." That aikido is primarily used for defense, though at times proactively, not as a tool to bully or oppress.
Does that help?
I agree with most of your post however, I would ask you, is Aikido a martial art? or is Aikido a martial way? Personally, I believe it started out as an art that was martial in nature, but as with most living things, through time and a lifelong refinement by it's discoverer (O'sensei), it evolved into what it is today. I also believe this process of evolution has led to the different "styles" we see today. People took home that which it was at that time.
Violence has it's place in Aikido as well as in all martial arts and ways, for If there were no violence, there would be little need to practice. Although violence is not intended, to someone watching you do ikkyo to someone that doesn't know how to take ukemi will appear very violent indeed. I believe it's really a matter of perception.
Just my two pennies in the pot.