Aikido being violent (and possibly damaging) is one thing, but that doesn't make it brutal. I wouldn't consider someone being assaulted who smashes his/her attackers face into the pavement or breaks their arm brutal. What they are doing is violent, but they aren't doing it because they like the smell of blood or the sound of bones breaking.
If you can defend yourself without harming your attacker, great, if you aren't confident in your ability to do it, stepping up the violence is completely neccessary. That doesn't make you brutal, it makes you concerned with your welfare and that of those around you.
I'd rather be brutal and alive than harmonious and dead, even if it means going for the throat, groin, eyes, liver, kneecaps or spine