Maybe something like the Youtube’s Flag feature.
I don’t know if I understand completely how it works there, but if you made something like that, where people can flag a comment if they think that it is spam, trolling etc. and maybe write an explanation on why they think it is.
Which then sends the comment to moderation and blocks it.
Of cause one guy who flags it shouldn’t send it there, but lets say, 10 users on the site Flag it then it automatically gets removed and send to moderation.
By only letting users flag comments you would be able to see who did it.
That way, if you see a comment in moderation that isn’t justified getting there, you could then see who flagged it, and then send them a warning by email with the guidelines for flagging, and if they keep flagging comments unjustified, remove the feature from their user/IP, whatever, or just ban them.
I understand that you all ready have to go through a gazillion comments every day, and it must at times be a super tiring part of running a site like this but you could, like others have said. Recruit co-mods to help ease the job of running the site, so you can focus more on writing the articles.
I am sure that, there would be plenty of qualified and trustworthy volunteers out there who would be happy to help running this site.