From Cnet News:

Google vet’s new Candid app for anonymous sharing puts trolls in their place.

Bindu Reddy wanted a social app where she could say what she really thought, hear unfiltered responses and tamp down the volume on trolls.

Joining a cohort of anonymous-sharing apps like Whisper and Yik Yak, Candid leans on artificial intelligence to moderate posts. To combat the typical occurrence of a small number of users contributing the majority of negative posts, Candid developed a “natural language processing” technology to analyze all posts. The tech is meant to remove objectionable content, flag negative posts, and redirect off-topic comments to appropriate groups. The app also has a “sentiment analysis” model that the company says identifies hate speech, slander and threats.

It’s great to see that sites are starting to think about this kind of thing, but I suspect it’s going to take a while to come up with solutions that work fairly well. Particularly if the smart programmers building these systems keep trying to do stuff that’s clever instead of easy and convenient.

Really, this is an epistemological problem: how do you know that something is abuse or off-topic? You have all of the standard AI techniques, most of which are used for spam-filtering: keyword analysis, thread analysis, response frequency, white-listing. There are lots of other tools that aren’t being used – for example, a poster might declare a thread to be about a certain topic (see the “Tags” field in FtB postings, which could then index into clusters of words, and postings could be scored based on matching clusters. Of course that, by itself, wouldn’t work – ideally you’d have a whole set of rules that the owner of a thread or forum could switch on/off to create a policy.

• Do not accept postings from accounts less than (threshhold) days old
• Do not accept postings from accounts that have had more than (threshhold) strikes for off-topic posts
  Note: this would mean that there’d be some kind of underlying catalog of types of offenses, and if someone’s account took a strike (e.g.: “racism”) it would be accessible in the rules system, site-wide
• Do not accept postings from accounts that participate in threads owned by (List of accounts)
• Do not accept postings from accounts that are denied by my friends in (List of accounts)
• If I delete a posting from an account, ban them and delete all their past postings (Yes/No)
• Adopt my posting rules from (account)
• Adopt my ban list from (List of accounts)

It’s tempting to put in rules like “do not accept postings from anyone who is a friend of Sam Harris” except then you’ve got a potential denial of service attack if trolls can get someone you want to hear from to be-friend Sam Harris.

I think the idea of a website that is devoted to blocking trolls is a nice idea, but they all should be. It ought to have been part of the basic design of the site. Also, being able to manage how and when posts get deleted should be built into the site. Basically, I think there’s less need for artificial intelligence and more usefulness in user-tunable rules systems and user-accessible moderation and management systems. It’s a lot easier to build rift-deepening control systems than an AI.*

The best part about having user-driven filtering rules-bases is that the site can return to ignoring policy while focusing on discriminating types of posting in the knowledge-base engine. “Oh, so-and-so doesn’t accept postings from you because you’re a friend of Pat Condell? Yeah? Let me put you on hold while I play the smallest violin “sympathy symphony”  in accordance with the terms of service…”)

I think that a big part of the problem is that moderating comments in a lot of sites looks like the horrible bletcherousness that WordPress implemented. It needs to be fast and have one-click access to deeper, more complete logic.

(* I coded a system such as this, back in 2007. The idea was simple: keep a collection of combinations of events and allow the user to assign point-scores to each event; if a combination of events went over a ceiling by accumulating too many points, flag it with a summary of the points it collected and how it collected them. It’s super easy, computation-wise and it doesn’t require fancy Bayesian statistics or anything more than addition. This straightforward approach and resembles some early game AIs including chess-playing AIs – you simply change the system’s behavior from flagging points to hypothesizing about how to gain them)

PS – YikYak. Does anyone but me remember this disaster?