I learn a lot from reading Ed Felton. In A Spoonful of Sugar he describes an absolutely brilliant method being used at Carnegie-Mellon to “label all the images on the web”.
… a pair of strangers, shown a photographic image, are each asked to guess the single word that the other will use to characterize the image. Get it right and you score valuable points. For an extra challenge, sometimes there are “taboo words” that you aren’t allowed to use. Players report that the game is semi-addictive.
The brilliant part is that the game “tricks” its players into doing an important and incredibly time-consuming job. By playing the game, you’re helping to build a giant index that associates each image on the internet with a set of words that describe it. It’s well known that indexing and searching a set of images requires the time-consuming manual step of assigning descriptive words to each image. Labeling all of the images on the internet is an enormous amount of work. When you play the ESP Game, you’re shown images randomly chosen from the internet. You’re doing the time-consuming manual work to index the whole internet’s images – and enjoying it! So far the group has collected over two million labels.
Oddly, according to their FAQ, the designers are trying to pre-filter the content: to remove porn. I would have thought that this filtering/labeling would be most desired by people trying to blacklist porn images. Of course, if the system were used that way, people might try to game the system by introducing bad data….