Artificial intelligence system with a sense of humor

The idea for the project arose at a different convention about a year ago. Dafna Shahaf, a researcher at Microsoft, attended a speech by Mankoff about the cartoon archive, and she left feeling excited. Shahaf wondered whether she could teach a computer to accurately assess how funny a caption might prove to be—and in the process, crack one of the most difficult challenges in machine learning. Sarcasm, wordplay, and other tools of humor have perplexed AI systems for decades. At Microsoft, teaching machines and software to get the joke is important for things like the Skype Translator, which is designed to let users speak to each other in different languages and hear translations on the fly.

For the study, Shahaf fed cartoons and captions from the New Yorker’s database into the system and trained it to find the funniest choices among captions that make similar jokes. She relied partly on crowdsourced input from contract workers, using Amazon.com’s Mechanical Turk. Then she moved to the harder task of ranking jokes. Because typical computer vision software is designed for photos, not drawings, the researchers had to manually describe what was pictured in each cartoon. They organized this into two categories: the context and the anomalies.

While Mankoff says a computer will probably never be able to beat his writers at being funny, Microsoft has successfully devised an AI system that could make cartoonists’ jobs easier. Mankoff is impressed by the software’s ability to winnow out lame submissions and narrow the list to the funnier ones. For example, when evaluating a cartoon depicting a car salesman hawking a hybrid animal-vehicle with hairy legs instead of wheels, the AI picked, “Just listen to that baby purr” as one of the top captions; “It runs on a 100 percent fuel efficient Paleo diet” was rated one of the worst. The winning caption in the New Yorker was, “Relax! It just smells the other car on you.”

The machine and the New Yorker editors don’t always align on shortlists. On average, though, all of the editors’ favorites appeared in the AI’s top 55.8 percent of choices, according to the study. That means the New Yorker could use the system to eliminate at least 2,200 submissions a week without missing the gems. “On average, we saved about 50 percent of his workload,” says Shahaf. It could also save Mankoff the time it takes to hire new assistants. “I do think the future is human-machine companionship,” Mankoff says. “Computers can be a great aid.”

That’s an unexpected admission coming from Mankoff, who has frequently mined the concept of humans submitting to robot overlords in his cartoons. (In 2013, Mankoff published an article titled “How to Deal with the Coming Robot Apocalypse.”) He professes to be optimistic about AI’s ability to aid humorists, although he says Microsoft’s system needs to improve its accuracy somewhat before he’d employ it. Mankoff doubts whether machines can ever completely replicate his sense of humor. Riffing on Hamlet (“There are more things in heaven and earth, Horatio, than are dreamt of in your philosophy”), Mankoff opines, “There are more things in humor and human beings than are dreamt of in even Microsoft’s algorithms.”

That hasn’t lessened Microsoft’s ambitions. The researchers say they hope to one day train computers to come up their own jokes based on situations, which would make digital assistants such as Cortana and Siri more pleasant to interact with. Eric Horvitz, the managing director of Microsoft’s research group and a co-author on the study, jokes that your computer might try to lighten the mood when you accidentally delete an important file with a carefully timed “oy vey.” More broadly, understanding what we find funny and how we come up with jokes is important to the study of how the brain works, which itself is crucial to artificial intelligence research. A side benefit: With all the consternation over the evils of AI, Horvitz says it’s nice to think of these systems doing something more pleasant than destroying the universe.

Or stealing someone’s job. But that, too, is unlikely to happen when computer analysis remains split on even the basic principles of humor. While Microsoft found that positive messages play better in captions, a second study of the same database of New Yorker cartoons by a University of Michigan researcher came to the opposite conclusion. Mankoff, who authored one of the New Yorker’s most famous cartoons about a brutally direct businessman trying to avoid scheduling a meeting, doesn’t foresee a computer replacing him soon. “How about never,” Mankoff says. “Is never good for you?”

Dina Bass