The Incredible AI That Can Watch Videos and Tell You What It’s Seeing

We’ve glimpsed the future of online search, and here it is: a 17-second video of a puppy brought to you by Clarifai, a tiny startup that specializes in artificial intelligence.

The video (above) shows the puppy looking very cute as it nuzzles with its female owner, but the interesting stuff is happening in the squiggly lines below. Using a database of 10,000 visual categories Clarifai has built over the past six months, the company’s software tracks the images that appear in the video, automatically describing it with words like “dog,” “female,” “eyes,” and even “cute.”

The idea is that you can then search for these words, and the software will tell you when the corresponding images appear.

It’s part of a trend in artificial intelligence, called deep learning, that’s sweeping through technology giants, giving us software that approaches human levels of perception. Google uses it to boost Android’s voice recognition. Microsoft uses it in a Star Trek-like instant language translator. Facebook is using it to improve its automatic tagging of everyone in your photos. And soon, deep learning will change how we search through videos, making it possible for machines to analyze clips and quickly understand what’s within them.

This is a big deal because current video search engines base their results on text that surrounds the video—its title, or comments, or the meta data we use to tag it. These search engines can find videos of car chases, but they can’t queue up the part where the chase starts.

Clarifai can, says company CEO Matthew Zeiler. We wrote about Zeiler last year when his AI algorithms won the prestigious ImageNet image recognition competition. Clarifai already is selling high-quality image search software that could be used on smartphones, by catalog companies, or anyone who needs a large number of images.

But the video search software could bring deep learning to video archives, or even companies like GoPro. Imagine searching through your ski vacation footage for all of the times you catch air. Or through your security camera footage for the moment someone grabbed your garden gnome. We’d be surprised if you couldn’t, within the next few years, do this type of search on YouTube videos, too.

By then, Clarifai’s software may be able to automatically process and summarize videos and even sort them by the type of activity that they contain. “This is just the tip of the iceberg,” says Zeiler.