There is an old project at U of Washington CS that does something similar:

http://grail.cs.washington.edu/projects/query/

That page describes the technique used to find similar images from the sketch. Very cool stuff.

• Posted by: Greg Linden
• January 2, 2006 9:51 AM

Numinous Technologies was founded in 1996 to develop graphics and interactive video software including the image query link above.

Numinous was acquired by Microsoft March 1999.

• Posted by: km4
• January 2, 2006 11:12 AM

It is not just at U Washington that this stuff has been done for years now. Here is another paper from some folks at MIT, in 1995, that talks about content-based image querying using things like shape, texture, color, relationship to other content-images and so on. They also suggest a Riya-like application in which you circle someone's face in an image, and then find other images or video with that same face in them.

http://citeseer.ist.psu.edu/pentland95photobook.html

And here's another one, from IBM Almaden in 1994:

http://citeseer.ist.psu.edu/faloutsos94efficient.html

So again, it has been over ten years ago now, that these ideas have been around.

And yet Google spends its time working on "Google Talk", rather than on content-based image search. What ever happened to search being the main focus? Shouldn't they have beaten this "retrievr" service by years and years?

• Posted by: JG
• January 2, 2006 11:30 AM

herez another one on QBE (Query By Example) by folks at CMU.

http://amp.ece.cmu.edu/Publication/Howard/icme2002_howard.pdf

I think this kind of stuff has been around for quite a few years but never really took off because of various pitfalls.

• Posted by: Bageshree Shevade
• January 2, 2006 1:18 PM

This is very cool. I drew a tree-like figure and it found a few. I've always wanted a music search that allows me to hum a part of a melody and then it will find the song. Anyone ever seen one of those?

• Posted by: Mark Johnson
• January 2, 2006 2:33 PM

Thanks for appreciating the concept (initial idea and development by Chris), we've added some background info. This is also the place to watch when we release our blog post similarity demo in a few weeks, which should be more useful.

• Posted by: Bruno
• January 2, 2006 2:50 PM

I'm not sure what this means, but I drew a circle and it gave me about 20 pictures of women. Hmm :)

I think the idea is good, would also be cool if somebody did something similar for music/sound.

• Posted by: Slashdoc Beta
• January 2, 2006 3:41 PM

Greg is (as usual) spot on - "Fast Multiresolution Image Querying" is the paper that guided the implementation (which is, by the way, one of my all-time favourites, and recommended reading for anyone with only a passing interest in image retrieval).

I first came across it when someone (I think Edd Dumbill) linked to imgSeek a couple of years back; imgSeek is a standalone image management application that incorporates the same algorithm. retrievr is a new implementation in pure Python (plus a host of great libraries: PIL, aggdraw and numarray).

In my experience, the results are usually fairly good, sometimes even stunning - considering the artistic sophistication most of us are able to come up with (gallery forthcoming); and in the cases they're not so stellar, they are at least entertaining ;-) But clearly, the approach has its limits.

One thing to keep in mind is that it doesn't do object/face/text recognition of any kind, so if you're drawing an outline sketch of a chair (or corporate logos like Tara Calishain has tried), it almost certainly won't get you one back (except your index only contains images of chairs). It helps to think of it as matching the most pronounced slabs of colors. Another thing to know is that there's currently no way to specify the aspect ratio, so you have to rescale the image in your head (things that are close to the borders of the image you're thinking of should be close to the borders of your sketches), but that's really just a missing feature of the drawing flashlet than an inherent problem. Sometimes it also helps to _remove_ detail instead of adding it. And finally, the index covers only about 85k of Flickr's "most interesting" images at the moment (I didn't want to use up even more of their resources before checking back with them; it's fantastic enough that Flickr isn't imposing any up-front limits on API usage like most everyone else is doing).

In a way, I see retrievr less as a "search" tool than an "exploration" tool, and it seems to work very well for that.

• Posted by: Chris Langreiter
• January 2, 2006 3:44 PM

"In a way, I see retrievr less as a "search" tool than an "exploration" tool, and it seems to work very well for that." of course, if you use it for this, it works very well. a interesting article Chris.

• Posted by: odp
• January 3, 2006 1:19 AM

My attempts at recreating Sowa's flying pig sadly came to naught. Perhaps this system needs some contemporary (well, some even more contemporary) cyber-Vettriano to give it its own identity!

BB

• Posted by: billkruse
• January 3, 2006 2:17 AM

This is very cool. I drew a tree-like figure and it found a few. I've always wanted a music search that allows me to hum a part of a melody and then it will find the song. Anyone ever seen one of those?

No, sorry :(

• Posted by: gerben
• January 3, 2006 7:15 AM

There have been three comments above asking about QBH (query by humming), for music retrieval applications. The first paper that I know of that talks about this is a 1993 paper:

Tetsuya Kageyama, Kazuhiro Mochizuki, and Yosuke Takashima: Melody Retrieval with Humming, Proceedings of the 1993 International Computer Music Conference, pp.349-351, September 1993.

Unfortunately, it is completely in japanese, so not a lot of people (myself included) have read it. After that, probably the most well-known work is a 1997 paper from the New Zealand Digital Library project. Here is the paper:

http://www.dlib.org/dlib/may97/meldex/05witten.html

And here is a website with the implementation (java required). Note (pun not intended) that input can be either by piano keyboard or hummed audio.

Since that time there has been an explosion of such systems. This list is large, but I can think offhand of a few systems it misses:

http://mirsystems.info/index.php?id=402

But other than that.. why limit your imagination to query by humming? There are so many different ways of doing content-based music retrieval. How about query by tapping for rhythm retrieval? How about query by genre for finding new music in the same style as music you already like? How about query by instrument timbre, so that you can find other songs that use that driving guitar you really like? Or query by vocal timbre, so you can find that Paul Weller song in which Paul McCartney sings backup, largely uncredited in most metadata sources available?

There is so much cool stuff being done right now in this area. The community has largely moved beyond query by humming onto more ambitious research and projects. Check it out: http://www.music-ir.org/

• Posted by: JG
• January 3, 2006 9:09 AM

Unfortunately, it is completely in japanese, so not a lot of people (myself included) have read it. After that, probably the most well-known work is a 1997 paper from the New Zealand Digital Library project. Here is the paper:

http://www.dlib.org/dlib/may97/meldex/05witten.html

old input, but very interesting. i must smile, if you compare may 1997 and today ;)

Andre

• Posted by: Andre
• January 3, 2006 10:42 AM

Didn't the new Alexa API have a demo that did song searches based on whistling?

Ah ha - is this it?
http://www.musipedia.org/whistle.0.html

• Posted by: rod edwards
• January 3, 2006 12:30 PM

Didn't the new Alexa API have a demo that did song searches based on whistling?

Yes it is: http://www.musipedia.org/whistle.0.html

• Posted by: Gerome
• January 3, 2006 1:05 PM