Google Lens can now search images and text at the same time

At I/O 2021, Google announced it was using a new technology called Multitask Unified Model to enhance the capabilities of its search engine. Then, in September of that same year, it showed off how MUM would make it possible to search images and text simultaneously within Google Lens. At the time, the company promised that multisearch would launch “in the coming months.” And while it’s not using MUM to enhance searches just yet, it has started beta testing multisearch.  

Provided you live in the US, you can try the feature out in the Google app on Android and iOS. To do so, tap the Lens icon and then swipe up after snapping a new photo or importing an existing image from your camera roll. Then tap the “Add to your search” icon.

You can use this field to either ask questions about the image before you or to refine your search. For instance, you could take a photo of your bike’s rear derailleur (the component that moves the chain from one gear to another) and then search for how to fix or adjust it on your own. By combining text and images, Google suggests it’s making it easier to complete searches where doing so with words alone might be tricky. After all, even most casual cyclists don’t know what a derailleur is or what it does.

As mentioned already, you can also use the feature to refine your searches. So say you see a shirt with a pattern you like but want to see if that same pattern is available on socks and other items of clothing. You could type “white floral Victorian socks” into Google, but again that would depend on you having the fashion vocabulary to know what you want from Google, and even if you accurately describe what you see, the search engine might not produce useful results. At the moment, Google suggests the feature works best when posed shopping-related searches such as the one pictured above.

“All this is made possible by our latest advancements in artificial intelligence, which is making it easier to understand the world around you in more natural and intuitive ways,” Google said. “We’re also exploring ways in which this feature might be enhanced by MUM – our latest AI model in Search – to improve results for all the questions you could imagine asking.”