Google needs Gemini to reinvent the mouse. I am skeptical

A signature characteristic of Google’s upcoming Googlebooks guarantees to place a contemporary AI twist on one of many oldest pc interfaces: the mouse pointer.

With the Magic Pointer, a product of Google’s DeepMind lab, you’ll have the ability to wave the pointer at an object or space on the pc display screen and easily inform Gemini what you need it to do–something from modifying the picture you’re pointing at to including elements from a recipe to a buying listing, with the AI-enabled mouse pointer performing as a shortcut for prompting.

The Magic Pointer is one in every of top-line options for Google’s new Googlebooks, the Gemini-powered successor to Chromebooks which might be due within the fall. However whereas we’ll have to attend till later this 12 months to get our fingers on a Googlebook, you’ll be able to check drive the Magic Pointer proper now.

One method to give Magic Pointer a attempt is through Gemini in Chrome, which helps you to use the pointer to simply ask Gemini about any a part of a given internet web page. Now, I couldn’t get Magic Pointer to work in Chrome (possibly it’s as a result of I’m utilizing Chrome on a Mac), however I did handle to get it working in Google AI Studio, which gives a few temporary Magic Pointer demos.

Within the first demo, you need to use Magic Pointer to edit a picture–on this case, a cartoony illustration of a seashore populated by a palm tree, a crab, a browsing penguin, a snowman, and a wood signal.

The demo steps you thru a collection of duties, together with one the place you progress the crab from one a part of the picture to a different. You simply wave the Magic Pointer on the crab, then transfer the pointer the place you need the crab to go, and say “transfer the crab right here.”

The factors on the display screen the place you waved the pointer ought to start to glow yellow as Gemini processes your immediate. After I tried it, Gemini chewed on the “transfer the crab right here” immediate for a number of seconds earlier than lastly shifting the crab the place I instructed. Crude, sure, nevertheless it bought the job carried out.

I then pointed on the snowman’s cap and stated, “make {that a} solar hat,” and growth, a solar hat appeared. I additionally used the Magic Pointer to show the penguin right into a turtle (“make {that a} turtle”) and altered the writing on the signal (“make this say Ben’s seashore,” though Gemini heard it as “Benz seashore”).

Within the second demo, you employ the Magic Pointer to search out locations on Google Maps. For this one, I pointed the mouse at a picture of London’s Hyde Park and requested “the place is that this?” Inside just a few moments, Gemini had pinpointed Hyde Park in Google Maps.

A a lot more durable activity concerned utilizing the Magic Pointer to ask Gemini for instructions from one level to a different. Within the demo, you’re imagined to circle a photograph of 1 location, then circle one other picture, after which say “how do I am going from right here to there?”

Ben Patterson/Foundry

However on the primary a number of makes an attempt, Gemini stored insisting that I used to be circling the identical photograph, despite the fact that I used to be fairly certain I wasn’t. By the point I did get the demo to work, it felt like typing the total immediate into Gemini would have been simpler.

After all, the early Magic Pointer I attempted was extraordinarily restricted, and Google is promising way more highly effective performance as soon as its new Googlebooks launch. For instance, you’ll have the ability to level at sections of a Google Doc and ask Gemini to rewrite them, or level on the picture of a restaurant and ask Gemini to ebook a desk.

Attention-grabbing, however will the Magic Pointer actually have the ability to reinvent the mouse? Colour me skeptical for now.