MOUNTAIN VIEW, Calif. — This year's Google I/O 2024 keynote with its relentless focus on AI proved the power of being some place in person. While I'm sure that people who watched the live stream — and followed the Tom's Guide live blog — got plenty out of the many different announcements Google made about its Gemini AI model. But being on site at the Shoreline Amphitheater afforded you the chance to see Gemini's multitude of skills up close, to really get a sense of what Google's driving toward with these tools.
I should know. Right after the keynote wrapped, I spent Tuesday afternoon running from demo to demo, illustrating a different part of Google's AI story. In many cases, these were repeats of what Google showed off on-stage, only with the benefit of me standing a few feet away instead of seated a dozen rows back from the stage. And in some cases, Google showed off other capabilities that were only hinted at in its keynote.
The end result? I have an appreciation for what the Gemini model can do that I may not have had at the start of the day. And while the final proof of the value of these features will come when they're put to use by folks like you and me, I think Google has some extremely useful features that it's in various stages of rolling out.
Here are seven Google I/O demos that really stood out for me as I learned more about what Gemini can do.
Summarize Emails in Gmail
@tomsguide ♬ original sound - Tom’s Guide
I was tasked with covering Gemini-powered features coming to Google Workspace during the keynote itself, so I was pretty familiar with the AI upgrades Google has planned for Gmail. Still, the specific demo for the email summary feature performed for my benefit hit pretty close to home.
Google's demo featured a very elaborate — and very lengthy — school newsletter that might wind up in any parents' inbox. And rather than skim through each line looking for information relevant to their student, Google showed how a new Summarize Emails feature can let you ask for something specific — say the pieces of information that are only relevant to seventh grade students — and summarize those in neat and tidy bullet points.
I happen to be the parent of a seventh grader who receives fairly regular school newsletters containing a lot of information with only some of it relevant to my interests. So yes, Summarize Emails can not get here soon enough. It's going to be available to workspace lab users this month, with Gemini for Workspace customers and Google One AI Premium subscribers gaining access in June.
I should note that the AI agent scanning through the letter in Google's demo can be a pretty clever customer. In addition to newsletter items that specifically mentioned seventh graders, it also pulled an item about an opportunity open to kids in the fourth through ninth grade — it knows enough to recognize that applies to seventh graders, too.
Gmail Q&A
Another tool coming to Gmail is a Q&A feature where you can task Gemini with pulling out specific data from a collection of emails. In that school newsletter example, the email summary noted that there was aan overnight trip for the seventh grade class. Using the new Gemini tools in the Workspace side panel that's now available for Workspace Labs and Gemini for Workspace Alpha users, we asked Gemini if there was a packing list for that trip. Indeed, there was, and the AI agent pulled the list from where it was buried in the email.
As helpful as it will be to have the Q&A feature search for specific text, Google sees this tool doing more than that. During the Google I/O keynote, Aparna Pappu, the vice president and general manager of Google Workspace, talked about a time later this year when Gemini will be able to recognize receipts and invoices, filing them into a folder on your Google Drive and even entering them into a tracking spreadsheet in Sheets.
Gemini Advanced
A demo showing off the Gemini Advanced model updated with 1.5 Pro may not have been the most compelling demo that I saw at Google I/O, but it did feature a rather helpful visual aide. The photo up above shows how big 1,500-page PDF would be if you printed it out. Gemini Advanced's 1 million token context window means it's able to make sense of a document that large.
That's not just summarizing gigantic documents, either. Gemini Advanced can also process any complex information contained in those pages. To demonstrate that, a Google employee took two large PDFs covering similar regulatory issues — one was 522 pages, the other 144 pages — and asked Gemini Advanced to identify topics covered in one document, but not in the other.
AI coaching
My favorite demo at Google I/O allowed me to kick a soccer ball in anger and use AI analysis to pinpoint where my wayward penalty kicks had gone so wrong. I had three attempts to kick the ball at goal while cameras measured my form to produce scores for power, accuracy and style.
The analysis came courtesy of Gemini 1.5 Pro, but it didn't just take the form of raw numbers. The AI also generated feedback voiced by a virtual football coach, with actual coaches having trained the model on what to evaluate. The verdict? I need to lock my ankle and strike through the ball with my laces to generate power consistently. Watch your back, Messi.
Project Astra
@tomsguide ♬ original sound - Tom’s Guide
My colleague Kate Kozuch has already detailed her experience with Project Astra at Google I/O, so I'll try not to repeat what she had to say about the real-time camera-based AI that draws on Gemini 1.5 to recognize objects and use that knowledge to do everything from play games to help you find things. While some Project Astra's skills certainly prove that the multimodal reasoning Gemini draws on works as advertised, but the AI tool's ability to recognize objects like apples and ice cream cones and identify them using alliterative descriptions like "scrumptious snacks" is not going to have you dreaming of an exciting AI future filled with aliteration.
Much more impressive was a game of Pictionary that we played with Project Astra where the AI had to identify a drawing. This was no canned demo, as I got to do some of the drawing, and Project Astra correctly identified my sketch of a pig, even though my drawing skills could best be described as "remedial." The AI ws less successful at identifying another reporter's sketch of Totoro, guessing that it was an owl or bear, but it was good to put Project Astra to test on our own.
There's one other aspect of Project Astra that caught my attention — it remembers things, whether it's the names of different stuffed animals it was shown in my demo or the part in the Project Astra video seen in the Google I/O keynote where Gemini can tell someone where they left their glasses. It's one thing to be able to recognize different objects and put them in context; it's quite another to retain that information for use later.
Imagen 3
My experience with generative image creation has largely been restricted to free tools with limited capabilities, so I was excited to be in the driver's seat for a turn with Imagen 3, Google's generative AI tool that can use text prompts to create images out of thin air.
Google's demo involved creating letters from the alphabet, which meant 26 different results, all drawing on different models. I got to provide the main text prompt — "pickles stacked on top of one another" was my contribution — with a Google employee providing other text prompts to tweak the look of my pickle-based alphabet. The result of my name spelled out in pickles is one I will long relish.
MusicFX DJ
I went into the MusicFX DJ demo with some trepidation, as I really didn't care for the pre-keynote demo/hype session by an improvisation musician. (I will tolerate a good many things but loud noises at 9:30 a.m. is not one of them.) Also, while I can see the merits of using AI to do repetitive grunt work, I'm a bit ambivalent about its use in creative endeavors.
Still, once you have a DJ board in front of you, and it's you controlling the beats, MusicFX DJ is certainly fun to use. My colleague Ryan Morrison detailed the project earlier this year, but basically you can use text prompts to create sounds — Vinyl Scratch, Groovy Bass and Horn Section Hits were some of the ones at my disposal. Using sliders, I could then make a particular sound more prominent than the others.
This isn't the sort of thing that sounds like it's going to replace actual music creation, but it does seem lik something musicians could use as a tool for inspiration or that teachers could use to teach their students about how to put together a song. And that's a lot more reassuring than having some improv musician shouting "Google!" at you over and over again, before the morning coffee has really taken effect.
