Google AI Overviews in Search Highlights Its Big AI Problem (Thinks Out Loud Episode 423)

May 30, 2024
Tim Peter
Artificial Intelligence (AI), Marketing, Podcast, Strategy

MidJourney generated image of a slice of pizza with glue on it to illustrate Google's big AI problem

Subscribe: Apple Podcasts | Spotify | Amazon Music | Youtube Music | RSS

In the immortal words of Omar Little, “You come at the king, you best not miss.” And we all know that Google is the king, right? But history shows what happens when kings move too slowly or make too many mistakes. Google’s new AI Overview feature in search might be one of the bigger missteps the king has taken. It’s also not their first (I’m looking at you, Gemini Launch earlier this year). These new AI products highlight a big problem for Google. And they might be an even bigger problem for you.

Google is the biggest website in the world. It’s also, in all likelihood, one of the the biggest drivers of traffic and revenues for your business. If the king stumbles, what happens to the folks who rely on the king for their success? Nothing good, that’s what.

Why does AI Overview represent a big AI problem for Google? How might that problem affect your business? And what can you do about it? That’s what this episode of Thinks Out Loud is all about.

Want to learn more? Here are the show notes for you.

Google AI Overviews in Search Highlights Its Big AI Problem — Headlines and Show Notes

Show Notes and Links

You might also enjoy this webinar I recently participated in with Miles Partnership that looked at "The Power of Generative AI and ChatGPT: What It Means for Tourism & Hospitality" here:

Free Downloads

We have some free downloads for you to help you navigate the current situation, which you can find right here:

A Modern Content Marketing Checklist. Want to ensure that each piece of content works for your business? Download our latest checklist to help put your content marketing to work for you.
Digital & E-commerce Maturity Matrix. As a bonus, here’s a PDF that can help you assess your company’s digital maturity. You can use this to better understand where your company excels and where its opportunities lie. And, of course, we’re here to help if you need it. The Digital & E-commerce Maturity Matrix rates your company’s effectiveness — Ad Hoc, Aware, Striving, Driving — in 6 key areas in digital today, including:
- Customer Focus
- Strategy
- Technology
- Operations
- Culture
- Data

Best of Thinks Out Loud

You can find our “Best of Thinks Out Loud” playlist on Spotify right here:

Subscribe to Thinks Out Loud

Contact information for the podcast: podcast@timpeter.com

Past Insights from Tim Peter Thinks

Technical Details for Thinks Out Loud

Recorded using a Shure SM7B Vocal Dynamic Microphone and a Focusrite Scarlett 4i4 (3rd Gen) USB Audio Interface into Logic Pro X for the Mac.

Running time: 21m 21s

You can subscribe to Thinks Out Loud in iTunes, the Google Play Store, via our dedicated podcast RSS feed (or sign up for our free newsletter). You can also download/listen to the podcast here on Thinks using the player at the top of this page.

Transcript: Google AI Overviews in Search Highlights Its Big AI Problem

Well hello again everybody and welcome back to Thinks Out Loud, your source for all the digital expertise your business needs. My name is Tim Peter. This is episode 423 of The Big Show. And thank you so much for tuning in. I think we’ve got a really cool show for you today.

In an episode a couple of weeks ago, I talked about the new models coming from Google and AI, OpenAI, and I said I would dive into them a little deeper in a subsequent episode. And I’ve tried out the new models. So we’re talking about ChatGPT featuring GPT 4o; that’s "4-lowercase-o." And I’ve tried out Google Gemini 1.5 Pro. They’re both excellent. They’re really, really good. If you’re not using them, and I suspect many of you are using them, but for those who aren’t, they’re definitely worth checking out. I’m going to come back to that in a bit, though.

Because during the time I’ve been playing with them, another situation has come up around AI, and especially around Google with AI, that highlights a big, big problem for Google. And that big, big problem might become a big, big problem for you, too. And that’s why I think it’s worth talking about today.

As part of its product announcements a couple of weeks ago, Google released its AI Overview in search. Now this is essentially what was Search Generative Experience. It’s just now incorporated into search in a more consistent way across a wide array of searches. It’s not terrible. It’s also… not great.

There have been some hilarious and a few truly atrocious errors. To get one of the ickier ones out of the way, it presented the obviously false summary in response to a question stating that Barack Obama was The first Muslim president in US history. Now, we all know that that is not true. We all know that that was something that was used against him a lot in his political career. But it’s factually inaccurate.

That’s a big problem for Google and potentially a big problem for you. The consensus for the funniest error Google’s AI overview made was it said that one way to keep cheese from falling off of homemade pizza is to, and I quote, "…add some glue. Mix about one eighth cup of Elmer’s glue in with the sauce."

Happily, it also says, "non-toxic glue will work."

Y’think?

For the record, don’t put glue on your pizza, please, I’m begging you. My personal favorite screw up, though, said that 13 U. S. presidents attended the University of Wisconsin Madison, which is obviously not true, receiving a total of 59 degrees among them. That one just keeps getting funnier to me, by the way. Andrew Johnson was particularly industrious, according to Google’s AI overview. Which claimed he earned 14 degrees as part of the classes of 1947, 1965, 1985, 1996, 1998, 2000, 2011, and 2012. That’s super impressive if it were true, especially given that Johnson, by 1947, to say nothing of 2012, Had been dead for over 70 years.

I mean, this isn’t a little false, this is super false. For the record, I fact checked Andrew Johnson’s date of birth and date of death on the White House website. I didn’t fully trust Google to give me the right answer.

And that’s a big deal for Google. Google has a problem that no one else in this space does. They’re not allowed to get the answer wrong.

Google is the number one site in the US and all around the world. They’re the number one site in the US and the number one site globally, according to SimilarWeb. It also says that Google got 250 billion visits over the last three months, and tends to average around 83 billion visits every month.

By contrast, over the last three months, OpenAI, for all the hype that it’s getting, got 5.3 billion visits. Now that’s a big number, but they’re averaging around 1.7 to 7 billion visits every month. They’re the 23rd most visited site on the internet globally, and only number 53 in the US. What’s truly amazing is when you compare them with Bing, Bing got just over 4.1 billion visits from February through April. They’re averaging about 1. 37 billion visits every month. They’re the 26th most visited site globally, and good for them, they made it all the way to number 19 in the US. But think about that. Bing launched almost exactly 15 years ago. Ironically, 15 years ago this week relative to when I’m recording this episode.

OpenAI has existed for a grand total of 8 years, and ChatGPT has only been around for 17 months. Their growth is unlike just about anything we’ve ever seen. Even with that, though, their traffic is roughly 2 percent of Google’s. Two! Two percent! That’s not very much.

Recent research from the Reuters Institute for the Study of Journalism and YouGov, this was a partnership, only 53% of Americans have heard of ChatGPT. Not used. Heard.

Another way to say that is that 47% of Americans haven’t heard of ChatGPT. I’m going to go out on a limb here. What percentage of people do you think have heard of Google? Now, only 24% have heard of Google Gemini, their AI, or a bit less than half as many as know about ChatGPT.

And let’s be fair, this data was captured between late March and early April 2024. So it’s not like we’re talking about earlier in ChatGPT’s life and people didn’t know about it.

As much as we all talk about ChatGPT and AI all the time, most people don’t. Seriously. Only about 18 percent of US respondents use ChatGPT at least weekly. Which was one of the largest usage patterns in the study, and that’s kind of Google’s problem.

They’re super, super big and that creates a whole separate issue for them. I mentioned they’re the number one site globally, they’re the number one site in the US. Even Facebook, which is the third largest site in the world, gets only about 60% of Google’s traffic monthly. Right? That’s a big difference and that creates a problem for Google and a benefit for folks like ChatGPT.

We frequently laugh and shake our heads at ChatGPT’s hallucinations. But think about those numbers we just talked about. Most people have never seen them. Ever. And now Google, with its 83 billion monthly visits is going to attach an AI overview to all kinds of its roughly eight and a half billion daily searches.

As we saw with Barack Obama and Andrew Johnson and Pizza Toppings answers, sometime it’s going to get those answers spectacularly, hilariously, frighteningly wrong. A lot! Right? Think about this. I use this analogy all the time. I use this thought exercise all the time. But let’s say that Google does an amazing job.

Let’s say that their AI overview gets it right 99.999% of the time. That’s the so called five nines many technologists refer to in terms of what getting it right looks like. That means that they’d provide a great AI overview 8,499,915,000 times every day. That’s fantastic, right? They’d also give a bad AI overview 85,000 times every day.

That’s actually a lot. If they get it right only 99% of the time, just to give this context, they’d be giving a bad result — every day — 85 million times a day.

Now, obviously I’m only talking about the frequency of errors, I’m not talking about the magnitude. Not all errors are created equally, and sometimes those errors might be fairly minor. But sometimes they might be huge if we’re talking about people’s health, if we’re talking about people’s finances.

This could be an enormous problem for them. I talked about why this is a problem a few episodes back when Google launched Gemini and it was creating imagery that was troubling. It too was a launch that went spectacularly, hilariously, frighteningly wrong. You’re noticing a pattern here, right?

Google’s scale, the very strength that allows them to create amazing artificial intelligence capabilities, is also an incredibly dangerous weakness for the company. What happens when they get it wrong? What happens to their user experience? What happens to their brand?

I believe bad answers. are a huge problem for Google, the biggest problem they face by far. Consider this, that Reuters/YouGov study that I talked about before shows that ChatGPT use is most common among younger users and declines among older cohorts. 43% of folks age 25-34 have used ChatGPT, as have fully 56% of those aged 18-24.

Over a quarter of respondents aged 18-24 use it at least weekly, which is 50% more than the general population in that study. And their top use? "Answering factual questions."

Funny, isn’t that, you know, Google’s job?

By the way, 60% of that rough cohort also have used TikTok as a search engine. This is according to data from Adobe. Younger users, millennials and Gen Z, are learning that Google doesn’t always give the best answer. And I’m concerned in the longer term that AI Overview might reinforce that belief.

Obviously, it’s not like Google can’t incorporate AI into its results. Remember, AI Overview only launched two weeks ago on May 14th.The Reuters/YouGov study occurred before that. And the same with the TikTok data that I referenced. Younger users are already looking for alternatives.

They, Google, need to be able to respond to what’s happening in the marketplace. The challenge for them is they appear to face trouble no matter which path they take.

If they keep AI out of search, they potentially get bypassed by folks like ChatGPT or TikTok or something else that might come along.
If they add AI to search, they potentially see their brand take a huge beating.

I don’t want to make it sound like they can’t figure this out. I do want to point out that they face a challenge that nobody else does. In their defense, Gemini is a brilliant product. I’m using it regularly, and I’m very impressed with it. Google is awesome at building these kinds of tools. I can’t say that it’s better at ChatGPT than ChatGPT for every use case. My experience suggests that ChatGPT is a little more creative in its responses, which is cool when you’re doing like brainstorming activities, or, you know, outlining writing, or discussing writing. However, Gemini, I think, is much better at summarizing documents and a handful of other use cases that I’ve tested it on.

Google’s technology leadership, their compute capabilities, their access to data, and their pile of money they’re sitting on is second to none. There’s a lot of ways they can figure this out.

They’ve also done a great job of clearing up some of the most egregious errors in AI overview. Quickly, the ones that I just talked about, you can’t reproduce now. Hopefully, they can keep up that pace of cleaning up after themselves. I’m seeing some reports, though, that some users are seeing AI overviews less frequently, maybe while Google’s trying to iron out some of the more egregious errors.

Also, Google is operating from position of strength. Remember the part where they’re 47 times bigger than OpenAI? And significantly larger than TikTok and Facebook, too? So, you know, Google might be in an okay position to weather this storm.

As marketers and business strategists, though, I keep coming back to the same questions. Starting with, how much of your traffic comes from Google. If you think about organic search, paid search, local search, meta search, things like that, you’re probably getting 40-70% of your traffic and a similar amount of business from Google.

How ready are you if customer behaviors suddenly shift?
What are you doing to diversify where your business comes from?
What are you doing to expand your sources of traffic and revenue?
Are you doing more to have customers search for you by name?
Are you working to improve the quality of your content and your customers experience?
Are you testing new social channels to reach a new set of customers or a wider array of customers or existing customers who might be looking elsewhere?

Remember, Google doesn’t have to fall from number one for any downturn to impact your business. They could lose a billion searches a day and a billion users per month and still be much larger than their closest competitors. The question is, can you? What would happen to your business if Google lost 10% or 15% or 20% of its traffic? They’d still be bigger than everybody else. But what would happen to your business?

Your job is to ensure that Google’s artificial intelligence problem doesn’t become your company’s actual business problem. We can laugh when Google gets it wrong, but if they get it wrong too often, there’s nothing funny about it for your business. That’s the challenge today. That’s where we are today.

Google has a big problem with AI. Your job is to make sure it doesn’t become a big problem for you.

Show Wrap-Up and Credits

Now, looking at the clock on the wall, we are out of time for this week.

And I want to remind you again that you can find the show notes for this episode. As well as an archive of all past episodes by going to timpeter.com/podcast. Again, that’s timpeter.com/podcast. Just look for episode 423.

Subscribe to Thinks Out Loud

Don’t forget that you can click on the subscribe link in any of the episodes that you find there to have Thinks Out Loud delivered to your favorite podcatcher every single week. You can also find Thinks Out Loud on Apple Podcasts, Spotify, YouTube Music, anywhere fine podcasts are found.

Leave a Rating or Review for Thinks Out Loud

I would also very much appreciate it if you could provide a positive rating or review for the show whenever you use one of those services.

If you like what you hear on Thinks Out Loud, if you enjoy what we talk about, if you like being part of the community that we’re building here, please give us a positive rating or review.

Reviews help other listeners find the podcast. Reviews help other listeners understand what Thinks Out Loud is all about. They help to build our community and they mean the world to me personally. So thank you so much for doing that. I very, very much appreciate it.

Thinks Out Loud on Social Media

You can also find Thinks Out Loud on LinkedIn by going to linkedin.com/tim-peter-and-associates-llc. You can find me on Twitter or X or whatever you want to call it this week by using the Twitter handle @tcpeter. And of course, you can email me by sending an email to podcast(at)timpeter.com. Again, that’s podcast(at)timpeter.com.

Show Outro

Finally, and I know I say this a lot, but I want you to know just how thrilled I am that you keep listening to what we do here. It means so very, very much to me. You are the reason we do this show.

You’re the reason that Thinks Out Loud happens every week. You’re the reason we produce a new episode. So please, please keep listening, keep your messages coming on LinkedIn, keep hitting me up on Twitter, sending things via email. I love getting a chance to talk with you, to hear what’s going on in your world, and to learn how we can do a better job building the types of information and insights and content and community that work for you and work for your business.

And with all that said, I hope you have a fantastic rest of your day. I hope you have a wonderful week, and I will look forward to speaking with you here on Thinks Out Loud next time. Until then, please be well, be safe, and as always, take care, everybody.

Tim Peter

Tim Peter is the founder and president of Tim Peter & Associates. You can learn more about our company's strategy and digital marketing consulting services here or about Tim here.

Google AI Overviews in Search Highlights Its Big AI Problem (Thinks Out Loud Episode 423)