The Future Is Here
We may earn a commission from links on this page

Google Definitely Had Its ‘Hands-On’ That Gemini AI Demo

The video exaggerated what Google’s AI is capable of doing today.

We may earn a commission from links on this page.
Image for article titled Google Definitely Had Its ‘Hands-On’ That Gemini AI Demo
Screenshot: Google

Google released Gemini, its newest generative AI chatbot that promises to be better than ChatGPT, alongside a splashy demo video showcasing its capabilities. A Google developer blog post reveals that the viral showcase was edited, sped up, and used different prompts than the ones shown in the video.

“For the purposes of this demo, latency has been reduced and Gemini outputs have been shortened for brevity,” said Google in the Gemini demo video description, but that doesn’t describe the full amount of editing here.

Advertisement

The release of Gemini was a big marketing moment for Google’s artificial intelligence team, which has faced pressure from investors to catch up with OpenAI. Google’s main AI chatbot Bard has lagged behind ChatGPT and other models in 2023, so Gemini needed to help Google return to the cutting edge of the AI conversation – and it did. However, Gemini’s demo video seems to have been an over-exaggeration of its actual capabilities. Google pushed back the original launch of Gemini because Sundar Pichai deemed it wasn’t ready, and now we’re seeing why.

Advertisement

The demo video appeared as if Gemini works with speech, video visuals, and hand gestures, but that’s not how Google got those answers. Behind the scenes, Gemini required more careful, longer prompts with text and images to get those answers in the video, according to a Google blog post. “Hands-on with Gemini” appeared as if the AI was reacting as a real-time assistant with video and voice capabilities, but that was misleading, and users will not have those features anytime soon. The video was likely filmed with Gemini Ultra, the most advanced version of Google’s AI that won’t be released until at least next year.

Advertisement

Google’s claims that Gemini is better than ChatGPT seem misleading as well, as Bloomberg points out. Google touted a model of Gemini that doesn’t exist yet (Ultra, coming 2024) against outdated models from OpenAI (ChatGPT-4, from March 2023). The benchmark doesn’t compare to ChatGPT-4 Turbo, OpenAI’s most advanced AI model that was released last month. Google’s model that’s available today, Gemini Pro, doesn’t beat ChatGPT-4 in Massive Multitask Language Understanding – it loses by over 7%.

Benchmarks comparing Gemini to top generative AI models
Benchmarks comparing Gemini to top generative AI models
Screenshot: Google
Advertisement

As users got their first chance to use Gemini Pro on Wednesday, many felt the experience lacked the luster of Google’s marketing campaign. Gemini Pro seems to ‘hallucinate’ a lot, a common problem where chatbots confidently give false answers. One user posted on X that Gemini Pro couldn’t answer who won best actor at the 2023 Oscars. Another user found that Gemini Pro told them to “Google it” when asked questions about Israel and Gaza.