Google released Gemini, its newest generative AI chatbot that promises to be better than ChatGPT, alongside a splashy demo video showcasing its capabilities. A Google developer blog post reveals that the viral showcase was edited, sped up, and used different prompts than the ones shown in the video.
“For the purposes of this demo, latency has been reduced and Gemini outputs have been shortened for brevity,” said Google in the Gemini demo video description, but that doesn’t describe the full amount of editing here.
The release of Gemini was a big marketing moment for Google’s artificial intelligence team, which has faced pressure from investors to catch up with OpenAI. Google’s main AI chatbot Bard has lagged behind ChatGPT and other models in 2023, so Gemini needed to help Google return to the cutting edge of the AI conversation – and it did. However, Gemini’s demo video seems to have been an over-exaggeration of its actual capabilities. Google pushed back the original launch of Gemini because Sundar Pichai deemed it wasn’t ready, and now we’re seeing why.
The demo video appeared as if Gemini works with speech, video visuals, and hand gestures, but that’s not how Google got those answers. Behind the scenes, Gemini required more careful, longer prompts with text and images to get those answers in the video, according to a Google blog post. “Hands-on with Gemini” appeared as if the AI was reacting as a real-time assistant with video and voice capabilities, but that was misleading, and users will not have those features anytime soon. The video was likely filmed with Gemini Ultra, the most advanced version of Google’s AI that won’t be released until at least next year.
Google’s claims that Gemini is better than ChatGPT seem misleading as well, as Bloomberg points out. Google touted a model of Gemini that doesn’t exist yet (Ultra, coming 2024) against outdated models from OpenAI (ChatGPT-4, from March 2023). The benchmark doesn’t compare to ChatGPT-4 Turbo, OpenAI’s most advanced AI model that was released last month. Google’s model that’s available today, Gemini Pro, doesn’t beat ChatGPT-4 in Massive Multitask Language Understanding – it loses by over 7%.
As users got their first chance to use Gemini Pro on Wednesday, many felt the experience lacked the luster of Google’s marketing campaign. Gemini Pro seems to ‘hallucinate’ a lot, a common problem where chatbots confidently give false answers. One user posted on X that Gemini Pro couldn’t answer who won best actor at the 2023 Oscars. Another user found that Gemini Pro told them to “Google it” when asked questions about Israel and Gaza.