Google Gemini: Everything you need to know about the generative AI apps and models

Google’s trying to make waves with Geminiits flagship suite of generative AI modelsappsand services. But what’s Gemini? How can you use it? And how does it stack up to other generative AI tools such as OpenAI’s ChatGPTMeta’s Llamaand Microsoft’s Copilot?

To make it easier to keep up with the latest Gemini developmentswe’ve put together this handy guidewhich we’ll keep updated as new Gemini modelsfeaturesand news about Google’s plans for Gemini are released.

What is Gemini?

Gemini is Google’s long-promisednext-gen generative AI model family. Developed by Google’s AI research labs DeepMind and Google Researchit comes in several flavors:

• Gemini Ultraa very large model.
• Gemini Proa large model — though smaller than Ultra. The latest versionGemini 2.0 Prois Google’s current flagship.
• Gemini Flasha speedier“distilled” version of Pro.
• Gemini Flash-Litea slightly smaller and faster version of Gemini Flash.
• Gemini Flash Thinkinga model with “reasoning” capabilities.
• Gemini Nanotwo small models: Nano-1 and the slightly more capable Nano-2which is meant to run offline.

All Gemini models were trained to be natively multimodal — that isable to work with and analyze more than just text. Google says they were pre-trained and fine-tuned on a variety of publicproprietaryand licensed audioimagesand videos; a set of codebases; and text in different languages.

This sets Gemini apart from models such as Google’s own LaMDAwhich was trained exclusively on text data. LaMDA can’t understand or generate anything beyond text (e.g.essaysemailsand so on)but that isn’t necessarily the case with Gemini models. For examplethe latest versions of Gemini Flash and Gemini Pro can natively output images and audio in addition to text.

We’ll note here that the ethics and legality of training models on public datain some cases without the data owners’ knowledge or consentare murky. Google has an AI indemnification policy to shield certain Google Cloud customers from lawsuits should they face thembut this policy contains carve-outs. Proceed with caution — particularly if you’re intending on using Gemini commercially.

What’s the difference between the Gemini apps and Gemini models?

Gemini is separate and distinct from the Gemini apps on the web and mobile (formerly Bard).

The Gemini apps are clients that connect to various Gemini models and layer a chatbot-like interface on top. Think of them as front ends for Google’s generative AIanalogous to ChatGPT and Anthropic’s Claude family of apps.

Gemini on the web lives here. On Androidthe Gemini app replaces the existing Google Assistant app. And on iOSthe Google and Google Search apps serve as that platform’s Gemini clients.

On Androidusers can bring up a Gemini overlay to ask questions about what’s on their screen (for examplea YouTube video). Pressing and holding a supported smartphone’s power button or saying“Hey Google” summons the overlay.

Gemini apps can accept images as well as voice commands and text — including files like PDFseither uploaded or imported from Google Drive — and generate images. As you’d expectconversations with Gemini apps on mobile carry over to Gemini on the web and vice versa if you’re signed in to the same Google Account in both places.

Gemini Advanced

The Gemini apps aren’t the only means of recruiting Gemini models’ assistance with tasks. Slowly but surelyGemini-imbued features are making their way into staple Google apps and services like Gmail and Google Docs.

To take advantage of most of theseyou’ll need the Google One AI Premium Plan. Technically a part of Google Onethe AI Premium Plan costs $20 a month and provides access to Gemini in Google Workspace apps like DocsMapsSlidesSheetsDriveand Meet. It also enables what Google calls Gemini Advancedwhich brings the company’s more sophisticated Gemini models to the Gemini apps.

Gemini Advanced users get extras here and theretoolike priority access to new features and models; the ability to run and edit Python code directly in Gemini; and increased limits for NotebookLMGoogle’s tool that turns PDFs into AI-generated podcasts. RecentlyGemini Advanced gained a memory feature that stores users’ preferences and allows Gemini to refer to old conversations as context for current chats.

One of the more compelling Gemini Advanced exclusivesDeep Researchleverages Gemini models with “advanced reasoning” to create detailed briefs. In response to a prompt (e.g. “How should I redesign my kitchen?”)Deep Research develops a multi-step research plan and searches the web to craft a comprehensive answer.

Gemini in GmailDocsChromedev toolsand more

In GmailGemini lives in a side panel that can write emails and summarize message threads. You’ll find the same panel in Docswhere it helps write and refine content and brainstorm new ideas. Gemini in Slides generates slides and custom images. And Gemini in Google Sheets tracks and organizes datacreating tables and formulas.

Gemini is in Google Mapswhere it can aggregate reviews about local businesses and offer recommendations like how to spend a day visiting a foreign city. The chatbot’s reach extends to Driveas wellwhere it can summarize files and folders and give quick facts about a project.

Gemini recently came to Google’s Chrome browser in the form of an AI writing tool. You can use it to write something completely new or rewrite existing text; Google says it’ll consider the web page you’re on to make recommendations.

Elsewhereyou’ll find hints of Gemini in Google’s database products, cloud security tools, and app development platforms (including Firebase and Project IDX)as well as in apps like Google Photos (where Gemini handles natural language search queries)YouTube (where it helps brainstorm video ideas)and Meet (where it translates captions).

Code Assist (formerly Duet AI for Developers)Google’s suite of AI-powered assistance tools for code completion and generationis offloading heavy computational lifting to Gemini. So are Google’s security products underpinned by Geminilike Gemini in Threat Intelligencewhich can analyze large portions of potentially malicious code and let users perform natural language searches for ongoing threats or indicators of compromise.

Gemini extensions and Gems

Gemini Advanced users can create Gemscustom chatbots on desktop and mobile powered by Gemini models. Gems can be generated from natural language descriptions — for instance“You’re my running coach. Give me a daily running plan” — and shared with other users or kept private.

The Gemini apps can tap into Google services via what Google calls “Gemini extensions.” Gemini integrates with DriveGmailYouTubeand more to respond to queries such as “Could you summarize my last three emails?”

Gemini Live in-depth voice chats

An experience called Gemini Live allows users to have “in-depth” voice chats with Gemini. It’s available in the Gemini apps on mobile and the Pixel Buds Pro 2where it can be accessed even when your phone’s locked.

With Gemini Live enabledyou can interrupt Gemini while the chatbot’s speaking to ask a clarifying questionand it’ll adapt to your speech patterns in real-time. Live is also designed to serve as a virtual coach of sortshelping you rehearse for eventsbrainstorm ideasand so on. For instanceLive can suggest which skills to highlight in an upcoming job interview and give public speaking pointers.

You can read our review of Gemini Live here.

Gemini for teens

Google offers a teen-focused Gemini experience for students.

The teen-focused Gemini has “additional policies and safeguards,” including a tailored onboarding process and an AI literacy guide. Otherwiseit’s nearly identical to the standard Gemini experiencedown to the “double-check” feature that looks across the web to see if Gemini’s responses are accurate.

What can the Gemini models do?

Because Gemini models are multimodalthey can perform a range of multimodal tasksfrom transcribing speech to captioning images and videos in real-time. Many of these capabilities have reached the product stageand Google is promising much more in the not-too-distant future.

Of courseGoogle offers no fix for some of the underlying problems with generative AI technology todaylike its encoded biases and tendency to make things up (i.e., hallucinate). Neither do its rivalsbut it’s something to keep in mind when considering using or paying for Gemini.

Gemini Pro’s capabilities

Google says that its latest Pro modelGemini 2.0 Prois its best yet for coding and complex prompts. 2.0 Pro outperforms its predecessorGemini 1.5 Proin benchmarks measuring programmingreasoningmathand factual accuracy.

In Google’s Vertex AI platformdevelopers can customize Gemini Pro to specific contexts and use cases via a fine-tuning or “grounding” process. For examplePro (along with other Gemini models) can be instructed to use data from third-party providers like Moody’sThomson ReutersZoomInfoand MSCIor source information from corporate datasets or Google Search instead of its wider knowledge bank. Gemini Pro can also be connected to externalthird-party APIs to perform particular actionslike automating a back-office workflow.

Google’s AI Studio platform offers templates for creating structured chat prompts with Pro. Developers can control the model’s creative range and provide examples to give tone and instructions — and also tune Pro’s safety settings.

Gemini Flash is lightweightwhile Gemini Flash Thinking adds reasoning

Gemini 2.0 Flashwhich can use tools like Google Search and interact with external APIsoutperforms some of the larger Gemini 1.5 models on benchmarks measuring coding and image analysis. An offshoot of Gemini ProFlash is small and efficient — built for narrowhigh-frequency generative AI workloads.

Google says that Flash is particularly well-suited for tasks like summarization and chat appsplus image and video captioning and data extraction from long documents and tables. MeanwhileGemini 2.0 Flash-Litea more compact version of Flashoutperforms Gemini 1.5 Flash but runs at the same price and speedaccording to Google.

Last DecemberGoogle released a “thinking” version of Gemini 2.0 Flash that’s capable of “reasoning.” The AI model takes a few seconds to work backward through a problem before it gives an answerwhich can improve its reliability.

Gemini Nano can run on your phone

Gemini Nano is a tiny version of Gemini efficient enough to run directly on (some) devices instead of sending the task off to a server somewhere. So farNano powers a couple of features on the Pixel 8 ProPixel 8Pixel 9 ProPixel 9and Samsung Galaxy S24including Summarize in Recorder and Smart Reply in Gboard.

The Recorder appwhich lets users push a button to record and transcribe audioincludes a Gemini-powered summary of recorded conversationsinterviewspresentationsand other audio snippets. Users get summaries even if they don’t have a signal or Wi-Fi connection — and in a nod to privacyno data leaves their phone in process.

Nano is also in GboardGoogle’s keyboard replacement. Thereit powers Smart Replywhich helps to suggest the next thing you’ll want to say when having a conversation in a messaging app such as WhatsApp.

A future version of Android will tap Nano to alert users to potential scams during calls. The new weather app on Pixel phones uses Gemini Nano to generate tailored weather reports. And TalkBackGoogle’s accessibility serviceemploys Nano to create aural descriptions of objects for low-vision and blind users.

Gemini UltraMIA for now

We haven’t seen much of Gemini Ultra in recent months. The model isn’t available in the Gemini appsand it isn’t listed on Google’s Gemini API pricing page. Howeverthat doesn’t mean Google won’t bring Ultra back at some point in the future.

How much do the Gemini models cost?

Gemini 1.5 Pro1.5 Flash2.0 Flashand 2.0 Flash-Lite are available through Google’s Gemini API for building apps and services. They’re pay-as-you-go. Here’s the base pricing — not including add-ons — as of February 225:

• Gemini 1.5 Pro: $1.25 per 1 million input tokens (for prompts up to 128K tokens) or $2.50 per 1 million input tokens (for prompts longer than 128K tokens); $5 per 1 million output tokens (for prompts up to 128K tokens) or $10 per 1 million output tokens (for prompts longer than 128K tokens)
• Gemini 1.5 Flash: 7.5 cents per 1 million input tokens (for prompts up to 128K tokens)15 cents per 1 million input tokens (for prompts longer than 128K tokens)30 cents per 1 million output tokens (for prompts up to 128K tokens)60 cents per 1 million output tokens (for prompts longer than 128K tokens)
• Gemini 2.0 Flash: 10 cents per 1 million input tokens40 cents per 1 million output tokens. For audio70 cents per 1 million input tokens.
• Gemini 2.0 Flash-Lite: 7.5 cents per 1 million input tokens30 cents per 1 million output tokens.

Tokens are subdivided bits of raw datalike the syllables “fan,” “tas,” and “tic” in the word “fantastic”; 1 million tokens is equivalent to about 750,000 words. Input refers to tokens fed into the modelwhile output refers to tokens that the model generates.

2.0 Pro pricing has yet to be announcedand Nano is still in early access.

Is Gemini coming to the iPhone?

It might. 

Apple has said that it’s in talks to put Gemini and other third-party models to use for a number of features in its Apple Intelligence suite. Following a keynote presentation at WWDC 2024Apple SVP Craig Federighi confirmed plans to work with models, including Geminibut he didn’t divulge any additional details.

This post was originally published February 162024and is updated regularly.