GPT-4o: OpenAI’s Native Multimodality

GPT-4o ("o" = omni) was presented by OpenAI on May 13, 2024. What’s new isn’t that GPT-4 can process image and audio (that existed via separate APIs), but that a single native model processes text, image, and audio for both input and output. Result: human-conversation latency (~320 ms), better multimodal understanding, and a price 50% lower than GPT-4 Turbo.

Key takeaways

• The three modalities (text, vision, audio) are fused in the same base model: the model hears tone, emotion, and interruptions, not just transcribed words.

• ~50% cheaper than GPT-4 Turbo: $5/1M input tokens vs $10.

• ~320 ms latency in audio mode: genuinely natural conversation.

• HumanEval 90%: best coding result among models available in May 2024.

• The Realtime API bidirectional WebSocket opens voice-first applications that previously required complex pipelines.

What’s different

All three modalities are now fused in the same base model:

• Text: quality comparable to, or slightly above, GPT-4 Turbo.

• Vision: more integrated image analysis than GPT-4V; advanced OCR, diagram analysis, and screenshot understanding for UI automation.

• Audio: native input (transcription plus emotional understanding) and output (synthesis). The model picks up tone, emotion, and interruptions.

• Video: video frames as inputs for screen-recording analysis or content moderation.

The difference vs the "Whisper to GPT-4 to TTS" pipeline is latency and information preservation: the previous pipeline lost voice tone, emotion, and pauses. GPT-4o processes them directly.

Pricing and speed

For real-time chat apps, the latency gap is noticeable to the end user.

Benchmarks

GPT-4o stands out especially in coding (HumanEval 90%) and multilingual tasks.

Unlocked use cases

GPT-4o makes practical what previously required a complex pipeline. That covers voice assistants with real conversational latency for phone-based customer service and interactive tutors, accessibility interfaces for people with visual or motor limitations, and simultaneous translation in meetings with tone preservation. It also covers complex document understanding (forms, invoices, technical drawings) in a single API call, and UI automation that describes screenshots for testing or support pipelines.

The real-time API

The most impactful post-launch novelty: a bidirectional WebSocket connection with audio streaming. The client can interrupt, the server detects the interruption and adjusts the response. Real 320 ms latency.

# Bidirectional WebSocket connection
# Audio stream in -> understanding + reasoning -> audio stream out
# The model detects interruptions and adjusts in real time

This opens patterns previously impractical: phone support bots, voice interfaces for IoT, interactive tutorials with instant feedback. The integration complexity (WebSocket plus audio streaming) is real, but the result is qualitatively different from any prior pipeline.

Honest limitations

With honesty:

• Audio bandwidth in current models is ~24 kHz: enough for voice, not for high-fidelity audio.

• Tool use plus audio in the same call is more complex than in text mode.

• Vision hallucinations: low-quality OCR or confusing text in images produces a higher error rate than plain text.

• The context window is 128k tokens, the same as GPT-4 Turbo, smaller than Claude 3 Opus or Gemini 1.5.

• Audio cost adds up fast: roughly $0.06/min of input, $0.24/min of output.

GPT-4o vs GPT-4 Turbo

This is what weeks of production use showed:

• Text quality: GPT-4o matches or slightly beats Turbo on most tasks.

• Complex reasoning in edge cases: Turbo is occasionally better.

• Latency: GPT-4o wins clearly.

• Cost: GPT-4o wins clearly.

• Multimodal: GPT-4o wins clearly.

Default for new applications: GPT-4o. Stay on Turbo only for reasoning-heavy cases you have verified empirically favour it.

Conclusion

GPT-4o is the starting point for evaluating what can now be built with multimodal LLMs. Fusing modalities into a single model changes the economics of applications that depend on voice or vision: you no longer need to orchestrate three separate services to get a natural conversation. The Realtime API for voice is the most real differentiator for voice-first applications.

Frontier models move fast: Claude 3.5 Sonnet answered a month later, and months after that GPT-5 merged GPT-4o and o3 into a single adaptive model. Even so, GPT-4o remains a genuine leap in price, quality, and modality for anyone building on the OpenAI API today.

This article is also available in Spanish: GPT-4o: multimodalidad nativa de OpenAI.

Sources:

1. OpenAI: Hello GPT-4o^[1]
2. TechCrunch: OpenAI debuts GPT-4o ‘omni’ model now powering ChatGPT^[2]
3. Wikipedia: GPT-4o^[3]
4. CNN Business: OpenAI unveils newest AI model, GPT-4o^[4]

Frequently asked questions

How much does GPT-4o cost compared with GPT-4 Turbo?

Half as much for text: $5 per million input tokens and $15 output, versus $10 and $30 for GPT-4 Turbo. GPT-4o mini drops to $0.15 and $0.60 per million. Audio is billed separately and adds up fast, roughly $0.06 per minute of input and $0.24 per minute of output. The first token arrives in about 500 ms versus ~1000 ms for Turbo.

Is it worth staying on GPT-4 Turbo instead of GPT-4o?

Only in specific cases. Weeks of production use put GPT-4o ahead: it matches or slightly beats Turbo on text quality and wins clearly on latency, cost and multimodality, so it is the default for new applications. Turbo is occasionally better at complex reasoning in edge cases; keeping it only makes sense where you have verified that empirically. Both share the 128k-token context window.

What are the limitations of GPT-4o's audio mode?

Audio bandwidth is about 24 kHz, enough for voice but not high-fidelity audio. Combining tool use and audio in the same call is more complex than in text mode. Audio cost also adds up fast. The Realtime API runs over a bidirectional WebSocket with audio streaming and real 320 ms latency, including interruption detection, but the integration requires handling WebSocket and audio streaming.

Sources

1. OpenAI: Hello GPT-4o
2. TechCrunch: OpenAI debuts GPT-4o ‘omni’ model now powering ChatGPT
3. Wikipedia: GPT-4o
4. CNN Business: OpenAI unveils newest AI model, GPT-4o