In March 2023, OpenAI launched GPT-4[1] with a presentation that promised "human-level performance on many benchmarks". Months later, with thousands of real integrations, I’m in a better position to judge which capabilities held up and which were oversold. And to see where gaps remain versus alternatives like Claude 2 and LLaMA 2. The Spanish version of this analysis is available at /gpt-4-capacidades-reales/.

Key Takeaways

  • GPT-4 consistently excels at chained reasoning, precise technical writing, medium-complexity code, and detailed instruction-following.

  • It systematically fails at arithmetic, post-cutoff information, cross-conversation consistency, and long contexts.

  • Claude 2 wins on long context (100k tokens) and conservative tone; LLaMA 2 wins on privacy, cost, and customisation.

  • The only benchmark that matters is yours: evaluate with 20–30 real prompts.

  • The model generates plausible text even when wrong: unsupervised flows with real impact are dangerous.

Where GPT-4 really excels

With the benchmark expanded to dozens of real cases, GPT-4 is consistently better at:

  • Complex chained reasoning. On problems requiring maintaining multiple variables, conditionals, and intermediate steps, GPT-4 hallucinates less and holds coherence better than any other model available at the time.

  • Precise technical writing. Generating documentation, paper summaries, or step-by-step explanations of complex concepts (especially in domains like programming, quantitative finance, or medicine) produces text requiring few editorial corrections.

  • Medium-complexity code. Not infallible, but on tasks like "refactor this function", "write tests for this component", or "explain what this legacy code does", it clearly beats GitHub Copilot when the IDE suggestion isn’t enough.

  • Following detailed instructions. A prompt with 15 specific constraints ("respond in JSON with these keys, don’t mention X, limit to 100 words") is followed much more faithfully than with GPT-3.5.

Where it remains frustrating

But there are areas where GPT-4, despite the marketing, still systematically fails:

Arithmetic

Surprisingly, GPT-4 gets trivial arithmetic wrong. "What is 2394 × 71?" gives wrong results ~30% of the time. This improves dramatically with Code Interpreter, which runs actual Python for calculations.

Post-training-cutoff information

The model was trained on data up to a specific cutoff date. Questions about recent events, current library versions, or fresh news result in outdated or fabricated information. With the browsing plugin this improves, but latency and reliability drop.

Consistency across conversations

The same question in two different conversations can get answers that differ in substance, not just in wording. For use cases requiring determinism (audits, reproducible validations), this forces more rigid prompting techniques or dropping temperature to 0, which sometimes degrades quality.

Long contexts

With 8k–32k tokens context (depending on version), GPT-4 stumbles when there’s a lot of text. The phenomenon Liu et al. (2023) call "lost in the middle"[2] shows models ignore information placed in the middle of a long context. Claude 2 with 100k tokens does slightly better, but the problem exists in both.

GPT-4 vs Claude 2

Claude 2[3], released by Anthropic in July 2023, brings some notable advantages:

  • 100k-token context window. Loads entire books, technical manuals, or long transcripts in a single prompt.

  • More conservative tone. Less prone to exaggeration or invention; when it doesn’t know, it typically admits so.

  • Stricter built-in safety. For applications where minimising problematic responses matters, Claude 2 fails in fewer edge cases.

Where GPT-4 wins: complex code, symbolic mathematics, multi-step reasoning on problems with more than 5 interacting entities.

GPT-4 vs LLaMA 2 70B

Compared with LLaMA 2 70B, the open-model family Meta released[4] in July 2023:

  • GPT-4 clearly wins on complex reasoning and code.

  • LLaMA 2 70B is competitive on summarisation, classification, simple Q&A.

  • LLaMA 2 has the absolute edge on privacy, cost at scale, and customisation.

For any task where LLaMA 2 70B gives "acceptable" results, it’s almost always the better choice: the quality gain rarely justifies the cost and privacy tradeoff of GPT-4.

Domain Evaluation

The only benchmark that matters is yours. A practical process for evaluating GPT-4 vs alternatives:

  • Select 20–30 representative prompts from your real application, with human-annotated "ideal" answers.

  • Run each prompt through GPT-4, Claude 2, LLaMA 2 and record responses.

  • Evaluate blindly (without knowing which model produced what): which came closest to the ideal?

  • Quantify total cost: per-token price × expected volume + operational overhead.

This process is where the surprises show up: sometimes Claude 2 wins where you expected GPT-4, or LLaMA 2 gives sufficient results at a tenth of the cost. The same process applies when evaluating Bard with PaLM 2 as an alternative from the Google ecosystem.

Responsible use

A dimension worth remembering: GPT-4 generates plausible text even when wrong. For applications with real impact (medical, legal, financial decisions), model output must go through human validation or independent verification systems. The model has no way to know when it’s confidently wrong, and that’s dangerous in unsupervised flows.

Conclusion

GPT-4 is the most capable general-purpose model available when this analysis is written, but "most capable" doesn’t mean "best choice for everything". Mature teams evaluate by use case, not by model reputation. There are scenarios where Claude 2 or LLaMA 2 offer better value/cost ratios; in others, GPT-4 remains the unsurpassed standard. Team sophistication is measured in knowing which is which.

Sources:

  1. OpenAI: GPT-4 research[1]
  2. Anthropic: Introducing Claude 2[3]
  3. Meta AI: Llama 2 official page[4]
  4. Liu et al. (2023), "Lost in the Middle: How Language Models Use Long Contexts", arXiv:2307.03172[2]

Frequently asked questions

Why does GPT-4 get simple arithmetic wrong?

Because the model generates plausible text rather than executing operations: a multiplication such as 2394 × 71 comes out wrong roughly 30% of the time. The practical fix is Code Interpreter, which runs actual Python to perform the calculation instead of predicting it, and improves numerical results dramatically.

When should I choose Claude 2 or LLaMA 2 instead of GPT-4?

Claude 2 wins when you need a 100k-token context window (entire books, technical manuals, long transcripts) or a more conservative tone that admits what it does not know. LLaMA 2 70B is the better choice whenever it gives acceptable results on summarisation, classification or simple Q&A, because it brings privacy, lower cost at scale and customisation. GPT-4 remains ahead on complex code, symbolic mathematics and multi-step reasoning.

How do I evaluate which model works best for my own use case?

Select 20–30 representative prompts from your real application with human-annotated ideal answers, run them through GPT-4, Claude 2 and LLaMA 2, and evaluate blindly which output comes closest to the ideal. Then quantify total cost: per-token price times expected volume plus operational overhead. This process often reveals that LLaMA 2 is sufficient at a tenth of the cost.

Sources

  1. launched GPT-4
  2. "lost in the middle"
  3. Claude 2
  4. Meta released