What LLMs can't do (yet)

LLMs predict tokens; they don't know things. Here's where they break: hallucinations, long documents, and novel reasoning.

The hype around large language models is real. So are the limitations. After building LLM systems across several industries, I've learned that setting the right expectations is half the battle.

Hallucinations

LLMs don't know things. They predict probable next tokens. That means they can produce plausible-sounding nonsense with full confidence. If accuracy matters in your application, you need a verification layer.

What helps

• Check outputs against trusted sources
• Use retrieval-augmented generation (RAG) so the model works from real documents
• Design the system to admit uncertainty instead of guessing

Context window constraints

Context windows keep growing, but LLMs still struggle with very long documents. They can't read a 200-page report the way a human can.

No true reasoning

Current LLMs are pattern matchers, not reasoners. They can imitate reasoning on familiar problem types. Novel logical problems trip them up.

Build accordingly

None of this makes LLMs useless. They work well when you apply them where they fit: design your system around their strengths and guard against their weaknesses. Know the tools. Respect the limits.