Why AI language models choke on too much text

Timothy B. Lee writing for Ars Technica:

Today’s LLMs are far more capable:

• OpenAI’s GPT-4o can handle 128,000 tokens (about 200 pages of text).
• Anthropic’s Claude 3.5 Sonnet can accept 200,000 tokens (about 300 pages of text).
• Google’s Gemini 1.5 Pro allows 2 million tokens (about 2,000 pages of text).

Still, it’s going to take a lot more progress if we want AI systems with human-level cognitive abilities.

Good explainer on context windows and how they impact the performance and usefulness of chatbots.