AI Reasoning Models Collapse Completely Once Puzzles Get Hard Enough

The study, authored by Apple researchers Parshin Shojaee, Iman Mirzadeh, Keivan Alizadeh, Maxwell Horton, Samy Bengio and Mehrdad Farajtabar, built a suite of controllable puzzles that let them dial compositional depth while keeping the logical rules constant. Running both the newest reasoning‑focused LLMs and their standard non‑thinking counterparts under identical token‑budget conditions, they observed three distinct performance zones. On trivially simple tasks, the plain LLMs actually out‑performed the “thinking” models, delivering correct answers with fewer tokens. As the puzzles grew moderately complex, the chain‑of‑thought models leveraged their self‑reflection mechanisms to gain an edge, albeit at higher token cost. Beyond a certain complexity point—far below the models’ maximum generation length—both families collapsed to near‑zero accuracy, and the reasoning models paradoxically reduced the length of their internal thought traces, indicating a hard scaling ceiling in their inference process.

These results arrive amid a wave of hype around LLMs that claim to reason like humans. Companies such as OpenAI, Anthropic, DeepSeek and Google have been marketing “thinking” variants as a leap toward general AI, often showcasing headline numbers on benchmarks like MATH or Codeforces. However, the Apple paper highlights that those benchmarks are vulnerable to data leakage and do not expose how models actually solve problems. By isolating the reasoning trace, the authors reveal that current LRMs still rely on pattern matching rather than systematic algorithmic execution, and they fail to maintain consistent logical depth when tasks demand deeper compositional planning. This undermines the narrative that chain‑of‑thought prompting alone yields robust, scalable reasoning.

The collapse observed suggests that future releases will need more than longer prompts or reinforcement‑learned self‑reflection; they will require architectures or training regimes that embed explicit procedural knowledge. Practitioners should be wary of deploying LRMs for high‑stakes reasoning tasks without independent verification of intermediate steps. Watch for upcoming research that integrates program synthesis or tool‑use capabilities, and for benchmark suites that stress controllable complexity rather than static test sets. Monitoring token‑efficiency curves and trace‑length behavior will be key to spotting whether new models truly overcome the scaling bottleneck identified here.