The Shape of the System - Engineering for Bounded Cognition

The article reveals that humans can only hold about four separate things in mind at once, with a narrow beam of attention, and for only a few seconds. This limitation is starkly contrasted with the tens of millions of lines of code in modern software systems. The author argues that the surprising thing about software is not that it breaks, but that it works at all, given the cognitive limitations of the minds building and interacting with it.

The article's insights have significant implications for the tech industry, particularly in the development and deployment of large language models like those used in GitHub's Copilot or Google's Bard. These models have a context window, a hard limit on how much they can consider at one time, which behaves similarly to human short-term memory. The study "Lost in the Middle" found that these models perform well when the required information is near the start or end of the input but are noticeably worse when it's buried in the middle. This boundedness is not unique to humans; it's also a characteristic of machines, and it affects their performance in predictable ways.

The article's central question – how to shape complex systems so that small minds can work on them without bringing them down – has practical implications for software engineering and design. By acknowledging the limitations of human cognition and the systems we build, engineers can develop strategies to mitigate these limitations, such as precise naming, boundary setting, and testing. These approaches can help ensure the stability and reliability of complex systems, even when built and interacted with by humans and machines with limited cognitive abilities.