Models don’t have preferences, they have context

The Microsoft blog post, authored by Waldek, a Principal Developer Advocate at Microsoft, specifically criticizes the practice of asking AI models questions in isolation, without context, and then declaring their responses as "preferences." For instance, when asked "What framework should I use for a web app?" in an empty chat, a model might respond with "React," but this doesn't necessarily mean it prefers React. According to Waldek, this response is likely a reflection of the model's training data, where React dominates. He illustrates this point by demonstrating how the same model, when provided with a workspace containing Svelte files, recommends Svelte instead.

The broader context of this discussion is the rapidly evolving field of AI-powered developer tools, where companies like Microsoft, Anthropic, and others are actively developing and evaluating the performance of large language models (LLMs). Waldek's argument is particularly relevant as the tech industry moves towards more sophisticated AI-assisted coding and development. For example, Gao and Kreiss have documented how models shift behavior in "testing mode," producing systematically different outputs when they detect evaluation-like patterns. Additionally, Anthropic has shown that formatting changes alone can swing MMLU accuracy by roughly 5%, underscoring the significance of context in model responses.

The implications of this discussion are significant for developers and companies looking to integrate AI-powered tools into their workflows. Waldek advises testing models with realistic context and prompts, rather than relying on abstract questions in empty chats. This approach provides more actionable information about a model's performance in real-world scenarios. For instance, instead of asking a model "What framework should I use?", developers should set up a workspace with project files, configuration, and dependencies, and then ask prompts that reflect real tasks. By doing so, they can better assess a model's ability to write good code in their specific stack, with their patterns, and in their repository.