Evolving how LLMs are measured for Android: the next era of Android Bench
In the July update, Google’s Android Bench leaderboard switched its evaluation engine from the custom mini‑swe‑agent v1 to the open‑source Harbor framework, re‑running every test to generate a fresh baseline. The change brings standardized tooling, easier reproducibility, and a new “cost and efficiency” dimension that lets developers compare not just raw capability but also inference expense. The refreshed run introduced eight fresh LLMs: Anthropic’s Claude Fable 5, Claude Sonnet 5, Claude Opus 4.8, the Chinese GLM 5.2, Kimi K2.7 Code, MiniMax M3, and two Qwen 3.7 variants. Claude Fable 5 now leads with an 84.5 score, outpacing GPT‑5.5’s 80.2, while among open‑weight models GLM 5.2 tops at 72.2. The leaderboard also displays efficiency metrics for tasks such as Jetpack Compose migrations and wearable networking, giving developers a clearer picture of trade‑offs between speed, cost, and accuracy.
The overhaul reflects a broader shift toward community‑driven, transparent AI evaluation in the mobile space. As LLMs become integral to “agentic” development—auto‑generating code, refactoring UI, or suggesting API updates—platform vendors need reliable, comparable data. By adopting Harbor, Android Bench aligns with industry moves toward open benchmark ecosystems (e.g., MLPerf, HELM) that encourage third‑party contributions and cross‑model scrutiny. Adding a diverse set of models from Western (Anthropic, OpenAI) and Asian (GLM, Qwen) providers also mirrors the increasingly global AI market, where developers must choose among a crowded field of proprietary and open‑weight options.
Looking ahead, the success of Android Bench will hinge on how actively the developer community supplies real‑world tasks and publishes independent runs. If contributions grow, the leaderboard could become a de‑facto standard for measuring LLM productivity on Android, influencing IDE integrations and pricing models. However, the added efficiency dimension may expose cost disparities that could push smaller teams toward open‑weight models, reshaping vendor strategies. Watch for updates to Harbor’s dataset, potential weighting changes for cost versus accuracy, and how major AI providers respond to their rankings.
Key Takeaways
Android Bench now uses the Harbor framework, delivering standardized, reproducible LLM evaluations with added cost‑efficiency metrics.
Claude Fable 5 tops the updated leaderboard with an 84.5 score, while GLM 5.2 leads the open‑weight segment at 72.2.
Google invites developers to submit custom Android tasks, turning the benchmark into a collaborative, community‑curated platform.
The shift may accelerate adoption of open‑weight models among cost‑sensitive developers and pressure proprietary providers to improve efficiency.
About the Source
This analysis is based on reporting by Android Developers. Here is a short excerpt for context:
Posted by Zoe Lopez-Latorre, Senior Developer Relations Engineer, Android Back in March, we introduced Android Bench—our LLM leaderboard for real-world Android development tasks. Our goal was to provide transparency around model capabilities in Android development and to encourage model improvements, to give you more helpful AI options for your everyday workflow. Since then, we have enhanced the benchmark based on your feedback, including evaluating open-weight models and adding cost and efficiency dimensions to the leaderboard. But AI capabilities are ever-evolving, and measurement needs to follow suit. As part of our July release, we have adopted the Harbor framework, which includes an updated version of the benchmarking agent used to evaluate models. Claude Fable 5, Claude Sonnet 5, Claude Opus 4.8, GLM 5.2, Kimi K2.7 Code, MiniMax M3, Qwen 3.7 Plus and Qwen 3.7 Max) to the leaderboard. We’re also sharing opportunities for you, the Android developer community, to contribute to the benchmark. Upgrading our methodology with the Harbor framework When we designed Android Bench, we anchored our methodology on leading industry standards available at the time. We used mini-swe-agent v1, a general-purpose benchmarking agent, and adapted it to the nuances of Android development to provide a baseline measurement for the capabilities of models for common Android development tasks. To continue providing you with state-of-the-art evaluations that accurately measure the latest model capabilities on Android development, we are standardizing our benchmark to the Harbor framework. Harbor defines standards and integrations that make it easy for anyone to run the benchmark, evaluate their preferred set-up, or share results – providing you with additional transparency and visibility. This upgrade enables us to more rigorously evaluate models and their capabilities, and we re-ran the benchmark on all models to establish an updated baseline. This means there is a minor shift in scoring, but you will still be able to view historical scores within the archive on our website. We want to ensure Android Bench is helpful for you, so we will continuously update it as our evaluations and the industry mature. Expanding the leaderboard with 8 new models As part of our commitment to keeping the leaderboard fresh, we have added Claude Fable 5, Claude Sonnet 5, Claude Opus 4.8, GLM 5.2, Kimi K2.7 Code, MiniMax M3, Qwen 3.7 Plus and Qwen 3.7 Max to the Android Bench leaderboard. You will see that Claude Fable 5 is at the top of the leaderboard with a score of 84.5, followed by GPT 5.5 with 80.2, with Claude Sonnet 5 in 3rd with a score of 76.2. When just comparing Open-weight models, GLM 5.2 is at the top with 72.2, followed by Kimi K2.7 Code with a score of 70.4. You can check out model performance and efficiency metrics on the updated leaderboard to see how these new and previous models navigate Android-specific challenges like Jetpack Compose migrations, wearable networking, and platform API updates. Opening Android Bench to community contributions From the beginning, we’ve valued an open and transparent approach, which is why we made our original methodology and test harness publicly available on GitHub. You’ve asked for a way to provide feedback on our dataset, so now we’re taking collaboration a step further by giving you, the Android developer community, a chance to shape Android Bench. Starting today, you can contribute to Android Bench in two ways: Design and submit your own Android development tasks to evaluate how models handle the scenarios that matter to you. Run and share benchmark evaluations firsthand, testing your preferred models against our dataset or your own custom tasks. We will be reviewing the submitted tasks and will be assessing if they get added to the benchmark. We hope to build a benchmark that truly reflects the diverse, day-to-day realities of the global Android developer community. Looking ahead With more and more options for agentic development, maintaining a cutting-edge benchmark ensures that the AI assistance you rely on keeps getting smarter, more helpful, and more effective. Head over to our GitHub repository to check out the tasks. We invite you to submit a task to our team for review, and you can check out Harbor Hub to explore the dataset or submit evaluations. As always, you can find the updated leaderboard, or read the methodology on our website.Read the original at Android Developers
