How to test agent skills without hitting real APIs
In the blog post, Waldek points out that testing a skill that invokes an external API quickly becomes expensive—running 50 scenarios across three models with five repeats already generates 750 calls per iteration, each billed to the developer. Even when the API is internal, repeated reads and writes (PATCH, DELETE) corrupt live data, forcing manual resets or tolerating noisy results. Traditional mock servers solve the cost problem but require a separate codebase that must be kept in sync with the real service, and swapping URLs for localhost changes the token sequence the LLM sees, contaminating the evaluation itself. Dev Proxy sidesteps both issues by intercepting traffic based on a simple JSON config, serving realistic payloads and resetting data automatically, so the skill continues to address its production endpoint while the proxy supplies deterministic responses.
The move reflects a broader shift in the AI‑agent ecosystem where developers treat prompt engineering and tool integration as software components that need the same CI/CD rigor as traditional code. Companies such as OpenAI and Anthropic are rolling out “sandbox” environments for function calling, yet Microsoft’s approach emphasizes a lightweight, configuration‑first proxy that can be dropped into any CLI‑based agent (GitHub Copilot, Claude Code, Codex). By keeping the URL unchanged, the solution respects the LLM’s context window, a nuance that many testing frameworks overlook, and it aligns with the industry’s push toward reproducible, cost‑effective evaluation pipelines for ever‑changing foundation models.
If teams adopt Dev Proxy or similar interceptors, they can embed systematic evaluation into their release cycles, catching regressions before a model update or API schema change hits users. However, reliance on a static JSON snapshot may mask edge‑case behavior that only appears with live data variability, so developers must complement proxy tests with occasional real‑API sanity checks. Watch for Microsoft’s upcoming tooling integrations in Azure DevOps and for competing proxy solutions that might add features like dynamic response generation or security policy enforcement.
Key Takeaways
Evaluating agent skills with real API URLs but a local proxy eliminates both monetary cost and production data mutation.
Mock servers introduce token‑level differences that can skew LLM reasoning, a problem Dev Proxy avoids.
The 750‑call per‑run figure illustrates how quickly evaluation budgets can balloon without emulation.
Teams should still schedule periodic real‑API runs to validate proxy fidelity against live edge cases.
About the Source
This analysis is based on reporting by Microsoft Blog. Here is a short excerpt for context:
Your agent skill calls an API. The moment you start evaluating it, every run either costs money or mutates production data. Learn how to mock APIs transparently so you can run evals without changing your skill or hitting real endpoints. The post How to test agent skills without hitting real APIs appeared first on Microsoft for Developers.Read the original at Microsoft Blog
