Speed Beat Relevance: What Broke When I Put an LLM in Front of Product Search

The engineer wired a large‑scale catalog—hundreds of millions of listings—in twelve languages to an LLM that rewrites natural queries into seller‑used terms and feeds them to the marketplace’s API. The conversion layer appeared to work, yet the price filter never excluded low‑priced junk because the min_sale_price field was supplied in cents while the code passed a dollar value, turning a “$0.15” floor into fifteen cents. Without an assert that an absurd floor yields an empty result set, the defect persisted for months. A second failure emerged when the model supplied numeric category identifiers; the IDs matched the expected format but many did not exist or pointed to unrelated departments. Because the API accepted any numeric ID, the system silently returned off‑topic products. The fix required cross‑checking every model‑generated ID against the authoritative category tree before use.

Beyond data‑level glitches, a regex that stripped sorting keywords also stripped substrings inside legitimate product terms—turning “newborn” into “born” and “cheaper” into “er”. The bug survived because JavaScript’s \b word‑boundary token only respects ASCII characters, breaking on Hebrew and Arabic. Replacing it with a custom pattern that matches whitespace or string edges eliminated the unintended truncation. The most consequential issue was performance: cold searches incurred a six‑to‑eight‑second round‑trip to the external catalog service, causing users to abandon the page before any results appeared. Introducing a 24‑hour cache collapsed latency to roughly 0.2 seconds, delivering a conversion uplift that dwarfed months of ranking refinements. A similar oversight left some category pages invoking the LLM before checking the cache, inflating response times from 2.4 seconds to 0.2 seconds once the order was corrected.

These incidents illustrate that LLM integration amplifies the impact of classic engineering oversights. While the language model solved the cross‑lingual vocabulary gap, it also introduced new failure surfaces—unvalidated identifiers, invisible hallucinations, and brittle text processing—that traditional QA pipelines often miss. The episode underscores that latency, observability, and defensive validation are decisive factors when LLMs sit in the critical path of high‑traffic commerce services. Future deployments must embed automated sanity checks (e.g., extreme‑value assertions), schema validation for any model‑generated keys, and language‑aware regex libraries, while keeping caching strategies front‑and‑center.