Every engineering team that has shipped a customer-facing AI agent into production eventually hits the same wall: reasoning latency kills conversion . In a demo environment, watching an autonomous agent spend 15 to 20 seconds reading a customer’s history, calling tools, and assembling a personalized recommendation feels impressive. In a real mobile commerce app, making a hungry or busy shopper stare at a loading spinner for 16 seconds causes them to abandon the flow and close the app.
DoorDash’s engineering team ran head-first into that exact bottleneck inside Ask DoorDash , its conversational shopping assistant. When a customer asked the assistant to build a recurring grocery reorder list, the live agent originally analyzed 90 days of past order history and user profile signals on the fly—taking roughly 16 seconds to return a shopping list.
By rethinking what actually needs to happen inside a live LLM loop versus what can be computed ahead of time, DoorDash slashed grocery reorder latency by 87.5%—from 16 seconds down to 2 seconds . The business impact was immediate and dramatic: cutting those 14 seconds of wait time doubled shopping list views (2x) and increased the flow’s completed order rate by approximately 34% .
Even better for software architects and product engineers, DoorDash achieved this speedup not by waiting for a faster frontier model, but by applying three practical architectural patterns—offline batch LLM inference , a three-tier deterministic catalog waterfall , and an authoritative editable state artifact —that any engineering team can replicate in their own production agent stack today.
87.5% Latency Reduction: End-to-end response latency for the AI grocery reorder flow dropped from ~16 seconds to ~2 seconds .Conversion & Engagement Lift: Dropping latency to 2 seconds doubled (2x) shopping list views and boosted the grocery reorder flow’s conversion/order rate by ~34% .Offline Batch LLM Precomputation: Instead of analyzing purchase history on every live prompt, DoorDash runs batch LLM inference offline over 90 days of purchase history and consumer profile signals, followed by deterministic validation , to pre-store reusable recurring shopping needs.Three-Tier Live Catalog Waterfall: At request time, the serving path resolves precomputed items against live store inventory in three deterministic steps: (1) original store item match $\rightarrow$ (2) catalog product ID match $\rightarrow$ (3) search term fallback .10x Faster Agent Onboarding: Built on DoorDash’s shared agent platform (providing centralized orchestration, memory, tracing, and evals), launching new domain agents—such as the company’s Reservations agent—now takes 1 week , roughly 10 times faster than building its initial standalone agents. The Core Lesson: Stop Running Offline Reasoning on the Live Request Path
As DoorDash engineer Twisha Jain summarized the design philosophy behind the overhaul: “Better agentic systems aren’t always about more reasoning; they’re about deciding what belongs in the agent loop and what can happen ahead of time.”
When a customer opens a grocery app on Sunday afternoon to restock milk, eggs, coffee, and produce, their past 90 days of purchase history did not magically change in the last five milliseconds. Running an expensive, high-latency LLM reasoning pass over three months of receipts after the user taps “Reorder my weekly groceries” is pure architectural waste.
Architectural Layer
Legacy Synchronous Agent (16s Latency)
DoorDash Hybrid Batch + Live Agent (2s Latency)
90-Day History Analysis Computed live via LLM on every user request
Precomputed offline via Batch LLM Inference
Hallucination / Schema Check Validated dynamically during live streaming
Verified ahead of time via Deterministic Validation
Live Inventory & Price Sync LLM tool calls querying store catalog mid-turn
Deterministic 3-tier waterfall (Store Item $\rightarrow$ Product ID $\rightarrow$ Search)
Quantity / Item Removal Edits Triggers a brand-new full LLM conversational turn
Mutates the structured list artifact directly (zero LLM call )
By moving the 90-day purchase analysis into an offline batch LLM job paired with deterministic validation, DoorDash precomputes each consumer’s recurring replenishment candidates before they even open the app. However, precomputing a shopping list introduces a classic e-commerce problem: grocery store inventory and prices change constantly. A carton of organic strawberries bought two weeks ago might be out of stock today.
DoorDash solved that without invoking a slow LLM on the live serving path by implementing a three-tier deterministic catalog resolution waterfall :
Tier 1 (Exact Store Item): The serving layer first checks whether the exact original store SKU is currently in stock at the selected merchant.Tier 2 (Catalog Product Identifier): If the specific store SKU changed, it resolves against the universal catalog product ID.Tier 3 (Search Term Fallback): If the product is unavailable, it executes a fast deterministic search-term match against live store inventory to slot in the closest available equivalent.
Because all three resolution steps are deterministic database and search lookups rather than multi-step LLM reasoning loops, the entire personalized, inventory-verified shopping list renders in 2 seconds .
Treating the UI State as an Authoritative Artifact (Not Just Chat History)
The second major architectural breakthrough in DoorDash’s implementation is how it coordinates between graphical UI actions and conversational LLM turns—a pattern DoorDash calls its grounded shopping interface .
In many poorly designed AI wrappers, every user action is forced through the chat box. If a shopper wants to remove bananas or change milk quantity from 1 to 2, the app sends a hidden prompt to the LLM, waits several seconds for the model to regenerate the state, and repaints the screen. Worse yet, if the user taps a UI button to delete an item and then types a follow-up question in chat, the LLM often re-adds the deleted item because the model is still reading its own earlier conversational transcript!
DoorDash eliminated both problems with two strict rules:
Skip the LLM for Deterministic UI Edits: Simple CRUD actions—incrementing quantities, checking off items, or deleting products—update the structured shopping list object directly in milliseconds without making an LLM round trip.Direct Edits Are Authoritative Over Chat History: When the user does invoke the live Grocery Agent for a task that genuinely requires reasoning—such as “Swap my snacks for cheaper high-protein alternatives on sale today” —the agent turn initializes from the currently edited shopping list artifact , treating the user’s direct UI edits as the single source of truth rather than relying on stale chat transcript history.
Practical Blueprint for Production AI Engineers
Whether you are building an e-commerce assistant, a B2B CRM copilot, a financial portfolio analyzer, or an automated customer-support workflow, DoorDash’s 16-second-to-2-second refactoring provides an immediately applicable checklist for cutting latency and lifting conversion.
The 4-Step Enterprise Implementation Playbook
Audit Your Live Agent Traces for Static Historical Reasoning: Inspect your slowest p95 agent endpoints this week. If your live LLM prompt is summarizing 30 to 90 days of user tickets, CRM notes, or purchase logs at request time, move that summarization into a nightly or event-triggered offline batch LLM job (which also cuts token costs by 50% on most provider batch APIs).Replace Live LLM Tool Lookups With a Deterministic Resolution Waterfall: Never use a slow LLM tool-calling loop to check whether an ID exists in a live catalog or database. Precompute the semantic candidates offline, and resolve live availability, pricing, or permissions on the serving path using a fast deterministic fallback chain (Exact ID $\rightarrow$ Canonical Entity ID $\rightarrow$ Keyword Search).Make the Structured UI State Authoritative Over the Chat Transcript: Stop forcing button clicks, quantity toggles, and filter changes through an LLM round trip. Mutate the structured state object directly in the frontend/backend, and inject that serialized state object as the authoritative system context whenever the user triggers a genuine multi-step reasoning turn.Centralize Orchestration and Evals in a Shared Agent Platform: Follow DoorDash’s platform model—where a core platform team maintains tracing, memory, model routing, and rollout gates while domain teams own prompts and tools—so launching a new specialized vertical agent takes one week instead of three months.
Next Steps
