Self-hosted support agent

Context

The platform's prior live-chat support relied on a per-seat, low-code commercial product priced on conversation volume. As traffic scaled, expenditure outpaced delivered value, and the product remained brittle on the dimensions that matter most for support: it did not read tone, did not draw on the platform's knowledge base with any fidelity, and did not hand off cleanly when out of its depth. A rebuild was warranted.

Design

• An edge worker sits between the chat widget and a model gateway, holding conversation state in a key-value store at the edge for low latency and predictable cost.
• A pluggable escalation pipeline composed of three layers: pre-inference keyword rules; model confidence checks; and post-inference sentiment evaluation. Any of these can trigger a clean handoff to a human operator.
• A curated knowledge base of platform articles, onboarding videos, and troubleshooting guides held in-repo as markdown and served to the model as cached grounding context. The knowledge base is the source of truth; the model has no other.
• Model routing on conversational class. Inexpensive, fast models answer high-frequency canonical questions; a stronger model is invoked only when the conversation becomes nuanced. The routing decision is itself a model call, but a cheap one.

Outcome

Roughly an order-of-magnitude reduction in monthly infrastructure cost relative to the prior vendor, with the same widget integration on the customer side and a cleaner escalation path on the operator side. Median first-response latency is sub-second: an immediate acknowledgement is dispatched while the substantive reply is composed asynchronously, which both improves perceived responsiveness and decouples the user experience from upstream model latency variance.

Stack

Cloudflare Workers · KV · TypeScript · multi-provider LLM gateway · structured JSON tool calls.