How We Built a Customer Support Bot That Handles 80% of Tickets

Our client — a mid-sized DTC e-commerce brand doing $8M ARR — was drowning in support tickets. Their team of 4 agents was spending 70% of their time on questions that had the same 12 answers. Average first response time: 6.5 hours. Cart abandonment from frustrated customers waiting for support answers: measurable in revenue.

Eight weeks later, the bot handles 80% of tickets autonomously, average first response is 47 seconds, and the team of 4 now handles only complex escalations — freeing them for proactive customer success work that's driving retention up 14%.

Here's exactly how we built it.

Architecture Overview

The system has four layers:

1. Intake: Zendesk webhook → n8n → ticket classification
2. Knowledge retrieval: Pinecone vector store with embedded FAQ, policy docs, and product catalog
3. Response generation: GPT-4o with a strict system prompt and retrieved context
4. Quality gate: Confidence scoring + escalation logic before the response is posted

Step 1: Building the Knowledge Base

We exported three sources of ground truth:

• Zendesk macros (their existing canned responses) — 47 of them
• Shopify store policies (returns, shipping, exchanges)
• Product FAQ document (size guides, materials, care instructions)

We chunked each document into 400-token segments with 50-token overlap, embedded them using text-embedding-3-small (fast, cheap, good for retrieval), and loaded them into Pinecone with metadata tags (source, category, product_ids).

Step 2: Classification First

Before calling GPT-4o for a response, we classify the ticket into one of 8 categories using a cheap GPT-4o-mini call:

• Order status inquiry
• Return/refund request
• Shipping issue
• Product question
• Discount/promotion inquiry
• Account/password issue
• Complaint (negative sentiment)
• Other / unclear

This classification drives two things: the retrieval query (more targeted than using the raw ticket text), and the escalation decision (complaints always go to a human, regardless of confidence).

Step 3: RAG Retrieval

Using the classification + key entities from the ticket, we construct a retrieval query and fetch the top-5 most similar chunks from Pinecone. For an order status question, the query might be: "order status tracking WISMO delayed shipping update".

We also pull live order data via a Shopify API call (order ID extracted from the ticket via regex) so the response can include real tracking information, not just generic instructions.

Step 4: Response Generation

The GPT-4o system prompt is the most important part. Key elements:

You are a helpful customer support agent for [Brand].
Respond in a warm but concise tone. Maximum 3 sentences unless the customer needs step-by-step instructions.
NEVER make up information not in the provided context.
If you're uncertain, say "Let me get our team to look into this for you" — do not guess.
Always end with a specific next step or question.
Context: {retrieved_chunks}
Order data: {shopify_order_json}

The "never make up information" instruction and the explicit uncertain-state behavior were critical. Without them, the model hallucinated shipping ETAs and return windows that didn't match actual policy.

Step 5: The Quality Gate

We don't post every response automatically. After generation, a second LLM call scores the response 0–10 on:

• Accuracy (does it match the retrieved context?)
• Helpfulness (does it answer the question?)
• Tone (is it on-brand?)

Score ≥ 8: post as solved. Score 6–7: post as draft for agent review. Score < 6 or complaint category: escalate immediately.

This gate is why we're comfortable at 80% auto-resolution rather than lower — we're not posting responses we're not confident in.

The 3 Mistakes We Made

1. Skipping classification and going straight to generation. Response quality was inconsistent, and retrieval was too broad. Adding the classification step improved auto-resolution rate from 62% → 80%.
2. Not including live order data. The bot gave accurate policy answers but couldn't answer "where is MY order" — the most common question. Adding the Shopify API call was the single highest-impact change.
3. Trusting LLM confidence scores instead of building our own quality gate. GPT-4o's own uncertainty estimates were poorly calibrated. The separate scoring call was more reliable.

Interested in deploying something similar for your support team? Book a free audit and we'll scope out your specific use case.