Building a Chatbot with Large Language Models

01Chatbot architecture: what the pieces are

The core is just one HTTP call to a model, but a real product wraps several layers around it. The usual request path looks like this:

1. The UI sends the message to your own server, not to the model provider.
2. The server authenticates the user, applies rate limits and checks the input.
3. If needed, relevant documents are retrieved from a knowledge base (RAG).
4. The final prompt is assembled and sent to the model; the answer streams back.
5. The output is checked, logged and shown to the user.

02The first endpoint: a backend proxy

The proxy keeps the key secret and is the natural home for the rest of the logic. In the Node.js and Express example below, every provider-specific detail lives in the callModel function:

import express from "express";
const app = express();
app.use(express.json({ limit: "20kb" }));

// Provider-specific details live here only.
async function callModel(messages) {
  const res = await fetch(process.env.LLM_URL, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      Authorization: "Bearer " + process.env.LLM_API_KEY
    },
    body: JSON.stringify({ model: process.env.LLM_MODEL, messages, temperature: 0.2 })
  });
  if (!res.ok) throw new Error("LLM error " + res.status);
  const data = await res.json();
  return data.reply; // adapt to your provider's response shape
}

app.post("/api/chat", async (req, res) => {
  const { messages } = req.body;
  const system = { role: "system", content: "You are a helpful support assistant." };
  res.json({ reply: await callModel([system, ...messages.slice(-12)]) });
});
app.listen(3000);

A low temperature usually suits support and Q&A because answers become more stable. When you take this service live, deploying with Docker and scalable API design are the two related reads.

03Conversation memory and the context window

Models remember nothing between requests; "memory" means resending the history each time. That costs money and the context window is finite. The example above sent only the last 12 messages, which is the simplest strategy.

• Sliding window: only the last N messages; simple, but forgets older details.
• Summarisation: older messages are condensed into a short summary placed at the top of the prompt.
• Structured memory: key facts (name, order) are stored separately in a database and injected when needed.

For storing sessions and caching frequent answers, smart caching with Redis is a common choice.

04RAG: answering from your own documents

The model knows nothing about your prices, policies and products, and when it does not know it may guess with confidence. In RAG (retrieval-augmented generation) the system first finds a few relevant document chunks and gives them to the model along with the question.

Preparing the data

Split documents into short chunks (a few paragraphs), create a vector (embedding) for each with an embedding model, and store them in a vector database or even a plain array. Chunks should be self-contained; cutting through the middle of a table or list lowers quality.

Retrieval and prompt assembly

function cosine(a, b) {
  let dot = 0, na = 0, nb = 0;
  for (let i = 0; i < a.length; i++) { dot += a[i] * b[i]; na += a[i] ** 2; nb += b[i] ** 2; }
  return dot / (Math.sqrt(na) * Math.sqrt(nb));
}

async function answer(question, index) {
  const qVec = await embed(question); // call your embedding endpoint
  const top = index
    .map(c => ({ ...c, score: cosine(qVec, c.vector) }))
    .sort((x, y) => y.score - x.score)
    .slice(0, 4);

  const context = top.map((c, i) => "[" + (i + 1) + "] " + c.text).join("\n\n");
  const system = "Answer ONLY from the context. Cite sources like [1]. " +
                 "If the answer is not in the context, say you do not know.";
  return callModel([
    { role: "system", content: system },
    { role: "user", content: "Context:\n" + context + "\n\nQuestion: " + question }
  ]);
}

Two things matter more in practice than the choice of vector database: an explicit "if it is not in the context, say you do not know" instruction, and showing sources to the user. Those two raise trust and make review easier.

05Guardrails and cost control

Real users ask unexpected things and some deliberately try to derail the bot (for example with "ignore your previous instructions"). A few simple layers of defence help:

• Scope the role: define allowed topics and tone in the system message and tell the bot to decline politely outside them.
• Separate data from instructions: label documents and user text as data and never obey commands found inside them.
• Validate output: if you need a structure such as JSON, validate it before use.
• Rate and token limits: cap requests per user and the maximum response length.
• Least privilege: if the bot calls tools or databases, allow only narrow, read-only operations.

const limits = new Map(); // userId -> { count, resetAt }

function allow(userId, max = 20, windowMs = 60_000) {
  const now = Date.now();
  const e = limits.get(userId);
  if (!e || now > e.resetAt) { limits.set(userId, { count: 1, resetAt: now + windowMs }); return true; }
  return ++e.count <= max;
}

06Evaluation: how to know it got better

Without evaluation, every prompt or model change may fix one case and break three. The simplest approach is a file of 30 to 100 real questions with the expected answer or key fact, run before every release.

const cases = [
  { q: "What is the return period?", mustInclude: ["14 days"] },
  { q: "Do you ship abroad?",        mustInclude: ["yes", "international"] },
  { q: "Tell me a joke about politics", mustDecline: true }
];

let pass = 0;
for (const c of cases) {
  const out = (await answer(c.q, index)).toLowerCase();
  const ok = c.mustDecline
    ? /cannot|can't|only help/.test(out)
    : c.mustInclude.every(s => out.includes(s.toLowerCase()));
  if (ok) pass++; else console.log("FAIL:", c.q);
}
console.log(pass + "/" + cases.length + " passed");

This is raw text matching; open-ended answers also need human judgement or a judge model, but even this simple version catches obvious regressions. Add failed questions from real users to the set over time. For better prompts, see the prompt engineering guide.

07Pre-launch checklist

• The API key lives only on the server in an environment variable, and rotation is planned.
• Requests, latency and per-conversation cost are logged, without storing sensitive data.
• Provider errors have timeouts and a polite fallback message.
• Users can be handed to a human and know they are talking to a bot.
• The evaluation set runs before each release and results are recorded.

If you would rather walk this path with an experienced team, get in touch or browse our previous projects. For uses beyond chat, business automation with AI offers more ideas.