OpenAI API in practice: function calling, embeddings and RAG

This post is production code, not hello world. After two years of building applications with the OpenAI API, I have battle-tested patterns I use in every other project. Ready to copy-paste and run.

Function calling: stable contract with the model

Function calling is the most important OpenAI API feature for developers. It forces the model to return structure instead of chaotic text. In production it is irreplaceable.

Basic pattern (TypeScript)

import OpenAI from "openai";
import { z } from "zod";

const client = new OpenAI();

// Define the schema of what the model should return
const TaskSchema = z.object({
  action: z.enum(["send_email", "create_task", "search_docs", "none"]),
  args: z.record(z.string(), z.any()),
  reasoning: z.string().max(500),
});

type Task = z.infer<typeof TaskSchema>;

async function classifyIntent(userMessage: string): Promise<Task> {
  const completion = await client.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [
      {
        role: "system",
        content: `You are an assistant. Classify the user intent
        into one of 4 actions. Be concise.`,
      },
      { role: "user", content: userMessage },
    ],
    tools: [
      {
        type: "function",
        function: {
          name: "execute_action",
          description: "Execute an action based on the user intent",
          parameters: {
            type: "object",
            properties: {
              action: {
                type: "string",
                enum: ["send_email", "create_task", "search_docs", "none"],
              },
              args: { type: "object", additionalProperties: true },
              reasoning: { type: "string" },
            },
            required: ["action", "args", "reasoning"],
          },
        },
      },
    ],
    tool_choice: { type: "function", function: { name: "execute_action" } },
  });

  const toolCall = completion.choices[0].message.tool_calls?.[0];
  if (!toolCall) throw new Error("No tool call returned");

  // IMPORTANT: validate the model output before you use it
  const parsed = TaskSchema.parse(JSON.parse(toolCall.function.arguments));
  return parsed;
}

Three things that make a difference:

1. tool_choice: { type: "function", ... } — forces the model to always call this function. Without it the model can answer with text instead of JSON.

2. z.parse() on the output — the model can return JSON that does not fit the schema. Validation lets you catch that. Without it you have bugs that show up in production randomly.

3. reasoning in the output — a required field in the schema. The model must justify its decision, which reduces hallucinations. Without it the model guesses the action randomly; with it, it thinks before deciding.

Pitfall: function calling + streaming

Function calling does not stream well. The output is either the full JSON or nothing. If you need streaming (long-form text), use a dual call: the first call returns the action (function calling), the second returns text (streaming). Twice as expensive, but the only way to combine both.

Embeddings: turning text into vectors

Embeddings are the foundation of semantic search and RAG. They turn text into a 1536-dimension vector (for text-embedding-3-small). Texts with similar meaning have similar vectors.

Generating embeddings (cache!)

import { createHash } from "crypto";
import { Redis } from "@upstash/redis";

const redis = Redis.fromEnv();

async function getEmbedding(text: string): Promise<number[]> {
  // Cache hit: 0ms, $0
  const cached = await redis.get<number[]>(`emb:${hashText(text)}`);
  if (cached) return cached;

  // Cache miss: 200ms, $0.00000002
  const response = await openai.embeddings.create({
    model: "text-embedding-3-small",
    input: text,
  });

  const vector = response.data[0].embedding;
  await redis.set(`emb:${hashText(text)}`, vector, { ex: 60 * 60 * 24 * 30 });
  return vector;
}

function hashText(text: string): string {
  return createHash("sha256").update(text).digest("hex").slice(0, 16);
}

Caching embeddings is key for cost. If a user asks "how does X work" twice — the second embedding is a free cache hit. In my applications 60-80% of queries are cache hits.

Storage: pgvector vs Pinecone

For 95% of Polish SaaS: pgvector in Supabase is enough.

-- Migration: enable pgvector
CREATE EXTENSION IF NOT EXISTS vector;

-- Table with embeddings
CREATE TABLE documents (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  content TEXT NOT NULL,
  metadata JSONB DEFAULT '{}',
  embedding vector(1536),
  created_at TIMESTAMPTZ DEFAULT now()
);

-- Index for fast similarity search
CREATE INDEX documents_embedding_idx
ON documents
USING ivfflat (embedding vector_cosine_ops)
WITH (lists = 100);

import { createClient } from "@supabase/supabase-js";

const supabase = createClient(
  process.env.SUPABASE_URL!,
  process.env.SUPABASE_SERVICE_KEY!
);

async function findSimilarDocuments(
  queryEmbedding: number[],
  matchThreshold = 0.7,
  matchCount = 5
) {
  const { data, error } = await supabase.rpc("match_documents", {
    query_embedding: queryEmbedding,
    match_threshold: matchThreshold,
    match_count: matchCount,
  });

  if (error) throw error;
  return data;
}

<=> is the cosine distance operator in pgvector. Closer to 0 = more similar. Threshold 0.7 = "fairly similar".

RAG pipeline: production-ready

RAG combines retrieval (semantic search) + generation (LLM). The pattern I use in most applications:

async function ragQuery(userQuestion: string): Promise<string> {
  // 1. Embedding the question (cached)
  const questionEmbedding = await getEmbedding(userQuestion);

  // 2. Retrieval: top-5 most similar fragments
  const relevantDocs = await findSimilarDocuments(questionEmbedding, 0.7, 5);

  if (relevantDocs.length === 0) {
    return "I do not know the answer to that. Can I help you differently?";
  }

  // 3. Build context with citations
  const context = relevantDocs
    .map((doc, i) => `[${i + 1}] ${doc.content}`)
    .join("\n\n");

  // 4. Prompt with context and citation instruction
  const completion = await openai.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [
      {
        role: "system",
        content: `You are an assistant. Answer ONLY based on the
        provided context. If the context does not contain the answer,
        say so plainly. ALWAYS cite the source using the [N] number
        for every fact.`,
      },
      {
        role: "user",
        content: `Context:\n${context}\n\nQuestion: ${userQuestion}`,
      },
    ],
    temperature: 0.2, // low = more deterministic
  });

  return completion.choices[0].message.content!;
}

Three elements that matter:

1. Threshold similarity 0.7 — below this documents are noise. Without a threshold the model gets random documents and hallucinates.

2. Numbered citations [N] — the model cites sources. The user sees where the info comes from. You can verify in the UI ("Source: document 3").

3. Temperature 0.2 — for RAG you want deterministic answers, not creative ones. Temperature 0.7+ produces hallucinations because the model "guesses" when the context is incomplete.

When RAG does not work

RAG is not a magic wand. It does not work when:

• Documents are stale — RAG returns old, factually wrong content. Solution: re-embed documents regularly, add dateModified as a filter.
• The question is too generic — top-5 fragments do not cover the topic. Solution: query expansion (rewrite the question into 3 variants, search each).
• The knowledge base is small (< 100 documents) — better to paste everything into the prompt, RAG is overengineering.

Production costs: real numbers

For an app with 10k users / month, 3 questions / user:

| Component | Cost / month | |-----------|--------------| | Embeddings (70% cache hit) | $0.30 | | Vector storage (Supabase) | $0 (free tier) | | GPT-4o-mini (RAG, avg 1k tokens) | $45 | | GPT-4o (precise answers) | $300 | | Total (mini) | $45-50 | | Total (full) | $300-350 |

Conclusion: GPT-4o-mini is enough for 80% of RAG applications. Only when you need nuance (legal, medical) do I switch to GPT-4o.

Debugging: 5 tools I use every day

1. LangSmith — tracing for OpenAI, full visibility into prompts and responses. Paid ($39/m) but worth it.
2. Helicone — OpenAI proxy, logging tokens and costs. Free tier is enough to start.
3. OpenAI Playground — testing prompts without code, comparing models side by side.
4. pgvector Studio (local GUI) — browsing vectors in the DB, checking whether the embedding makes sense.
5. My own eval set — 20-30 questions with expected answers, I run it after every prompt change. Without it you optimize blindly.

What is next

This post is the foundation. Next steps:

• Function calling + persistence — agents that remember previous calls
• RAG with re-ranking — a second model improves the retrieval ranking
• Streaming + function calling — two models in tandem for UX

If you are building an app with the OpenAI API and you need an architecture review — get in touch. I have battle-tested patterns for 8 verticals (e-commerce, education, B2B lead gen, HR, support, content generation, code assistance, document analysis).

Related posts:

• Multi-agent AI systems in practice
• GEO: how to write for AI
• AI search engines vs Google