Real-Time RAG

Inject live user data and documentation into a generative prompt using edge functions.

Overview

Retrieval adds latency before your LLM starts streaming. With SwarmV's index at the edge, next to your API routes, retrieval stays fast, and searching with the user's own token means each user only retrieves documents they're allowed to see.

Store each chunk's text in its metadata when you index it, for example { container_id: "container:handbook", text: chunk }. Metadata is flat (string, number or boolean values) and up to 8 KB per record, and comes back as strings. For longer documents, store a short chunk plus a reference to the full source.

Next.js / Vercel AI SDK

import { SwarmV } from "@swarmv/sdk";
import { generateText } from "ai";
import { openai } from "@ai-sdk/openai";

export async function POST(req: Request) {
  const user = await getSignedInUser(req);           // your own auth
  const { prompt, embedding } = await req.json();

  // Search as this user, so they only get documents they're allowed to see.
  const swarmv = new SwarmV({ token: await swarmvTokenFor(user) });
  const hits = await swarmv.search(embedding, { topK: 3 });

  // Each hit carries the metadata stored with it, e.g. the chunk text (values are strings).
  const context = hits.map((h) => h.metadata?.text ?? "").join("\n\n");

  const { text } = await generateText({
    model: openai("gpt-4o"),
    system: `You are a helpful assistant. Use the following context to answer:\n\n${context}`,
    prompt,
  });

  return Response.json({ text });
}

LlamaIndex Custom Retriever

SwarmV can be used as a custom retriever for LlamaIndex.

Python / LlamaIndex

from llama_index.core import Document
from swarmv import SwarmV

swarmv = SwarmV(token_provider=fetch_swarmv_token)  # a token for the end user

class SwarmVRetriever:
    def retrieve(self, query):
        text = getattr(query, "query_str", query)
        vector = embed_model.get_text_embedding(text)
        hits = swarmv.search(vector, top_k=3)
        # Each hit carries its stored metadata, e.g. the chunk text
        return [Document(text=(h.metadata or {}).get("text", ""), id_=h.id) for h in hits]

retriever = SwarmVRetriever()
context = "\n".join(d.text for d in retriever.retrieve("How does edge RAG work?"))
response = llm.complete(f"Use this context to answer the question:\n{context}\n\nQuestion: How does edge RAG work?")
print(response)