Build AI Agent Memory with RAG: Complete Guide

Introduction

Retrieval-Augmented Generation, commonly abbreviated as RAG, is a widely adopted LLM technical framework. Its name breaks down into three core actions: Retrieval, Augmented, and Generation. The standard operating procedure follows two primary phases. First, the system retrieves relevant materials from a knowledge repository during the retrieval step. Second, the model generates a final answer, augmented with the retrieved context. This two-stage pipeline solves a fundamental limitation of large language models: the constraint of fixed context windows.

To illustrate the practical value of RAG, we can examine the example of an SVG generation platform. While interacting with this application, users can share detailed project requirements over multiple turns of dialogue. A user may specify that the tech stack should use TypeScript, describe the product as an SVG editor with live preview, code editing and export functions, and note that all SVG modifications must sync to a defaultSVG variable.

Without a dedicated memory mechanism, every new request forces the agent to reload the full conversation history from scratch. The agent has no persistent memory; it simply re-reads the entire chat log on each invocation. This naive approach quickly hits performance and cost limits as conversations grow longer. RAG offers a more efficient way to store, index and recall long-term user memories for AI agents.

Why Adopt RAG for User Memory

Developers who have built systems like Vibe Coding will quickly recognize the scaling problem. As users interact with AI agents, the volume of persistent user data accumulates rapidly. This dataset includes user identity, technical preferences, project specifications, and past decisions.

A naive alternative is injecting the full conversation history directly into the system prompt. This strategy has severe drawbacks. The complete history expands the prompt drastically, most of the loaded content is irrelevant to the current query. Longer prompts slow down model inference, increase token consumption, and dilute signal. If you inject months of chat records, dozens of project notes and unrelated background facts into context, the model struggles to identify critical information.

RAG uses a much simpler workflow for memory management. After a conversation segment ends, the system extracts long-term memory entries and writes them into user_memories. Before processing a new user request, the system searches this memory store, fetches relevant records, and inserts those records into a dedicated [RUSH_MEMORY] block inside the system prompt. The model receives only memories relevant to the current question instead of the full chat log. This is the core of the RAG workflow: retrieve relevant records (Retrieval), augment the prompt (Augmented), and then produce the response (Generation).

For this SVG platform example, the agent may recall that the user prefers TypeScript, works on SVG preview components, and needs all changes synced to defaultSVG. Only these relevant memories are inserted into the prompt, rather than loading every prior message.

Core RAG Pipeline for User Memory

The production-grade RAG workflow for user memory can be split into two major stages:

1. Post-conversation processing: Chunking and Indexing

2. Pre-response preparation: Retrieval, Reranking, and Generation

These steps form a repeatable cycle. The pipeline first writes new memories after chat sessions finish. Then it fetches and filters memories before each new user query to enrich the prompt for model inference.

Chunking

When a conversation finishes, the system collects roughly the last 10 messages and sends this batch to the GLM model. With carefully designed prompt engineering, GLM extracts high-value information from the dialogue and outputs structured memory records in JSON format.

import { generateText } from 'ai';

const EXTRACTION_SYSTEM_PROMPT = 'You are a long-term memory assistant. Extract high-value facts from the dialogue and output a JSON array.';

async function extractConversationMemory(messages) {
  const conversationText = buildConversationText(messages);
  const result = await generateText({
    model: createFastModel(),
    system: EXTRACTION_SYSTEM_PROMPT,
    prompt: conversationText,
  });
  return result;
}

The project delegates chunking work to GLM for three practical reasons. First, manual rule-based chunking requires heavy prompt tuning and maintenance. Second, extraction runs after the conversation completes, so inference latency does not impact user-facing response speed. Third, using an LLM for chunking reduces engineering overhead.

Indexing

Indexing consists of two key tasks: converting text chunks into embedding vectors, and storing the raw text alongside vector embeddings in a vector-enabled database. This implementation uses the embedding-3 model for vector conversion, which produces 1024-dimensional embeddings.

function createEmbeddingModel() {
  const apiKey = process.env.ZHIPU_EMBEDDING_API_KEY;
  return zhipu.textEmbeddingModel('embedding-3', { dimensions: 1024 });
}

const embeddingModel = createEmbeddingModel();

export async function embedText(value: string) {
  const model = embeddingModel;
  const result = await withTimeout(
    model.doEmbed({
      values: [value],
    })
  );
  return result;
}

This system uses PostgreSQL with the pgvector extension rather than a standalone specialized vector database. Vector data is stored in a dedicated embedding column. The pgvector extension adds vector type support directly inside PostgreSQL, eliminating the requirement to deploy and maintain an independent vector database service.

export async function upsertMemory(userId: string, memoryContent: string, metadata) {
  const pool = getPgPool();
  await pool.query(`
    INSERT INTO user_memories (user_id, content, category, source, is_evergreen, project_id)
    VALUES ($1, $2, $3, $4, $5, $6)
    ON CONFLICT ...
  `, [userId, memoryContent, metadata.category, metadata.source, metadata.is_evergreen, metadata.project_id]);
  
  const embeddingVector = await embedText(memoryContent);
  await pool.query(`
    UPDATE user_memories SET embedding = $1::vector WHERE id = $2
  `, [embeddingVector, memoryId]);
}

Retrieval

Retrieval triggers before processing each new user message. The system extracts a search query from the incoming user prompt, then searches the user_memories table for matching records. This project uses hybrid search, combining vector similarity search and full-text search, with a result limit of 10 top memory entries.

const query = extractUserQuery(messages);
const results = await hybridSearchMemories({
  query,
  limit: 10,
});

The hybrid search implementation uses the same embedding-3 model to convert the user query into a vector. It then runs two parallel search operations inside PostgreSQL:

1. Vector search: Using vector cosine distance <-> to calculate similarity scores between the query embedding and stored memory embeddings.

2. Full-text search: Using PostgreSQL ts_rank to match lexical text overlap between query and memory content.

The final score blends the two results, using a weight of 0.7 for vector similarity and 0.3 for full-text matching, sorted from highest total score to lowest.

SELECT 
  content,
  (0.7 * (1 - (embedding <-> $1::vector))) AS vec_score,
  (0.3 * ts_rank(to_tsvector('simple', content), to_tsquery('simple', $2))) AS fts_score,
  (0.7 * (1 - (embedding <-> $1::vector))) + (0.3 * ts_rank(to_tsvector('simple', content), to_tsquery('simple', $2))) AS total_score
FROM user_memories
ORDER BY total_score DESC
LIMIT $3;

Reranking

Many RAG implementations use cross-encoder rerankers to reorder retrieved candidates. This project skips cross-encoder reranking for latency reasons. Instead, it applies two lightweight reranking filters locally after fetching results from PostgreSQL.

The pipeline fetches an expanded candidate pool (30 records, using an over-fetch factor of 3 × the final limit of 10). Two filters reduce the candidate set:

1. Time decay scoring: Older memories gradually lose weight over time. The score decays as the memory ages.

2. MMR (Maximal Marginal Relevance): Reduce redundancy by penalizing memories that are semantically similar to records already selected in the result list.

The pipeline retains only the top 10 memories after these filters. Avoiding cross-encoder models removes extra network calls and inference delay, which is critical for keeping user response latency low.

Generation

After reranking, the 10 selected memories are formatted into a [RUSH_MEMORY] block and appended to the system prompt. This block sits at high priority inside the prompt. The model reads these relevant memory records and uses them to augment its response generation.

function buildMemoryPromptBlock(memories) {
  if (memories.length === 0) return "";
  return `[RUSH_MEMORY]
Relevant memories:
${memories.map((m, idx) => `${idx+1}. ${m.content}`).join("\n")}
[/RUSH_MEMORY]`;
}

The model now sees relevant long-term facts about the user. In the SVG platform example, the model remembers the user’s preference for TypeScript, the requirement for live SVG preview, and the defaultSVG sync rule. The model can reference these details without loading the entire conversation history.

Production Architecture Considerations

This RAG memory system separates memory writing and memory reading into asynchronous workflows. Memory extraction and embedding run after conversation turn completes, so these heavy tasks do not add latency to user requests. Retrieval and reranking execute synchronously right before model generation, so relevant facts are available in the prompt for each user query.

The use of PostgreSQL and pgvector simplifies operational complexity. Teams do not need to learn, deploy and monitor a separate vector database. The same database that stores user business data can also host vector embeddings. This reduces infrastructure costs and operational burden for small to medium AI agent applications.

For teams running multiple LLM endpoints and embedding services, routing API requests securely and monitoring usage across different model providers creates operational overhead. 4sapi, an API gateway, helps centralize authentication, request throttling and logging for diverse model backends.

Limitations and Practical Tradeoffs

This RAG memory design has clear tradeoffs. The chunking step relies on an LLM to extract facts, which introduces occasional extraction noise. The hybrid search with static weighting (0.7 / 0.3) may not be optimal for all memory types. The system also does not automatically prune outdated memories; stale facts can remain in the memory store unless manually marked or updated.

Time decay and MMR partially mitigate these issues. Time decay reduces the influence of older memories. MMR prevents redundant memories from dominating the retrieved results. Developers can further tune the weight coefficients, over-fetch factor and memory retention policy according to their application’s use case.

Compared with putting full conversation history into context, RAG memory drastically cuts prompt token usage. In long-running multi-session projects, token savings accumulate quickly. The model focuses only on relevant facts, improving response quality and reducing hallucinations caused by noisy, irrelevant context.

Conclusion

RAG provides a practical and scalable way to implement persistent user memory for AI agents. By separating memory storage and on-demand retrieval, developers avoid the pitfalls of stuffing complete chat logs into the LLM context window. The pipeline demonstrated here uses GLM for memory extraction, embedding-3 for vector encoding, PostgreSQL with pgvector for storage, hybrid search and lightweight reranking to deliver relevant user facts to the model.

The workflow is well-suited for AI coding assistants, generative UI platforms and multi-turn agent systems. It balances implementation complexity, latency, token cost and response quality. Developers can adapt this pattern to store user preferences, project context and long-term facts across sessions, without rewriting core model logic.

International access: https://4sapi.com

Domestic access: https://4sapi.cn