How to Reduce Token Cost for LLM Agents: A Practical Guide

原创

gavin1024

发布于 2026-10-22 16:58:11

1370

If you've been building with LLM agents for a while, you've probably noticed the pattern: conversations grow, context windows fill up, and token costs climb steadily. A single long session can burn through more tokens than your first hundred short ones combined.

This guide walks through practical strategies to cut token costs without sacrificing agent quality.

Where Tokens Go

Source	Typical Share	Why It Grows
Accumulated conversation history	40-60%	Every turn adds to the context
Repeated context (docs, code snippets)	20-30%	Users re-paste what agents should already know
Tool call results	10-20%	API responses and search results are verbose

Strategy 1: Conversation Summarization

Periodically summarize older conversation turns and replace them with the summary.

Works when: Single agent, manageable turn count.

Limitation: Summarization can drop critical details — a constraint mentioned in turn 3 might vanish from the summary in turn 15.

Strategy 2: Layered Memory with On-Demand Retrieval

Instead of stuffing all history into context, use a layered memory system that stores at different granularities and retrieves only what's relevant.

TencentDB Agent Memory implements this as a four-layer architecture:

Layer	Stores	Purpose
L0 Conversation	Raw conversations with full context	Source verification
L1 Atom	Extracted facts, preferences, constraints	Precise recall
L2 Scenario	Knowledge blocks organized by project	Fast context recovery
L3 Persona	Long-term user profile	Quick user understanding

Retrieval uses BM25 + vector search + RRF (Reciprocal Rank Fusion), with hard constraints on entry count, character budget, and timeout — so memory never blows up the context window.

Result: Only relevant memory slices enter context, not the full history.

Strategy 3: Structured Document and Code Indexing

If agents repeatedly reference the same documents or codebase, stop pasting them. Index them once.

TencentDB Agent Memory provides two structured index assets:

Wiki: Turns documents into structured pages with a link graph. Agents search and retrieve relevant sections instead of loading entire files.
CodeGraph: Indexes code symbols, files, call relationships, and impact paths. Agents can check "what does changing this function affect?" without loading the whole codebase.

Strategy 4: Team-Level Memory Reuse

When multiple agents serve the same team, each maintaining independent memory leads to redundant accumulation.

TencentDB Agent Memory's team memory: one agent's validated Skill (e.g., "Release Checklist") gets reviewed, shared with the team, and equipped by other agents — no re-learning required.

Quick Comparison

Strategy	Best For	Token Reduction	Setup Effort
Summarization	Single agent, quick fix	30-50%	Low
Layered memory	Long-term	50-70%	Medium
Doc/code indexing	Repeated reference	60-80%	Medium
Team memory reuse	Multi-agent teams	Depends on reuse	Medium-high

On the PersonaMem benchmark, TencentDB Agent Memory improved agent user-understanding accuracy from 48% to 76% — compressing context without sacrificing quality.

MIT licensed. One command to start:

git clone https://github.com/TencentCloud/TencentDB-Agent-Memory.git
cd TencentDB-Agent-Memory/deploy/global-images
./start-all.sh

Repo: https://github.com/TencentCloud/TencentDB-Agent-Memory

原创声明：本文系作者授权腾讯云开发者社区发表，未经许可，不得转载。

如有侵权，请联系 cloudcommunity@tencent.com 删除。

LLM