Stop Burning API Credits on Dumb Agent Loops: How Smart Orchestration Cut Our LLM Costs by 52%

Let me be blunt. Most multi-agent systems I see in production have a dirty secret.

They’re hemorrhaging money on API calls.

Outsourcing Software Development? Here’s What Every CTO Needs to Know in 2025

TL;DR: Outsourcing software isn’t just about cutting costs—it’s about access to talent. In this guide, I break down… ...

Read more

You build a nice workflow — a research agent, a summarizer, a fact-checker. Each one calls the LLM independently. Each one passes around bloated context. And somewhere in that chain, two agents ask the exact same question to the model.

We recently audited a client’s multi-agent pipeline built in Ho Chi Minh City. The raw numbers were ugly: 63% of all LLM calls were redundant. Same input, same output, different agent.

How to Build a Custom AI Code Review Agent: A Step-by-Step Tutorial with ECOA AI Platform ACP

How to Build a Custom AI Code Review Agent: A Step-by-Step Tutorial with ECOA AI Platform ACP Let’s… ...

Read more

That’s not orchestration. That’s burning cash.

Here’s what we did about it.

The Anatomy of a Dumb Agent Loop

Most orchestrators treat every agent request as a fresh, unique event. They don’t check if the answer already exists in the system.

python
# Typical naive approach — every agent calls the LLM fresh
class NaiveOrchestrator:
    def run_workflow(self, user_query):
        research = self.research_agent.analyze(user_query)  # LLM call #1
        summary = self.summarizer_agent.summarize(research)  # LLM call #2
        fact_check = self.fact_checker.verify(summary)       # LLM call #3
        return fact_check

This pattern is everywhere. It’s simple. It’s clean. And it’s wasting 30-60% of your API budget.

The fix isn’t rocket science. You need three things:

1. A shared, semantic cache that stores agent outputs keyed by intent + context hash
2. A deduplication layer that intercepts identical or near-identical requests
3. A cost-aware router that decides *which* model to call based on task complexity

Building a Cost-Aware Orchestrator

We built this for a logistics client with a team in Can Tho. Their pipeline handled shipment tracking, exception handling, and customer comms. The naive version was making 12-18 LLM calls per transaction.

Here’s the core architecture we shipped.

Step 1: Semantic Caching with Redis

Don’t just cache exact string matches. Two different agents might ask “What’s the status of order 44921?” and “Is order 44921 delayed?” — same intent, different phrasing.

We hash the *semantic intent* using a lightweight embedding model and store the response.

python
import hashlib
import numpy as np
from redis import Redis
from sentence_transformers import SentenceTransformer

class SemanticCache:
    def __init__(self, redis_client: Redis, threshold: float = 0.92):
        self.redis = redis_client
        self.model = SentenceTransformer('all-MiniLM-L6-v2')
        self.threshold = threshold

    def _get_embedding(self, text: str) -> np.ndarray:
        return self.model.encode(text)

    def _hash_embedding(self, embedding: np.ndarray) -> str:
        return hashlib.sha256(embedding.tobytes()).hexdigest()

    def lookup(self, query: str) -> str | None:
        query_emb = self._get_embedding(query)
        query_hash = self._hash_embedding(query_emb)
        
        # Check exact semantic hash first (fast path)
        cached = self.redis.get(f"sem_cache:{query_hash}")
        if cached:
            return cached.decode()
        
        # Fallback: scan recent cache keys for similarity
        for key in self.redis.scan_iter("sem_cache:*", count=100):
            stored_emb = np.frombuffer(bytes.fromhex(key.decode().split(":")[1]), dtype=np.float32)
            similarity = np.dot(query_emb, stored_emb) / (np.linalg.norm(query_emb) * np.linalg.norm(stored_emb))
            if similarity > self.threshold:
                return self.redis.get(key).decode()
        return None

    def store(self, query: str, response: str):
        emb = self._get_embedding(query)
        emb_hash = self._hash_embedding(emb)
        self.redis.setex(f"sem_cache:{emb_hash}", 3600, response)  # 1-hour TTL

Honestly, this single component cut our cache hit rate from 12% to 41% in the first week. The exact-match cache was useless — agents rarely phrase things identically.

Step 2: Deduplication at the Router Level

Before any agent gets invoked, the orchestrator checks a request registry. If a pending request with the same semantic hash exists, the new agent subscribes to that result instead of firing a new LLM call.

python
import asyncio
from collections import defaultdict

class DedupRouter:
    def __init__(self, cache: SemanticCache):
        self.cache = cache
        self._pending_requests: dict[str, asyncio.Future] = defaultdict(asyncio.Future)

    async def route(self, agent_name: str, query: str) -> str:
        # Check cache first
        cached = self.cache.lookup(query)
        if cached:
            return cached
        
        query_hash = self.cache._hash_embedding(self.cache._get_embedding(query))
        
        # If another agent already requested this, wait for it
        if query_hash in self._pending_requests:
            result = await self._pending_requests[query_hash]
            return result
        
        # We're the first — create a future, call LLM, resolve
        future = asyncio.Future()
        self._pending_requests[query_hash] = future
        
        try:
            result = await self._call_llm(agent_name, query)
            self.cache.store(query, result)
            future.set_result(result)
            return result
        except Exception as e:
            future.set_exception(e)
            raise
        finally:
            del self._pending_requests[query_hash]

This eliminated the “three agents asking the same question” pattern entirely. In our production logs, we saw cases where Agent A and Agent B would fire identical lookups within 200ms of each other. The dedup layer caught every one.

Step 3: Cost-Aware Model Selection

Not every agent call needs GPT-4o. A simple status lookup? That’s a GPT-4o-mini job. A complex legal document analysis? That’s worth the premium.

We added a complexity scorer that evaluates the agent’s task and routes to the appropriate model tier.

We also set a budget per workflow — if an agent chain exceeds $0.05, the orchestrator starts routing subsequent calls to cheaper models or falls back to cached responses.

The Results After 8 Weeks in Production

We deployed this on a real logistics pipeline handling 50,000+ transactions per day. The team in Can Tho maintained the system.

• LLM API costs dropped 52% (from $14,200/month to $6,800/month)
• P95 latency improved 38% (cached responses are instant)
• Cache hit rate stabilized at 47% after the first week
• Zero regressions in output quality — the cheaper models handled the simple tasks just as well

More importantly, the orchestrator now *learns* which queries are cacheable. After a week, it started pre-fetching common patterns during idle cycles.

Why Most Teams Don’t Do This

It’s not that the technique is hard. It’s that most orchestrators are built for *correctness* first, *cost* never.

You see this all the time. A startup builds a multi-agent demo. It works. They ship it. Then the API bill arrives and everyone panics.

The fix is architectural. You can’t bolt cost optimization onto a naive orchestrator after the fact. You need the cache, the dedup, and the router built in from day one.

The ECOA AI Platform Approach

This is exactly why we built the ECOA AI Platform ACP with cost-aware orchestration as a first-class feature. When our developers in Vietnam build multi-agent systems for clients, they don’t start from scratch. They configure:

• Intent-based caching with configurable similarity thresholds
• Request deduplication with automatic timeout handling
• Model routing rules per agent and task type

The result? Our clients consistently see 40-60% cost reductions compared to their previous orchestration setups.

Frequently Asked Questions

Q: Won’t semantic caching return stale or incorrect answers for time-sensitive queries?

A: Yes, if you don’t set TTLs properly. We use per-agent TTLs — a weather agent might have a 5-minute cache, while a document summarizer can cache for 24 hours. You can also force a cache bypass for specific queries by setting a `no_cache` flag in the request.

Q: How do you handle near-duplicate queries that should return different results?

A: The similarity threshold is your control knob. We default to 0.92, which catches semantic duplicates but avoids false positives. For domains where precision is critical (like medical or legal), we drop the threshold to 0.98 or disable semantic matching entirely.

Q: Does this work with streaming LLM responses?

A: Yes, but you need to buffer the stream in the cache layer. We store the complete response as it streams in, then serve it from cache for subsequent requests. The first caller gets streaming; everyone after gets the cached result.

Q: What’s the performance overhead of the embedding lookup?

A: With `all-MiniLM-L6-v2` on a modern CPU, each embedding takes ~5ms. The Redis lookup adds another 1-2ms. Total overhead is under 10ms — a fraction of the 500-2000ms an LLM call takes. The cache hit savings more than compensate.

Related reading: Why Smart CTOs Hire Vietnamese Developers: A 2025 Playbook for Offshore Software Engineering

Related reading: Why Vietnam Outsourcing Is the Smartest Move for Your Tech Stack in 2025