From Batch to Real-Time: How a Logistics Company Orchestrated a Live Data Pipeline with AI Agents

Batch processing is the silent killer of modern logistics.

Your nightly cron jobs fail silently at 3 AM. Your customers see stale tracking data. Your ops team manually reconciles spreadsheets because the warehouse management system and the billing platform are 12 hours out of sync.

AI Coding Tools in 2026: Benchmarking Claude Code, OpenAI Codex CLI, Cline, Aider, and Hermes Agent on Real-World Development Tasks

TL;DR We benchmarked 5 leading AI coding tools — Claude Code, OpenAI Codex CLI, Cline, Aider, and Hermes… ...

I’ve seen this pattern a dozen times. But last quarter, we tackled it with a different playbook.

A mid-sized logistics company in the US moves about 8,000 containers annually between Long Beach and Ho Chi Minh City. Their legacy pipeline was a Frankenstein of Python cron jobs, SQL Server stored procedures, and a single Kafka topic that everyone was afraid to touch. Data freshness averaged 30 minutes—and spiked to 4+ hours during peak season.

Vietnam Outsourcing: Why Smart CTOs Are Betting on Southeast Asia’s Rising Tech Hub

TL;DR: Vietnam outsourcing is quickly becoming the preferred destination for cost‑effective, high‑quality software development. With engineering talent growing… ...

Their CTO came to us with one question: *”Can you make my tracking data feel real-time without rewriting everything?”*

The Problem Wasn’t the Database. It Was the Orchestration.

Let’s be honest about what “batch” really means. It’s not about the technology. It’s about coordination.

Their pipeline had 17 discrete stages:

Ingest EDI 214 messages from carriers
Normalize carrier-specific fields into a common schema
Enrich with customs data from a third-party API
Update the warehouse slot reservation system
Trigger billing for completed legs
Push tracking updates to their customer portal

Each stage was a separate Lambda function or a SQL job. And each one was triggered by a cron that assumed the previous stage had finished. When one stage failed (which happened roughly 3 times per week), the entire chain stalled until someone manually replayed it.

The real issue? No shared state. No error recovery. No visibility into what actually happened.

The Agentic Approach: One Orchestrator, Seven Specialized Agents

We didn’t rip out their existing infrastructure. That would’ve taken six months and nobody had the budget. Instead, we built an orchestration layer on top—using the ECOA AI Platform ACP to deploy a set of specialized AI agents.

Here’s the architecture:

Agent	Responsibility	Tool Access
Ingest Agent	Parse incoming EDI 214 files	S3, Postgres
Normalize Agent	Map carrier fields to canonical schema	Embedding store, schema registry
Enrich Agent	Call customs API, merge results	REST endpoints, cache
Slot Agent	Reserve warehouse slots	Warehouse API, Redis
Billing Agent	Calculate line-haul charges	Pricing DB, Stripe
Status Agent	Push to customer portal	WebSocket, Firebase
Watchdog Agent	Monitor all agents, handle failures	Log stream, alert webhook

The key insight? Each agent is a task-specific actor, not a generic LLM stuffed into a prompt. The orchestrator (a state machine, not a DAG) manages the flow. When the Enrich Agent fails on a rate-limited API call, the orchestrator retries with exponential backoff. When the Billing Agent returns a price that’s 15% above historical average, it flags for human review instead of silently committing.

*But doesn’t that just shift the complexity from cron jobs to an agent system?*

Actually, no. The difference is observability and recovery. The cron job failure was a black hole. The agent failure produces a structured error, a trace, and an automatic reroute to the Watchdog Agent.

The Configuration That Changed Everything

Let me show you what the orchestrator config looked like for the core pipeline segment:

yaml
pipeline:
  id: "tracking-sync-prod"
  trigger: event_stream
  source: s3://edi-inbound/raw/
  
  agents:
    - role: ingest
      model: claude-sonnet-4
      instructions: "Parse EDI 214 messages. Extract shipment_id, carrier_code, event_type, timestamp, and location fields. Return structured JSON."
      retry_policy:
        max_attempts: 3
        backoff: exponential
        initial_delay: 1s
    
    - role: normalize
      model: claude-sonnet-4
      instructions: "Map carrier-specific field names to canonical schema v3.2. If unknown carrier, escalate to watchdog."
      context:
        - vector_store: "schema_registry"
          query: "Canonical mapping for {carrier_code}"
    
    - role: enrich
      model: claude-haiku
      instructions: "Query customs API for shipment {shipment_id}. Cache results for 24 hours."
      error_handler:
        on_429:
          - wait: 30s
          - retry
        on_500:
          - fallback_to: "cache_delayed"
          - notify_watchdog: true
    
    - role: watchdog
      model: gpt-4o
      instructions: "Review errors flagged by other agents. Generate recovery plan. Post to #ops-alerts if manual intervention required."

This isn’t a pipeline in the traditional sense. It’s a conversation between agents, coordinated by a state machine. Each agent has a clear role, access to specific tools, and a defined error path.

The Results That Made the CFO Happy

We deployed this with a team of three senior Vietnamese developers based in Can Tho. Total timeline: 7 weeks from kickoff to production.

After 60 days in production:

Data latency dropped from 30 minutes to 1.8 seconds (p99)
Incident response time went from 4+ hours to 11 minutes (agent auto-recovery)
Operational cost reduced by 82% — fewer on-call rotations, fewer manual replays
Error rate dropped from 3.2% to 0.04% of all EDI files processed

The biggest win nobody expected? The customer NPS for the tracking portal jumped 17 points. Turns out, when your customers see real-time container updates instead of 4-hour-old data, they actually trust you.

Why This Worked (And Why Most Batch Migrations Fail)

Most companies try to solve the batch problem by buying a streaming platform or hiring a team to rewrite everything in Flink. That’s expensive, risky, and takes a year.

We took a different bet: keep the legacy systems, but orchestrate them with agents that can reason about failures.

The Vietnamese team didn’t just write code. They configured the agent behaviors. They tuned the retry policies. They built the shared state layer (a managed Postgres instance with logical replication) that let agents see each other’s outputs without tight coupling.

*Can you replicate this without an agent orchestration platform?*

Technically, yes. But you’ll end up building your own state machine, your own error recovery, your own tool registry. That’s a multi-month detour. We used the ECOA AI Platform ACP specifically because its state-machine orchestration handles exactly these failure modes without custom code.

Lessons Learned

Don’t build agents for tasks you’ve already solved. The Ingest Agent didn’t rewrite the EDI parser. It just called the existing parser and handled the JSON output.

Give agents memory, not just context. We used a shared Postgres table as the agent memory layer. Each agent wrote its decisions and confidence scores. The Watchdog Agent could inspect the full history.

Expect your orchestrator config to evolve. We changed agent instructions 14 times in the first two weeks. That’s normal. Don’t over-engineer upfront.

—

Frequently Asked Questions

Q: Does this approach work for non-logistics domains?

Absolutely. The same pattern applies to any domain with batch-driven workflows — fintech settlement, healthcare claims processing, supply chain procurement. The agents don’t need deep domain knowledge of logistics. They just need clear instructions and the right tool access.

Q: How do you handle PII and data privacy with AI agents?

We kept the agent orchestration layer on the same VPC as the existing infrastructure. No data left the environment. The AI models accessed only anonymized field mappings and never saw raw PII. The ECOA platform supports on-prem or VPC deployment for this exact reason.

Q: What happens when the orchestrator itself fails?

The orchestrator is stateless and horizontally scaled behind a load balancer. If an instance crashes, the next instance picks up the in-flight events from the shared state in Postgres. We’ve tested this — zero data loss in failover scenarios.

Q: Can a small team manage this without a dedicated DevOps person?

Our team in Can Tho had three developers — no dedicated DevOps. The ECOA platform abstracts deployment, scaling, and monitoring. The team managed it via GitHub-based configuration changes. You’ll need someone comfortable with YAML and basic networking, but you don’t need a Kubernetes expert.