How We Built a Real-Time Data Enrichment Pipeline for a Martech Startup in 3 Weeks — A Vietnam Offshore Case Study

I’ve seen a lot of data pipelines in my time. Most of them are held together with duct tape and cron jobs. But this one was different.

A US-based martech startup came to us with a painful problem. They were ingesting user events from CRM, ad platforms, and webhooks — about 50,000 events per hour. Their existing batch pipeline took 4 hours to enrich and deliver personalized email campaigns. By the time the email went out, the user had already churned.

How AI Is Reshaping Every Stage of the Software Development Lifecycle

TL;DR: AI tools are now automating code generation, testing, debugging, and deployment across the entire software lifecycle. This… ...

They needed real-time. Sub-second latency. And they needed it fast.

We built it in 3 weeks with a team of 5 engineers in Ho Chi Minh City, using the ECOA AI Platform ACP for intelligent agent orchestration. Here’s exactly how.

Vietnam Outsourcing: The Silent Revolution in Offshore Software Development

TL;DR: Vietnam outsourcing is rapidly becoming the go‑to destination for companies wanting quality software development at a fraction… ...

The Problem: Batch Processing Killed Personalization

The old pipeline was a mess:

Events landed in a Postgres table every 15 minutes via a cron job.
A monolithic Python script did all enrichment: CRM lookups, geocoding, sentiment analysis, and ad attribution.
The script ran sequentially. One slow API call blocked everything.
Average end-to-end latency: 4 hours 12 minutes.

The CTO told me: “We can’t personalize if we don’t know who the user is until tomorrow.”

Why not just throw more servers at it? Because the bottleneck wasn’t compute — it was the enrichment logic itself. Each event needed to call 4-6 external APIs, and those APIs had unpredictable latency.

The Architecture: Streaming + Agent Orchestration

We switched from batch to streaming. Here’s the high-level stack:

Component	Technology	Purpose
Event ingestion	Apache Kafka (Confluent Cloud)	50K events/min peak
Stream processing	Apache Flink (Dataflow)	Lightweight transforms, dedup
Enrichment agents	ECOA AI Platform ACP	Dynamic routing to enrichment functions
State store	Redis + PostgreSQL	Enrichment cache + audit trail
Output	Kafka → Webhook → Customer.io	Real-time campaign triggers

The key innovation was the enrichment agent layer built on ECOA AI Platform ACP. Instead of a hardcoded sequence of API calls, we defined a set of specialized agents:

CRM Lookup Agent – queries Salesforce or HubSpot for user metadata
Geocoding Agent – resolves IP to city/country
Sentiment Agent – runs a local HuggingFace model on event text
Ad Attribution Agent – matches event to ad campaign via a Redis lookup

Each agent is a stateless function wrapped in an ECOA agent definition. The orchestrator decides which agents to invoke based on the event type and the enrichment rules.

Sample Agent Definition (YAML)

yaml
agents:
  - name: crm_lookup
    type: function
    timeout: 2s
    retry: 2
    fallback: cache
    input:
      user_id: event.user_id
    output:
      crm_segment: segment
  - name: sentiment
    type: local_llm
    model: "cardiffnlp/twitter-roberta-base-sentiment"
    timeout: 500ms
    input:
      text: event.description
    output:
      sentiment: label

The orchestrator runs these agents in parallel where possible, but respects dependencies (e.g., geocoding must finish before ad attribution can use the country code). ECOA’s DAG engine handles that automatically.

The 3-Week Sprint

We split the work:

Week 1: Set up Kafka, Flink job for basic parsing, and the ECOA agent definitions. The Vietnamese team had prior experience with Kafka on a logistics project, so they moved fast.
Week 2: Built the enrichment agents and connected them to the external APIs. We hit a snag with the CRM API rate limits — solved by adding a Redis cache layer with a 5-minute TTL.
Week 3: Integration testing, performance tuning, and deployment. We benchmarked at 50,000 events per minute with p99 latency of 890ms.

Results: From 4 Hours to Sub-Second

Metric	Before	After
End-to-end latency	4h 12m	< 1s (p99 890ms)
Events processed per minute	800	50,000
Campaign conversion rate	1.2%	4.7%
Infrastructure cost	$4,200/mo	$3,800/mo

The startup saw a 4x lift in email conversion within two weeks. Why? Because the email now arrived when the user was still engaged.

Lessons Learned

Don’t over-engineer the orchestrator. We started with a complex state machine. ECOA’s built-in DAG with parallel execution was simpler and faster.
Cache aggressively. The CRM lookup agent was the slowest. A 5-minute Redis cache cut its average latency from 1.2s to 4ms.
Test with realistic traffic. Our initial load test used uniform event sizes. Production events varied wildly. We had to tune Flink’s checkpointing intervals twice.
Vietnamese engineers are fast, but they need clear specs. We wrote detailed RFCs for each agent. The team in Ho Chi Minh City delivered on time because they knew exactly what to build.

Why This Worked

Honestly, the technology was only half the battle. The other half was the team.

We’ve worked with offshore teams across multiple countries. The Vietnamese engineers on this project stood out for their ownership mentality. When the CRM API rate limit broke the pipeline, they didn’t wait for instructions — they proposed the Redis cache solution and implemented it overnight.

That’s the real advantage of hiring developers from Vietnam. They don’t just execute tickets. They solve problems.

And with ECOA AI Platform ACP, they could focus on business logic instead of wiring up message queues and retry policies. The platform handled the orchestration, observability, and fault tolerance. We just wrote the agents.

Frequently Asked Questions

Q: What’s the difference between this pipeline and a standard Kafka Streams setup?

A: Kafka Streams handles the plumbing, but it doesn’t know how to orchestrate heterogeneous enrichment tasks with different timeouts, retries, and fallbacks. ECOA AI Platform ACP adds a semantic layer that decides *which* agents to call and *when*, based on event metadata. You’d need to build that logic yourself with Kafka Streams.

Q: Can this architecture handle 500,000 events per minute?

A: Yes, with horizontal scaling. We tested up to 50K events/min on a single Flink worker with 4 vCPUs. To go to 500K, you’d add more Kafka partitions and Flink parallelism. The ECOA orchestrator is stateless, so it scales linearly.

Q: How long does it take to train a Vietnamese team on the ECOA AI Platform ACP?

A: About 3 days for a senior engineer. The platform abstracts away most infrastructure concerns. Our team in Ho Chi Minh City was productive after a 2-day workshop and a day of pair programming.

Q: What happens if one enrichment agent fails?

A: ECOA allows per-agent retry policies and fallback strategies. In our pipeline, if the CRM lookup fails after 2 retries, the orchestrator uses a cached value. If no cache exists, it marks the event as “partial enrichment” and continues. You define the behavior in the agent YAML.