· Manfredi Miraula · Case Study · 2 min read
From hours to minutes: near real-time data pipeline for a leading pharma company
Operational data was arriving hours late because of slow batch processes. We rebuilt the pipeline to sync data in near real time, including binary files up to 4GB.
At a leading pharmaceutical company, operational decisions were being made on data that was hours old. The teams were looking at numbers that described a situation that had already moved on.
The cause was a batch architecture that had worked in the past and could no longer keep up with the business.
The problem
The operational teams needed data in time to act on it. The existing pipeline, though, processed it in large, slow batches that only refreshed at intervals. The lag in between made any real-time analysis impossible.
There was also a technical constraint. Besides ordinary tabular records, the system had to handle binary files of up to 4GB, and standard streaming tools aren’t built for that.
The solution
We redesigned the pipeline from scratch with a near real-time architecture on Azure Service Bus:
- Event-driven ingestion: data moves when the source changes, not on a fixed schedule.
- Incremental processing: only changed records are processed, which cuts both compute load and latency.
- Binary file sync: a custom layer handles large files (up to 4GB) and syncs them in near real time.
- Full monitoring: alerts and metrics on pipeline health, latency and data quality at every stage.
We built on the infrastructure the client already had, and the systems consuming the data kept running throughout the migration.
The results
- Data available within minutes instead of hours.
- Binary files up to 4GB synced in near real time.
- Better-informed operational decisions, based on current, reliable data.
- No disruption to downstream systems during the migration.
The principle that applies everywhere
Batch processing is often the right place to start, and it stays right until the business outgrows it. When teams are deciding on stale data, the people are rarely the problem. The limit is usually in the architecture.
A targeted redesign makes things possible that weren’t before.
Is slow data holding your team back? Get in touch and we’ll look together at whether a similar approach makes sense for you.