When Enterprise Data Platforms Become Too Complex
Enterprise data platforms usually start with a sensible goal:
- Connect the data
- Make it trustworthy
- Make it useful
The problem is that, over time, the platform itself can become part of the complexity.
More services are added. More specialist skills are needed. More workloads become dependent on one environment.
Eventually, the technology introduced to simplify data can become another major system the organisation has to manage.
The better question for a CTO is:
Do we really need an entire enterprise data platform to solve what is mainly a data integration problem?
When the solution becomes bigger than the problem
Take healthcare.
A large healthcare organisation may need to connect:
- Patient administration systems
- Waiting lists
- Theatre schedules
- Bed occupancy
- Staffing
- Medical supplies
- Clinical and operational systems
The business need is simple:
Bring the right information together, keep it current and make it available where it is needed.
The value is obvious.
But if this gradually requires:
- A large proprietary platform
- Specialist developers
- Platform-specific processes
- Expensive support
- Increasing vendor dependency
then the architecture may have become heavier than the original problem.
Defence organisations face a similar challenge.
They may need to connect:
- Asset information
- Logistics
- Engineering systems
- Personnel data
- Supply chains
- Operational systems
- Legacy databases
Again, the requirement is straightforward:
Get trusted, current data to the right place quickly enough to support a decision.
These are difficult data problems.
But difficult data problems do not always require an increasingly complex platform.
What is the platform actually doing?
Strip away the product names.
For many organisations, the daily workload comes down to:
- Connect data
- Capture changes
- Clean it
- Transform it
- Join it
- Validate it
- Maintain history
- Track lineage
- Deliver it somewhere else
These are essential capabilities.
But they are fundamentally data engineering and data integration capabilities.
If this is where most of the value is coming from, it is reasonable to ask whether the organisation still needs the much larger platform around them.
A simpler model
The alternative is not necessarily another large platform.
It can be a smaller, independent production data layer sitting between the systems the organisation already owns.
For example:
- SAP stays where it is
- Oracle stays where it is
- Salesforce stays where it is
- Microsoft Fabric stays in place
- Databricks stays in place
- Snowflake stays in place
- Existing cloud and on-premise infrastructure stays in place
The missing layer simply connects them reliably.
That is the approach behind IOblend .
IOblend handles:
- Real-time integration
- Batch integration
- Change Data Capture
- Transformation
- Data quality
- Deduplication
- Historical data management
- Lineage
- Production pipeline management
The organisation does not need to rebuild its whole estate around IOblend.
Keep business logic portable
The hardest thing to migrate is often not the data.
It is the business logic.
Over time, organisations build rules around questions such as:
- How is a customer matched?
- How are duplicates handled?
- Which record wins?
- How is history maintained?
- What happens when a source changes?
- Which values are valid?
If that logic becomes trapped inside one platform, future technology decisions become harder.
IOblend takes a more portable approach.
Organisations can continue using:
- SQL
- Python
- Visual pipeline logic
- Existing engineering skills
The aim is to keep business logic useful even when the surrounding architecture changes.
Today’s target may be Fabric.
Tomorrow it may be Databricks.
The integration layer should make that change easier, not harder.
Reliability should be built in
Data teams often assemble several separate tools around a pipeline just to make it production-ready.
That can include:
- Development tools
- Testing frameworks
- Debugging tools
- Version control
- Data quality tooling
- Deployment scripts
- Monitoring
IOblend integrates much of this into the development workflow.
Developers can:
- Test components as they build
- Inspect data visually
- Validate pipeline logic
- Validate SQL
- Validate Python
- Run developer tests
- Compare pipeline versions
- Prevent invalid logic from executing
There is no need to write separate testing code for normal pipeline validation.
The principle is simple:
Production reliability should be built into the pipeline, not bolted on afterwards.
Migration should not be a leap of faith
Many organisations stay with complex platforms because moving away feels risky.
Critical pipelines cannot simply be switched off and replaced overnight.
They do not need to be.
A safer approach is:
- Build
- Run in parallel
- Reconcile
- Approve
- Cut over
The existing pipeline continues running.
The replacement processes the same source data.
Teams compare:
- Record counts
- Transformations
- Data quality
- Outputs
- Missing records
- Duplicates
- Processing behaviour
Only when the results match expectations does the organisation move production across.
That turns migration into a controlled engineering process rather than a high-risk platform replacement programme.
Simplicity can be an architectural advantage
For years, enterprise technology strategy has focused on adding more.
More platforms. More services. More layers.
The next advantage may come from knowing what can be removed.
That does not mean giving up:
- Reliability
- Performance
- Governance
- Control
It means asking a simpler question:
What is the smallest architecture capable of delivering the outcome properly?
For many organisations, the answer may not be another all-encompassing enterprise data platform.
It may simply be an independent data integration layer capable of keeping trusted data moving between the technologies the organisation has already chosen.
Keep the infrastructure that works.
Keep the business logic that matters.
Remove the complexity you no longer need.
How Operational Analytics power Ground Handling
The Ground Handling journey – today and tomorrow In today’s blog we are discussing how Operational Analytics (OA) enables the aviation Ground Handling industry to deliver their services to airlines. Aviation is one of the most complex industries out there, so it offers a wealth of examples (plus it’s also close to our hearts). OA
Airline safety management: enhance your SMS with IOblend
Today we are looking at the data aspect of flight safety management in the aviation industry.
Unlock new capabilities with real time ACARS data
In this short article we are looking at one of the key data sources for the aviation industry – ACARS – and how IOblend helps to unlock new analytical capabilities from it.
Time to automate your airline’s DOC data
How to automate Direct Operating Cost (DOC) data collection, processing and serving with IOblend.
Automate airline fuel data collection & management
Collecting and managing airline fuel data is complex and time consuming. IOblend can greatly streamline the process and enable real-time decisioning.
The Data Mesh Gotchas!
I think most practitioners in the data world would agree that the core data mesh principles of decentralisation to improve data enablement are sound. Originally penned by Zhamak Dehghani, Data Mesh architecture is attracting a lot of attention, and rightly so. However, there is a growing concern in the data industry regarding how the data
