Build production data pipelines without rebuilding the production plumbing.
IOblend gives data engineers a metadata-driven development and execution layer on Apache Spark. Build batch, CDC and streaming pipelines with visual composition, SQL and Python, then test, debug, version, validate and run them with production DataOps controls already attached.
Use low-code where it removes repetition. Use SQL and Python where the logic deserves code. IOblend is not a no-code replacement for engineering judgement.
Design, validate and operate the same pipeline model from desktop to cluster.
The useful unit is not a drag-and-drop diagram. It is a versioned production pipeline with explicit dependencies, executable logic, testable components, runtime configuration and DataOps behaviour that survives deployment.
Portable JSON playbooks separate pipeline intent from runtime infrastructure.
IOblend stores the pipeline as metadata: components, dependencies, configuration, transformation behaviour and operational controls. That makes the graph portable, versionable and reusable instead of binding the engineering logic to one visual session or one target platform.
Illustrative playbook metadata
Change the arrival pattern without rebuilding the entire engineering stack.
IOblend uses a Kappa-style architecture so batch, database changes and event streams can participate in the same transformation and DataOps model. That matters when a pipeline needs a static reference table, live events and a CDC feed in the same logical flow.
Three arrival patterns, one transform graph
Window, deduplicate, upsert and handle late events as part of the pipeline.
Real-time engineering gets difficult when the flow becomes stateful. IOblend supports production patterns such as windowed processing, chained aggregations, deduplication, SCD logic, real-time upserts and late-arriving data while keeping the state behaviour visible in the same pipeline model.
Illustrative event-time window
Capture database changes, transform them in flight and write the result where it belongs.
IOblend supports hybrid CDC approaches including log-based, trigger-based and query-based patterns. Engineers can apply transforms, quality logic and schema controls to the change stream before upserting into a lakehouse, database, application or synchronised target.
Illustrative change sequence
lsn:8812lsn:8813lsn:8814lsn:8815lsn:8816Treat schema evolution as a controlled change, not a 02:00 production surprise.
IOblend can generate and version schemas, validate incoming records against expected contracts and isolate records that violate the contract. The goal is to make structural change explicit while allowing healthy records to continue where the pipeline policy permits it.
Illustrative schema diff
LONGSTRINGSTRINGLONGINT?STRINGTest-as-you-build, then treat the playbook like software.
IOblend validates the pipeline during construction. Components can be run and inspected before deployment, invalid pipeline logic is blocked, and SQL or Python syntax problems are surfaced before execution. Pipeline versions are stored and the JSON playbook can participate in normal repository and deployment practices.
Integrated validation path
Carry lineage and exception context through the transformation path.
Pipeline-level logs tell you that a job failed. Record-level lineage helps answer the harder questions: which record changed, which component changed it, what source it came from and why it was routed to an exception path.
Illustrative record trace
The deployment boundary should not change the pipeline logic.
Developer Edition installs the Designer and a local IOblend Engine with a local Spark environment. Enterprise Edition uses a remote IOblend Engine packaged for customer cloud or on-premises Spark infrastructure. The Designer can connect to local or remote engines for development and testing.
Promotion path
Use model or agent logic as a controlled pipeline component when the dataflow needs it.
IOblend can embed Python and Agentic AI logic inside the ETL path for tasks such as extracting fields from documents, classifying content or validating unstructured inputs. The output can then pass through the same structured quality, quarantine and lineage model as the rest of the pipeline.
Controlled AI step inside the graph
Start locally, follow the pipeline tutorials, then move into production behaviour.
The Developer Edition includes the Designer, local Engine and local Spark environment. The documentation already provides a practical sequence from installation and run parameters through static pipelines, streaming, event management and JDBC sinks.
IOblend should fit your engineering practices, not pretend the rest of the ecosystem does not exist.
Work with your existing stack and augment where you have gaps and performance issues. Use your preferred schedulers, ontology and observability tools as needed.
What IOblend actually changes in the engineering workflow.
These answers are written for engineers evaluating the product, not for a generic platform comparison.
What is IOblend for a data engineer?
IOblend is a metadata-driven data integration and DataOps development layer on Apache Spark. Engineers use the Designer, SQL and Python to build batch, CDC and streaming pipelines, then execute those pipelines with built-in quality, schema, lineage, state and exception controls.
Does IOblend generate Apache Spark pipelines?
Yes. The IOblend Engine turns the playbook definition into distributed Spark execution so engineers do not have to hand-build the surrounding Spark application for every production pipeline.
Is IOblend no-code?
No. IOblend supports low-code visual composition to remove repetitive engineering, while SQL and Python remain available for custom transformation and processing logic.
Can IOblend handle batch and streaming in the same pipeline?
Yes. Static, batch, CDC and streaming inputs can participate in the same pipeline model, including joins between live events and reference data.
How does IOblend handle Change Data Capture?
IOblend supports hybrid CDC patterns including log, trigger and query approaches, then allows engineers to transform and validate change events before writing target state.
How does IOblend handle duplicate events and idempotency?
Stateful deduplication and upsert patterns can be used so retries or duplicate events do not create repeated target outcomes.
How are schema drift and data contracts handled?
IOblend can generate and version schemas, validate data against defined contracts and isolate invalid records while valid data continues where the pipeline policy allows it.
What does test-as-you-build mean?
Pipeline components can be executed and inspected during construction. IOblend also validates pipeline logic and surfaces SQL or Python syntax errors before the pipeline is treated as executable.
Can IOblend pipelines be version controlled?
Yes. Pipeline definitions are stored as JSON metadata files and can be stored in a normal source-code repository. IOblend also stores pipeline versions for comparison.
How does visual debugging work?
Engineers can inspect components and intermediate data through the development environment, which reduces the need to infer the entire pipeline state only from distributed runtime logs.
What is record-level lineage in IOblend?
IOblend attaches lineage context at record level as data moves through the flow, allowing engineers to trace source, transformation and exception context for individual records.
Where does IOblend run?
Developer Edition runs the Designer and local Engine on the developer machine. Enterprise Edition can run a remote Engine on customer-controlled cloud, on-premises or hybrid Spark infrastructure.
Can I use Airflow with IOblend?
Yes. Enterprise run files can be scheduled by external scheduling software such as Apache Airflow, allowing IOblend to participate in existing orchestration practices.
Can I try IOblend before an enterprise deployment?
Yes. Developer Edition is available for local development and the documentation includes installation, first-pipeline, streaming and JDBC tutorials.
Start with the source, the state problem and the production controls you are tired of rebuilding.
Use Developer Edition to build locally, or bring us a representative CDC, streaming, migration or batch pipeline and we can work through how the playbook, Engine and DataOps model would apply.