Informatica Cloud Data Integration: Scaling Spark on Kubernetes

Architectural Evolution of Cloud Data Integration (CDI)

The Informatica Cloud Data Integration (CDI) platform has evolved from a traditional single-node engine to a high-scale, distributed architecture built on Spark and Kubernetes. Supporting over 5,500 corporate clients and 250,000 daily tasks, the platform required a transition to handle massive, multi-terabyte datasets while maintaining backward compatibility.

Solving for Scale and Compatibility

Transitioning to a distributed model presented a significant engineering constraint: preserving existing graphical data mappings. The team achieved this by separating the design-time abstraction from the execution runtime:

• Abstraction Layer: Engineers define data pipelines via graphical interfaces.
• Execution Engine: The runtime compiles these mappings into distributed Spark execution plans.
• Spark++: Since open-source Spark lacked necessary enterprise features, the team extended it to include lineage tracking, robust connector support, and internal governance frameworks.

Reliability Principles

To function as a mission-critical backbone, the architecture enforces strict reliability standards to achieve a 99.9% control-plane availability target:

• Data Integrity: Row-level execution tracking allows the isolation of failed records without failing the entire pipeline.
• Tenant Isolation: Compute clusters are deployed within strict Virtual Private Cloud (VPC) boundaries with ephemeral nodes that terminate after task completion.
• Infrastructure Resilience: Deployment across multiple availability zones ensures service continuity during localized outages.

FinOps and Infrastructure Optimization

Scaling distributed systems often leads to ballooning infrastructure costs. CDI mitigates this using automated FinOps systems that analyze workloads to determine optimal cluster configurations:

1. Cluster Lifecycle Manager: Predicts job demand to scale clusters up or down automatically.
2. Cluster Tuner: Dynamically selects instance types, storage, and networking configurations based on the specific job requirements.
3. Job Tuner: Adjusts Spark runtime parameters (CPU and memory allocation) based on historical performance data.

This automated approach has demonstrated a 1.65x reduction in infrastructure costs compared to manual cluster management.

Maintaining Throughput

To ensure consistent performance across hundreds of thousands of jobs, the architecture utilizes a decoupled design:

• Control Plane: Manages orchestration and scheduling independent of the data processing layer.
• Data Plane: Executes Spark workloads in isolated environments, preventing compute-heavy jobs from starving the control plane of resources.

Key Takeaways

• Decoupling: Separate orchestration from execution to ensure control plane stability during heavy data throughput.
• Logical Abstraction: Retain user-defined mappings while swapping the underlying runtime to Spark to support legacy pipelines at scale.
• FinOps Automation: Automate cluster lifecycle, instance selection, and runtime tuning to balance performance against cloud infrastructure spend.
• Tenant Isolation: Utilize VPC boundaries and ephemeral nodes to maintain security and reliability in multi-tenant cloud environments.