SDE III
SDE III/IV - Data Platform Engineering
Mission
Architect a self-serve Data "Platform-as-a-Product" powering InMobi's global-scale data ecosystem. Integrate OSS tools, proprietary services, and Cloud/SaaS into unified infrastructure. Requires deep data engineering
expertise (batch/streaming pipelines, data modeling, query optimization, governance) combined with platform engineering to build production-grade solutions for data engineers and analysts.
Core Responsibilities
Design & Development - Bridge OSS tools (Spark, Flink, Airflow, Iceberg), internal services, and cloud offerings into cohesive data platform infrastructure. Build intuitive platform integrations enabling push-button data workflows.
Scale Engineering - Operate distributed systems processing petabytes of data daily. Own multi-region Kubernetes infrastructure with elastic scalability and fault tolerance.
Performance Optimization - Optimize compute utilization (Spark/Flink clusters, Velox/Gluten acceleration) for large-scale batch and real-time streaming with sub-second latency.
Observability & Data Quality - Build comprehensive telemetry (metrics, logs,traces) and data quality frameworks for 24/7 uptime. Enforce SLAs/SLOs with automated incident response and data validation.
Required Skills & Experience (Must-Have)
7-10 years building, optimizing, and operating production data platforms
Deep data engineering fundamentals: data modeling, partitioning strategies, query optimization
Distributed compute: Spark (PySpark/Scala), Flink streaming, performance tuning at petabyte scale
Data lake architecture: Iceberg table format, Polaris catalog, schema evolution, time travel
Orchestration: Airflow DAG development, dependency management, SLA monitoring
Data transformation: DBT modeling, testing, documentation, incremental builds
Data quality: Great Expectations, dqueue validation frameworks, drift detection
Query acceleration: Velox, Gluten integration, columnar formats (Parquet, ORC)
Data governance: OpenMetadata catalog, lineage tracking, access control
Kubernetes platform development: operators (Spark/Flink), Yunikorn scheduler, multi-tenancy, autoscaling
Cloud infrastructure: GKE multi-region clusters, GCS object storage, hybrid cloud/on-prem architecture
Programming: Python, PySpark, Scala for data pipelines and platform tooling
IaC: Terraform, Helm, GitOps for reproducible deployments
CI/CD: Automated testing, deployment pipelines for data platform components
Good-to-Have
Experience building cloud data platform / control plane development
Advanced observability: Prometheus/Grafana, Loki, Firehydrant integration
Real-time streaming: Kafka integration, exactly-once semantics, backpressure handling
Cost optimization: Resource allocation, query optimization, storage tiering strategies
