SDE 2

SDE II - Data Platform - Lucknow

Mission

Contribute to building a self-serve Data Platform-as-a-Product enabling scalable, reliable, and easy-to-use data infrastructure across batch and streaming workloads.\

Core Responsibilities

• Build and maintain scalable data pipelines (batch & streaming) using Spark/Flink

• Develop integrations across Airflow, Iceberg, and cloud platforms

• Apply best practices in data modeling, partitioning, and query optimization

• Support Kubernetes-based data platform workloads and troubleshoot issues

• Optimize pipelines for performance, cost, and reliability

• Implement monitoring, logging, and data quality checks

• Participate in on-call and incident resolution

Required Skills

• 3-6 years in data/platform engineering

• Strong fundamentals: SQL, data modeling, partitioning, optimization

• Hands-on with Spark (PySpark/Scala); exposure to Flink is a plus

• Experience with Airflow (or similar orchestration tools)

• Familiarity with data lake formats (Iceberg/Delta/Hudi)

• Working knowledge of Kubernetes & cloud (GCP/AWS/Azure)

• Programming: Python (must-have), Scala (good-to-have)

• Exposure to CI/CD, Git, and IaC (Terraform/Helm basics)

Good to Have

• Kafka / streaming systems experience

• DBT, Great Expectations, or similar tools

• Observability tools (Prometheus, Grafana)

• Metadata/catalog tools (OpenMetadata/DataHub)

Success Metrics

• Delivers reliable, production-ready pipelines

• Improves performance and platform usability

• Strong ownership, debugging, and collaboration skills