Posted 20 May, 2026
Software Engineer 3_India
eTeam
Bangalore,Karnataka,560001
Full Time
Reference: 365_621352_26-02624
Job Title: Senior Data Engineer
Role summary
We are seeking a Data Engineer with deep experience building reliable, high-throughput data pipelines using PySpark Structured Streaming, batch PySpark, AWS, and Databricks. You will design, implement, and operate streaming and batch ETL that feed our data lake and analytics platforms, drive data quality improvements, and produce clear technical documentation for operational teams.
Key responsibilities:
Spark Structured Streaming:
Design and develop production streaming applications using PySpark Structured Streaming.
Integrate with streaming sources such as Kafka, Kinesis, Event Hub, and similar systems.
Parse and process JSON payloads at scale.
Implement serverless logic where appropriate using AWS Lambda.
Deploy and operate streaming workloads on Databricks when applicable, leveraging Databricks job scheduling and cluster management.
Batch applications:
Build and maintain batch ETL pipelines using PySpark on Databricks and/or AWS compute.
Improve data quality through validation, monitoring, and automated remediation.
Lake catalog and SQL analytics:
Manage S3 layout and partitioning to support efficient reads and cost-aware storage.
Integrate Kinesis Data Firehose for durable delivery into the data lake.
Use AWS Glue jobs and crawlers to maintain an accurate Data Catalog and support Athena queries.
Support Athena-based analysis and reporting, collaborating with Analytics on workgroups, cost controls, and governance.
Leverage Databricks features such as Delta Lake, Unity Catalog, and Databricks SQL to support governed, performant analytics and ACID-compliant pipelines.
Role summary
We are seeking a Data Engineer with deep experience building reliable, high-throughput data pipelines using PySpark Structured Streaming, batch PySpark, AWS, and Databricks. You will design, implement, and operate streaming and batch ETL that feed our data lake and analytics platforms, drive data quality improvements, and produce clear technical documentation for operational teams.
Key responsibilities:
Spark Structured Streaming:
Design and develop production streaming applications using PySpark Structured Streaming.
Integrate with streaming sources such as Kafka, Kinesis, Event Hub, and similar systems.
Parse and process JSON payloads at scale.
Implement serverless logic where appropriate using AWS Lambda.
Deploy and operate streaming workloads on Databricks when applicable, leveraging Databricks job scheduling and cluster management.
Batch applications:
Build and maintain batch ETL pipelines using PySpark on Databricks and/or AWS compute.
Improve data quality through validation, monitoring, and automated remediation.
Lake catalog and SQL analytics:
Manage S3 layout and partitioning to support efficient reads and cost-aware storage.
Integrate Kinesis Data Firehose for durable delivery into the data lake.
Use AWS Glue jobs and crawlers to maintain an accurate Data Catalog and support Athena queries.
Support Athena-based analysis and reporting, collaborating with Analytics on workgroups, cost controls, and governance.
Leverage Databricks features such as Delta Lake, Unity Catalog, and Databricks SQL to support governed, performant analytics and ACID-compliant pipelines.
