Databricks, Python
Description
We are looking for a Senior Databricks Engineer to design, develop, and optimize scalable data solutions using Databricks for large enterprise data platforms. The role requires strong hands on expertise in Spark, PySpark, SQL, and cloud ecosystems, with ownership of end to end data pipelines and mentoring of junior team members in an offshore delivery model.
Key Responsibilities
Data Engineering & Development
Design and develop scalable data pipelines using Azure Databricks / Databricks on GCP
Build and optimize ETL/ELT pipelines using Spark, PySpark, and SQL
Implement Delta Lake for ACID transactions, data versioning, and performance optimization
Handle structured and semi structured data (JSON, Parquet, Avro)
Cloud & Integration
Integrate Databricks with Cloud Storage, ADLS / GCS, BigQuery, Snowflake
Implement secure data access using IAM, service principals, secrets
Work with CI/CD pipelines for Databricks deployments
Collaborate with Data Scientists, BI teams, and downstream consumers
Delivery & Offshore Ownership
Own offshore delivery for Databricks workstreams
Provide technical guidance and code reviews to junior engineers
Work closely with onshore counterparts and clients across time zones
Participate in sprint planning, backlog grooming, and technical estimations
Required Skills
Technical Skills
6-10 years of experience in Data Engineering
3+ years of strong hands on experience with Databricks
Strong expertise in Apache Spark, PySpark, Spark SQL
Experience with Delta Lake
Proficiency in SQL (query optimization, complex joins)
Experience with Azure or GCP cloud platforms
Experience with Git, CI/CD, and DevOps practices
Preferred Skills
Experience with Azure Data Lake / BigQuery / Snowflake
Exposure to streaming frameworks (Kafka, Spark Structured Streaming)
Knowledge of data governance, data quality, and security
Experience supporting production workloads
Soft Skills
Strong analytical and problem solving skills
Ability to own modules independently in an offshore model
Good communication skills for client and onshore coordination
Mentoring mindset and ability to guide junior resources
Mandatory skills* Databricks, Python
