Databricks Engineer

Experience: 6+ Years
Location: : All USI locations i.e, Hyderabad, Bengaluru, Chennai, Gurgaon, Mumbai, Pune & Kolkata.
Salary: As per market standards

Job Summary

We are looking for an experienced Databricks Engineer with strong expertise in designing, developing, and optimizing scalable data engineering solutions on the Databricks platform. The ideal candidate should have hands-on experience with Apache Spark, Delta Lake, Azure Databricks/AWS Databricks, and modern data pipelines. The candidate should be capable of working with large-scale data processing, ETL development, and cloud-based analytics solutions.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks.
• Develop ETL/ELT workflows using PySpark and Spark SQL.
• Build and optimize data ingestion pipelines from multiple structured and unstructured data sources.
• Implement Delta Lake architecture and optimize data storage and processing.
• Work with Azure Data Factory (ADF) or equivalent orchestration tools for workflow scheduling.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Collaborate with business stakeholders, architects, and data analysts to understand data requirements.
• Perform code reviews and follow best practices for data engineering.
• Troubleshoot production issues and provide ongoing support.
• Ensure data quality, governance, and security standards are followed.

Required Skills

• 5+ years of experience in Data Engineering.
• Strong hands-on experience with Databricks.
• Expertise in Apache Spark (PySpark/Spark SQL).
• Strong SQL programming skills.
• Experience with Delta Lake.
• Experience in Azure Databricks or AWS Databricks.
• Experience with Azure Data Factory (ADF), Airflow, or similar orchestration tools.
• Hands-on experience with Git and CI/CD pipelines.
• Good understanding of data warehousing concepts and ETL processes.
• Experience working with cloud platforms (Azure/AWS).

Preferred Skills

• Experience with Unity Catalog.
• Knowledge of Azure Data Lake Storage (ADLS Gen2) or Amazon S3.
• Experience with Kafka or Event Hub.
• Exposure to Azure Synapse, Snowflake, or Redshift.
• Knowledge of Data Governance and Data Quality frameworks.
• Experience in Agile/Scrum methodology.

Nice to Have

• Azure Data Engineer Associate Certification (DP-203).
• Databricks Certified Data Engineer Associate/Professional.
• Experience working with large enterprise clients.

Mandatory Skills

• Databricks
• PySpark
• Spark SQL
• SQL
• Delta Lake
• Azure Databricks / AWS Databricks
• ETL Development
• Azure Data Factory (ADF) / Airflow
• Data Lake
• Git

Soft Skills

• Excellent communication and stakeholder management skills.
• Strong analytical and problem-solving abilities.
• Ability to work independently and within cross-functional teams.
• Flexible to work in a fast-paced project environment.