Azure Databricks + PySpark

Job Summary
Cognizant is looking for experienced Azure Databricks + PySpark Developers to join its Data Engineering practice. The ideal candidate should have strong expertise in Azure cloud technologies, Databricks, PySpark, and SQL, with hands-on experience building scalable data pipelines and modern data lake solutions.
Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Azure Databricks and PySpark.
• Build and optimize data processing solutions for large-scale datasets.
• Develop data pipelines using Azure Data Factory and Azure Data Lake Storage (ADLS).
• Write optimized Spark SQL and SQL queries for high-performance data processing.
• Implement Delta Lake architecture and optimize Databricks workloads.
• Collaborate with business analysts, data scientists, and application teams to deliver data solutions.
• Monitor, troubleshoot, and optimize production data pipelines.
• Ensure data quality, security, governance, and compliance.
• Participate in code reviews and follow Agile development practices.

Required Skills

• Strong hands-on experience with Microsoft Azure.
• Expertise in Azure Databricks.
• Strong experience with PySpark and Apache Spark.
• Advanced SQL programming.
• Experience with Azure Data Factory (ADF).
• Experience with Azure Data Lake Storage (ADLS Gen2).
• Knowledge of Delta Lake.
• Strong Python programming skills.
• Experience designing ETL/ELT pipelines.
• Good understanding of data warehousing concepts.

Preferred Skills

• Azure Synapse Analytics.
• Azure Event Hubs or Azure Service Bus.
• Azure DevOps and CI/CD.
• Git version control.
• Docker and Kubernetes.
• Snowflake experience.
• Terraform or Infrastructure as Code.

Qualifications

• Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.
• 5.5–9 years of relevant experience in Data Engineering.
• Strong communication and stakeholder management skills.