Azure Data Engineer

Job Summary:

We are looking for a highly skilled and motivated Azure Databricks Developer to join our data engineering team. The successful candidate will be responsible for designing, developing, and optimizing large-scale data processing pipelines using Azure Databricks, Apache Spark, and other tools in the Azure Data Services ecosystem. You will collaborate with data scientists, analysts, and business stakeholders to deliver reliable and scalable data solutions.

Key Responsibilities:

• Develop and maintain data pipelines and ETL/ELT processes using Azure Databricks (Spark).
• Design and implement solutions for batch and streaming data using tools such as Delta Lake, Structured Streaming, and Kafka/Event Hubs.
• Collaborate with architects and analysts to translate business requirements into scalable data solutions.
• Optimize performance of Spark jobs and manage cluster configurations.
• Integrate Databricks with other Azure services such as Azure Data Lake Storage (ADLS), Azure Synapse Analytics, Azure Data Factory (ADF), Azure SQL DB, and Azure Key Vault.
• Implement data quality, monitoring, and logging processes.
• Maintain CI/CD pipelines using DevOps tools (e.g., Azure DevOps, Git).
• Ensure data security, compliance, and governance according to company standards.
• Participate in code reviews and knowledge-sharing sessions.

Required Qualifications:

• Strong hands-on experience with Azure Databricks and Apache Spark (PySpark/Scala).
• Proficiency in programming with Python (or Scala) for data processing.
• Solid understanding of Azure cloud services and modern data architectures (e.g., Lakehouse, Delta Lake).
• Experience with Azure Data Factory, Azure Data Lake Gen2, and Azure Synapse Analytics.
• Good knowledge of SQL and working with structured and unstructured data.
• Experience with Git-based development and CI/CD automation.
• Bachelor's degree in Computer Science, Engineering, or related field.

Preferred Qualifications:

• Microsoft Certified: Azure Data Engineer Associate or similar.
• Knowledge of machine learning workflows and integration with MLflow or Azure Machine Learning.
• Experience in big data performance tuning and cluster management.
• Familiarity with Databricks REST API and automation scripting.
• Understanding of data governance, data lineage, and metadata management.