Azure Data Engineer

This role is for one of Weekday's clients
Salary range: Rs 400000 - Rs 2600000 (ie INR 4 - 26 LPA)

Min Experience: 6+ years
Location: Bengaluru, Hyderabad, Chennai, Tamil Nadu, India
JobType: full-time

We are looking for an experienced Azure Data Engineer with 6-15 years of hands-on experience in designing, developing, and maintaining scalable data engineering solutions on Microsoft Azure. The ideal candidate will have strong expertise in Azure Data Factory, Azure Databricks, PySpark/Scala, Azure SQL, ADLS Gen2, and CI/CD pipelines.

You will be responsible for building robust data pipelines, developing data transformation workflows, integrating cloud-based data services, and ensuring data platforms are secure, scalable, reliable, and optimized for performance.

Requirements

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines using Azure Data Factory.
• Build and optimize large-scale data processing solutions using Azure Databricks and PySpark/Scala.
• Develop data ingestion and transformation workflows across structured and unstructured data sources.
• Work extensively with Azure Data Lake Storage Gen2 (ADLS Gen2) for data storage, organization, and processing.
• Develop and optimize solutions using Azure SQL, T-SQL, and Stored Procedures.
• Design and implement integrations and event-driven workflows using Azure Logic Apps.
• Implement secure access and credential management using Azure Key Vault.
• Create and maintain CI/CD pipelines for data engineering applications, notebooks, workflows, and infrastructure.
• Monitor data pipelines and troubleshoot performance, reliability, and production issues.
• Collaborate with data architects, software engineers, analysts, and business stakeholders to translate requirements into scalable technical solutions.
• Follow best practices for data security, governance, performance optimization, and cloud architecture.
• Participate in code reviews, technical documentation, deployment planning, and production support activities.

Must-Have Skills

• 6-15 years of professional experience in data engineering, with significant experience working on Microsoft Azure.
• Strong hands-on expertise with Azure Data Factory (ADF).
• Strong experience with Azure Databricks.
• Proficiency in PySpark and/or Scala for large-scale data processing.
• Strong knowledge of Azure Data Lake Storage Gen2 (ADLS Gen2).
• Hands-on experience with Azure SQL and T-SQL.
• Strong experience developing and optimizing Stored Procedures.
• Experience designing and implementing CI/CD pipelines for data engineering workloads.
• Practical experience with Azure Logic Apps and cloud-based integrations.
• Experience using Azure Key Vault for secrets, credentials, and secure configuration management.
• Strong understanding of data engineering principles, ETL/ELT architecture, data transformation, and distributed processing.

Good-to-Have Skills

• Experience with Azure Synapse Analytics.
• Hands-on experience with Azure DevOps.
• Familiarity with Azure Log Analytics and cloud monitoring solutions.
• Experience with infrastructure creation and automation using Bicep, Terraform, or Ansible.
• Understanding of Infrastructure as Code (IaC) practices.
• Experience working in Agile/Scrum development environments.
• Knowledge of data security, governance, and Azure cloud best practices.