Mantle Solutions - Data Engineer

Data Engineer

Job Summary

We are seeking an experienced Azure Data Engineer to design, develop, and maintain scalable data solutions on Microsoft Azure. The ideal candidate will have strong expertise in data engineering, ETL/ELT development, cloud technologies, and big data processing frameworks. This role involves building and optimizing data pipelines, integrating data from multiple sources, and enabling analytics and reporting across the organization.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines and data integration solutions using Azure services.
• Build and optimize ETL/ELT workflows using Azure Data Factory (ADF), Azure Databricks, and other Azure-native tools.
• Develop and manage data lakes and data warehouses using Azure Data Lake Storage (ADLS), Azure Synapse Analytics, and Azure SQL.
• Implement data ingestion frameworks for batch and real-time data processing.
• Work with structured and unstructured datasets from various internal and external sources.
• Ensure data quality, integrity, security, and governance across the data platform.
• Optimize data processing jobs for performance, scalability, and cost efficiency.
• Collaborate with business analysts, data scientists, architects, and application teams to understand data requirements.
• Monitor, troubleshoot, and support production data pipelines.
• Implement CI/CD processes and DevOps best practices for data engineering solutions.
• Create and maintain technical documentation, data lineage, and operational procedures.

Required Skills & Qualifications

• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.
• 3+ years of experience in data engineering or a related role.
• Strong experience with Microsoft Azure cloud services.
• Hands-on experience with:
  • Databricks
  • Azure Data Lake Storage (ADLS Gen2)
  • Azure SQL Database
• Strong SQL development and query optimization skills.
• Experience with Python, PySpark.
• Knowledge of data modeling, data warehousing, and dimensional modeling concepts.
• Understanding of data security, governance, and access control principles.
• Familiarity with Agile development methodologies.

Preferred Qualifications

• Experience with Delta Lake and Medallion Architecture.
• Knowledge of streaming technologies such as Azure Event Hub or Azure Stream Analytics.
• Experience with Power BI integration and analytics solutions.
• Azure certifications such as:
  • Microsoft Certified: Azure Data Engineer Associate (DP-203)

Technical Skills

• Azure Data Factory (ADF)
• Azure Databricks
• PySpark
• SQL
• Python
• Delta Lake
• Azure DevOps

Soft Skills

• Strong analytical and problem-solving skills
• Excellent communication and stakeholder management
• Ability to work independently and within cross-functional teams
• Strong attention to detail and commitment to data quality

Nice to Have

• Experience with Kafka or Event Hub
• Experience with Databricks Applications
• Exposure to machine learning data pipelines
• Experience in retail, finance, healthcare, or e-commerce domains