Lead Cloud Data Engineer

We are looking for an accomplished Lead Data Engineer to drive the architectural design, development, and optimization of our enterprise-scale data ecosystem. In this senior role, you will spearhead the build-out of highperformance data pipelines, enforce rigorous data quality and validation frameworks, and ensure endtoend reliability, scalability, and integrity of data flows across the organization. You will play a pivotal role in shaping our data engineering strategy, enabling advanced analytics and missioncritical, datadriven decision-making within our insurance-focused business domains.

• 8-12 years experience on Data Engineering role working with Databricks & Azure Cloud technologies.
• Bachelor's degree in computer science, Information Technology, or related field.
• Strong proficiency in PySpark, Python, SQL.
• Strong experience in data modeling, ETL/ELT pipeline development, and automation
• Hands-on experience with performance tuning of data pipelines and workflows
• Proficient in working on Azure cloud components Azure Data Factory, Azure DataBricks, Azure Data Lake etc.
• Experience with data modeling, ETL processes, Delta Lake and data warehousing.
• Experience on Delta Live Tables, Autoloader & Unity Catalog.
• Preferred - Knowledge of the insurance industry and its data requirements.
• Strong analytical skills with the ability to collect, organize, analyze, and disseminate significant amounts of information with attention to detail and accuracy.
• Excellent communication and problem-solving skills to work effectively with diverse teams
• Excellent problem-solving skills and ability to work under tight deadlines.

• Lead the design and implementation of scalable data architectures on Azure, leveraging Databricks, Delta Lake, and related Azure data services.
• Build and optimize highvolume ETL/ELT pipelines using PySpark, SQL, Databricks Workflows, and Delta Live Tables.
• Define and enforce best practices for data engineering across notebooks, jobs, CI/CD, Unity Catalog, security, and workspace governance.
• Integrate and orchestrate data pipelines using Azure Data Factory, Azure Synapse pipelines, or Azure Databricks Workflows.
• Drive performance tuning for Spark jobs, cluster configurations, and data storage layers to balance speed and cost efficiency.
• Implement robust data quality and validation frameworks using tools like Great Expectations, DLT expectations, or custom PySpark checks.
• Ensure compliance, governance, and lineage tracking through Unity Catalog, Purview integration, and RBAC/ABAC policies.
• Architect endtoend data solutions supporting analytics, ML, actuarial models, and business-critical reporting workloads.
• Evaluate new Databricks features, MLflow enhancements, Photon execution, serverless compute, and recommend adoption strategies.
• Familiar with working on Agile methodologies - scrum, sprint planning, backlog refinement etc.