Azure Databricks - Technical Lead-Data Engg

Area(s) of responsibility

Job Title: Senior Data Engineer / Senior Developer (Azure Databricks)

Location: Noida / Pune

Experience: 9+ years overall, with 5+ years handson in Azure Databricks & Data Engineering

Role Overview - Senior Data Engineer / Senior Developer with strong handson expertise in Azure Databricks, Azure SQL, and Azure Data Integration services. The role requires deep technical ownership, ability to design and build scalable data pipelines, and support modern data architectures while working closely with architects, product owners, and downstream consumers.

Key Responsibilities

Data Engineering & Development

• Design, develop, and maintain endtoend batch and nearrealtime data pipelines using Azure Databricks (PySpark/SQL).
• Implement scalable data processing using Medallion Architecture (Bronze / Silver / Gold layers).
• Develop and optimize complex TSQL queries, stored procedures, and schemas on Azure SQL Database.
• Perform performance tuning, query optimization, and data model refinement across Azure SQL and Databricks.
• Design and monitor complex ETL / ELT pipelines using Azure Data Factory (ADF).
• Build reusable PySpark frameworks for ingestion, transformation, and validation.
• Implement operational logging, monitoring, and data quality checks.
• Analyze existing legacy data architectures and Design and implement mirrored or modernized equivalents on Azure Databricks.

Security, Governance & Compliance

• Experience on secure data access using: Azure Key Vault, Managed Identities, Service Principals and VNETintegrated services
• Follow enterprise data security, RBAC, and governance standards.
• Ensure compliance with data privacy and audit requirements.

DevOps & Engineering Discipline

• Use Git / Azure DevOps for: Version control, Pull requests & code reviews CI/CD pipelines for data workloads

Required Skills & Qualifications:

• Handson experience on - Azure SQL Database, Advanced TSQL, Schema design, Query & performance tuning , Azure Data Factory (ADF), Complex ETL / ELT pipeline design, Monitoring and troubleshooting, Azure Databricks, PySpark & SQL, Data Modeling, Medallion Architecture (Bronze / Silver / Gold), Dimensional and analytical models, Data Integration

Nice to Have

• Experience analyzing or migrating legacy data platforms (e.g., Rosettastyle architectures).
• Exposure to Insurance domain data (Claims, Policy, Underwriting, Reinsurance).
• Experience with Delta Lake optimizations and cost management.
• Azure / Databricks certifications.