Lead Data Engineer

A. Data Engineering & Pipeline Development

• Design, develop, and optimize large-scale ETL/ELT pipelines using PySpark on Databricks, processing structured and unstructured data at scale.
• Build and maintain Lakehouse architecture (Bronze/Silver/Gold medallion layers) using Delta Lake, ensuring reliability, scalability, and schema evolution support.
• Develop reusable, parameterized, metadata-driven pipeline frameworks for ingestion, transformation, and curation of data from diverse source systems.
• Optimize Spark jobs for performance and cost (partitioning, caching, cluster sizing/auto-scaling, Photon engine, Z-ordering, file compaction).
• Implement data quality checks, validation rules, and monitoring/alerting to ensure pipeline reliability and data trust.

B. Databricks Platform & Cloud Engineering

• Configure and manage Databricks workspaces, clusters, jobs, and workflows; tune cluster policies for cost and performance.
• Implement data governance, access control, and lineage using Unity Catalog.
• Integrate Databricks with cloud services such as Azure Data Lake Storage (ADLS Gen2), Azure Data Factory, Event Hub/Kafka, and Key Vault (or AWS S3/Glue/EMR equivalents).
• Build and maintain CI/CD pipelines for Databricks notebooks, jobs, and Delta Live Tables using Azure DevOps/GitHub Actions and Databricks Repos.

C. Data Modeling, Architecture & Governance

• Design and maintain data models (dimensional, medallion, canonical) to support analytics, reporting, and downstream consumption.
• Partner with Data Architects on target-state data platform design, migration strategy, and platform standards.
• Ensure adherence to data governance, security, and privacy requirements (data masking, encryption, access controls, regulatory compliance).

D. Leadership & Collaboration

• Lead and mentor a team of data engineers; conduct code reviews and enforce engineering best practices and coding standards.
• Partner with Business Analysts, Data Scientists, and Product Owners to translate requirements into scalable technical solutions.
• Participate in Agile ceremonies, provide effort estimates, and manage delivery timelines and technical risks.

Tools & Environment (Typical)

• Big Data & Processing: PySpark, Databricks, Delta Lake, Spark SQL, Apache Airflow, Delta Live Tables.
• Cloud: Azure (ADLS Gen2, Data Factory, Synapse, Event Hub, Key Vault) or AWS (S3, Glue, EMR).
• Languages: Python, SQL; Scala (nice to have).
• DevOps: Git, Azure DevOps/GitHub Actions, Databricks CLI/API, Terraform (nice to have).
• Orchestration & Monitoring: Databricks Workflows, Airflow, Prometheus/Grafana, Datadog.