Databricks lead

We are hiring a Databricks Lead to drive a major enterprise migration program focused on transitioning large-scale datasets, ETL pipelines, and analytics workloads from AWS Glue + Redshift to AWS Databricks Lakehouse.
This role requires handson leadership in migration strategy, architecture modernization, pipeline redesign, performance optimization, and governance on Databricks.

---

Key Responsibilities

• Lead the migration of ETL pipelines from AWS Glue (PySpark) to Databricks workflows / PySpark notebooks / Delta Live Tables.
• Architect the migration of data warehouses from Amazon Redshift to Databricks Lakehouse (Delta Lake).
• Design the Medallion architecture to replace legacy Glue jobs and Redshift models.
• Lead the design and endtoend implementation of data engineering solutions using Databricks, Apache Spark, Delta Lake, and Unity Catalog.
• Redesign Redshift SQL logic (SPs, UDFs, views) into scalable Lakehouse patterns.
• Partner with solution architects, product owners, and business stakeholders to define technical requirements and solution roadmaps.
• Establish coding standards, reusable frameworks, and Databricks best practices across teams.
• Manage Databricks workspace governance, cluster policies, and cost optimization strategies.
• Oversee CI/CD automation using tools such as GitHub Actions, Azure DevOps, or Terraform.
• Lead technical troubleshooting, performance tuning, and platform-level optimizations.
• Provide mentorship, technical coaching, and performance guidance to data engineering teams.
• Ensure all solutions adhere to data governance, security, and compliance frameworks.

---

Required Skills & Experience

• 5+ years handson Databricks experience (Spark, Delta Lake, Unity Catalog).
• Proven experience migrating from AWS Glue ETL and Redshift to cloud-native platforms.
• Strong PySpark, SQL, and Spark optimization capabilities.
• Experience with Databricks-native features: Delta Lake, Delta Live Tables, Workflows, SQL Warehouses.
• Deep understanding of Lakehouse architecture and enterprise data modeling.
• Experience with AWS services: S3, Glue, Lambda, IAM, CloudWatch.
• Hands-on experience with CI/CD pipelines (GitHub Actions, Azure DevOps, or similar).
• Strong leadership, architecture thinking, and stakeholder management.

---

Nice to Have

• Knowledge of Fivetran, DBT, Airflow, or enterprise ETL tools.
• Experience with MLOps and AI/ML integration through Databricks.
• Exposure to security frameworks (SOC2, GDPR, PCI, HIPAA).
• Experience with data mesh and modern architectural patterns.
• Experience with Azure or GCP in addition to AWS
• Knowledge of DevOps practices in data engineering