Databricks COE Lead

Key Role & Responsibilities

CoE Leadership & Strategy

• Establish and lead the Databricks CoE: define vision, roadmap, governance, and operating model.
• Build reusable frameworks, accelerators, and solution blueprints for scalable lakehouse adoption.
• Drive standardisation of architecture patterns, coding practices, and delivery methodologies.
• Partner with sales, solutioning, and delivery teams to enable Databricks-led transformations across clients.

Architecture & Solutioning

• Lead end-to-end Databricks Lakehouse architecture (bronze/silver/gold layers, medallion architecture).
• Translate business requirements into customised data and AI solutions (batch, streaming, ML, GenAI).
• Define best practices for ingestion, transformation, orchestration, and consumption layers.
• Architect enterprise-scale solutions integrating Databricks with cloud ecosystems (Azure/AWS/GCP).

Data Engineering & Platform Enablement

• Design and implement scalable data pipelines using Spark, Delta Lake, and Databricks workflows.
• Establish CI/CD, DevOps, and DataOps practices for Databricks platforms.
• Define data modelling approaches (lakehouse models, marts, feature engineering layers).
• Build accelerators for ingestion, transformation, and ML/GenAI pipelines.

AI / GenAI & Advanced Analytics Enablement

• Drive adoption of ML, MLOps, and GenAI capabilities on Databricks (MLflow, Feature Store, Mosaic AI).
• Enable use cases such as intelligent automation, conversational analytics, and agentic workflows.
• Define patterns for integrating LLMs, vector databases, and knowledge retrieval architectures.

Governance, Security & Performance

• Define governance models including data lineage, access control, and compliance frameworks.
• Implement security best practices (RBAC, IAM, encryption, secrets management).
• Establish performance and cost optimisation strategies across compute and storage layers.

Stakeholder Management & Delivery Governance

• Lead client engagements: workshops, architecture reviews, solution roadmaps, and advisory.
• Collaborate with business stakeholders to ensure solutions drive measurable outcomes.
• Provide leadership across large delivery programmes and ensure quality and scalability.

People & Capability Building

• Build and mentor a high-performing Databricks engineering and architecture team.
• Conduct technical reviews, capability assessments, and skill development initiatives.
• Drive capability building across Data Engineering, ML, and GenAI domains.

Must Have

• 12+ years of experience with a strong background in Data Engineering, Big Data, and platform architecture.
• 5+ years of hands-on experience with Databricks (Spark, Delta Lake, Lakehouse architecture).
• Proven experience in leading CoE / practice / large-scale data platform initiatives.
• Strong expertise in PySpark/Scala/SQL, distributed data processing, and large-scale pipeline design.
• Experience with batch and streaming architectures (Kafka, Event Hubs, Structured Streaming, etc.).
• Strong understanding of ML/AI ecosystem on Databricks (MLflow, Feature Store, MLOps).
• Demonstrated customer-facing experience: solutioning, stakeholder discussions, and executive communication.
• Strong leadership and mentoring capabilities.

Good to Have

• Experience with GenAI / LLM ecosystems (OpenAI, Claude, vector DBs, RAG architectures).
• Exposure to Databricks Mosaic AI, Unity Catalog, Delta Live Tables.
• Cloud expertise on Azure/AWS/GCP (IAM, networking, storage, monitoring).
• Knowledge of tools such as dbt, Airflow, ADF, Terraform.
• Domain experience in BFSI / Insurance / Healthcare / Analytics services.