Databricks Administrator

KPMG is one of the world’s leading professional services firms and the fastest growing Big Four accounting firm

About The Job:

• 2-4 Years
• We are looking for a Databricks Administrator with 2–3 years of hands-on experience to own the administration, security, and automation of our Databricks platform on AWS and Azure.
• You will manage workspaces, govern compute and access, maintain Unity Catalog, and drive infrastructure-as-code practices that keep our Lakehouse environment stable, secure, and cost-efficient.
• This is a platform administration role — not a data engineering or pipeline development position.

Essential Job Functions:
Workspace Administration:

• Administer Databricks workspaces on AWS (E2 architecture) and Azure: user and group management, entitlements, and IP access lists.
• Configure and maintain SSO (SAML 2.0) and SCIM-based automated user provisioning with Entra ID or Okta.
• Manage workspace settings, feature flags, and runtime upgrades; track LTS versions and coordinate library dependency changes.
• Handle workspace onboarding and environment promotion across dev, QA, and production.
• Tag based RBAC implementation.

• Define and enforce cluster policies, instance pool configurations, and autoscaling rules across all workspaces.
• Monitor cluster utilization, DBU consumption, and job run costs; implement optimizations and budget alerts.
• Govern spot/on-demand instance usage on AWS and reserved/spot VM strategies on Azure to manage spend.
• Perform regular rightsizing reviews and retire unused clusters, pools, and scheduled jobs.

• Set up and administer Unity Catalog: metastore configuration, catalog and schema structure, and fine-grained access controls.
• Manage data lineage visibility, audit logging, and object tagging.
• Configure and maintain external locations, storage credentials, and Delta Sharing for secure data access.
• Administer Databricks Secrets and integration with AWS Secrets Manager or Azure Key Vault.

• Set up monitoring and alerting for workspace availability, job failures, cluster errors, and DBU overruns using Datadog, Grafana, or Azure Monitor.
• Act as first responder for platform incidents; triage, resolve, and document root cause analyses.
• Conduct regular health checks and produce platform availability and cost reports for stakeholders.

• Provision and manage Databricks resources using Terraform (Databricks provider) for repeatable, version-controlled deployments.
• Maintain modular and reusable Terraform configurations across multiple workspaces and environments.

• Build and maintain CI/CD pipelines (GitHub Actions or Azure DevOps) for workspace configs, cluster policies, and permission deployments.
• Write Python scripts for operational automation: user provisioning, cost reporting, health checks, and alerting utilities.
• Maintain platform runbooks, change logs, and incident documentation.

• Manage cloud IAM, Key Vault / Secrets Manager, storage accounts (S3, ADLS Gen2), and network configurations supporting Databricks workspaces.
• Ensure adherence to cloud security best practices including least-privilege access, encryption, and network isolation.

Qualifications:

• ZS welcomes candidates with degrees in computer science, information technology or a related discipline.
• A demonstrated record of success and significant experience are required.

• Bachelor’s degree in computer science, Engineering, or a related field.
• 2–3 years of hands-on Databricks workspace administration on AWS and/or Azure.
• Strong understanding of Unity Catalog: metastore setup, permissions model, lineage, and external locations.
• Experience with SSO/SAML and SCIM-based user provisioning (Entra ID, Okta, or equivalent).
• Proficiency with Terraform for Databricks resource provisioning (Databricks provider).
• Working knowledge of CI/CD pipelines using GitHub Actions or Azure DevOps.
• Python scripting skills for automation, provisioning, and operational tooling.
• Hands-on experience with AWS (IAM, S3, Secrets Manager) and/or Azure (Entra ID, ADLS Gen2, Key Vault).
• Familiarity with monitoring and alerting tools: Datadog, Grafana, or Azure Monitor.
• Strong communication, documentation, and cross-functional collaboration skills.
• Self-motivated, detail-oriented, and comfortable managing multiple platform requests simultaneously.
• Databricks Certified Associate — Platform Administrator certification is a plus.
• Experience administering Databricks ML workspaces, MLflow tracking servers, or Mosaic AI from a platform governance perspective is desirable.
• Familiarity with Python-based AI/ML frameworks (LangChain, OpenAI SDK) for platform integration context is advantageous.