Design and implement scalable data pipelines and ETL/ELT workflows using Apache Spark on Databricks.
Develop, deploy, and maintain Delta Lake tables and optimize for performance.
Integrate data from various structured and unstructured sources (e.g., SQL, NoSQL, APIs, files).
Collaborate with Data Scientists, Analysts, and Business stakeholders to understand data requirements.
Monitor data pipelines and troubleshoot performance and reliability issues.
Leverage Databricks features such as Jobs, Workflows, Delta Live Tables, Unity Catalog, and MLflow.
Ensure data quality, security, governance, and compliance.
Work with cloud-native tools on Azure (Azure Data Lake, ADLS, ADF) or AWS (S3, Glue, Lambda, etc
