Design and develop scalable ETL/ELT pipelines using Azure Databricks (Spark) to process large-scale structured and unstructured data.
Build and maintain data lakes, data warehouses, and other analytics platforms using Azure services.
Optimize data workflows for performance, cost, and reliability.
Integrate Databricks with Azure Data Factory, ADLS Gen2, Synapse, Event Hub, and other Azure components.
Write PySpark or Scala code within Databricks notebooks to transform and process data.
Implement and enforce data governance, quality, and security policies.
Collaborate with data scientists, analysts, and business stakeholders to understand data needs and deliver solutions.
Monitor and troubleshoot production data pipelines and jobs.
