Design and build scalable data pipelines and ETL/ELT workflows using Azure Databricks (Apache Spark).
Collaborate with data architects and analysts to transform business requirements into technical solutions.
Optimize and maintain data processing performance, including tuning Spark jobs and managing clusters.
Integrate data from various structured and unstructured sources (Azure Data Lake, Blob Storage, SQL DB, etc.).
Implement best practices for code versioning, CI/CD pipelines, and data security.
Work with Azure services such as Data Factory, Data Lake, Synapse Analytics, and Event Hubs.
Troubleshoot data pipeline issues and provide production support as needed.
Develop notebooks in Python/Scala/SQL for data processing and analytics.
