Design, develop, and optimize large-scale data pipelines using PySpark in Azure Databricks.
Build scalable ETL processes for structured and unstructured data from various sources.
Work closely with data architects and business stakeholders to understand data requirements and deliver high-quality data solutions.
Implement and manage data lake storage layers using Delta Lake.
Optimize Spark jobs for performance and cost-efficiency.
Integrate Azure services such as Azure Data Lake Storage (ADLS), Azure Data Factory (ADF), Azure Synapse, and Azure Key Vault.
Ensure data quality, integrity, security, and governance in all data processes.
Troubleshoot and resolve production issues related to data pipelines and processing.
Participate in code reviews, best practice sharing, and knowledge sessions with the team.
