| Roles and Responsibilities |
Design, develop, and optimize data pipelines using Azure Databricks and Apache Spark. Develop scalable ETL/ELT workflows to ingest data from various structured and unstructured sources. Work with large-scale datasets, ensuring performance and reliability in processing and transformation. Implement data lake and data warehouse solutions using Azure Data Lake Storage (ADLS), Azure Synapse, and Azure SQL. Collaborate with data scientists, analysts, and BI developers to support ML/AI models and analytics use cases. Ensure data quality, consistency, and security across all pipelines and systems. Implement CI/CD for data pipeline deployments using Azure DevOps. Monitor, troubleshoot, and optimize Databricks jobs and clusters. Write clean, scalable, and maintainable code in Python, SQL, Scala, or PySpark. |
