Data Pipeline Development: Design, develop, and maintain robust, scalable data pipelines and ETL (Extract, Transform, Load) processes using Azure Databricks and related Azure services.
Data Processing & Optimization: Implement and optimize data processing workflows, Spark jobs, and data transformations within the Databricks environment.
Machine Learning Integration: Develop and optimize machine learning models and workflows using Databricks' integrated capabilities, including MLflow.
Collaboration & Requirements Gathering: Work with data scientists, analysts, and other stakeholders to understand data needs and deliver effective data solutions.
