Design and implement ETL/ELT pipelines in Databricks using PySpark, SQL, and Delta Lake.
Collaborate with data analysts and data scientists to support advanced analytics and machine learning workflows.
Optimize and troubleshoot Spark jobs for performance, scalability, and cost efficiency.
Integrate Databricks with various data sources such as Azure Data Lake, AWS S3, SQL databases, and APIs.
Manage and monitor data pipelines using Databricks Workflows, Job Clusters, and Notebooks.
Implement data quality checks, error handling, and monitoring for robust data pipelines
