Develop and maintain data pipelines, ETL/ELT processes using Apache Spark on Databricks.
Design and implement scalable data models and architecture to support analytics and machine learning workloads.
Collaborate with data scientists, analysts, and other engineers to ingest, transform, and serve data.
Optimize Spark jobs for performance and cost-efficiency.
Manage data security, governance, and compliance in the Databricks environment.
Automate deployment and orchestration using tools like Azure Data Factory, Airflow, or similar.
Monitor, troubleshoot, and resolve production issues related to Databricks clusters and jobs.
Stay current with industry trends and best practices in big data and cloud analytics
