Design, develop, and optimize scalable ETL/ELT pipelines using Databricks and Apache Spark.
Work with Delta Lake to manage data lakes and ensure data quality, consistency, and performance.
Collaborate with data scientists, analysts, and stakeholders to build data models and pipelines that support analytics and ML workloads.
Implement data governance, security, and compliance best practices in the data engineering workflow.
Tune Spark jobs and optimize cluster performance on Databricks runtime.
Integrate with data sources such as Azure Data Lake, AWS S3, Kafka, or Snowflake.
Monitor and troubleshoot pipeline performance and failures using Databricks monitoring tools.
Document data processes and contribute to overall data architecture and strategy
