| Detailed job description - Skill Set: |
- Design, build, and maintain scalable and efficient data pipelines using Databricks Notebooks, Workflows, and Delta Live Tables (DLT).
- Leverage Databricks SQL, PySpark, and Python for data transformation, enrichment, and analytics.
- Implement and manage Delta Lake architectures to support real-time and batch data processing.
- Use Unity Catalog for data governance, access control, and metadata management.
- Orchestrate complex workflows using Databricks Workflows and integrate with external schedulers if needed.
- Optimize performance of Spark jobs and manage cluster configurations for cost and efficiency.
- Collaborate with data scientists, analysts, and business stakeholders to deliver high-quality, production-grade data solutions.
- Ensure data quality, lineage, and observability using Databricks tools and best practices.
- Integrate with AWS S3 or Google Cloud Storage (GCS) for data ingestion and storage.
- Follow Agile methodologies and use Git for version control and CI/CD pipelines.
Qualifications
- Bachelor's degree in Computer Science, Information Technology, or a related field.
- 6+ years of experience as a Data Engineer, with 3+ years of hands-on experience with Databricks.
- Strong expertise in Databricks Lakehouse Platform, including:
- Delta Lake
- Databricks SQL
- Unity Catalog
- Databricks Workflows
- Delta Live Tables (DLT)
- Proficiency in PySpark, Python, and SQL.
- Experience with cloud storage services like AWS S3 or Google Cloud Storage.
- Solid understanding of data modeling, data warehousing, and ETL/ELT best practices.
- Exposure to BI tools like Tableau, Power BI, or Databricks SQL Dashboards.
- Experience with CI/CD pipelines, Git, and automated testing for data pipelines.
Strong problem-solving skills and ability to work independently or in a team.
|
