Sr. Data Engineer

Job Overview:

We are seeking a talented Data Engineer with over 5 years of experience in Databricks, Delta Lake, and Azure Data Services. The ideal candidate will have hands-on expertise in building scalable data ingestion, transformation, and AI-driven data pipelines using our agents, contributing to our data-driven decision-making processes.

Key Responsibilities:

• Build and maintain data ingestion and transformation pipelines.
• Implement ingestion, transformation, and validation agents.
• Optimize Spark workloads for improved performance and efficiency.
• Implement data quality measures and testing hooks to ensure data integrity.
• Support Continuous Integration/Continuous Deployment (CI/CD) deployments.

Qualifications:

• 5+ years of experience in Databricks, Delta Lake, and Azure Data Services.
• Proven expertise in building scalable data pipelines.
• Strong understanding of data quality and testing methodologies.
• Familiarity with CI/CD processes and tools.
• Excellent problem-solving and analytical skills.
• Strong communication and collaboration skills.