Senior Data Engineer (Airflow Specialist)

Ltd. is a global technology and healthcare IT services provider delivering advanced digital solutions for healthcare payers, providers, and life sciences organizations. Our expertise spans enterprise platforms, healthcare interoperability, AI-enabled analytics, cloud-native architectures, and digital member engagement platforms\n\n.\nRole Overvi\newWe are seeking a versatile Senior Data Engineer to lead the design, implementation, and scaling of our enterprise data orchestration platform using Apache Airflow.

This role requires a \"builder\" mindset, combining the agility to create robust pipelines from the ground up with the maturity to ensure they are scalable, secure, and integrated into a complex cloud ecosystem. As the primary architect for workflow automation, you will move beyond simple task scheduling to build a resilient, self-healing data infrastructur\n\ne.\nKey Responsibili\ntiesOrchestration & Pipeline Engineer ing: Author complex, modular, and idempotent DAGs using Python and the Airflow TaskFlow\nAPI.Framework Developm ent: Build custom Operators, Hooks, and Sensors to standardize data integrations across the organizat\nion.High-Compute Integrat ion: Integrate Airflow with heavy-compute environments including Azure Databricks, Spark clusters, and cloud data warehou\nses.Platform Managem ent: Optimize Airflow executors (Celery or Kubernetes) to manage high-concurrency workloads and handle backend database tuning (PostgreSQL/MyS\nQL).Security & Complia nce: Implement enterprise-grade security protocols, including Role-Based Access Control (RBAC), Secret Management (Azure Key Vault/HashiCorp Vault), and OAuth integrat\nion.DevOps & Reliabil ity: Build and maintain automated CI/CD deployment pipelines for DAGs and infrastructure-as-code using Terraform or Bi\ncep.Observabil ity: Implement advanced monitoring and alerting using Prometheus, Grafana, or the ELK stack to track pipeline health and SLA breac\nhes.Leadership & Mentors hip: Establish data engineering best practices across the team and conduct rigorous code reviews to ensure architectural integr\n\nity.\nTechnical Requir\nementsOrchestr ation: Expert knowledge of Apache Airflow 3.x+ (specifically XComs, Providers, and Dynamic Task Map\nping).Lang uages: Advanced proficiency in Python and expert-leve\nl SQL.Cloud Plat forms: Proficiency in Azure (Data Factory, Blob Storage, AKS) or equivalent services in AW\nS/GCP.D evOps: Hands-on experience with CI/CD tools such as GitHub Actions, Azure DevOps, or Je\n\nnkins.\nQualif\nicationsExp erience: 5+ years in Data Engineering, with a minimum of 2-3 years focused specifically on scaling Airflow in production envir\nonments.Ed ucation: Bachelor’s or Master’s degree in Computer Science, Information Systems, or a relate\nd field.Preferred\nSkills:Experience migrating legacy schedulers to\nAirflow.Background in software development (Java o\nr .NET).Knowledge of data quality frameworks like Great Expectations\n\nor dbt.\nCo\nmpetenciesStrong focus on long-term maintenance and architectural\nintegrity.Ability to move fast in an agile environment without compromising on cod\ne quality.Exceptional problem-solving skills for debugging complex data orchestrati\n\non issues.\nApplicat i\non Process -Interested candidates may apply at: or WhatsApp at +9\n\n1 7024041115