Detailed JD (Roles and Responsibilities) |
Astronomer Developer
Key Responsibilities:
• Design, develop, and maintain data orchestration pipelines leveraging the Astronomer platform to automate and scale complex enterprise data workflows.
• Build, configure, and manage custom Airflow DAGs, operators, and hooks within Astronomer to support source acquisition, transformation, and data delivery across multiple environments.
• Apply expert-level knowledge of Astronomer to configure deployments, manage worker queues, optimize scheduling, and ensure high availability and scalability.
• Lead the hands-on implementation of data ingestion pipelines, integrating with diverse data sources such as APIs, databases, cloud storage, and SaaS applications.
• Collaborate with data engineering, DevOps, and platform teams to standardize and automate orchestration processes across environments (Dev, Test, Prod).
• Contribute to the implement robust CI/CD workflows for DAG versioning, testing, and deployment using GitHub Actions or similar frameworks.
Required Skills & Experience:
• Extensive hands-on expertise with the Astronomer platform, including setup, configuration, deployment management, and environment tuning.
• Proven ability to design and implement source acquisition pipelines that extract and load data from multiple systems into enterprise data platforms.
• Deep understanding of Apache Airflow fundamentals — DAG design, dependency management, task retries, dynamic task mapping, and SLA monitoring.
• Strong Python development skills for creating custom operators, hooks, and Airflow plugins.
• Experience integrating Astronomer pipelines with DBT, Snowflake, Databricks, and cloud storage (Azure).
• Demonstrated experience implementing CI/CD automation for Astronomer DAG deployments using GitHub Actions.
• Solid understanding of containerization (Docker) and orchestration concepts for scalable pipeline execution.
• Hands-on experience monitoring, troubleshooting, and optimizing DAG performance using Astronomer's observability and Airflow UI tools.
• Strong knowledge of data orchestration best practices, including modular pipeline design, error handling, and recovery mechanisms.
• Excellent problem-solving, debugging, and communication skills, with a proactive, ownership-driven mindset.
|
