- Design and manage AWS infrastructure for the IIA Data Lake including S3 storage, Glue data catalog, Athena query engine, and EMR compute clusters.
- Manage cross-account connectivity, VPC networking, security groups, and IAM roles/policies to enable secure data flow between IIA, upstream data providers, and downstream consumers.
- Build and maintain CI/CD pipelines for AI agent deployments using GitLab CI/CD, Docker, and Artifactory.
- Support and maintain existing ETL pipelines written in Scala/Spark, including troubleshooting, enhancements, and onboarding new data sources.
- Ensure production environment stability through monitoring, alerting, and incident response. Maintain SLAs for data pipeline availability and agent uptime.
- Strong experience with infrastructure-as-code (Terraform preferred, CloudFormation acceptable)
- Experience managing cross-account AWS architectures, VPC peering, PrivateLink, and transit gateway configurations
- Experience with IAM policy design, least-privilege access patterns, and service account management
- Experience with containerization (Docker) and container orchestration
- Experience with CI/CD pipelines (GitLab CI preferred)
- Experience with event-driven architectures and messaging systems (e.g., Kafka, SNS/SQS, EventBridge)
- Proficiency with Linux-based operating systems and shell scripting
- Experience with monitoring and alerting tools (CloudWatch, Prometheus, Grafana, or similar)
- Understanding of networking fundamentals: DNS, CIDR, NAT, VPC endpoints, firewalls, security groups
- Demonstrated ability to work across teams and coordinate with external platform owners on connectivity and access requirements
- Proficiency in Python (or a comparable general-purpose language) for building automation, tooling, and applications
- Solid software engineering fundamentals: Git-based workflows, code review, modular and reusable design, dependency management, and writing maintainable, documented code
- Experience writing automated tests (unit/integration) for application and infrastructure code, and integrating those tests into CI/CD
- Ability to write integration code against REST APIs and cloud SDKs (e.g., AWS SDK / boto3)
|
