SRE - Observability Expert

Data Infrastructure & Operations

Offer flexible and secure data ingestion, streaming, transformation, analytics and data lake storage paired with self-service compute & ML workspaces so that In-house data teams can spin-up services and create pipelines as per their business requirements

Job description

Looking for 4-6 years of hand-ons experience with production-level development and operations on AWS or Azure Cloud

Develop and maintain infrastructure-as-code using Terraform to deploy and manage Kubernetes clusters (AKS) and Databricks environments

Hands-on experience with data pipeline orchestration tools like Azure Data Factory, Amazon data Pipeline, Apache Spark, Databricks

Hands-on experience with one or more of stream & batch processing systems: Kafka (Confluent cloud, open source), Apache Storm, Spark-Streaming, Apache Flink, Kappa architecture

Experience in architecting right storage strategy for use-cases, keeping data processing, data accessibility, data availability and cloud cost considerations

Proficiency in Data transformation using Kstreams App/KSQL/Processor Libraries

Data ingestion and data distribution integration experience using managed connectors such as Event Hubs, kafka topics, ADLS2, REST APIs

Proficiency to set-up & manage open-source stack, including Airflow, Druid, Kafka (open source) OpenSearch, and Superset

Proficiency in Python scripting for automation and integration tasks

Utilize FastAPI for building and deploying high-performance APIs

Handling requirements of managed services, IAM, auto-scaling, High availability, elasticity, networking options

Handle federated access to cloud computing resource (or set of resources) based on a user's role within the organization

Proficiency with Git, including branching/merging strategies, Pull Requests, and basic command line functions

Proficiency in DevSecOps practices throughout the product lifecycle including fully managed Day 2 Ops leveraging Datadog

Shared access controls to support multi-tenancy and self-service tooling for customers

Manage data catalog per topic or domain based on services & use-cases offered

Research, investigate and bring new technologies to continually evolve data platform capabilities

Experience in working under Agile scrum Methodologies