Staff Software Engineer - Data Platform

Position Summary

The Unified Data Platform (UDP) is a comprehensive data infrastructure solution designed to provide a common storage, ingestion, processing, and query layer for all product modules within the Harness ecosystem. The platform enables real-time and batch data processing, unified querying across heterogeneous data sources, and provides a semantic layer for consistent data modeling across modules.. The platform will support some of the core Harness AI initiatives like Knowledge Graph and AI powered dashboarding to name a few.

About the Role

• You will participate in the design, architecture & development of this platform alongside principal engineers & architects, solving deep problem statements like declarative ingestion & processing frameworks OR DSL based query expressions - all while ensuring sub-second read latency , sub-minute freshness SLA with security & compliance baked in.
• You will mentor senior/junior engineers in the form of peer reviews(code/design) or complex debugging.
• Author high level & low level design documents.
• Design and implement automation of infrastructure setup using helm, terraform and similar technologies.
• Work alongside Site Reliability Engineers and cross functional teams to diagnose/troubleshoot any production performance related issues

About You

• Bachelor's or Master's degree in Computer Science or related field.
• 6-12 years of backend engineering experience, with strong focus on data platforms and analytics systems.
• Proven expertise in building low-latency, highly concurrent microservices in the Data Platform space.
• Proficiency in JVM-based languages (Java, Scala, Kotlin) is a MUST.
• Strong understanding of various Data Platform technologies that are fundamental to ingestion, processing or querying like Kafka & Kafka connect, Apache Spark, Object Storage, Iceberg, Query engines like Trino or StarRocks or even OLAP stores like Pinot or Druid. We will prefer deep exposure to some of the above mentioned technologies.
• Good understanding of OLAP modelling meant for DataLake or DataWarehouse.