Big Data Developer

This role is for one of the Weekday's clients

Min Experience: 3+ years

Location: Bengaluru
JobType: full-time

We are looking for a highly skilled Big Data Engineer with 3+ years of experience in building scalable data pipelines and distributed systems. The ideal candidate will have strong expertise in Apache Spark (Scala), experience working across on-premise and AWS environments, and a solid understanding of large-scale data processing in AdTech ecosystems.

This role involves working on high-volume datasets (billions of records), optimizing distributed jobs, and contributing to the design of robust data infrastructure powering analytics and identity-driven use cases.

Requirements

Key Responsibilities:

• Design, develop, and optimize large-scale batch data pipelines using Apache Spark (Scala)
• Process and transform high-volume datasets (TBs of data) in distributed environments
• Build and maintain data pipelines across hybrid infrastructure (On-Prem + AWS)
• Work with object storage systems such as S3 and MinIO for efficient data access and storage
• Develop reusable and scalable data processing frameworks
• Optimize Spark jobs for performance (memory tuning, partitioning, shuffling, etc.)
• Manage and orchestrate workloads using HashiCorp Nomad
• Integrate data pipelines with PostgreSQL and other downstream systems
• Ensure data quality, consistency, and reliability across pipelines
• Troubleshoot production issues and perform root cause analysis
• Contribute to system design discussions, especially for high-scale AdTech use cases (identity resolution, user profiling, etc.)

Required Skills:

• Strong programming experience in Scala
• Good working knowledge of Python (for auxiliary tasks, scripting, or ML integration)
• Deep expertise in Apache Spark (Core and SQL)
• Strong understanding of distributed data processing and large-scale systems
• Experience working with AWS (S3, EMR or equivalent ecosystem) and on-prem clusters
• Hands-on experience with object storage systems (S3 / MinIO)
• Experience with HashiCorp Nomad or similar orchestration tools
• Solid understanding of data modeling and ETL pipeline design
• Experience working with PostgreSQL or similar relational databases
• Strong debugging and performance tuning skills for Spark jobs
• Familiarity with Unix/Linux environments and shell scripting

Good to Have:

• Experience in AdTech, Identity Graph, or User Profiling systems
• Exposure to machine learning pipelines or feature engineering workflows
• Experience with data lake architectures
• Understanding of cost optimization and resource management in AWS

Tech Stack Summary:

• Languages: Scala (Primary), Python (Secondary)
• Processing: Apache Spark (Core and SQL)
• Infrastructure: AWS and On-Prem
• Storage: S3, MinIO
• Orchestration: HashiCorp Nomad
• Database: PostgreSQL

Must-have skills

Spark, SQL, Scala

Good-to-have skills

Python, AWS, Big Data