Skip to main content
Posted 17 July, 2026

Senior Big Data Engineer

Sky Systems, Inc
Hyderabad, Telangana, IN Full Time
Reference: 26-00734-1793-1
Role: Senior Big Data Engineer
Position Type: Full-Time Contract (40hrs/week)
Contract Duration: 6+ months
Work Schedule: 8 hours/day (Mon-Fri)
Location: Hybrid 2x days onsite (Wed/Thur) - Hyderabad, India

Job Description

We are seeking an experienced Senior Big Data Engineer with 7+ years of expertise in designing, developing, and supporting large-scale data processing applications on AWS Cloud. The ideal candidate will have strong hands-on experience with Big Data technologies, distributed computing frameworks, cloud-native microservices, batch processing, and CI/CD automation. This role will be responsible for building scalable data platforms, optimizing high-volume batch workloads, and developing API-driven data solutions that support critical business initiatives.

Key Responsibilities

  • Design, develop, implement, and maintain scalable Big Data applications and data processing pipelines on AWS Cloud.
  • Build and support both API-based microservices and batch processing solutions using Java, Scala, Kotlin, and Python.
  • Develop and optimize large-scale distributed data processing applications utilizing:
    • Apache Spark
    • Hadoop Ecosystem
    • HDFS
    • MapReduce
    • Cassandra
  • Create and maintain RESTful APIs and SOAP Web Services for enterprise data integration.
  • Develop cloud-native solutions leveraging AWS services including:
    • EMR
    • EC2
    • ECS
    • S3
    • Airflow
    • AWS Step Functions
    • AWS API Gateway
  • Perform performance tuning and optimization of Spark, Hadoop, EMR, Java, and Python-based applications.
  • Develop Linux-based automation scripts using Shell Scripting and Python.
  • Collaborate with cross-functional teams including Data Engineering, Architecture, DevOps, and Product teams.
  • Implement and maintain CI/CD pipelines to automate application deployment and infrastructure provisioning.
  • Participate in code reviews, troubleshoot production issues, and ensure adherence to engineering best practices.

Required Qualifications

Experience

  • 7+ years of experience in software development, data engineering, and Big Data application development.
  • Proven experience building scalable data platforms and distributed processing systems on AWS Cloud.

Technical Skills

Programming Languages

  • Strong hands-on experience with:
    • Scala
    • PySpark
    • Java
    • Kotlin
    • Python

Big Data Technologies

  • Apache Spark
  • Hadoop Ecosystem
  • HDFS
  • MapReduce
  • Cassandra

API & Microservices

  • REST API development
  • SOAP Web Services
  • Microservices Architecture

AWS Cloud Services

  • Amazon EMR
  • EC2
  • ECS
  • S3
  • AWS API Gateway
  • AWS Step Functions
  • Apache Airflow

Operating Systems & Scripting

  • Linux Administration
  • Shell Scripting
  • Python Automation

CI/CD & DevOps

  • Maven
  • Gradle
  • Jenkins
  • Artifactory
  • AWS CodeCommit
  • AWS CloudFormation
  • CI/CD Pipeline Development and Automation

Preferred Qualifications

  • Experience with Snowflake for data warehousing and analytics.
  • Experience developing automated data ingestion and transformation pipelines using Spark and Snowflake.
  • Familiarity with DevOps and source control tools including:
    • Git
    • GitHub
    • GitLab
    • Bitbucket
    • IntelliJ IDEA
    • PyCharm
  • Experience working within Agile/Scrum development environments.
We are seeking an experienced Senior Big Data Engineer with 7+ years of expertise in designing, developing, and supporting large-scale data processing applications on AWS Cloud. The ideal candidate will have strong hands-on experience with Big Data technologies, distributed computing frameworks, cloud-native microservices, batch processing, and CI/CD automation. This role will be responsible for building scalable data platforms, optimizing high-volume batch workloads, and developing API-driven data solutions that support critical business initiatives.

Key Responsibilities

  • Design, develop, implement, and maintain scalable Big Data applications and data processing pipelines on AWS Cloud.
  • Build and support both API-based microservices and batch processing solutions using Java, Scala, Kotlin, and Python.
  • Develop and optimize large-scale distributed data processing applications utilizing:
    • Apache Spark
    • Hadoop Ecosystem
    • HDFS
    • MapReduce
    • Cassandra
  • Create and maintain RESTful APIs and SOAP Web Services for enterprise data integration.
  • Develop cloud-native solutions leveraging AWS services including:
    • EMR
    • EC2
    • ECS
    • S3
    • Airflow
    • AWS Step Functions
    • AWS API Gateway
  • Perform performance tuning and optimization of Spark, Hadoop, EMR, Java, and Python-based applications.
  • Develop Linux-based automation scripts using Shell Scripting and Python.
  • Collaborate with cross-functional teams including Data Engineering, Architecture, DevOps, and Product teams.
  • Implement and maintain CI/CD pipelines to automate application deployment and infrastructure provisioning.
  • Participate in code reviews, troubleshoot production issues, and ensure adherence to engineering best practices.

Required Qualifications

Experience

  • 7+ years of experience in software development, data engineering, and Big Data application development.
  • Proven experience building scalable data platforms and distributed processing systems on AWS Cloud.

Technical Skills

  • Programming Languages

Strong hands-on experience with:
  • Scala
  • PySpark
  • Java
  • Kotlin
  • Python

Big Data Technologies

  • Apache Spark
  • Hadoop Ecosystem
  • HDFS
  • MapReduce
  • Cassandra

API & Microservices

  • REST API development
  • SOAP Web Services
  • Microservices Architecture

AWS Cloud Services

  • Amazon EMR
  • EC2
  • ECS
  • S3
  • AWS API Gateway
  • AWS Step Functions
  • Apache Airflow

Operating Systems & Scripting

  • Linux Administration
  • Shell Scripting
  • Python Automation

CI/CD & DevOps

  • Maven
  • Gradle
  • Jenkins
  • Artifactory
  • AWS CodeCommit
  • AWS CloudFormation
  • CI/CD Pipeline Development and Automation

Preferred Qualifications

  • Experience with Snowflake for data warehousing and analytics.
  • Experience developing automated data ingestion and transformation pipelines using Spark and Snowflake.
  • Familiarity with DevOps and source control tools including:
    • Git
    • GitHub
    • GitLab
    • Bitbucket
    • IntelliJ IDEA
    • PyCharm
  • Experience working within Agile/Scrum development environments.
Apply to this Job

Sign up for Job Alerts

By clicking Create Alert, you agree to the Global Terms of Use Agreement and acknowledge that you have read and understand the Global Privacy Policy.