Posted 23 June, 2026
Big Data Engineer
NR Consulting - India
Hyderabad, IN
Full Time
Reference: 26-07335-2220-1
What To Expect We are seeking a skilled and collaborative Big Data Engineer to join our team. As a Senior Big Data Engineer, you will play a critical role in designing and delivering scalable, high-quality data solutions across Housing (RentBureau and Mortgage) products. You will leverage your expertise to design and implement large-scale data pipelines and integrated data solutions using the latest technologies, ensuring efficient data storage, processing, and presentation. You will also build and utilize advanced data modeling frameworks and adopt modern architectural patterns such as Medallion Architecture to deliver well-structured, efficient, and future-ready data solutions. This role requires expertise in big data architectures, data processing, and the ability to build modular, reusable components that can scale.
What You'll Do " Design and implement robust, scalable solutions for processing huge volume of data " Develop and maintain high-performance data pipelines using Spark, Glue, Airflow, SageMaker, and other modern data technologies " Build and optimize batch data processing systems " Develop aggregate and summary tables for various engineering teams that span across multiple product lines and geographies " Design and develop new systems and tools to enable cross-functional teams to consume and understand data faster " Implement and maintain CI/CD pipelines for data applications " Research and incorporate emerging data infrastructures, tools, and technologies
Primary Responsibilities " Experience in Requirement Gathering, Estimating, Managing large scale Data Engineering Projects. " Data Modeling, Data Pipeline Design and Integration. Create robust and scalable data pipelines and data products in a variety of domains. " Testing: Define and manage the data load procedures to reject or discard datasets or data points that do not meet the defined business rules and quality thresholds. " Operations: Able to independently manage the data load operations of the solutions or products in scope and be able to debug " Understanding of and ability to Implement Data Engineering principles and best practices
Continuous Integration and Continuous Deployment (CI/CD): Implement and maintain CI/CD pipelines for data solutions, ensuring rapid, reliable, and streamlined updates to the data environment. " Leverage AI on day to day tasks to bring in efficiency in implementation and quicker time to market. " Use Github copilot or Claude Code to generate and refactor boilerplate PySpark jobs, SQL, Airflow DAGs, and documentation " Define and maintain agents and skills for recurring workflows such as requirement intake, code review, PySpark job generation " Build and evolve data quality checks, observability signals, and operational runbooks to improve reliability of the datasets " Drive engineering best practices code reviews, automated tests, schema controls.
Qualifications Preferred Qualifications " 5 or more years of work experience with a bachelor's degree " Hands-on data engineering experience building and operating production data pipelines. " Extensive experience in Big Data ecosystem and associated technologies, (For e.g. Spark, Python, AWS Glue, Athena, Redshift), Open table formats (e.g. Iceberg) " Proficiency in Python, SQL, and PySpark. " Experience with data pipeline and workflow management tools like Airflow, AWS Step Functions etc. " Experience with relational SQL and NoSQL databases, such as Postgres, mySQL " Practical experience with AWS data engineering services, especially S3, EMR, and Glue Catalog. " Experience with building processes supporting data transformation, data structures, metadata, dependency, and workload management.
Experience designing for data quality, resiliency, and observability, including validation checks, monitoring, alerting, and runbooks " Practical experience using generative AI tools such as Cursror, Claude code or similar systems to accelerate coding, documentation, or incident analysis while applying appropriate review controls. " Strong communication skills and the ability to convert ambiguous business requirements into clear technical implementation plans. " Experience with platforms such as databricks or snowflake
What You'll Do " Design and implement robust, scalable solutions for processing huge volume of data " Develop and maintain high-performance data pipelines using Spark, Glue, Airflow, SageMaker, and other modern data technologies " Build and optimize batch data processing systems " Develop aggregate and summary tables for various engineering teams that span across multiple product lines and geographies " Design and develop new systems and tools to enable cross-functional teams to consume and understand data faster " Implement and maintain CI/CD pipelines for data applications " Research and incorporate emerging data infrastructures, tools, and technologies
Primary Responsibilities " Experience in Requirement Gathering, Estimating, Managing large scale Data Engineering Projects. " Data Modeling, Data Pipeline Design and Integration. Create robust and scalable data pipelines and data products in a variety of domains. " Testing: Define and manage the data load procedures to reject or discard datasets or data points that do not meet the defined business rules and quality thresholds. " Operations: Able to independently manage the data load operations of the solutions or products in scope and be able to debug " Understanding of and ability to Implement Data Engineering principles and best practices
Continuous Integration and Continuous Deployment (CI/CD): Implement and maintain CI/CD pipelines for data solutions, ensuring rapid, reliable, and streamlined updates to the data environment. " Leverage AI on day to day tasks to bring in efficiency in implementation and quicker time to market. " Use Github copilot or Claude Code to generate and refactor boilerplate PySpark jobs, SQL, Airflow DAGs, and documentation " Define and maintain agents and skills for recurring workflows such as requirement intake, code review, PySpark job generation " Build and evolve data quality checks, observability signals, and operational runbooks to improve reliability of the datasets " Drive engineering best practices code reviews, automated tests, schema controls.
Qualifications Preferred Qualifications " 5 or more years of work experience with a bachelor's degree " Hands-on data engineering experience building and operating production data pipelines. " Extensive experience in Big Data ecosystem and associated technologies, (For e.g. Spark, Python, AWS Glue, Athena, Redshift), Open table formats (e.g. Iceberg) " Proficiency in Python, SQL, and PySpark. " Experience with data pipeline and workflow management tools like Airflow, AWS Step Functions etc. " Experience with relational SQL and NoSQL databases, such as Postgres, mySQL " Practical experience with AWS data engineering services, especially S3, EMR, and Glue Catalog. " Experience with building processes supporting data transformation, data structures, metadata, dependency, and workload management.
Experience designing for data quality, resiliency, and observability, including validation checks, monitoring, alerting, and runbooks " Practical experience using generative AI tools such as Cursror, Claude code or similar systems to accelerate coding, documentation, or incident analysis while applying appropriate review controls. " Strong communication skills and the ability to convert ambiguous business requirements into clear technical implementation plans. " Experience with platforms such as databricks or snowflake
