Big Data Engineer

We're looking for a Big Data Engineer who expects more from their career. It's a chance to extend and improve dunnhumby's Data Engineering Team. It's an opportunity to work with a market-leading business to explore new opportunities for us and influence global retailers.

What we expect from you

• 2-5 years of experience in Data Engineering or a related field
• Strong experience with Python, PySpark, and SQL (mandatory)
• Experience working in Linux environments and writing shell scripts
• Hands-on experience with big data technologies such as Spark, Hive, Hadoop, Airflow, Oozie, HBase, or MapReduce
• Experience with data pipeline development and data flow management
• Familiarity with Git and process automation
• Experience with relational database management systems (RDBMS) and Data Flow Development

Key Responsibilities

• Build data solutions, including data lakes, data warehouses, ETL/ELT pipelines, APIs, and analytics platforms.
• Build scalable and low-latency data pipelines using tools like Apache Kafka, Flink, or Spark Streaming to handle high-velocity data streams.
• Automate data pipelines and processes end-to-end using orchestration frameworks such as Apache Airflow to manage complex workflows and dependencies.
• Develop pipeline for real-time data processing.
• Implement data governance, metadata management, and data quality standards.
• Explore appropriate tools, platforms, and technologies aligned with organizational standards.
• Ensure security, compliance, and regulatory requirements are addressed in all data solutions.