Senior Big Data Engineer

We're looking for a Sr Big Data Engineer who expects more from their career. It's chance to extend and improve dunnhumby's Data Engineering Team. It's an opportunity to work with a market-leading business to explore new opportunities for us and influence global retailers.

Key Responsibilities

• Design end-to-end data solutions, including data lakes, data warehouses, ETL/ELT pipelines, APIs, and analytics platforms.
• Architect scalable and low-latency data pipelines using tools like Apache Kafka, Flink, or Spark Streaming to handle high-velocity data streams.
• Design /Orchestrate end-to-end automation using orchestration frameworks such as Apache Airflow to manage complex workflows and dependencies.
• Design intelligent systems that can detect anomalies, trigger alerts, and automatically reroute or restart processes to maintain data integrity and availability.
• Define and implement data governance, metadata management, and data quality standards.
• Lead architectural reviews and technical design sessions to guide solution development.
• Partner with business and IT teams to translate business needs into data architecture requirements.
• Ensure security, compliance, and regulatory requirements are addressed in all data solutions.
• Evaluate and recommend improvements to existing data architecture and processes.
  
  

Technical Expertise

• Bachelors or masters degree in computer science, Information Systems, Data Science, or related field.
• Extensive experience with high level programming languages - Python, Java or Scala
• 5+ years of experience in data architecture, data engineering, or a related field.
• Proficient in data pipeline tools such as Apache Spark, Kafka, Airflow, or similar.
• Experience with data governance frameworks and tools (e.g., Collibra, Alation, OpenMetadata).
• Strong knowledge of cloud platforms (Azure or Google Cloud), especially with cloud-native data services.
• Experience working in Agile or DevOps environments.
• Experience with modern data stack tools (e.g., dbt, Snowflake, Databricks).
• Experience with Hive, Oozie, Airflow, HBase, MapReduce, Spark along with working knowledge of Hadoop/Spark Toolsets.
  
• Extensive Experience working with Git and Process Automation
  
• In depth understanding of relational database management systems (RDBMS) and Data Flow Development.