FinOps Data Engineer

Your Impact

The Data Engineer will be responsible for building and maintaining robust data pipelines, managing and transforming data, and ensuring the quality and integrity of data across various platforms. The candidate will have hands-on experience with Google Data Products and AWS Data Products, metadata management tools, orchestration tools and strong expertise in SQL, Python, and ETL processes.

Key Responsibilities:

• Google Data Products: Use Google Data Products including BigQuery, Dataflow, and Cloud Composer/ Apache Airflow to design, develop, optimize and maintain data pipelines and workflows.
• AWS Data Services: Use AWS Data Products such as S3, Glue, Athena, Redshift, API Gateway to design, develop, optimize and maintain data pipelines and workflows.
• Metadata Management & Data Quality: Implement and manage metadata, data quality, and data lineage using tools from AWS, GCP and Azure to ensure data integrity, relevance and compliance.
• E2E Data Engineering: Provide input into the end-to-end data engineering lifecycle, as well as project/ programme lifecycle including managing non-functional requirements, operations, and performance tuning.
• Solution Design: Collaborate with stakeholders to design E2E solutions, including prototyping, usability testing, and data visualization, to meet business needs.
• SQL & NoSQL Databases: Develop, optimize, and manage data storage solutions using SQL and NoSQL databases/virtual databses, ensuring data is efficiently structured and accessible.
• Python Programming: Work with semi-structured and unstructured data, including making ReST API calls, subscribing to message topics, using Python for data manipulation and transformation.
• ETL Tools & Data Modeling: Develop and maintain ETL processes using tools from AWS, GCP and Azure, and create data models to support business requirements.
• Collaboration: Work closely with FinOps team, Engineering, Corporate BI Teams, and as needed with data scientists/ analysts, and other stakeholders to deliver high-quality data products and solutions.

Your Qualifications

• Python: 3 to 4 years of hands-on experience with Python, especially in handling semi/unstructured data and REST API integration.
• Relational Databases: 3 to 4 years of experience working with relational databases like Google Big Query, AWS Athena, AWS Redshift, or at least MySQL/RDBMS, with a strong focus on writing optimized queries.
• ETL Tools: 4 to 5 years of working knowledge with ETL tools (e.g., Apache Airflow, Google Composer, AWS Glue or Informatica, MuleSoft, SSIS) and data modeling.
• Google Cloud Experience: 2 years of experience with Google BigQuery, Dataflow, and Cloud Composer.
• AWS Experience: 2 years of experience with AWS S3, Glue, Athena, Redshift, API Gateway
• Data Engineering Lifecycle: Experience managing the complete data engineering lifecycle, including solution design, development, and operations.
• Familiarity with Agile project approaches (Scrum/ Kanban) and tools such as Jira.

Qualification:

• Education: Bachelor's degree in Computer Science, Information Technology, or a related field.
• Experience: 4-5 years of relevant experience in data engineering, particularly with cloud platforms like Google Cloud.
• Certifications: Preferred certifications in Google Cloud Platform or related data engineering tools.
• Skills: Proficiency in Python, SQL/NoSQL, ETL tools, and Google Data Products (BigQuery, Dataflow, Dataproc).