GCP Engineer --Bangalore, Hyderabad, Chennai

Experience: 4+ Years

About Us:Strong hands-on experience with Google Cloud Platform, data engineering best practices, and the ability to design, build, and optimize scalable data pipelines. You will collaborate with cross-functional teams to deliver innovative solutions that meet business objectives.

Responsibilities:

• Data Pipeline Development & Management:
  • Design, develop, and manage scalable and reliable data pipelines on Google Cloud Platform (GCP).
  • Use Google Cloud Storage, BigQuery, Dataflow, and Pub/Sub to build high-performing, real-time data ingestion and ETL pipelines.
  • Implement data quality checks and ensure the integrity and consistency of data flowing through the system.
• Data Warehousing:
  • Design, implement, and maintain BigQuery data warehouses, including schema design, partitioning, and optimization.
  • Create and maintain automated scripts for data loading, processing, and validation.
• Data Integration & Transformation:
  • Integrate and transform data from diverse sources (cloud storage, relational databases, APIs, etc.) into GCP services.
  • Work with the team to design and implement data models, data marts, and other reporting structures in BigQuery.
• Performance Optimization:
  • Optimize queries, database performance, and cloud resource usage (e.g., BigQuery, Dataflow) to reduce cost and increase efficiency.
  • Utilize Cloud Composer for orchestration of data pipelines.
• Collaboration & Stakeholder Management:
  • Work closely with data scientists, analysts, and business stakeholders to define data needs and provide data-driven solutions.
  • Document workflows, design patterns, and best practices for the team.
• Monitoring & Troubleshooting:
  • Set up monitoring, alerting, and logging for data pipelines and infrastructure using tools like Stackdriver and Cloud Monitoring.
  • Troubleshoot and resolve issues related to data quality, performance, and reliability in the pipeline.
• Tools & Technologies:
  • Proficiency in BigQuery, Cloud Dataflow, Cloud Pub/Sub, Cloud Storage, Cloud Composer.
  • Experience with Apache Beam, Kafka, and other streaming technologies is a plus.
  • Strong working knowledge of SQL for querying and transforming large datasets.
• Soft Skills:
  • Strong problem-solving and troubleshooting skills.
  • Excellent communication skills, with the ability to work collaboratively in a cross-functional environment.
  • Ability to document processes and technical decisions for both technical and non-technical stakeholders