Sr Lead Data Engineer With GCP

Area(s) of responsibility

Job Title: Sr. Data Engineer / Jr Solution Architect
Location - Noida/Bengaluru/Hyderabad
Shift - 12Pm-10 PM
Mode - Hybrid - 3 days a week
Notice Period - Immediate-30 Days

Total Experience - 8-10 Years

Primary Skill: - PySpark / Apache Spark, GCP, Dataproc jobs, APIGEE / API Integration(Knowledge should be there), Should be experience in developing ETL and ELT pipelines.

Key Roles & Responsibilities:
Design and implement scalable solutions on Google Cloud Platform based on business requirements
Build and manage data pipelines using Dataproc and PySpark for batch processing
Develop, optimize, and manage datasets and queries in BigQuery
Create and manage Dataproc jobs, ensuring performance, reliability, and cost efficiency
Develop backend and event-driven components using Cloud Functions
Deploy and manage containerized applications on Google Kubernetes Engine
Support development of AI/ML and Generative AI solutions using Vertex AI
Implement monitoring, logging, and alerting using Cloud Monitoring
Troubleshoot production issues, debug data pipelines, and ensure system stability
Contribute to solution design documents (HLD/LLD) and architecture discussions
Participate in client interactions, requirement gathering, and solution walkthroughs
Collaborate with cross-functional teams including data, DevOps, and application teams

Skill Requirement:
4-8 years of experience with at least 2-3 years hands-on in GCP
Strong programming skills in Python
Hands-on experience with PySpark / Apache Spark (mandatory)
Experience working with Dataproc jobs and cluster management
Strong SQL skills and experience with BigQuery
Understanding of data pipeline design (batch and basic streaming)
Experience with serverless and microservices-based architecture
Hands-on exposure to GKE and container-based deployments
Basic knowledge of Vertex AI, ML concepts, and Generative AI (LLMs, RAG)
Familiarity with monitoring, logging, and debugging distributed systems
Understanding of cloud fundamentals (IAM, networking, security basics)
Exposure to CI/CD pipelines and Infrastructure as Code (Terraform preferred)
Good problem-solving and analytical skills
Ability to communicate technical concepts clearly to stakeholders
Experience in production support and handling real-world system issues
Willingness to learn, adapt, and grow into a Solution Architect role