Senior Big Data Consultant

As an SME, you will bring deep technical expertise and strategic vision to our cloud data initiatives, guiding the team on best practices, ensuring the successful delivery of data projects, and improving operational efficiency. You’ll work closely with engineers, architects, and leadership to ensure the implementation of cutting-edge solution\n\ns.\nKey Responsibiliti\n\nes:\nArchitect and Design Scalable Data Platforms: Leverage your expertise in Azure, AWS, and GCP to design and build scalable, high-performance data platforms for analytics and real time process\ning.Cloud Technologies Expertise: Drive the adoption of cloud-native tools and services like Azure Synapse, AWS Glue, Redshift, and Google BigQuery, and ensure alignment with industry best practi\nces.Big Data & Streaming Solutions: Provide hands-on support and guidance on big data technologies including Apache Spark (PySpark, Scala, Java), Hadoop (HDFS, YARN, Hive, Impala), and streaming frameworks like Kafka, Kinesis, and Fl\nink.Data Pipelines & ETL Frameworks: Lead efforts in creating and optimizing ETL pipelines using tools like DBT, Apache Airflow, and cloud-native data orchestration tools (e.g., AWS Glue, Azure Data Facto\nry).Containerization & Orchestration: Lead efforts in containerizing applications using Docker and orchestrating them with Kubernetes (AKS, EKS, GKE) for scalable and reliable deploym\nent.Data Security & Governance: Ensure data security, governance, and compliance using IAM, RBAC, and data lineage tracking tools (Azure Purview, AWS Glue Data Catal\nog).CI/CD & DevOps: Contribute to the development of CI/CD pipelines with Terraform, Jenkins, Azure DevOps, and GitHub Actions to automate the deployment of cloud infrastructure and data workfl\nows.Cost Optimization: Implement strategies for optimizing cloud resource utilization and reducing operational costs, particularly in storage (AWS S3, Azure Blob, GCS) and compute resour\nces.Team Collaboration & Mentorship: Provide mentorship and guidance to junior engineers, and lead initiatives to build and maintain high-performing teams through knowledge-sharing sessions and best practice adopt\n\nion.\nRequirem\nents:Experience: 9+ years in Data Engineering or related roles, with a focus on cloud technologies and big data solut\n\nions.\nTechnical Expe\nrtise:Deep knowledge of cloud platforms such as Azure, AWS, and GCP. Hands-on experience with Big Data technologies like Apache Spark, Hadoop, Scala, Mapreduce, Kafka, and\nFlink.Solid understanding of ETL frameworks (DBT, Apache Airflow, AWS Glue, etc.).

Expertise in containerization with Docker and Kubernetes (AKS, EKS, GKE). Proven experience in Data Warehousing & Modeling (Snowflake, Redshift, BigQuery, Syn\napse).Strong background in Data Security and Governance (IAM, RBAC, encryption, data lineage). Experience with CI/CD pipelines using Terraform, GitHub Actions, and Azure DevOps.

Programming & Scripting S\nkills:Proficiency in Python, Bash, or PowerShell for automation tasks. Cloud Architecture: Experience designing hybrid/multi-cloud architectures to ensure high availability and fault tolerance across Azure, AWS, an\nd GCP.Leadership & Mentorship: Proven ability to lead teams, mentor junior engineers, and collaborate effectively with cross-functional\n\nteams.\nPreferred\nSkills:Familiarity with Machine Learning pipelines and predictive ana\nlytics.Experience with Serverless Computing (AWS Lambda, Azure Functions, Google Cloud Func\n\ntions).