Agentic QA Engineer

This role is for one of the Weekday's clients

Salary range: Rs 2500000 - Rs 4000000 (ie INR 25 - 40 LPA)

Min Experience: 5+ years

Location: Bengaluru

JobType: full-time

We are looking for an innovative Agentic QA Engineer to define and build the next generation of quality assurance practices for AI-native platforms. This is far more than a traditional QA role-you will combine software testing, automation engineering, and AI system validation to ensure platform reliability, scalability, and intelligent decision-making.

You'll be responsible for validating the platform's behavior, including its AI-driven orchestration, routing, and multi-agent workflows. Rather than evaluating customer-built AI agents, your focus will be on ensuring the platform consistently makes reliable, secure, and efficient decisions under real-world conditions.

Requirements

Key Responsibilities

Platform Functional Testing

• Design and execute end-to-end functional tests covering agent onboarding, deployment, orchestration, and request routing.
• Validate platform behavior across multiple supported agent frameworks and deployment scenarios.
• Test failure handling, recovery mechanisms, and edge cases throughout the platform lifecycle.

QA Automation & Reliability

• Build and maintain automated regression test suites integrated into CI/CD pipelines.
• Develop automated quality gates that prevent deployments when reliability metrics fall below acceptable thresholds.
• Perform resilience testing by intentionally introducing failures such as service outages, database interruptions, network failures, and high traffic loads.
• Define and monitor Service Level Objectives (SLOs) for platform reliability and availability.

Agentic Testing

• Develop testing methodologies for AI-driven platform behavior where deterministic outputs are not possible.
• Validate intelligent routing decisions across multiple AI agents.
• Test multi-agent orchestration to ensure correct information flow, execution limits, cost controls, and successful task completion.
• Verify platform safeguards against infinite agent loops, resource overconsumption, security risks, and cross-tenant data leakage.
• Build and maintain a suite of deterministic test agents with predictable behaviors to evaluate platform decision-making accurately.

Quality Engineering

• Create adversarial ("red team") testing strategies to uncover weaknesses in AI workflows and orchestration logic.
• Convert execution traces into measurable platform health metrics such as routing accuracy, task success rate, execution cost, failure rates, and stuck-loop detection.
• Investigate failure patterns and develop clear troubleshooting guides and operational playbooks.
• Own quality assurance processes for platform releases and continuously improve testing strategies as the platform evolves.
• Contribute toward developing platform testing capabilities that customers can leverage for validating their own AI agents.

Required Qualifications

• Strong experience in QA Engineering, SDET, Software Testing, or Site Reliability Engineering.
• Proven expertise in building automated testing frameworks and integrating them into CI/CD pipelines.
• Hands-on experience with test automation using Python or similar scripting languages.
• Strong understanding of software quality principles, distributed systems, and cloud-native architectures.
• Familiarity with AI/ML concepts and an understanding of how AI-powered systems behave in production environments.
• Excellent analytical and problem-solving skills with an adversarial mindset for identifying edge cases and failure scenarios.
• Strong communication skills with the ability to explain technical quality issues to engineering, product, and leadership teams.

Preferred Qualifications

• Experience working with Large Language Models (LLMs), AI agents, or autonomous AI systems.
• Familiarity with agent frameworks such as LangGraph, CrewAI, AutoGen, or similar orchestration platforms.
• Knowledge of distributed systems reliability and control plane architectures.
• Experience with observability platforms such as OpenTelemetry, Grafana, Datadog, or similar monitoring tools.
• Contributions to open-source QA, AI, or developer tooling projects.

Why Join Us?

• Help define an entirely new discipline at the intersection of Quality Engineering, AI Systems, and Platform Reliability.
• Build innovative testing methodologies for intelligent, AI-native platforms.
• Work in a highly collaborative environment with significant ownership and technical autonomy.
• Make a direct impact on the quality, scalability, and reliability of next-generation AI products.
• Enjoy a flexible, remote-friendly work environment with competitive compensation and growth opportunities.

Must-Have Skills

• QA Automation
• Agentic Testing
• Test Automation
• Python

Good-to-Have Skills

• Large Language Models (LLMs)
• LangGraph
• CrewAI
• AutoGen
• Site Reliability Engineering (SRE)
• Distributed Systems
• OpenTelemetry
• Grafana
• Datadog