Senior Software Development Test Enigneer

Roles & Responsibilities

Generative AI & LLM Evaluation

• Build automated testing suites to detect hallucinations, bias, toxicity, and prompt injection vulnerabilities across LLM-powered products

• Implement automated evaluations for RAG systems measuring context relevance, groundedness, and answer faithfulness using frameworks like RAGAS or DeepEval

• Design test beds to validate multi-agent workflows - tool-calling accuracy, multi-step reasoning, memory, and autonomous decision loops

• Build and run automated conversation simulations - scripted and synthetic user journeys - to stress-test agent behaviour across intents, edge cases, and multi-turn dialog flows

• Create prompt regression frameworks to assess how changes in system prompts, temperature, and sampling parameters impact output consistency

Data Quality Assurance

• Statistically validate AI data outputs - distributions, precision/recall, error pattern analysis - to catch silent data quality failures before production

• Programmatically audit data ingestion, transformation, and feature store pipelines for schema drift and data corruption

• Validate vector DB indexing, embedding semantic similarity accuracy, and retrieval latency

• Verify quality, diversity, and privacy compliance of synthetic datasets used for model training and evaluation

Classical ML & Deep Learning Validation

• Maintain automated suites tracking ML metrics - Precision, Recall, F1, ROC-AUC - and deep learning loss curves across model versions

• Implement continuous monitoring scripts to detect data and concept drift on live inference endpoints

Automation Engineering & CI/CD

• Build and maintain scalable test automation frameworks for APIs, backend services, and model endpoints

• Embed AI evaluation and data QA suites into MLOps and CI/CD pipelines so quality failures block releases automatically

• Define and track AI quality KPIs and communicate release readiness to engineering and product teams

Experience of 5+ years SDET role

Technical Skills & Frameworks

Core Programming

• Python - expert level; test automation, eval pipelines, data analysis (Pandas, NumPy, Pytest)

• SQL - data output validation, ground truth querying, pipeline data quality checks

GenAI & Evaluation

• RAGAS / TruLens / DeepEval / Promptflow etc - LLM evaluation frameworks for measuring faithfulness, hallucination rate, and task success

• LangChain / LangSmith / LlamaIndex - agent workflow testing, prompt tracing, and LLM response debugging

• OpenAI / Anthropic / Hugging Face APIs - direct LLM endpoint testing and output consistency validation

• Vector DBs - retrieval quality testing, embedding validation, and latency benchmarking

• Pandas / NumPy etc. - statistical analysis for output validation and error pattern investigation, data profiling, schema validation, and pipeline integrity checks

API & Automation

• Pytest - modular, reusable test framework for AI eval and automation suites

• Postman / REST Assured / Requests - API contract validation and service-level integration testing

MLOps & CI/CD

• MLflow - tracking model versions and eval runs to detect regressions across updates

• Docker / GitHub Actions / Jenkins - containerised test environments and deployment pipeline automation

Observability

• Grafana / Kibana / OpenTelemetry - monitoring AI system health, output drift, and distributed tracing across agent pipelines

Good to Have

• Cloud AI Services - AWS Bedrock, Azure OpenAI, or GCP Vertex AI for testing managed model endpoints and cloud-deployed agents

• MLOps Platforms - MLflow, Kubeflow, Weights & Biases, or Feast feature stores for experiment tracking and model governance

• ML Frameworks - Scikit-learn, TensorFlow, or PyTorch familiarity for understanding model internals and validating training pipelines

• Infrastructure as Code - Docker, Kubernetes, Terraform for managing containerised test environments at scale

• UI Automation - Playwright or Cypress for end-to-end conversational AI application testing

• Performance Engineering - Locust or JMeter for load testing heavy AI inference endpoints under peak traffic

• Statistical Hypothesis Testing - t-tests, confidence intervals, significance testing to distinguish real quality signal from noise

• Synthetic Data Generation - using LLMs to generate diverse test cases and evaluation datasets at scale