Principal AI Automation Engineer

Principal Engineer – AI Quality & Evaluation Architecture
Role Overview
Vocera (Stryker) is seeking a Principal Engineer to own end-to-end AI quality across the lifecycle — data, models, prompts, evaluation, deployment, and monitoring. This role will define and scale reliable, measurable, production-grade AI systems across speech, NLP, and GenAI in healthcare.

Key Responsibilities
AI Quality Ownership

• Own AI quality across the full lifecycle

• Define SLAs, KPIs, release gates, and production readiness decisions

Evaluation & Reliability

• Build evaluation frameworks for ASR (WER, latency), NLP (intent/entity), and LLMs/RAG (hallucination, safety, groundedness)

• Develop benchmarking, regression pipelines, and golden datasets

• Drive adversarial testing, edge case handling, and failure analysis

AI Testing Platform

• Architect scalable evaluation platforms (offline, regression, A/B, shadow testing)

• Integrate with CI/CD and MLOps pipelines

• Implement monitoring, observability, and drift detection

Data Governance

• Define standards for data curation, annotation, and versioning

• Ensure reproducibility and feedback loops from production

• Maintain healthcare data compliance

MLOps & Continuous Quality

• Establish AI MLOps standards for evaluation, retraining, and deployment

• Enable continuous evaluation and performance monitoring at scale

Leadership

• Act as AI quality authority across the organization

• Mentor teams and align with product and business goals


Qualifications

• 12+ years in software/AI engineering; 5+ years in LLMs, NLP, RAG, or speech

• Experience building scalable AI evaluation frameworks

Expertise in:

• LLM evaluation (hallucination, safety, groundedness)

• Golden datasets, regression testing, adversarial testing

• Prompt validation, Python, data analysis, automation

• CI/CD, MLOps, distributed systems


Nice to Have

• RAG evaluation & retrieval benchmarking

• Speech/ASR evaluation

• Azure ML / OpenAI / AI Search

• Responsible AI & compliance

Travel Percentage: 10%