Senior Applied AI Engineer -- Evaluation & Reliability

Senior Applied AI Engineer — Evaluation & Reliability

15Rock · Toronto (remote-friendly across North American hours)

About 15Rock

15Rock builds financial intelligence that institutions can act on. Our platform produces deep analytical research in a day — the kind of work that once took a team of analysts weeks — grounded in real sources and built to withstand professional scrutiny. Our clients are among the world's largest asset owners, asset managers, consulting and advisory firms, and the corporates they serve. They make consequential decisions and answer for them. They don't need analysis that sounds right. They need analysis they can defend and drives expected outcomes.

Our ambition is larger than producing good reports. We want to be the analytical infrastructure institutionals relies on, and getting there means solving problems no one has fully solved yet.

About the role

This role exists to solve the hardest of those problems: making AI-generated analysis trustworthy at an institutional standard.

A serious report is long, dense, and full of numbers that must agree with one another and trace back to their sources. Language models don't do this reliably on their own. They produce work that is fluent and often subtly wrong, and across a long document those errors compound. You would own the layer that catches them — the evaluation, grounding, and quality systems that decide whether what we produce can be trusted. This is central engineering, not a side function.

What you'll do

• Build the evaluation systems that show, continuously and precisely, where our output is wrong and why.
• Design the architecture that keeps generated analysis grounded in its sources and consistent across a long document.
• Define what \"ready to deliver\" means in terms a machine can enforce, and build the gates that hold that line as volume grows.
• Work alongside the engineering team — tech lead, engineers, scrum master — owning the reliability layer in particular.
  

You may be a good fit if you

• Have several years of production engineering experience, strong in Python, on systems others relied on.
• Have built and operated LLM systems in production, including multi-step or agentic ones.
• Have designed evaluation or validation for systems that don't return the same answer twice — not unit tests, but ways to measure how and where a model fails.
• Are bothered by a wrong answer in a way that's hard to switch off.
  

Nice to have

• A data-engineering background: provenance, normalization, retrieval, grounding.
• Time in a domain where a wrong answer has real consequences — finance, medicine, anything safety-critical.
• Judgment about when a problem needs a fixed rule and when it needs a model in the loop.
  

Working here

We are a small, senior team, Institutional investor funded, based in Toronto and comfortable with remote work across North American hours. You would own the reliability of the product directly. The package includes meaningful equity, and we are open to contract-to-hire for the right person.

Every application is read by someone on the team. If this is the problem you want to spend the next few years on, email with a short note and something that shows how you think about correctness in real systems — a project, a write-up, a repo.