Role Summary
Project A is ALX’s AI learning platform — a set of LLM products used by learners. Every one generates a stream of LLM data, and every one has hypotheses baked into it about what “working” means. The LLMOps Engineer owns the analyzer function: turning that stream into an honest answer about whether the products work. Take RAG as one example — documents must be stored accurately, fetched accurately, and fetched in the right mixture: three separate failure modes, each needing its own eval. Every product decomposes like that. This is a junior-to-mid role with a deliberate growth path: you start close to the technical lead’s designs and grow into full ownership of the function.
You will work in collaboration with Anthropic Engineers, a cross functional team of AI engineers, product managers and data scientists to design world class learning experiences.
Specific Responsibilities
Evaluation Suites
- Build and run eval suites per product, decomposed by failure mode, running on schedule and on every release — regression testing so nothing ships if it broke what worked.
- Keep evals cost-effective as the product line grows.
Reporting, Data & Collaboration
- Own the reporting loop — findings from evals and platform data in front of the team and stakeholders, including surfacing unintended or problematic model behaviour before learners do.
- Steward the core datasets the team depends on, including classified customer-support data.
- Partner with the AI Product Manager on instrumentation — they instrument the product, you build the evals over what is captured. This is a measurement role, not infrastructure — no model hosting or serving.
Skill Requirements - Essential
- Python & data: solid Python and a data inclination, comfortable shaping and analysing messy LLM-generated data.
- Decomposition: the ability to look at an AI product and decompose it into success and failure metrics.
- Eval landscape: familiarity with Langfuse, RAGAS, DSPy, or similar — depth in one, awareness of the rest. These tools are learnable; we hire the fundamentals underneath them.
- Desirable (not required): experience keeping evals cheap at scale; dashboarding and reporting; classical statistics.
Essential Traits for Success
- You want to own a function, not execute tickets.
- You communicate well and like collaborating, you will support every builder on the team.
- You can point to any project, even a small one, where you measured an AI system honestly.
Qualifications
- BA/BSc/HND degree in a relevant field.
How to Apply
Interested and qualified candidates should apply online through the ALX Application Portal.
