Agreed, though the gap is a bit narrower than a full black box. An evaluator — a second model or a person scoring the output — can only check what it was handed. If its prompt contains just the AI output and no reference facts to compare against, it can judge tone or fluency but not whether the claim is true. The score log then answers “what did we decide?” and not “was it right?” For accountability you need the inputs stored alongside the verdict: which facts went in, which came back. Otherwise the trail records a number whose basis nobody can reconstruct.
AI Evals are especially important since they bridge the gap between what engineering prioritizes (MMLU scores) and what users expect (quality, reliability, safety, and performance). And AI PMs can decide the next step based on them: whether to train the model, pivot, or put it on hold.
The "ammunition" part is spot on. I couldn't have described it better. Even if some countries have more severe laws than others, as a customer, I'd be furious if I were told "yes" when it's actually "no."
We need better methods for catching all of these before things burn, as you said. I completely agree!
What reference facts reach the judge, whose prompt shows only [AI OUTPUT]?
Ah that is the scariest part of automation in high stakes settings! If the audit trail is just a black box, accountability goes out the window. 🫣
Agreed, though the gap is a bit narrower than a full black box. An evaluator — a second model or a person scoring the output — can only check what it was handed. If its prompt contains just the AI output and no reference facts to compare against, it can judge tone or fluency but not whether the claim is true. The score log then answers “what did we decide?” and not “was it right?” For accountability you need the inputs stored alongside the verdict: which facts went in, which came back. Otherwise the trail records a number whose basis nobody can reconstruct.
This article is right on spot between user focus and technical accuracy. Very helpful!
Thanks Benedikt glad you’ve found it useful! Hope this helps others too :)
Yes, I've actually been through this... not fun, but fixing it asap is the key... and not making the same mistakes.
I love it! Learn from your mistakes and document them so you don't repeat them, for everything not just AI.
Documentation isn't fun, either, but it's certainly better than having errors like this in production!
Amen to that. Making the same mistakes is a killer of progress.
Great insight, Elena.
AI Evals are especially important since they bridge the gap between what engineering prioritizes (MMLU scores) and what users expect (quality, reliability, safety, and performance). And AI PMs can decide the next step based on them: whether to train the model, pivot, or put it on hold.
Yes, Vishal! That's a perfect summary of everything. You put it perfectly! 👏
The "ammunition" part is spot on. I couldn't have described it better. Even if some countries have more severe laws than others, as a customer, I'd be furious if I were told "yes" when it's actually "no."
We need better methods for catching all of these before things burn, as you said. I completely agree!