Blogs
Notes on building, connecting, and deploying agents.
Engineering deep-dives, product updates, and field notes from teams shipping agents in production.
Back Now
Why CII Evaluations Matter
Why CII Evaluations Matter
Why CII Evaluations Matter
Why CII Evaluations Matter
Why CII Evaluations Matter
Every team that ships agents eventually builds a dashboard. Latency, cost, success rate a wall of charts that looks reassuring right up until a customer reports something the charts never caught. The problem isn’t the dashboard. It’s when it tells you.
Dashboards are a trailing indicator. By the time a regression shows up as a dip in your success-rate chart, it has already reached production and, worse, real users. The fix is to move quality checks to the same place we catch every other regression: continuous integration.
The case for evals-as-tests
An eval is just a test with a fuzzier assertion. Instead of expect(x).toBe(2), you assert that an agent’s output satisfies a rubric: did it resolve the ticket, stay in policy, and avoid hallucinating an order number? Run that assertion on every pull request and quality stops being a vibe.
Why CII Evaluations Matter
Every team that ships agents eventually builds a dashboard. Latency, cost, success rate a wall of charts that looks reassuring right up until a customer reports something the charts never caught. The problem isn’t the dashboard. It’s when it tells you.
Dashboards are a trailing indicator. By the time a regression shows up as a dip in your success-rate chart, it has already reached production and, worse, real users. The fix is to move quality checks to the same place we catch every other regression: continuous integration.
The case for evals-as-tests
An eval is just a test with a fuzzier assertion. Instead of expect(x).toBe(2), you assert that an agent’s output satisfies a rubric: did it resolve the ticket, stay in policy, and avoid hallucinating an order number? Run that assertion on every pull request and quality stops being a vibe.
Every team that ships agents eventually builds a dashboard. Latency, cost, success rate a wall of charts that looks reassuring right up until a customer reports something the charts never caught. The problem isn’t the dashboard. It’s when it tells you.
Dashboards are a trailing indicator. By the time a regression shows up as a dip in your success-rate chart, it has already reached production and, worse, real users. The fix is to move quality checks to the same place we catch every other regression: continuous integration.
The case for evals-as-tests
An eval is just a test with a fuzzier assertion. Instead of expect(x).toBe(2), you assert that an agent’s output satisfies a rubric: did it resolve the ticket, stay in policy, and avoid hallucinating an order number? Run that assertion on every pull request and quality stops being a vibe.
