Gate pull requests on evals
Connect
SPIRITT
HoneyHiveHoneyHive
Ship AI you can measure
SPIRITT runs your HoneyHive evaluation loop. It instruments your app, turns production traces into datasets, defines metrics, runs experiments and compares results before a release goes out.
What you can do with HoneyHive.
Traces from every run
Log sessions, model calls and tool events, then attach feedback and metrics as results come in.
Datasets from real traffic
Turn production events into datapoints with ground truth, history and metadata for repeatable tests.
Releases backed by evidence
Run evaluations, compare runs side by side and keep metrics and configurations current.
Works with your other apps.
Get started in three steps.
01
Open your workspace
Start a SPIRITT workspace for the AI product you want measured and improved.
02
Connect HoneyHive
Connect HoneyHive with access to your projects, plus the repository and data sources the mission needs.
03
Delegate an outcome
Name the behavior you care about. SPIRITT instruments, evaluates and reports, then acts on what the numbers show.
Hand over a mission.
Choose a mission. Make it yours.
Put HoneyHive to work.
Ship AI you can measure
Questions
HoneyHive, with SPIRITT.
01What can SPIRITT log to HoneyHive?+
Sessions, model calls, tool calls and chain steps, individually or in batches, plus feedback, metrics and outputs attached to existing events.
02Can it build evaluation datasets?+
Yes. It creates datasets, adds datapoints with inputs, ground truth and history, and bulk-imports production events.
03How does it compare releases?+
It starts and closes evaluation runs, then compares two runs by metrics, datapoints and matched event pairs.
04What kinds of metrics can it define?+
Python code checks, LLM judges, human review and composite metrics, with thresholds and production toggles.
05Can it manage configurations?+
Yes. It creates and updates model and pipeline configurations by environment, so a winning setup reaches production cleanly.
06Can it add HoneyHive to our codebase?+
Yes. With your repository connected, SPIRITT writes the instrumentation, ships it and confirms traces arrive.