Human data and AI training glossary
17 terms buyers meet when scoping human data, evaluation, and security requirements.
- RLHF (reinforcement learning from human feedback)
- Training a model using human judgments of which outputs are better, usually through a reward model.
- DPO (direct preference optimization)
- A method that trains directly on preference pairs without a separate reward model.
- SFT (supervised fine-tuning)
- Training on example inputs paired with ideal outputs written or approved by people.
- Preference data
- Human comparisons or rankings of model outputs.
- Red teaming
- Deliberate adversarial testing to find unsafe, insecure, or unwanted model behavior.
- Evaluation (eval)
- A structured test of model performance against tasks and grading criteria.
- Benchmark
- A fixed, reusable evaluation set used to compare models over time.
- RL environment
- A task setting with tools and automatic checks where a model or agent can act and be scored.
- Agent trajectory
- The recorded sequence of steps, tool calls, and results an agent takes on a task.
- Chain-of-thought (CoT) data
- Worked reasoning steps written or reviewed by people.
- Gold set
- Items with known correct answers used to check annotator quality.
- Inter-annotator agreement
- How consistently different people label the same item.
- Domain expert
- A contributor with verified professional or academic expertise in the task’s subject.
- Off-the-shelf (OTS) dataset
- An existing dataset licensed as-is.
- SOC 2 Type II
- An independent auditor’s report on how a company’s security controls operated over a period.
- ISO/IEC 27001
- An international certification for an information security management system.
- ISO/IEC 42001
- An international certification for an AI management system.