How to choose a human data vendor for AI training

01

Define the job

• Preference data (pairwise comparisons, rankings) for RLHF or DPO
• Supervised fine-tuning data (expert-written demonstrations)
• Evaluations and benchmarks (rubric grading, held-out test sets)
• Red teaming (adversarial testing for safety and misuse)
• RL environments and agent trajectories (tasks with automatic checks)
• Collection and annotation (speech, image, video, LiDAR, text)

Write one sentence on what “done” looks like, including volume, turnaround, and acceptance criteria.

02

Choose a buying model

• Managed service: the vendor recruits, trains, QA’s and delivers. Best when you lack an ops team.
• Expert network or marketplace: you get access to vetted people; more of the task design is on you.
• Self-serve platform: you run studies or labeling yourself, pay per participant or seat.
• Off-the-shelf datasets: licensed data available now; check license terms and exclusivity.

03

Match the expertise

Ask how experts are verified (credentials, tests, interviews), how many qualified people exist for your exact domain and language, and whether the same people stay on your project across rounds.

04

Inspect quality control

Ask for the QA design in writing: gold questions, consensus or adjudication, reviewer tiers, inter-annotator agreement reporting, and how disagreements are resolved. Run a paid pilot with two vendors on the same small batch and compare what survives your own review.

05

Security and compliance evidence

Ask for current reports, not badges: SOC 2 Type II report, ISO/IEC 27001 certificate and scope, HIPAA or other sector terms if relevant, data residency, subprocessors, retention and deletion, and whether workers access data on managed devices or in secure facilities. A certification on a website may cover only part of a company.

06

Commercials

Clarify unit of pricing (per task, per hour, per participant, per dataset), minimums, pilot pricing, rework policy, IP ownership of outputs, exclusivity, and termination. Only a few vendors publish prices; most quote per project.

07

Conflicts and ownership

Check ownership and major investors, and whether the vendor or its workforce also serves your direct competitors. Ask how project data is segregated.