How to write effective AI data instructions
Create annotation and evaluation instructions people can apply consistently at production scale.
4 min readInstructions are part of the dataset. If contributors interpret the task differently, adding more people increases inconsistency rather than throughput.
State the objective first
Explain what the output is used for, then define the decision in plain language. Contributors make better edge-case choices when they understand the purpose.
Make the standard observable
Replace broad words like good or safe with criteria a reviewer can identify.
- Define every label
- Show positive and negative examples
- Explain precedence when rules conflict
- Provide a skip or escalation path
- Version material changes
Test with disagreement
Run the same pilot items through multiple contributors. Review where they differ and improve the instruction before treating disagreement as worker error.
Write for the decision point
Put the rule beside the moment it applies. Contributors should not have to remember an exception from a distant introduction while making a complex choice. Use short sections, descriptive headings and a consistent order: objective, input, required action, criteria, exceptions and submission checks.
Use examples as executable specifications
Examples reveal boundaries that prose hides. Include straightforward positives and negatives, then add near-boundary cases with explanations. Avoid examples that merely repeat the rule; show why a tempting alternative is wrong and which evidence changes the answer.
Design an escalation path
Instructions cannot anticipate every input. Define when contributors should abstain, flag sensitive material or request adjudication. Penalising appropriate uncertainty encourages guessing and contaminates the dataset with confident but unsupported decisions.
- Missing or corrupted source material
- Conflicting rules or examples
- Specialist knowledge beyond the stated role
- Sensitive content requiring restricted handling
- Cases that expose a new taxonomy gap
Maintain instructions as a product
Version every material change and record which batches used it. Analyse questions, disagreement and review corrections to find weak sections. Test revisions on held-out examples before switching production, then communicate the change instead of silently editing the standard underneath active work.