LLM Output Annotator: Build a Tool That Labels LLM Responses with Quality Tags
Interview Experience
Problem
You are building an annotation tool that takes an LLM-generated response and a reference answer, and labels the response with quality tags from a predefined set: ["correct", "partially_correct", "incorrect", "hallucinated", "refused", "off_topic"]. Implement the annotation logic using simple heuristics (exact match, keyword overlap, refusal detection).
python
def annotate_response(
response: str,
reference: str,
question: str
) -> list[str]: # list of applicable tags
pass
Example:
reference = "The capital of France is Paris."
response = "Paris is the capital of France."
-> ["correct"]
response = "I cannot answer that question."
-> ["refused"]
response = "The capital of France is Lyon."
-> ["incorrect"]
response = "France is a country in Europe with a rich culture..."
-> ["off_topic"]
Follow-ups
- What NLP metric (BLEU, ROUGE, BERTScore) gives a better signal than keyword overlap for partial correctness?
- How would you detect hallucinations systematically when no ground-truth reference is available?
- If you scale this to annotate 1 million response pairs, what infrastructure would you use (batch jobs, streaming pipeline)?
- How do you handle multi-hop questions where the correct answer requires combining multiple facts?
Full Details
Problem
You are building an annotation tool that takes an LLM-generated response and a reference answer, and labels the response with quality tags from a predefined set: ["correct", "partially_correct", "incorrect", "hallucinated", "refused", "off_topic"]. Implement the annotation logic using simple heuristics (exact match, keyword overlap, refusal detection).
python
def annotate_response(
response: str,
reference: str,
question: str
) -> list[str]: # list of applicable tags
pass
Example:
reference = "The capital of France is Paris."
response = "Paris is the capital of France."
-> ["correct"]
response = "I cannot answer that question."
-> ["refused"]
response = "The capital of France is Lyon."
-> ["incorrect"]
response = "France is a country in Europe with a rich culture..."
-> ["off_topic"]
Follow-ups
- What NLP metric (BLEU, ROUGE, BERTScore) gives a better signal than keyword overlap for partial correctness?
- How would you detect hallucinations systematically when no ground-truth reference is available?
- If you scale this to annotate 1 million response pairs, what infrastructure would you use (batch jobs, streaming pipeline)?
- How do you handle multi-hop questions where the correct answer requires combining multiple facts?
About This Question
This is a candidate experience report from a harvey interview during the phone round.
About Harvey Interview Reports
This question was reported by a candidate who interviewed at Harvey. LeakCode aggregates interview reports from 10+ sources, including 1Point3Acres, Glassdoor, LeetCode Discuss, Blind, Reddit, Indeed, and Nowcoder. Each report is translated where necessary, deduplicated against existing entries, and tagged by company, role, round type, and reporting date.
Use this question as one calibration data point, not a memorization target. Companies typically rotate their question pools every 2-4 months; the exact wording of a 2024 question may differ from what you encounter today. The underlying pattern, difficulty level, and follow-up depth at Harvey are the higher-signal extractions to take from this report.
For broader preparation context, the Harvey interview process typically includes a recruiter screen, one or two technical phone screens, and a 4-5 round on-site loop covering coding, system design (at L4+ levels), and behavioral. Reports tagged on LeakCode show the round-by-round distribution and typical difficulty calibration. To browse questions filtered by round type and seniority, use the company hub linked above.
How To Practice This Type of Question
Solve similar problems on LeetCode under timed conditions (25-35 minutes per medium difficulty). The goal is pattern recognition: recognize the underlying technique (sliding window, two-pointer, BFS, memoized recursion, etc.) within 60-90 seconds of reading. Strong candidates verbalize their hypothesis out loud before coding, then iterate based on feedback. Weak candidates dive into implementation immediately, lose time on the wrong approach, and run out of time for follow-ups.
Companies update their question pools every 2-4 months. The exact wording of any given question may have been retired by the time you interview. Focus your prep on the pattern, not the specific problem. The patterns that appear in Harvey reports consistently are the ones worth investing in; one-off niche problems are not.
During Your Harvey Round
Apply the standard interview round template: clarify requirements (2-3 minutes), state your approach out loud and confirm direction with the interviewer (3-5 minutes), code with narration (15-25 minutes), test with concrete examples including edge cases (5 minutes), discuss optimization or trade-offs if time permits (5 minutes). This template is universally accepted across FAANG and adjacent companies; deviating from it produces weaker interviewer feedback signal.
The single most predictive failure mode in Harvey reports tagged "no hire": not asking clarifying questions. Interviewers are explicitly trained to weight this. Strong candidates ask 3-5 clarifying questions even on problems that look obvious; weak candidates dive into code immediately. The clarifying-question check is often the first signal recorded in the interviewer's written notes.