LLM Evaluator (Model Response Analyst)

About the role

We are looking for a detail‑oriented LLM Evaluator to assess and improve the performance of large language models. You will work remotely with a global team, reviewing AI‑generated content for accuracy, safety, bias, and alignment with our guidelines.

Key responsibilities

• Evaluate and rank model‑generated text using complex rubrics covering factuality, coherence, safety, instruction‑following, and creativity.
• Compare multiple responses to the same prompt, select the preferred output, and justify your choice.
• Provide concise feedback to modeling and training teams about recurring failure patterns.
• Craft adversarial prompts to expose biased, harmful, or insecure model behavior.
• Collaborate with QA to refine evaluation guidelines for ambiguous cases.
• Participate in cross‑checking sessions to ensure inter‑rater reliability.
• Analyze underperforming outputs, hypothesize root causes, and flag novel behaviors to researchers.

Required profile

• Minimum 2 years of professional experience in computational linguistics, data analysis, technical writing, NLP QA, or cognitive science.
• Bachelor’s degree in Computer Science or a related field.
• Strong ability to explain why a model output is good or bad based on logic, tone, factuality, and instruction adherence.
• Experience with Reinforcement Learning from Human Feedback (RLHF) data collection and evaluation‑team consistency.

Required skills

• Prompt engineering
• RLHF data collection
• Dataset sourcing, cleaning, and annotation
• A/B testing for AI models
• IAA score analysis and calibration

What we offer

• Fully remote work with flexible hours
• Collaboration with leading AI researchers
• Opportunities to influence the safety and reliability of cutting‑edge language models