Multi-Turn Evaluation Support
Evaluate conversational AI across full dialogue sessions with turn-level and conversation-level metrics. This release introduces support for multi-turn datasets where each item contains a sequence of conversation turns rather than a single input/output pair.
New Metrics
- Conversation Completeness - Measures whether the assistant addressed all user questions and requests across the full dialogue
- Knowledge Retention - Evaluates whether the assistant maintains context and remembers earlier information throughout the conversation
- Role Adherence - Checks if the assistant stays within its defined role and constraints across all turns
- Turn Relevancy - Scores each individual response for relevance to the immediate user message
G-Eval Custom Metrics
Define custom evaluation metrics using natural language criteria without writing raw prompts. The system automatically generates chain-of-thought scoring prompts based on your criteria description.
How It Works
- Describe your evaluation criteria in plain language
- The engine generates a CoT prompt for consistent scoring
- Metrics are powered by the existing LLM-as-judge engine
- Results appear alongside built-in metrics in the eval dashboard
Example
metric:
name: "Tone Consistency"
criteria: "The assistant maintains a professional and helpful tone throughout all conversation turns"
score_type: "numeric"
scale: [1, 5]
LLM-as-Judge Engine Updates
The underlying judgment engine now supports turn-level scoring, allowing granular analysis of individual conversation turns within a multi-turn evaluation run.