Text2QASampleEvaluator
About 608 wordsAbout 2 min
2025-10-09
📘 Overview
The Text2QASampleEvaluator is an operator designed to score and evaluate generated Question-Answer (QA) pairs. It assesses multiple quality dimensions, including question quality, answer alignment, answer verifiability, and downstream value, by leveraging a Large Language Model (LLM) to provide both a numerical grade and textual feedback for each dimension.
__init__ function
def __init__(self, llm_serving: LLMServingABC)| Parameter | Type | Default Value | Description |
|---|---|---|---|
| llm_serving | LLMServingABC | Required | Large language model serving instance for executing inference and generation. |
Prompt Template Descriptions
| Prompt Template Name | Primary Purpose | Applicable Scenarios | Feature Description |
|---|---|---|---|
run function
def run(
self,
storage: DataFlowStorage,
input_question_key: str = "generated_question",
input_answer_key: str = "generated_answer",
output_question_quality_key: str = "question_quality_grades",
output_question_quality_feedback_key: str = "question_quality_feedbacks",
output_answer_alignment_key: str = "answer_alignment_grades",
output_answer_alignment_feedback_key: str = "answer_alignment_feedbacks",
output_answer_verifiability_key: str = "answer_verifiability_grades",
output_answer_verifiability_feedback_key: str = "answer_verifiability_feedbacks",
output_downstream_value_key: str = "downstream_value_grades",
output_downstream_value_feedback_key: str = "downstream_value_feedbacks"
)| Parameter | Type | Default Value | Description |
|---|---|---|---|
| storage | DataFlowStorage | Required | DataFlow storage instance for reading and writing data. |
| input_question_key | str | "generated_question" | Input column name for the question field. |
| input_answer_key | str | "generated_answer" | Input column name for the answer field. |
| output_question_quality_key | str | "question_quality_grades" | Output column name for question quality scores. |
| output_question_quality_feedback_key | str | "question_quality_feedbacks" | Output column name for question quality feedback. |
| output_answer_alignment_key | str | "answer_alignment_grades" | Output column name for answer alignment scores. |
| output_answer_alignment_feedback_key | str | "answer_alignment_feedbacks" | Output column name for answer alignment feedback. |
| output_answer_verifiability_key | str | "answer_verifiability_grades" | Output column name for answer verifiability scores. |
| output_answer_verifiability_feedback_key | str | "answer_verifiability_feedbacks" | Output column name for answer verifiability feedback. |
| output_downstream_value_key | str | "downstream_value_grades" | Output column name for downstream value scores. |
| output_downstream_value_feedback_key | str | "downstream_value_feedbacks" | Output column name for downstream value feedback. |
🧠 Example Usage
🧾 Default Output Format (Output Format)
| Field | Type | Description |
|---|---|---|
| generated_question | str | Input question text. |
| generated_answer | str | Input answer text. |
| question_quality_grades | float | Model-generated score for question quality. |
| question_quality_feedbacks | str | Model-generated feedback for question quality. |
| answer_alignment_grades | float | Model-generated score for answer alignment. |
| answer_alignment_feedbacks | str | Model-generated feedback for answer alignment. |
| answer_verifiability_grades | float | Model-generated score for answer verifiability. |
| answer_verifiability_feedbacks | str | Model-generated feedback for answer verifiability. |
| downstream_value_grades | float | Model-generated score for downstream value. |
| downstream_value_feedbacks | str | Model-generated feedback for downstream value. |
Example Input:
{
"generated_question": "What is the primary function of the mitochondria in a cell?",
"generated_answer": "The mitochondria is known as the powerhouse of the cell because it generates most of the cell's supply of adenosine triphosphate (ATP), used as a source of chemical energy."
}Example Output:
{
"generated_question": "What is the primary function of the mitochondria in a cell?",
"generated_answer": "The mitochondria is known as the powerhouse of the cell because it generates most of the cell's supply of adenosine triphosphate (ATP), used as a source of chemical energy.",
"question_quality_grades": 5.0,
"question_quality_feedbacks": "The question is clear, specific, and directly addresses a core concept in cell biology. It is well-formed and unambiguous.",
"answer_alignment_grades": 5.0,
"answer_alignment_feedbacks": "The answer directly and accurately addresses the question, explaining the main function of the mitochondria and providing context with the term 'powerhouse of the cell'.",
"answer_verifiability_grades": 5.0,
"answer_verifiability_feedbacks": "The provided answer is a well-established scientific fact that can be easily verified in any standard biology textbook or reliable scientific source.",
"downstream_value_grades": 4.0,
"downstream_value_feedbacks": "This is a fundamental knowledge question that is very useful for educational purposes, such as in a biology quiz or study guide. Its value is high in a learning context."
}
