📘 Overview

The Text2QASampleEvaluator is an operator designed to score and evaluate generated Question-Answer (QA) pairs. It assesses multiple quality dimensions, including question quality, answer alignment, answer verifiability, and downstream value, by leveraging a Large Language Model (LLM) to provide both a numerical grade and textual feedback for each dimension.

__init__ function

def __init__(self, llm_serving: LLMServingABC)

Parameter	Type	Default Value	Description
llm_serving	LLMServingABC	Required	Large language model serving instance for executing inference and generation.

Prompt Template Descriptions

Prompt Template Name	Primary Purpose	Applicable Scenarios	Feature Description

run function

def run(
    self,
    storage: DataFlowStorage,
    input_question_key: str = "generated_question",
    input_answer_key: str = "generated_answer",
    output_question_quality_key: str = "question_quality_grades",
    output_question_quality_feedback_key: str = "question_quality_feedbacks",
    output_answer_alignment_key: str = "answer_alignment_grades",
    output_answer_alignment_feedback_key: str = "answer_alignment_feedbacks",
    output_answer_verifiability_key: str = "answer_verifiability_grades",
    output_answer_verifiability_feedback_key: str = "answer_verifiability_feedbacks",
    output_downstream_value_key: str = "downstream_value_grades",
    output_downstream_value_feedback_key: str = "downstream_value_feedbacks"
)

Parameter	Type	Default Value	Description
storage	DataFlowStorage	Required	DataFlow storage instance for reading and writing data.
input_question_key	str	"generated_question"	Input column name for the question field.
input_answer_key	str	"generated_answer"	Input column name for the answer field.
output_question_quality_key	str	"question_quality_grades"	Output column name for question quality scores.
output_question_quality_feedback_key	str	"question_quality_feedbacks"	Output column name for question quality feedback.
output_answer_alignment_key	str	"answer_alignment_grades"	Output column name for answer alignment scores.
output_answer_alignment_feedback_key	str	"answer_alignment_feedbacks"	Output column name for answer alignment feedback.
output_answer_verifiability_key	str	"answer_verifiability_grades"	Output column name for answer verifiability scores.
output_answer_verifiability_feedback_key	str	"answer_verifiability_feedbacks"	Output column name for answer verifiability feedback.
output_downstream_value_key	str	"downstream_value_grades"	Output column name for downstream value scores.
output_downstream_value_feedback_key	str	"downstream_value_feedbacks"	Output column name for downstream value feedback.

🧠 Example Usage

🧾 Default Output Format (Output Format)

Field	Type	Description
generated_question	str	Input question text.
generated_answer	str	Input answer text.
question_quality_grades	float	Model-generated score for question quality.
question_quality_feedbacks	str	Model-generated feedback for question quality.
answer_alignment_grades	float	Model-generated score for answer alignment.
answer_alignment_feedbacks	str	Model-generated feedback for answer alignment.
answer_verifiability_grades	float	Model-generated score for answer verifiability.
answer_verifiability_feedbacks	str	Model-generated feedback for answer verifiability.
downstream_value_grades	float	Model-generated score for downstream value.
downstream_value_feedbacks	str	Model-generated feedback for downstream value.

Example Input:

{
  "generated_question": "What is the primary function of the mitochondria in a cell?",
  "generated_answer": "The mitochondria is known as the powerhouse of the cell because it generates most of the cell's supply of adenosine triphosphate (ATP), used as a source of chemical energy."
}

Example Output:

{
  "generated_question": "What is the primary function of the mitochondria in a cell?",
  "generated_answer": "The mitochondria is known as the powerhouse of the cell because it generates most of the cell's supply of adenosine triphosphate (ATP), used as a source of chemical energy.",
  "question_quality_grades": 5.0,
  "question_quality_feedbacks": "The question is clear, specific, and directly addresses a core concept in cell biology. It is well-formed and unambiguous.",
  "answer_alignment_grades": 5.0,
  "answer_alignment_feedbacks": "The answer directly and accurately addresses the question, explaining the main function of the mitochondria and providing context with the term 'powerhouse of the cell'.",
  "answer_verifiability_grades": 5.0,
  "answer_verifiability_feedbacks": "The provided answer is a well-established scientific fact that can be easily verified in any standard biology textbook or reliable scientific source.",
  "downstream_value_grades": 4.0,
  "downstream_value_feedbacks": "This is a fundamental knowledge question that is very useful for educational purposes, such as in a biology quiz or study guide. Its value is high in a learning context."
}