📘 Overview

AlpagasusFilter is an operator designed to filter data based on scores generated by the AlpagasusSampleEvaluator. It leverages a Large Language Model (LLM) to assess the quality (or another specified dimension) of an instruction-following sample and retains only those samples that fall within a specified score range.

__init__

def __init__(self, min_score=3, max_score=5, llm_serving: LLMServingABC = None, dimension='quality')

Parameter	Type	Default	Description
min_score	int	3	The minimum score threshold for a sample to be kept.
max_score	int	5	The maximum score threshold for a sample to be kept.
llm_serving	LLMServingABC	None	The Large Language Model serving instance used for scoring.
dimension	str	'quality'	The evaluation dimension for scoring (e.g., 'quality', 'relevance').

Prompt Template Descriptions

Prompt Template Name	Primary Use	Applicable Scenarios	Feature Description

run

def run(self, storage: DataFlowStorage, input_instruction_key: str, input_input_key: str, input_output_key: str, output_key: str='AlpagasusScore')

Parameter	Type	Default	Description
storage	DataFlowStorage	Required	The DataFlow storage instance for reading and writing data.
input_instruction_key	str	Required	The column name for the instruction field.
input_input_key	str	Required	The column name for the input field.
input_output_key	str	Required	The column name for the output field.
output_key	str	'AlpagasusScore'	The column name where the generated score will be stored.

🧠 Example Usage

from dataflow.operators.text_sft.filter import AlpagasusFilter
from dataflow.utils.storage import FileStorage
from dataflow.utils.llm_serving import APILLMServing_request

# Prepare storage with SFT-style data
storage = FileStorage(first_entry_file_name="sft_data.jsonl")

# Initialize LLM serving
llm_serving = APILLMServing_request(
    api_url="http://<your_llm_api_endpoint>",
    model_name="<your_model_name>",
)

# Initialize and run the filter
alp_filter = AlpagasusFilter(
    min_score=3,
    max_score=5,
    llm_serving=llm_serving,
    dimension="quality",
)
alp_filter.run(
    storage.step(),
    input_instruction_key="instruction",
    input_input_key="input",
    input_output_key="output",
    output_key="AlpagasusScore",
)

🧾 Output Format

The operator appends a new column (specified by output_key) containing the generated score to the input DataFrame and filters the rows based on min_score and max_score.

Field	Type	Description
input_fields	-	All original fields from the input data are preserved.
AlpagasusScore	int	The quality score generated by the LLM, by default.

Example Input:

{
    "difficulty":"Easy",
    "instruction":"Imagine you are a school counselor tasked with promoting mental health among high school students. What are some basic steps you can take to create an environment that reduces the risk of psychological disorders among students? Consider actions such as organizing workshops or creating a supportive atmosphere.",
    "output":"Creating an environment that supports mental health and reduces the risk of psychological disorders among high school students requires a thoughtful and collaborative approach. Here are some steps a school counselor might take, along with practical examples and considerations for implementation:\n\n1. **Educate and Raise Awareness:**\n   - Organize workshops and seminars using successful models like the “Mind Matters Program,” which provides engaging sessions on stress management and coping skills. Partnering with local mental health organizations for expertise can enhance effectiveness.\n   - Use real-life stories and testimonials to normalize discussions around mental health and demonstrate that seeking help is a sign of strength.\n\n2. **Promote a Supportive Atmosphere:**\n   - Coordinate with teachers to implement inclusive classroom strategies, such as restorative circles or regular \"check-in\" moments, where students can openly share their feelings.\n   - Establish a peer support program, like a “Student Circle” where senior students mentor juniors, fostering a community of encouragement.\n\n3. **Provide Access to Resources:**\n   - Set up easy-access resources, such as a digital platform or app where students can find mental health information and local services. Collaborate with tech-savvy students or local developers for creation and maintenance.\n   - Introducing a virtual counseling option could cater to students who prefer more privacy when reaching out for help.\n\n4. **Develop and Implement Policies:**\n   - Work closely with the school's administration to craft and promote anti-bullying and mental health policies. Regularly update students on these policies through assemblies and classroom discussions.\n   - Creating a taskforce involving teachers, admin, and student leaders can ensure diverse perspectives and smoother policy implementation.\n\n5. **Incorporate Mental Health into the Curriculum:**\n   - Advocate for integrating emotional and social learning into existing classes. The use of programs like CASEL's SEL (Social Emotional Learning) framework can offer structure and resources for this integration.\n   - Providing teachers with training sessions on SEL can help them seamlessly incorporate into their teaching practices.\n\n6. **Engage Parents and Guardians:**\n   - Hold educational workshops for parents, focusing on understanding adolescent mental health and enhancing family communication. Using case studies from successful family engagement initiatives can illustrate impact and provide reassurance.\n   - Implement regular updates via newsletters that provide ongoing tips and strategies for parents to support their children.\n\n7. **Create Relaxation Spaces:**\n   - Explore partnerships with local businesses or artists to create inviting relaxation zones using minimal resources. For example, a student-led group could organize a \"Zen Corner\" with donated items.\n   - Share data on the mental health benefits of such spaces to justify allocation of existing school space.\n\n8. **Regular Check-ins:**\n   - Plan regular one-on-one or group check-ins. Using structured formats, such as a “Mood Meter” or guided journal prompts, can help facilitate meaningful dialogue.\n   - Encourage teachers to adopt short weekly emotional check-ins in classes to create consistency across student experiences.\n\nPrioritizing steps involves assessing the school’s current needs and resources, and initially focusing on foundational actions like early education and resource access. Collaborating with school leadership ensures alignment and fosters support, and addressing potential challenges fortifies initiatives against setbacks.\n\nBy implementing these strategies and involving the broader community, a school counselor can effectively cultivate a nurturing environment that champions mental health and mitigates psychological distress among students."
}

Example Output (if it passes the filter):

{
    "difficulty":"Easy",
    "instruction":"Imagine you are a school counselor tasked with promoting mental health among high school students. What are some basic steps you can take to create an environment that reduces the risk of psychological disorders among students? Consider actions such as organizing workshops or creating a supportive atmosphere.",
    "output":"Creating an environment that supports mental health and reduces the risk of psychological disorders among high school students requires a thoughtful and collaborative approach. Here are some steps a school counselor might take, along with practical examples and considerations for implementation:\n\n1. **Educate and Raise Awareness:**\n   - Organize workshops and seminars using successful models like the “Mind Matters Program,” which provides engaging sessions on stress management and coping skills. Partnering with local mental health organizations for expertise can enhance effectiveness.\n   - Use real-life stories and testimonials to normalize discussions around mental health and demonstrate that seeking help is a sign of strength.\n\n2. **Promote a Supportive Atmosphere:**\n   - Coordinate with teachers to implement inclusive classroom strategies, such as restorative circles or regular \"check-in\" moments, where students can openly share their feelings.\n   - Establish a peer support program, like a “Student Circle” where senior students mentor juniors, fostering a community of encouragement.\n\n3. **Provide Access to Resources:**\n   - Set up easy-access resources, such as a digital platform or app where students can find mental health information and local services. Collaborate with tech-savvy students or local developers for creation and maintenance.\n   - Introducing a virtual counseling option could cater to students who prefer more privacy when reaching out for help.\n\n4. **Develop and Implement Policies:**\n   - Work closely with the school's administration to craft and promote anti-bullying and mental health policies. Regularly update students on these policies through assemblies and classroom discussions.\n   - Creating a taskforce involving teachers, admin, and student leaders can ensure diverse perspectives and smoother policy implementation.\n\n5. **Incorporate Mental Health into the Curriculum:**\n   - Advocate for integrating emotional and social learning into existing classes. The use of programs like CASEL's SEL (Social Emotional Learning) framework can offer structure and resources for this integration.\n   - Providing teachers with training sessions on SEL can help them seamlessly incorporate into their teaching practices.\n\n6. **Engage Parents and Guardians:**\n   - Hold educational workshops for parents, focusing on understanding adolescent mental health and enhancing family communication. Using case studies from successful family engagement initiatives can illustrate impact and provide reassurance.\n   - Implement regular updates via newsletters that provide ongoing tips and strategies for parents to support their children.\n\n7. **Create Relaxation Spaces:**\n   - Explore partnerships with local businesses or artists to create inviting relaxation zones using minimal resources. For example, a student-led group could organize a \"Zen Corner\" with donated items.\n   - Share data on the mental health benefits of such spaces to justify allocation of existing school space.\n\n8. **Regular Check-ins:**\n   - Plan regular one-on-one or group check-ins. Using structured formats, such as a “Mood Meter” or guided journal prompts, can help facilitate meaningful dialogue.\n   - Encourage teachers to adopt short weekly emotional check-ins in classes to create consistency across student experiences.\n\nPrioritizing steps involves assessing the school’s current needs and resources, and initially focusing on foundational actions like early education and resource access. Collaborating with school leadership ensures alignment and fosters support, and addressing potential challenges fortifies initiatives against setbacks.\n\nBy implementing these strategies and involving the broader community, a school counselor can effectively cultivate a nurturing environment that champions mental health and mitigates psychological distress among students.",
    "AlpagasusScore":5.0
}