Skip to content

DataFlow Documentation

Main Navigation

Basic Info

Start with Dataflow

Case: PromptedGenerator

Case: Batch PDF Extracting

Pipelines

Knowledge Base Cleaning Pipeline

Function Calling Data Synthesis Pipeline

API Reference Developer Guide

English

简体中文

English

简体中文

Appearance

Sidebar Navigation

home

serving

serving_main

operators

agentic_rag

eval

AgenticRAGQAF1SampleEvaluator

generate

AgenticRAGAtomicTaskGenerator

AgenticRAGDepthQAGenerator

AgenticRAGWidthQAGenerator

chemistry

eval

EvaluateSmilesEquivalence

generate

ExtractSmilesFromText

code

eval

CodeAutoGeneratedSampleEvaluator

CodeDocumentQualitySampleEvaluator

CodeEncodedDataSampleEvaluator

CodeLengthSampleEvaluator

CodeQualitySampleEvaluator

CodeSandboxSampleEvaluator

CodeTextCompositionSampleEvaluator

filter

CodeAutoGeneratedFilter

CodeDocumentQualityFilter

CodeEncodedDataFilter

CodeFileTypeContentFilter

CodeGenericScoreFilter

CodeLengthSampleFilter

CodeQualityScoreFilter

CodeTextCompositionFilter

generate

CodeCodeToInstructionGenerator

CodeInstructionToCodeGenerator

conversations

generate

AtomTaskGenerator

CompositionTaskFilter

ConsistentChatGenerator

FunctionGenerator

MultiTurnConversationGenerator

ParaSeqTaskGenerator

ScenarioExpander

ScenarioExtractor

SequentialTaskGenerator

core_speech

generate

Speech2TextGenerator

core_text

eval

BenchDatasetEvaluator

PromptedEvaluator

Text2QASampleEvaluator

filter

GeneralFilter

KCenterGreedyFilter

PromptedFilter

generate

PairedPromptedGenerator

PromptedGenerator

RandomDomainKnowledgeRowGenerator

Text2MultiHopQAGenerator

Text2QAGenerator

refine

PandasOperator

PromptedRefiner

core_vision

generate

PromptedVQAGenerator

db

generate

DBOperator

general_text

eval

BertSampleEvaluator

BleuSampleEvaluator

CiderSampleEvaluator

LangkitSampleEvaluator

LexicalDiversitySampleEvaluator

NgramSampleEvaluator

PerspectiveSampleEvaluator

PresidioSampleEvaluator

Task2VecDatasetEvaluator

VendiDatasetEvaluator

filter

AlphaWordsFilter

BlocklistFilter

CapitalWordsFilter

CharNumberFilter

ColonEndFilter

ContentNullFilter

CurlyBracketFilter

HashDeduplicateFilter

HtmlEntityFilter

IDCardFilter

LLMLanguageFilter

LangkitFilter

LanguageFilter

LexicalDiversityFilter

LineEndWithEllipsisFilter

LineStartWithBulletpointFilter

LineWithJavascriptFilter

LoremIpsumFilter

MeanWordLengthFilter

MinHashDeduplicateFilter

NgramFilter

NgramHashDeduplicateFilter

NoPuncFilter

PerspectiveFilter

PresidioFilter

SemDeduplicateFilter

SentenceNumberFilter

SimHashDeduplicateFilter

SpecialCharacterFilter

StopWordFilter

SymbolWordRatioFilter

UniqueWordsFilter

WatermarkFilter

WordNumberFilter

refine

HtmlEntityRefiner

HtmlUrlRemoverRefiner

LowercaseRefiner

NERRefiner

PIIAnonymizeRefiner

ReferenceRemoverRefiner

RemoveContractionsRefiner

RemoveEmojiRefiner

RemoveEmoticonsRefiner

RemoveExtraSpacesRefiner

RemoveImageRefsRefiner

RemoveNumberRefiner

RemovePunctuationRefiner

RemoveRepetitionsPunctuationRefiner

RemoveStopwordsRefiner

SpellingCorrectionRefiner

StemmingLemmatizationRefiner

TextNormalizationRefiner

knowledge_cleaning

generate

FileOrURLToMarkdownConverterBatch

KBCChunkGenerator

KBCChunkGeneratorBatch

KBCMultiHopQAGeneratorBatch

KBCTextCleaner

KBCTextCleanerBatch

MathBookQuestionExtract

rare

generate

RAREBM25HardNegGenerator

RAREDoc2QueryGenerator

RAREReasonDistillGenerator

reasoning

eval

ReasoningCategoryDatasetEvaluator

ReasoningDifficultyDatasetEvaluator

ReasoningQuestionCategorySampleEvaluator

ReasoningQuestionDifficultySampleEvaluator

ReasoningTokenDatasetEvaluator

filter

ReasoningAnswerFormatterFilter

ReasoningAnswerGroundTruthFilter

ReasoningAnswerModelJudgeFilter

ReasoningAnswerNgramFilter

ReasoningAnswerPipelineRootFilter

ReasoningAnswerTokenLengthFilter

ReasoningQuestionFilter

generate

ReasoningAnswerExtractionQwenMathEvalGenerator

ReasoningAnswerGenerator

ReasoningPretrainFormatConvertGenerator

ReasoningPseudoAnswerGenerator

ReasoningQuestionGenerator

text2sql

eval

SQLComponentClassifier

SQLExecutionClassifier

filter

SQLConsistencyFilter

SQLExecutionFilter

generate

SQLByColumnGenerator

SQLGenerator

SQLVariationGenerator

Text2SQLCoTGenerator

Text2SQLPromptGenerator

Text2SQLQuestionGenerator

text_pt

eval

DebertaV3SampleEvaluator

FineWebEduSampleEvaluator

MetaSampleEvaluator

PairQualSampleEvaluator

PerplexitySampleEvaluator

QuratingSampleEvaluator

TextbookSampleEvaluator

filter

CCNetDeduplicateFilter

DebertaV3Filter

FineWebEduFilter

PairQualFilter

PerplexityFilter

QuratingFilter

TextbookFilter

generate

Phi4QAGenerator

text_sft

eval

AlpagasusSampleEvaluator

DeitaComplexitySampleEvaluator

DeitaQualitySampleEvaluator

InstagSampleEvaluator

RMSampleEvaluator

SuperfilteringSampleEvaluator

TreeinstructSampleEvaluator

filter

AlpagasusFilter

DeitaComplexityFilter

DeitaQualityFilter

InstagFilter

RMFilter

SuperfilteringFilter

TreeinstructFilter

generate

CondorGenerator

SFTGeneratorSeed

refine

CondorRefiner

On this page

Home
home

home

About 15 wordsLess than 1 minute

2025-06-13

This is API Reference

Contributors: Ma, Xiaochen

Next pageserving_main

Powered by VuePress & vuepress-theme-plume