The QuratingSampleEvaluator evaluates text quality across four dimensions using the Qurating model (princeton-nlp/QuRater-1.3B): writing_style, required_expertise, facts_and_trivia, and educational_value. Each dimension returns a score between 0 and 1, providing a comprehensive assessment of overall text quality.

init

__init__(self, map_batch_size: int = 512, num_workers: int = 1, device_batch_size: int = 16, device: str = 'cuda', labels: list = ['writing_style', 'required_expertise', 'facts_and_trivia', 'educational_value'], model_cache_dir: str = './dataflow_cache')

Parameter	Type	Default	Description
map_batch_size	int	512	Batch size for the mapping function during data processing.
num_workers	int	1	Number of worker processes for data processing.
device_batch_size	int	16	Batch size for model inference on the specified device.
device	str	'cuda'	The device to run the model on (e.g., 'cuda', 'cpu').
labels	list	['writing_style', 'required_expertise', 'facts_and_trivia', 'educational_value']	A list of quality dimensions to evaluate.
model_cache_dir	str	'./dataflow_cache'	The directory to cache the downloaded model.

Prompt Template Descriptions

Prompt Template Name	Primary Use	Applicable Scenarios	Features

run

run(self, storage: DataFlowStorage, input_key: str, output_key: str)

Parameter	Type	Default	Description
storage	DataFlowStorage	Required	DataFlow storage instance for reading and writing data.
input_key	str	Required	The name of the input column containing the text to be evaluated.
output_key	str	Required	The name for the output column. Note: This operator adds new columns for each score based on the labels parameter.

🧠 Example Usage

from dataflow.operators.text_pt.eval import QuratingSampleEvaluator
from dataflow.utils.storage import FileStorage

# Prepare data and storage
storage = FileStorage(first_entry_file_name="pt_input.jsonl")

# Initialize and run the operator
qurating_evaluator = QuratingSampleEvaluator()
qurating_evaluator.run(
    storage.step(),
    input_key='raw_content'
)

🧾 Default Output Format

Field	Type	Description
...	...	Original columns from the input data.
QuratingWritingStyleScore	float	The score for the writing style dimension.
QuratingRequiredExpertiseScore	float	The score for the required expertise dimension.
QuratingFactsAndTriviaScore	float	The score for the facts and trivia dimension.
QuratingEducationalValueScore	float	The score for the educational value dimension.

Example Input:

{
    "raw_content": "AMICUS ANTHOLOGIES, PART ONE (1965-1972)\nFebruary 23, 2017 Alfred Eaker Leave a comment\nWith Dr. Terror's House of Horrors (1965, directed by Freddie Francis and written by Milton Subotsky) Amicus Productions (spearheaded by Subotsky and Max Rosenberg, who previously produced for Hammer and was a cousin to Doris Wishman) established itself as a vital competitor to Hammer Studios. Rather than imitating Hammer's modernization of Gothic classics, Amicus developed its own niche with omnibus films. They were successful enough to be in full-fledged production for a decade, establishing a reputation as the go-to studio for horror anthologies. This, their introductory portmanteau film, clearly influenced by EC Comics, sets a pattern of to-be-expected unevenness. Still, Amicus installs themselves as a horror studio to be reckoned with, sparing no expense in procuring Hammer's top actors: Peter Cushing (who would star in all but one of the Amicus anthologies) and Christopher Lee. For its wraparound segment, Dr. Terror's House of Horrors opens to the duo (among other passengers) on a train. Dr. Schreck ((\"Schreck\" is German for \"terror,\" and a nod to the famous star of F.W. Murnau's Nosferatu.)) (Cushing, saddled with a terrible German accent and glued on brows) pulls out a deck of tarot cards. \"Pick a card, any card, and tape it three times,\" Schreck tells his fellow passengers. Each participant will hear of a fate that may await them. Among the passengers is Christopher Lee who will, of course, factor into one of the five narratives.\nIn \"Werewolf,\" Neil McCallum is an architect renovating an old dark house, which turns out to be cursed. The title monster is featured in this pedestrian tale of ancestral revenge with a \"twist.\"\nWith Alan Freeman (better known as the U.K D.J. for \"Pick of the Pops\") served up as a snack for a venus fly trap, \"The Creeping Vine,\" thankfully doesn't take itself so seriously. It is refreshingly lightheaded hokum.\n\"Voodoo\" is the worst of the lot; badly dated in its stereotypes, with Kenny Lynch belting out a stolen voodoo tune.\n\"Disembodied Hand,\" has elitist art critic Franklin Marsh (Lee) driving artist Eric Landor (Michael Gough) to suicide. Landor's severed hand returns to exact revenge on the mean critic. It's in the spirit of The Beast with Five Fingers, among others, and chock-full of two-dimensional caricatures of both artists and critics. It holds no surprises, but with Lee and Gough engaged in a bit of whistling-while-they-work fun, it's easily the best episode.\n\"Vampire\" feature a young Donald Sutherland who discovers he is married to… a vampire! It barely raises a pulse.\nSeen today, Dr. Terror's House of Horrors is more camp than horror, and its appeal is one of genre nostalgia. Still, the phenomenal box office success of Dr. Terror green-lighted a second portmanteau film in 1967, entitled Torture Garden (directed by Freddie Francis and written by Robert Bloch). It contains no torture nor any garden. Burgess Meredith (in a preposterous disguise, reminding us of the Penguin) is Old Nick himself, going by the pseudonym of Dr. Diablo and moonlighting as a carnival barker who promises a tortuous exhibit that can reveal the future. \"You'll shake, you'll shiver, but it's all good fun,\" Diablo hammily tells his patrons. Unfortunately, only one of the four tales lives up to that promise.\n\"Enoch,\" is the opening narrative. Michael Bryant's inheritance money (from an uncle who took his time dying) is going to be spoiled by a mean ol' puddy tat with a lot of doubloons.\n\"Over Hollywood\" has Beverly Adams discovering the fountain of youth in Hollywood with robotic consequences.\n\"Mr. Steinway\" might be seen as a poor precursor to Stephen King's \"Christine,\" replacing a killer car with a killer piano. It's as absurd as it sounds.\nThe first three segments are sloppily written and executed with little enthusiasm; each progressively worse, but the final segment single-handedly salvages the anthology.\n\"The Man Who Collected Poe\" finds Jack Palance (playing against type) as an Edgar Allan Poe-obsessed geek who may have found his soulmate in fellow fanatic Peter Cushing. However, somebody's got something—or someone—hidden in the basement and … somebody's got the fever, which leads to a fiery finale. Cushing and Palance clearly enjoyed playing opposite one another and their chemistry, along with clever writing, making one wish the previous segments had been as enjoyable.\n1970's The House That Dripped Blood (directed by Peter Duffell and written by Robert Bloch) is a considerable improvement over its predecessors. Duffell lacks the visual astuteness of Freddie Francis, but he has superior stories to work with and a top notch cast. The connecting theme is the titular house, which has a bit of baggage left over from all who have resided there.\nIn \"Method For Murder,\" Denholm Elliott is a horror author who writes a character that becomes a tad too three-dimensional, much to his wife's peril.\n\"Waxworks\" stars Cushing as an uptight retired stockbroker and lifelong bachelor who visits a wax museum, only to see a figure of a woman whom he once was in love with. Obsession and unrequited love naturally go hand-in-hand, or head-on-plate.\nIn \"Sweets to the Sweet,\" Nyree Dawn Porter is hired to tutor a young, motherless child (Chloe Franks) who is unloved by her cold-hearted father, Christopher Lee. Without giving too much away, let's just say the underlying theme is one few filmmakers would dare tackle today.\n\"The Cloak\" is the most famous of the four episodes, remembered fondly for its absurd humor. It stars John Pertwee (best known for his portrayal of Dr. Who) as an actor who mantles the cloak of a purported actual vampire. Hammer favorite Ingrid Pitt bares her fangs and, of course, a bit more.\nAll four episodes feature strong acting, which is a rarity in contemporary horror and should be a model for genre filmmakers. Elliot's restrained performance in \"Method For Murder\" is admirable enough to forgive the predictable \"twist.\" The stylish \"Waxworks\" features an equally stylish performance from Cushing, although narratively it is the thinnest episode. \"Sweets to the Sweet\" is psychologically intense with three powerhouse performances, making it the strongest entry. Although John Pertwee is a bit on-the-sleeve in \"The Cloak,\" his performance suits the tone; but, he's no match for Pitt.\nAsylum (1972, directed by Roy Ward Baker and written by Robert Bloch) is often cited as the best of the Amicus anthologies. It opens on Dr. Martin (Robert Powell, best known as the blue-eyed Anglo-Saxon savior plopped into the Middle East in Franco Zeferelli's Jesus of Nazareth) showing up for his scheduled job interview with Dr. Starr for a position at the Dunsmoor Asylum. Martin is met by Dr. Rutherford (Patrick Magee), however, and informed that Dr. Starr is now a patient after going insane and becoming violent. Rutherford devises a test for Martin: he will interview four patients and if he can guess which one is Dr. Starr, then he will be hired. Naturally, this segues into four tales from Mr. Bloch.\nIn \"Frozen Fear,\" Walter (Richard Todd) is having a sordid affair with Bonnie (Barbara Parkins). When his wife Ruth (Sylvia Syms) won't give him a divorce, Walter grabs an axe and fills his basement freezer with prime ex-wife cold cuts. Little does Walter know that the wifey was a voodoo priestess, and when that freezer thaws, big things come a-crawling in small packages—lots of them. This vignette is the most blatantly indebted to EC comics and, as such, it's probably Amicus' finest twenty minutes.\n\"The Weird Tailor\" opens with tailor Bruno (Barry Morse) on the verge of being evicted. As luck would have it, Mr. Smith (Peter Cushing) walks into Bruno's shop and orders a unique suit. With the promise of a hefty commission, Bruno obsessively begins working according to Mr. Smith's very specific instructions. Unknown to Bruno, the suit is meant to resurrect Mr. Smith's recently deceased son. Things don't go according to plan. Previously adapted for Boris Karloff's \"Thriller,\" this one can't match the TV effort. Given a shorter running time for Asylum, Bloch was forced to excise the prologue, and with it gone, the suspense and menace are diminished. The original thriller was actually more perverse in suggesting Bruno's wife's sexual attraction to a mannequin. Additionally, Bruno's character was less sympathetic, bringing a pronounced, and weird, abusive quality that is merely sketched here. Cushing is superb, bringing a sense of pathos to the character, but his part is little more than a cameo. Being more compressed, the schlock quality of the ending is more pronounced. Yet, for all of its comparative flaws, this is a strong second episode.\n\"Lucy Comes To Stay\" is the weakest of the four episodes. Barbara (Charlotte Rampling) has been released from the asylum to the care of her brother, George (James Valliers) and nurse Higgins (Megs Jenkins). However, Barbara has an imaginary friend named Lucy (Britt Ekland) who doesn't care for George or the nurse. Lucy is also handy with a knife. Disappointingly, it plays out exactly as expected, and isn't helped by lackluster performances (Rampling being the exception).\nSurprisingly, \"Mannequins of Horror\" is an extension of the wraparound, with the arrival of a new doctor named Byron (Herbert Lom) who has the demonic hobby of placing spirits within miniature robots and imbuing them with life, which serves as a potential gateway to immortality. Dr. Martin returns to uncover Dr. Starr's identity in a delightfully unpleasant ending. It's something of a mini-masterpiece that clearly proved an inspiration to later, albeit inferior films.\nPART TWO will begin with Tales From The Crypt (1972) and take us to the final Amicus anthology: From Beyond The Grave (1974).\nAnthologyChristopher LeeFreddie FrancisHorrorPeter CushingPeter DuffellRoy Ward Baker\nPrevious Post271. THE HOURGLASS SANATORIUM (1973)Next PostWEIRD HORIZON FOR THE WEEK OF 2/24/2017"
}

Example Output:

{
    "raw_content": "AMICUS ANTHOLOGIES, PART ONE (1965-1972)\nFebruary 23, 2017 Alfred Eaker Leave a comment\nWith Dr. Terror's House of Horrors (1965, directed by Freddie Francis and written by Milton Subotsky) Amicus Productions (spearheaded by Subotsky and Max Rosenberg, who previously produced for Hammer and was a cousin to Doris Wishman) established itself as a vital competitor to Hammer Studios. Rather than imitating Hammer's modernization of Gothic classics, Amicus developed its own niche with omnibus films. They were successful enough to be in full-fledged production for a decade, establishing a reputation as the go-to studio for horror anthologies. This, their introductory portmanteau film, clearly influenced by EC Comics, sets a pattern of to-be-expected unevenness. Still, Amicus installs themselves as a horror studio to be reckoned with, sparing no expense in procuring Hammer's top actors: Peter Cushing (who would star in all but one of the Amicus anthologies) and Christopher Lee. For its wraparound segment, Dr. Terror's House of Horrors opens to the duo (among other passengers) on a train. Dr. Schreck ((\"Schreck\" is German for \"terror,\" and a nod to the famous star of F.W. Murnau's Nosferatu.)) (Cushing, saddled with a terrible German accent and glued on brows) pulls out a deck of tarot cards. \"Pick a card, any card, and tape it three times,\" Schreck tells his fellow passengers. Each participant will hear of a fate that may await them. Among the passengers is Christopher Lee who will, of course, factor into one of the five narratives.\nIn \"Werewolf,\" Neil McCallum is an architect renovating an old dark house, which turns out to be cursed. The title monster is featured in this pedestrian tale of ancestral revenge with a \"twist.\"\nWith Alan Freeman (better known as the U.K D.J. for \"Pick of the Pops\") served up as a snack for a venus fly trap, \"The Creeping Vine,\" thankfully doesn't take itself so seriously. It is refreshingly lightheaded hokum.\n\"Voodoo\" is the worst of the lot; badly dated in its stereotypes, with Kenny Lynch belting out a stolen voodoo tune.\n\"Disembodied Hand,\" has elitist art critic Franklin Marsh (Lee) driving artist Eric Landor (Michael Gough) to suicide. Landor's severed hand returns to exact revenge on the mean critic. It's in the spirit of The Beast with Five Fingers, among others, and chock-full of two-dimensional caricatures of both artists and critics. It holds no surprises, but with Lee and Gough engaged in a bit of whistling-while-they-work fun, it's easily the best episode.\n\"Vampire\" feature a young Donald Sutherland who discovers he is married to… a vampire! It barely raises a pulse.\nSeen today, Dr. Terror's House of Horrors is more camp than horror, and its appeal is one of genre nostalgia. Still, the phenomenal box office success of Dr. Terror green-lighted a second portmanteau film in 1967, entitled Torture Garden (directed by Freddie Francis and written by Robert Bloch). It contains no torture nor any garden. Burgess Meredith (in a preposterous disguise, reminding us of the Penguin) is Old Nick himself, going by the pseudonym of Dr. Diablo and moonlighting as a carnival barker who promises a tortuous exhibit that can reveal the future. \"You'll shake, you'll shiver, but it's all good fun,\" Diablo hammily tells his patrons. Unfortunately, only one of the four tales lives up to that promise.\n\"Enoch,\" is the opening narrative. Michael Bryant's inheritance money (from an uncle who took his time dying) is going to be spoiled by a mean ol' puddy tat with a lot of doubloons.\n\"Over Hollywood\" has Beverly Adams discovering the fountain of youth in Hollywood with robotic consequences.\n\"Mr. Steinway\" might be seen as a poor precursor to Stephen King's \"Christine,\" replacing a killer car with a killer piano. It's as absurd as it sounds.\nThe first three segments are sloppily written and executed with little enthusiasm; each progressively worse, but the final segment single-handedly salvages the anthology.\n\"The Man Who Collected Poe\" finds Jack Palance (playing against type) as an Edgar Allan Poe-obsessed geek who may have found his soulmate in fellow fanatic Peter Cushing. However, somebody's got something—or someone—hidden in the basement and … somebody's got the fever, which leads to a fiery finale. Cushing and Palance clearly enjoyed playing opposite one another and their chemistry, along with clever writing, making one wish the previous segments had been as enjoyable.\n1970's The House That Dripped Blood (directed by Peter Duffell and written by Robert Bloch) is a considerable improvement over its predecessors. Duffell lacks the visual astuteness of Freddie Francis, but he has superior stories to work with and a top notch cast. The connecting theme is the titular house, which has a bit of baggage left over from all who have resided there.\nIn \"Method For Murder,\" Denholm Elliott is a horror author who writes a character that becomes a tad too three-dimensional, much to his wife's peril.\n\"Waxworks\" stars Cushing as an uptight retired stockbroker and lifelong bachelor who visits a wax museum, only to see a figure of a woman whom he once was in love with. Obsession and unrequited love naturally go hand-in-hand, or head-on-plate.\nIn \"Sweets to the Sweet,\" Nyree Dawn Porter is hired to tutor a young, motherless child (Chloe Franks) who is unloved by her cold-hearted father, Christopher Lee. Without giving too much away, let's just say the underlying theme is one few filmmakers would dare tackle today.\n\"The Cloak\" is the most famous of the four episodes, remembered fondly for its absurd humor. It stars John Pertwee (best known for his portrayal of Dr. Who) as an actor who mantles the cloak of a purported actual vampire. Hammer favorite Ingrid Pitt bares her fangs and, of course, a bit more.\nAll four episodes feature strong acting, which is a rarity in contemporary horror and should be a model for genre filmmakers. Elliot's restrained performance in \"Method For Murder\" is admirable enough to forgive the predictable \"twist.\" The stylish \"Waxworks\" features an equally stylish performance from Cushing, although narratively it is the thinnest episode. \"Sweets to the Sweet\" is psychologically intense with three powerhouse performances, making it the strongest entry. Although John Pertwee is a bit on-the-sleeve in \"The Cloak,\" his performance suits the tone; but, he's no match for Pitt.\nAsylum (1972, directed by Roy Ward Baker and written by Robert Bloch) is often cited as the best of the Amicus anthologies. It opens on Dr. Martin (Robert Powell, best known as the blue-eyed Anglo-Saxon savior plopped into the Middle East in Franco Zeferelli's Jesus of Nazareth) showing up for his scheduled job interview with Dr. Starr for a position at the Dunsmoor Asylum. Martin is met by Dr. Rutherford (Patrick Magee), however, and informed that Dr. Starr is now a patient after going insane and becoming violent. Rutherford devises a test for Martin: he will interview four patients and if he can guess which one is Dr. Starr, then he will be hired. Naturally, this segues into four tales from Mr. Bloch.\nIn \"Frozen Fear,\" Walter (Richard Todd) is having a sordid affair with Bonnie (Barbara Parkins). When his wife Ruth (Sylvia Syms) won't give him a divorce, Walter grabs an axe and fills his basement freezer with prime ex-wife cold cuts. Little does Walter know that the wifey was a voodoo priestess, and when that freezer thaws, big things come a-crawling in small packages—lots of them. This vignette is the most blatantly indebted to EC comics and, as such, it's probably Amicus' finest twenty minutes.\n\"The Weird Tailor\" opens with tailor Bruno (Barry Morse) on the verge of being evicted. As luck would have it, Mr. Smith (Peter Cushing) walks into Bruno's shop and orders a unique suit. With the promise of a hefty commission, Bruno obsessively begins working according to Mr. Smith's very specific instructions. Unknown to Bruno, the suit is meant to resurrect Mr. Smith's recently deceased son. Things don't go according to plan. Previously adapted for Boris Karloff's \"Thriller,\" this one can't match the TV effort. Given a shorter running time for Asylum, Bloch was forced to excise the prologue, and with it gone, the suspense and menace are diminished. The original thriller was actually more perverse in suggesting Bruno's wife's sexual attraction to a mannequin. Additionally, Bruno's character was less sympathetic, bringing a pronounced, and weird, abusive quality that is merely sketched here. Cushing is superb, bringing a sense of pathos to the character, but his part is little more than a cameo. Being more compressed, the schlock quality of the ending is more pronounced. Yet, for all of its comparative flaws, this is a strong second episode.\n\"Lucy Comes To Stay\" is the weakest of the four episodes. Barbara (Charlotte Rampling) has been released from the asylum to the care of her brother, George (James Valliers) and nurse Higgins (Megs Jenkins). However, Barbara has an imaginary friend named Lucy (Britt Ekland) who doesn't care for George or the nurse. Lucy is also handy with a knife. Disappointingly, it plays out exactly as expected, and isn't helped by lackluster performances (Rampling being the exception).\nSurprisingly, \"Mannequins of Horror\" is an extension of the wraparound, with the arrival of a new doctor named Byron (Herbert Lom) who has the demonic hobby of placing spirits within miniature robots and imbuing them with life, which serves as a potential gateway to immortality. Dr. Martin returns to uncover Dr. Starr's identity in a delightfully unpleasant ending. It's something of a mini-masterpiece that clearly proved an inspiration to later, albeit inferior films.\nPART TWO will begin with Tales From The Crypt (1972) and take us to the final Amicus anthology: From Beyond The Grave (1974).\nAnthologyChristopher LeeFreddie FrancisHorrorPeter CushingPeter DuffellRoy Ward Baker\nPrevious Post271. THE HOURGLASS SANATORIUM (1973)Next PostWEIRD HORIZON FOR THE WEEK OF 2/24/2017",
    "QuratingWritingStyleScore": 2.4186381508,
    "QuratingRequiredExpertiseScore": 1.512241278,
    "QuratingFactsAndTriviaScore": 2.3265771873,
    "QuratingEducationalValueScore": -0.38782312
}