Benchmarks Overview
About 450 wordsAbout 2 min
A RayOrch Benchmark is not a second execution framework. It packages a regular Pipeline as a configurable, repeatable, submit-ready experiment with standard results. Every built-in follows the same path: construct typed configuration, load inputs, create the Pipeline, run against Ray locally or through Ray Jobs, and emit a BenchmarkReport.
Choose a workload
| Benchmark | Type | Core topology | What to observe |
|---|---|---|---|
MinerUScaleBench | Real workload / scale | local or HDFS PDF → Page → MinerU → atomic Document | real end-to-end execution, large actor pools, fractional GPUs, batching, resume, shared-output behavior |
MinerUBench | Real model | PDF → Page → MinerU → Document | page throughput, multi-GPU scaling, cross-PDF batching, ordered reconstruction |
Panda70MBench | Real model | Panda source → clips → four frame teachers → source JSON | source ownership, temporal prompt fusion, persistent GPU actors, ordered clip reduction |
YoloSamBench | Real model | Image → YOLO → SAM → Save | balance between persistent model pools, stage bottlenecks, end-to-end image throughput |
DualVllmBench | Real model | Prompt → vLLM A → vLLM B | two-engine composition, model-stage time, batching efficiency |
SglangVllmBench | Real model | Prompt → SGLang → vLLM | cross-Conda scheduling overhead and stability |
DocumentTopologyBench | Dependency-free topology | Document → Page → TableJob → Document | two dynamic fan-outs, empty groups, ordered reduction, scheduler overhead |
VideoCaptionTopologyBench | Dependency-free topology | Video → Frame → Caption → Video | 1 → M → 1, cross-video batching, frame-level parallelism |
VideoMultimodalTopologyBench | Dependency-free topology | Video → Audio + Frame → Merge | independent heterogeneous branches joined by parent lineage |
Use the real-model cases to study complete workloads. The three topology references use deterministic UDFs and need no dataset, model, or GPU, making them suitable for first runs, CI, scheduling-semantics regression, and framework-overhead experiments. They are not production Docling, VLM, or ASR adapters.
Shortest path
from rayorch.benchmark import DocumentTopologyBench
report = DocumentTopologyBench(output_dir="./results").run(profile=False)
report.print_summary()Importing rayorch.benchmark does not import vLLM, SGLang, MinerU, or vision-model dependencies. The registry stores class paths and loads an implementation only when that Benchmark is used.
What the report can answer
Every run records startup_s, measured_wall_s, end_to_end_wall_s, actor and RPC counts, peak active input batches, and per-Call Grain, batch-size, and execution-time statistics. Workloads add business work such as pages, frames, detections, masks, or generated character counts. Read Interpret Performance Results before treating workload counts as throughput or driver-visible GPU samples as cluster-wide monitoring.
Next steps
- First run: Run a Benchmark
- Understand results: Interpret Performance Results
- Publish an experiment: Write a Benchmark