Audio Caption Generation

Step 1: Prepare the DataFlow environment

conda create -n myvenv python=3.12
pip install open-dataflow
pip install open-dataflow[vllm]

Step 2: Install the DataFlow audio module

pip install open-dataflow[audio]

Step 3: Launch the local model service

How to Call the Local Model Service:

llm_serving = LocalModelLLMServing_vllm(
    hf_model_name_or_path="./models/Qwen2-Audio-7B-Instruct", # set to your own model path
    vllm_tensor_parallel_size=2,
    vllm_max_tokens=8192,
    vllm_gpu_memory_utilization=0.7
)

Step 4: Fill in the audio path in the specified format and prepare the data that needs audio caption.

{"audio": ["your_audio_path"]}

Step 5: Set the data path in FileStorage using the format below.

storage = FileStorage(
    first_entry_file_name="your_path",
    cache_path="./cache",
    file_name_prefix="audio_caption",
    cache_type="jsonl",
    media_key="audio",
    media_type="audio"
)

Step 6: Initialize the CaptionGenerator operator

generator = CaptionGenerator(llm_serving)

Step 7: Run the operator

generator.run(storage=storage.step(), output_key="caption")