Batch Inferencing¶
Info
The guide below is only meant for reference only and not meant to be followed verbatim. You may need to generate your own guide site if you require guidance specifically for your own project.
Some problem statements do not warrant the deployment of an API server but instead methods for conducting batched inferencing where a batch of data is provided to a script and it is able to churn out a set of predictions, perhaps exported to a file.
This template provides a Python script (src/batch_infer.py) and a configuration file (conf/batch_infer.yaml) for this purpose.
Let's first create some sample data on our Coder workspace for us to conduct batch inferencing on:
mkdir -p /<NAME_OF_DATA_SOURCE>/workspaces/<YOUR_HYPHENATED_NAME>/project/data/batch-infer && cd $_
echo -n "Output1" > in1.txt
echo -n "Output2" > in2.txt
echo -n "Output3" > in3.txt
runai submit \
--job-name-prefix <YOUR_HYPHENATED_NAME>-download-batch-data \
-i alpine \
--working-dir /<NAME_OF_DATA_SOURCE>/workspaces/<YOUR_HYPHENATED_NAME>/project/data/batch-infer \
--existing-pvc claimname=<NAME_OF_DATA_SOURCE>,path=/<NAME_OF_DATA_SOURCE> \
--cpu 2 --cpu-limit 2 --memory 4G --memory-limit 4G --backoff-limit 1 \
--command -- /bin/bash -c "echo -n 'Output1' > in1.txt && \
echo -n 'Output2' > in2.txt && \
echo -n 'Output3' > in3.txt"
To execute the batch inferencing script:
# Navigate back to root directory
cd "$(git rev-parse --show-toplevel)"
conda activate project
python src/batch_infer.py
runai submit \
--job-name-prefix <YOUR_HYPHENATED_NAME>-batch-inference \
-i alpine \
--working-dir /home/aisg/project \
--existing-pvc claimname=<NAME_OF_DATA_SOURCE>,path=/<NAME_OF_DATA_SOURCE> \
--cpu 2 --cpu-limit 2 --memory 4G --memory-limit 4G --backoff-limit 1 \
--command -- python src/batch_infer.py \
output_path=/<NAME_OF_DATA_SOURCE>/workspaces/<YOUR_HYPHENATED_NAME>/project/batch_infer_res.jsonl \
input_data_dir=/<NAME_OF_DATA_SOURCE>/workspaces/<YOUR_HYPHENATED_NAME>/project/data/batch-infer
The script will log to the terminal the location of the .jsonl file (batch-infer-res.jsonl) containing predictions that look like such:
...
{"time": "2024-02-29T10:09:00+0000", "text_filepath": "./data/batch-infer/in1.txt", "prediction": "Output1"}
{"time": "2024-02-29T10:09:00+0000", "text_filepath": "./data/batch-infer/in2.txt", "prediction": "Output2"}
{"time": "2024-02-29T10:09:00+0000", "text_filepath": "./data/batch-infer/in3.txt", "prediction": "Output3"}
...
The hydra.job.chdir=True flag writes the .jsonl file containing the predictions to a subdirectory within the outputs folder. See here for more information on outputs generated by Hydra.