Batch Inferencing¶

Some problem statements do not warrant the deployment of an API server but instead methods for conducting batched inferencing where a batch of data is provided to a script and it is able to churn out a set of predictions, perhaps exported to a file.

This template provides a Python script (src/batch_inferencing.py) for this purpose.

Let's first download some data on our local machine for us to conduct batch inferencing on:

Linux/macOSWindows PowerShell

$ cd data
$ wget https://storage.googleapis.com/aisg-mlops-pub-data/kapitan-hull/batched-mnist-input-data.zip
$ unzip batched-mnist-input-data.zip
$ rm batched-mnist-input-data.zip

$ cd data
$ wget https://storage.googleapis.com/aisg-mlops-pub-data/kapitan-hull/batched-mnist-input-data.zip
$ Expand-Archive -Path batched-mnist-input-data.zip -DestinationPath .
$ rm batched-mnist-input-data.zip

To execute the batch inferencing script locally:

Linux/macOSWindows PowerShell

# Navigate back to root directory
$ cd "$(git rev-parse --show-toplevel)"
$ export PRED_MODEL_UUID=<MLFLOW_RUN_UUID>
$ export PRED_MODEL_PATH="$PWD/models/$PRED_MODEL_UUID/artifacts/model/model.pt"
$ conda activate {{cookiecutter.repo_name}}
$ python src/batch_inferencing.py \
    batch_infer.model_path=$PRED_MODEL_PATH \
    batch_infer.input_data_dir="$PWD/data/batched-mnist-input-data"

# Navigate back to root directory
$ cd "$(git rev-parse --show-toplevel)"
$ $Env:PRED_MODEL_UUID=<MLFLOW_RUN_UUID>
$ $Env:PRED_MODEL_PATH="$(Get-Location)\models\$PRED_MODEL_UUID\artifacts\model\model.pt"
$ conda activate {{cookiecutter.repo_name}}
$ python src/batch_inferencing.py `
    hydra.job.chdir=True `
    batch_infer.model_path=$Env:PRED_MODEL_PATH `
    batch_infer.input_data_dir="$(Get-Location)\data\batched-mnist-input-data"

The script will log to the terminal the location of the .jsonl file (batch-infer-res.jsonl) containing predictions that look like such:

...
{"time": "YYYY-MM-DDThh:mm:ss+0000", "image_filepath": "/home/aisg/{{cookiecutter.repo_name}}/data/XXXXXX.png", "prediction": "X"}
{"time": "YYYY-MM-DDThh:mm:ss+0000", "image_filepath": "/home/aisg/{{cookiecutter.repo_name}}/data/XXXXXX.png", "prediction": "X"}
{"time": "YYYY-MM-DDThh:mm:ss+0000", "image_filepath": "/home/aisg/{{cookiecutter.repo_name}}/data/XXXXXX.png", "prediction": "X"}
{"time": "YYYY-MM-DDThh:mm:ss+0000", "image_filepath": "/home/aisg/{{cookiecutter.repo_name}}/data/XXXXXX.png", "prediction": "X"}
{"time": "YYYY-MM-DDThh:mm:ss+0000", "image_filepath": "/home/aisg/{{cookiecutter.repo_name}}/data/XXXXXX.png", "prediction": "X"}
...

The hydra.job.chdir=True flag writes the .jsonl file containing the predictions to a subdirectory within the outputs folder. See here for more information on outputs generated by Hydra.