using fine tuned model

[Atefeh197]

I fine-tuned a whisper model with my train dataset. I have a folder with a checkpoint folder and some json files, how can I use that to transcribe the audio file?

I cannot distinguish whether I got a correct fine-tuned model or not.

[gongouveia]

@Atefeh197 Hello you do it with two different ways.
Using CLI,
whisper audio.wav --model checkpoint_model_path

or using python:

import whisper

model = whisper.load_model("checkpoint_model_path")

result = model.transcribe("audio.wav")

[Atefeh197]

Thank you for your response,

but I got this error,

RuntimeError: Model C:/.../checkpoint-400 not found; available models = ['tiny.en', 'tiny', 'base.en', 'base', 'small.en', 'small', 'medium.en', 'medium', 'large-v1', 'large-v2', 'large-v3', 'large']

I have only these files in the checkpoint-400 folder:

optimizer.pt,
trainer_state.json,
config.json,
preprocessor_config.json
generation_config.json
training_args.bin
mg_state.pth

[jantoortiz]

Were you able to get this working? I am encountering the same issue.

[gongouveia]

@Atefeh197 Sorry you are right, you have to specify with model_dir.
Specify --model_dir {path to the directory to download models}. In Python

whisper.load_model(..., download_root="{path to the directory to download models}")

On CLI you can access all whisper arguments using ´whisper --help' where you find

 --model MODEL         name of the Whisper model to use (default: small)

  --model_dir MODEL_DIR
                        the path to save model files; uses ~/.cache/whisper by default (default: None)

Plus if you trained with HuggingFace Transformers you may need to convert back the checkpoint model back to a whisper acceptable format

[mrmuminov]

@gongouveia How to convert to a whisper acceptable format? Is there any article?

[Hu-nie]

@gongouveia i want too

[Hu-nie]

I found it! Referring to #830 should be helpful.

[gongouveia]

@Hu-nie Sorry, I don´t understand, how I can help you?

[Hu-nie]

@gongouveia I would like to integrate or replace the fine-tuned model (safetensor) shown at the link above into Whisper for use. Is there a way to do this?

https://huggingface.co/gcasey2/whisper-large-v3-ko-en-v2

[gongouveia]

@Hu-nie Why do you have your model in safetensors?

It is definitely possible to convert, please follow the following link that is very well explained: safetensors to Whisper model

[Hu-nie]

@gongouveia Thank you, I will give it a try.