Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Model loading doesn't work for SentencePieceTokenizer #12

Open
siin-lab opened this issue Apr 6, 2024 · 0 comments
Open

Model loading doesn't work for SentencePieceTokenizer #12

siin-lab opened this issue Apr 6, 2024 · 0 comments

Comments

@siin-lab
Copy link

siin-lab commented Apr 6, 2024

This code fails with an error:

import tkseem as tk

tokenizer_path = 'model.pl'
tokenizer = tk.SentencePieceTokenizer()
tokenizer.train(dataset_file)

# save the tokenizer to a file
tokenizer.save_model(tokenizer_path)

# load the tokenizer from a file
tokenizer = tk.SentencePieceTokenizer()
tokenizer.load_model(tokenizer_path)

# test the tokenizer
a = tokenizer.tokenize("السلام عليكم")

Error message is:

Traceback (most recent call last):
  File "/Users/user/Desktop/Projects/train-tokenizer.py", line 15, in <module>
    a = tokenizer.tokenize("السلام عليكم")
  File "/Users/user/.pyenv/versions/3.10.0/lib/python3.10/site-packages/tkseem/sentencepiece_tokenizer.py", line 50, in tokenize
    return self.sp.encode(text, out_type=str)
AttributeError: 'bool' object has no attribute 'encode'

The solution to this issue is updating the "load_model" to:

    def load_model(self, file_path):
        """Load a saved sp model

        Args:
            file_path (str): file path of the trained model
        """
        self.sp = spm.SentencePieceProcessor(model_proto=open(file_path, "rb").read())
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant