You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
importtkseemastktokenizer_path='model.pl'tokenizer=tk.SentencePieceTokenizer()
tokenizer.train(dataset_file)
# save the tokenizer to a filetokenizer.save_model(tokenizer_path)
# load the tokenizer from a filetokenizer=tk.SentencePieceTokenizer()
tokenizer.load_model(tokenizer_path)
# test the tokenizera=tokenizer.tokenize("السلام عليكم")
Error message is:
Traceback (most recent call last):
File "/Users/user/Desktop/Projects/train-tokenizer.py", line 15, in <module>
a = tokenizer.tokenize("السلام عليكم")
File "/Users/user/.pyenv/versions/3.10.0/lib/python3.10/site-packages/tkseem/sentencepiece_tokenizer.py", line 50, in tokenize
return self.sp.encode(text, out_type=str)
AttributeError: 'bool' object has no attribute 'encode'
The solution to this issue is updating the "load_model" to:
defload_model(self, file_path):
"""Load a saved sp model Args: file_path (str): file path of the trained model """self.sp=spm.SentencePieceProcessor(model_proto=open(file_path, "rb").read())
The text was updated successfully, but these errors were encountered:
This code fails with an error:
Error message is:
The solution to this issue is updating the "load_model" to:
The text was updated successfully, but these errors were encountered: