Add a way to pass expected language to FastTextLangId filter #565
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
shuoyangd
changed the title
Signed-off-by: Shuoyang Ding <[email protected]>
shuoyangd
force-pushed
the
expected_langid_filter
branch
from
7b1e380 to
4f164a1
Compare
ryantwolf
reviewed
Mar 6, 2025
| @@ -68,14 +68,14 @@ def _load_model(self): | |||
|
|
|||
| class FastTextLangId(DocumentFilter): | |||
|
|
|||
| def __init__(self, model_path=None, min_langid_score=0.3): | |||
| def __init__(self, model_path=None, min_langid_score=0.3, lang=None): | |||
There was a problem hiding this comment.
Can you add a docstring for this function?
| @@ -91,14 +91,17 @@ def _score_document(text): | |||
| pp = text.strip().replace("\n", " ") | |||
| label, score = model.predict(pp, k=1) | |||
| score = score[0] | |||
| lang_code = label[0][-2:].upper() | |||
| lang_code = label[0][-2:].lower() | |||
There was a problem hiding this comment.
Can we keep it as upper for backwards compatibility? Or, is there some reason to prefer lower?
There was a problem hiding this comment.
I'm trying to follow ISO-639 set 1 here which is 2-letter lowercase: https://en.wikipedia.org/wiki/List_of_ISO_639_language_codes
| if model_path is None: | ||
| raise ValueError( | ||
| "Must provide a valid path to a FastText model " | ||
| "to identify languages with this filter" | ||
| ) | ||
| self._model_path = model_path | ||
| self._lang_code = None | ||
| self._lang_code = lang |
There was a problem hiding this comment.
regardless of what we decide on with upper vs lower, can you also manually set self._lang_code = lang.upper() so the user doesn't have to remember which case to specify?
| @@ -67,6 +72,15 @@ def filter_dataset(dataset: ParallelDataset, gpu: bool = False) -> ParallelDatas | |||
| ] | |||
| ) | |||
|
|
|||
| if FAST_TEXT_MODEL_DIR: | |||
There was a problem hiding this comment.
Instead of making this a hardcoded directory, how do you feel about making this a CLI arg with argparse?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Currently, FastTextLangId filter only supports filtering by a language ID filter, but sometimes, we know what the language the data is supposed to be, and it would be a useful addition to keep only the data that matches with the expected language.
We use two-letter ISO-639 code to denote languages.
Usage
Passing an extra argument when initializing the filter will make it check against expected language, for example:
If
langargument is not passed, it falls back to the old behavior of filtering by minimum language ID score.bitext_filteringtutorial is updated to demonstrate how this is used in a pipeline.Checklist
(FastTextLangId filter is currently only tested with a fake emulator class. Not sure how to best cover this change with test.)