@jbesomi Should it be checking line by line and if a line is duplicate remove it?
or no removal just state that there is duplicates.

The idea here is to compare long text of documents and try to find if there are some of them too similar; in this case, it might mean that documents are indeed duplicates. There are many applications for that, for instance, to detect plagiarism in papers.

A naive approach is to apply TF-IDF and look at the distance between vectors.

conI suggest having several methods for handling duplicated content.

In the very simplest form, you might just need to chech againsta a hash (sha1, for instance) to be sure you don't have exact duplicates (ok, this might be a preprocessing job).

The inteface might look like Pandas.Series.unique() but specifying a method / way to do the deduplication: unique( method='hash | jaquard | etc.' , threshold=xx).

Hey @igponce,

Exactly, the interface would look like hero.unique(df['text]).

A simple-yet-powerful solution is to simply compute a good representation of each text and remove documents that have very similar vectors.

Right, as you point the function will have as argument threshold. We will need to do some tests and pick a good one as default. This will largely depend on the underline algorithm.

Would you be interested in implementing this solution? Jaccard might work as well but it's easier to do better and to use word vectors instead of just counting.

Food for thoughts: what if the input must already be a representation? This would be even a better solution. In this case, the arguments might be the distance function as well as the threshold parameters.