Importer 1.6.0 create duplicate transactions

[Rueblikuchen]

Hey,

I have just updated to Firefly 6.2.1 and the data import to 1.6.0.

Now I have the problem that after a manual import everything was imported - including duplicates. I am importing CSV files. I have compared the later duplicates in the csv and there they are 100% identical

I have read that the release notes warned about this, but I would like to know if this is a known bug that will be fixed or not.

I took a closer look at one of the duplicates and compared both entries with /transactions/debug/. The only differences apart from “created_at”, “id”, “transaction_journal_id” and “updated_at” are: “date_tz” now with null instead of "Europe/XYZ" and "balance_after" as well as "balance_before" now with null instead of a "123.45". And the “hash” is also different but I don't know if that's the way it should be.

When I import the file a second time, duplicates are recognized correctly. Now it would be interesting to know if the problem is to be solved or not, otherwise I see the possibility to empty the database and re-import everything with the new importer.

[JC5]

There is no bug, so there is nothing to fix. It all works as expected.

It's just that sometimes information is ordered differently, tags are added or removed, or meta-data is added or removed. If this is different from a previous version, overlapping imports may result in duplicates.

[Rueblikuchen]

I understand - but it is a pity that old entries are not recognized as duplicates. I had hoped that removed tags or other changes would not be used to match duplicates.

But probably this is not (easily) possible.

But many thanks anyway

[dudu7731]

Hi,

Can you tell us how to handle this ?

perhaps a tabula rasa of transactions ? is duplicate mechanism will work after that ?

could you advise us how to have a clean and working imports for the future ?

in this state, i can't upgrade my main firefly because of this duplicates ...

[JC5]

Do not import transactions you know exist already, because the importer may create them as duplicate. Then again, it may not. Either way, import small batches.

If a duplicate is created, simply delete it. It will not be created again.

[dudu7731]

I import from gocardless so i don't control what is imported or not.

My gocardless import 30 days ago, I will modify this parameter, i that Ok ?

[JC5]

Yes that's ok.