Electrum: optimize merkle proof validation with batching

[Keerthi421]

This PR optimizes the Electrum client's performance by improving Merkle proof validation, addressing the significant performance regression in BDK 1.1.0 where full sync time increased from 4s to 26s.

Key improvements:

• Implemented batch processing for Merkle proof validations
• Added Merkle proof caching to prevent redundant network calls
• Optimized header handling with pre-fetching and reuse
• Modified core functions to use batch operations instead of individual calls

Notes to the reviewers

The optimization approach focuses on three main areas:

1. Reducing network round trips through batched Merkle proof requests
2. Minimizing redundant operations with a new Merkle proof cache
3. Improving header handling efficiency with pre-fetching

The batch size is set to 100 as a balance between performance and memory usage. This value can be adjusted based on testing results.

Changelog notice

Added

• New Merkle proof cache to prevent redundant network calls
• Batch processing for Merkle proof validations
• Performance tests to verify sync time improvements
  Solves issue Electrum client Performance issues #1891

[ValuedMammal]

Concept ACK

In 3ef0c83:

I don't think rustup-init.exe should be included.

[evanlinjin]

Let's say a tx is confirmed in block of height 100 and hash A. We then cache it's merkle proof. Then a reorg happens and the tx is now in block of hash B (but still height 100).

With the current implementation, we will never fetch the merkle proof for block B because we already have a merkle proof cached for a txid at the same height as block B.

Is there a way to cache merkle proofs against (txid, blockhash)?

I apologize in advance for how bad the Electrum API is designed.

[Keerthi421]

@evanlinjin I will work on it

[Keerthi421]

@evanlinjin , @notmandatory Please Review the changes made

[evanlinjin]

@Keerthi421 thank you for your attempt but you have not addressed the problem.

See if a transaction with txid get's confirmed in a block of height:hash 100:A and then we do a sync. We will have an entry: (txid, 100) : merkle_res_for_txid_in_A.

If there is a reorg, and the block at height 100 is now A', the entry will stay as it is and we will never fetch an anchor for "txid in A'". This will mean that this transaction will appear to never confirm on the wallet (until we restart and clear the cache).

[Keerthi421]

@notmandatory please review the changes