feat(event cache store): add an IndexedDB implementation #4617

ospfranco · 2025-02-04T09:16:22Z

This PR is WIP.

This PR is meant to add a EventCache store for the the matrix-sdk-indexeddb crate of the SDK meant to be used in WASM contexts. Similar to the sqlite implementation is supposed to provide a disk persisted implementation for storing Events in LinkedChunks.

It follows the general structure of matrix-sdk-indexeddb/src/crypto_store and matrix-sdk-indexeddb/src/state_store, a new module matrix-sdk-indexeddb/src/event_cache_store has been added.

There are still many TODOs and questions in the codebase to which I would appreciate some guidance from the SDK maintainers.

Public API changes documented in changelogs (optional)

Signed-off-by: Oscar Franco

crates/matrix-sdk-indexeddb/src/event_cache_store/idb_operations.rs

ospfranco · 2025-02-04T09:17:46Z

crates/matrix-sdk-indexeddb/src/event_cache_store/indexeddb_serializer.rs

+
+/// Handles the functionality of serializing and encrypting data for the
+/// indexeddb store.
+pub struct IndexeddbSerializer {


Copy and paste from crypto_store not sure if a different serializer is needed or they can be merged and re-used by both modules.

Did we end up having to change this much, or is it a straight copy? If straight copy, is it possible to pull it into a shared mod?

Not sure yet, I will hold on for now since I don't know what changes might I need to implement the rest of the methods

crates/matrix-sdk-indexeddb/src/event_cache_store/migrations.rs

ospfranco · 2025-02-04T09:19:17Z

crates/matrix-sdk-indexeddb/src/event_cache_store/mod.rs

+    migration_conflict_strategy: MigrationConflictStrategy,
+}
+
+impl IndexeddbEventCacheStoreBuilder {


Follows the same pattern as state_store, not sure if this is wanted or not.

crates/matrix-sdk-indexeddb/src/event_cache_store/mod.rs

ospfranco · 2025-02-04T09:20:14Z

crates/matrix-sdk-indexeddb/src/event_cache_store/mod.rs

+        Ok(())
+    }
+
+    async fn try_take_leased_lock(


Copy and paste from crypto_store. I could not find any info on what this "leased_lock" is.

This is related to the cross-process lock, the doc is missing a high-level overview I think… Tell me if you need more info.

I just copied the same from the crypto_store, if you think it is fine as it is, then no need. It should just work, right?

Possible candidate to try to put into a shared mod as well.

sure, I need to get the thing to run first to confirm it is working as intended first

ospfranco · 2025-02-04T09:20:29Z

crates/matrix-sdk/src/client/builder/mod.rs

    };

+    // let store_config = {


Will be removed later

codecov · 2025-02-04T17:15:26Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 85.74%. Comparing base (3f71d9a) to head (9e34ac0).
Report is 205 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4617      +/-   ##
==========================================
+ Coverage   85.71%   85.74%   +0.02%     
==========================================
  Files         291      291              
  Lines       33323    33322       -1     
==========================================
+ Hits        28563    28571       +8     
+ Misses       4760     4751       -9

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Hywan · 2025-02-11T09:12:11Z

Be aware that #4632 is likely to create a non-negligible conflict with your patches. The new methods to load chunk from the event cache are going to be simpler and will allow to incrementally load the chunks.

ospfranco · 2025-02-11T09:34:20Z

Be aware that #4632 is likely to create a non-negligible conflict with your patches. The new methods to load chunk from the event cache are going to be simpler and will allow to incrementally load the chunks.

Ah, interesting... well... maybe it's worth to put this a bit on hold until that is merged. If the trait will be easier to implement, then it's worth it

Hywan · 2025-02-11T09:35:45Z

The trait will gain 2 new methods: load_last_chunk and load_previous_chunk, and the method used to load the entire LinkedChunk is for testing-purposes only. So yeah, it's a bit simpler but it needs the whole package :-/.

zzorba · 2025-02-13T13:59:43Z

crates/matrix-sdk-indexeddb/src/event_cache_store/error.rs

+    fn from(e: IndexeddbEventCacheStoreError) -> Self {
+        match e {
+            IndexeddbEventCacheStoreError::Json(e) => StoreError::Json(e),
+            IndexeddbEventCacheStoreError::StoreError(e) => e,


Does this compile, I'm surprised it doesn't need a wrapper for StoreError here?

I'm using the command Jonas gave and it compiles with that, it will not pass the tests but the compiler shows no errors left:

cargo xtask ci wasm

zzorba · 2025-02-13T14:00:03Z

crates/matrix-sdk-indexeddb/src/event_cache_store/error.rs

+                EventCacheStoreError::Backend(Box::new(e))
+            }
+            IndexeddbEventCacheStoreError::Encryption(e) => EventCacheStoreError::Encryption(e),
+            _ => EventCacheStoreError::backend(e),


backend => Backend, right?

This was not implemented by me, comes from the sdk-base and the implemented traits

https://github.com/ospfranco/matrix-rust-sdk/blob/610860bd7163c34b822c1a1ed64e78765a2a492d/crates/matrix-sdk-base/src/event_cache/store/mod.rs#L166

zzorba · 2025-02-13T14:29:24Z

crates/matrix-sdk-indexeddb/src/event_cache_store/error.rs

+        match frm {
+            IndexeddbEventCacheStoreError::Json(e) => CryptoStoreError::Serialization(e),
+            IndexeddbEventCacheStoreError::CryptoStoreError(e) => e,
+            _ => CryptoStoreError::backend(frm),


s/backend/Backend/

Nope, same as above, this is the way it's implemented in the base crate

zzorba · 2025-02-13T14:29:47Z

crates/matrix-sdk-indexeddb/src/event_cache_store/error.rs

+    fn from(frm: IndexeddbEventCacheStoreError) -> CryptoStoreError {
+        match frm {
+            IndexeddbEventCacheStoreError::Json(e) => CryptoStoreError::Serialization(e),
+            IndexeddbEventCacheStoreError::CryptoStoreError(e) => e,


I would think this needs a wrapper as well, right?

Don't know, copied what was there from the state_store, this all were required there I think

zzorba · 2025-02-13T14:46:30Z

crates/matrix-sdk-indexeddb/src/event_cache_store/indexeddb_serializer.rs

+
+/// Handles the functionality of serializing and encrypting data for the
+/// indexeddb store.
+pub struct IndexeddbSerializer {


Did we end up having to change this much, or is it a straight copy? If straight copy, is it possible to pull it into a shared mod?

zzorba · 2025-02-13T15:01:01Z

crates/matrix-sdk-indexeddb/src/event_cache_store/mod.rs

+    /// Return all the raw components of a linked chunk, so the caller may
+    /// reconstruct the linked chunk later.
+    async fn reload_linked_chunk(&self, _room_id: &RoomId) -> Result<Vec<RawChunk<Event, Gap>>> {
+        Ok(vec![])


todo!, or this is just for tests?

this is todo but don't want to use it for now since my workflow is so slow anyways, seeing hundred of errors because of pending todos() is confusing whenever I have to run the cargo command manually

crates/matrix-sdk-indexeddb/src/event_cache_store/mod.rs

zzorba · 2025-02-13T15:02:56Z

crates/matrix-sdk-indexeddb/src/event_cache_store/mod.rs

+        Ok(())
+    }
+
+    async fn try_take_leased_lock(


Possible candidate to try to put into a shared mod as well.

zzorba · 2025-02-13T15:05:22Z

crates/matrix-sdk/src/client/builder/mod.rs

-    let store_config = {
-        tracing::warn!("The IndexedDB backend does not implement an event cache store, falling back to the in-memory event cache store…");
-        store_config.event_cache_store(matrix_sdk_base::event_cache::store::MemoryStore::new())
+        let event_cache_store = matrix_sdk_indexeddb::open_event_cache_store(name, None).await?;


Seems like sqlite uses the same passphrase for both stores, should we do the same here instead of passing None?

This is when the e2e encryption is not turned on, then None is required

Hywan · 2025-02-25T10:17:25Z

EventCacheStore is more stable now, I think it's fine.

ospfranco added 11 commits January 30, 2025 12:49

Modify the matrix-sdk and add feature flag for the event cache store

6955428

Create the basic struct to initialize an event_cache_store

b8bfd3e

Almost working instantiation of event cache store

c2761d5

Fix deps on feature indexeddb

3c6ca6a

Add license text at top

7eb8946

Copy try_take_leased_lock from crypto_store impl

64e68ba

Add missing methods from EventCacheTrait

1328003

Fix a bunch of errors, move error to it's own file

eebe00e

Fix final errors around creating the event cache store

e49bda6

First insertion into indexeddb

1b1029d

Merge branch 'main' into oscar/indexeddb-event-cache-store

c317bd8

ospfranco commented Feb 4, 2025

View reviewed changes

Finish implementing insert_chunk

0f8ebf1

ospfranco added 15 commits February 4, 2025 19:06

Comment unused code, finish implementing insert_chunk

ee40ddd

Get rid of name param

73713ac

Insert GAP chunk

d3b6f87

Tried to implement insert gap, but missing encode_value on serializer

8528522

Implement insert gap

7622be4

remove chunk deletes from LINKED_CHUNKS store, missing related entities

01cb5ba

Fix some compilation errors

8f2b427

PushItems

722bac2

PushItems

4ca221f

RemoteItem

0e05ac8

Half implementation of DetachLastItems

28647b6

Remove deletion

eee0bad

Add placeholder for deletion

0b0e15e

Change previous implementations to use serializer

993283c

Change previous implementations to use serializer

d755cc0

Do not interate through get_all_with_key

abee3ab

ospfranco added 4 commits February 13, 2025 09:25

Fix encoded room id as ref

06b0395

finish DetachLastItems

ae25b35

PushItems now uses TimeEventForCache struct

4dcf22a

Implement Update::Clear

610860b

zzorba reviewed Feb 13, 2025

View reviewed changes

ospfranco added 7 commits February 17, 2025 15:00

Fixes on initialization of event-cache store

ebda277

WIP, fixes for creating serializable objects

1af016f

More fixes on serialization and deserialization

6d8b405

Working retrieving of events

4179750

Update key handling

6c7b729

Fix ranges

c686f90

Add stub for tests

8cfe19c

bnjbvr changed the title ~~IndexedDB Event Store~~ feat(event cache store): add an IndexedDB implementation Feb 24, 2025

ospfranco added 2 commits February 25, 2025 06:56

Add indexeddbeventcachestoreerror to OpenStoreError

18c6287

Implement clear_all_rooms_chunks

b6a8e08

ospfranco added 3 commits February 25, 2025 13:38

Working media storage

533b1c3

Working media storage

4b7f73d

Fixed clear command

9e34ac0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(event cache store): add an IndexedDB implementation #4617

feat(event cache store): add an IndexedDB implementation #4617

ospfranco commented Feb 4, 2025

ospfranco Feb 4, 2025

zzorba Feb 13, 2025

ospfranco Feb 13, 2025

ospfranco Feb 4, 2025

ospfranco Feb 4, 2025

Hywan Feb 5, 2025 •

edited

Loading

ospfranco Feb 5, 2025

zzorba Feb 13, 2025

ospfranco Feb 13, 2025

ospfranco Feb 4, 2025

codecov bot commented Feb 4, 2025 •

edited

Loading

Hywan commented Feb 11, 2025

ospfranco commented Feb 11, 2025

Hywan commented Feb 11, 2025

zzorba Feb 13, 2025

ospfranco Feb 13, 2025

zzorba Feb 13, 2025

ospfranco Feb 13, 2025

zzorba Feb 13, 2025

ospfranco Feb 13, 2025

zzorba Feb 13, 2025

ospfranco Feb 13, 2025

zzorba Feb 13, 2025

zzorba Feb 13, 2025

ospfranco Feb 13, 2025

zzorba Feb 13, 2025

zzorba Feb 13, 2025

ospfranco Feb 13, 2025

Hywan commented Feb 25, 2025

feat(event cache store): add an IndexedDB implementation #4617

Are you sure you want to change the base?

feat(event cache store): add an IndexedDB implementation #4617

Conversation

ospfranco commented Feb 4, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Hywan Feb 5, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Feb 4, 2025 • edited Loading

Codecov Report

Hywan commented Feb 11, 2025

ospfranco commented Feb 11, 2025

Hywan commented Feb 11, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Hywan commented Feb 25, 2025

Hywan Feb 5, 2025 •

edited

Loading

codecov bot commented Feb 4, 2025 •

edited

Loading