Headless scaling features

[r4mmer]

Summary

Some features are being designed on the wallet-lib that can be very interdependent, meaning their usefulness are more evident when combining them.

Here we can understand why each feature is being designed and how they interact together.

Features

Persistent storage

As the name sugests, this is saving the storage on disk or externally with a database.
This is a way to keep the history and calculations made and saved on storage even after a reset, e.g. when running on kubernetes the container can be moved to another instance and the whole history would have to be downloaded and processed again.

By persisting the storage on disk we can also aleviate the memory usage of the service by offloading some information on disk, but the way the wallet-lib works internally does not allow for this offloading, which is why this feature requires a new storage scheme which will help the inner workings of how the wallet operates and handles data.

Reliable integration

This feature is a new form of fullnode connection. The new websocket interface introduces a way to identify missing events, either by network issues or any other reason. Along with the new event api we can recover from network errors and websocket disconnects without having to reload the entire wallet.
By reducing the need to make a full wallet reload on network errors we can assume it will increase service availability and uptime.

Together with the persistent storage we can recover even after a wallet is stopped, since we can recover the events lost in the meantime.
The reliable integration also comes with some issues the client needs to address, for instance the event filtering since we receive all network events we need to filter which ones are of interest, i.e. which events affect one of the wallets addresses, this will require that all addresses are loaded on memory since it would be too much overhead to search the addresses on every event, this is a reason why the new storage scheme will help this feature work with more efficience.

New storage scheme

Design

The new storage scheme is an abstraction of storage which will help make queries without the wallet needing to understand how the storage is saved (or where).
This includes a major refactor on the wallet since the storage holds the addresses and transaction history most wallet features interact with it and many utilities also use it.
We should also deprecate many methods that directly use the storage since the main feature is storage abstraction.

The storage abstraction allows for many storage implementations, i.e. persistent storage.
There are many storage methods and implementations that can benefit the reliable integration but since it is a new feature we should not make any assumptions and design the storage access on the reliable integration design

New address scanning policy

Design

The current address scanning policy is called gap-limit, on which we search addresses until we found a certain number of addresses without any transaction, this works perfectly for personal wallets since it is completely automated and provides a good functionalit.
When using this scanning policy with services that can have many addresses without transaction, e.g. when a service dedicates an address for an user, it does not perform well, the solution is having an alternative address scanning policy.

The new scanning policy is completely manual making the client responsible for address generation.
This particular feature does not have any notable synergy with the other features, but it will make our wallets work better on some services which rely on the wallet to work.