Inter-process communication (#585)

* Refactor transport types to different module.

* Prepare SyncDirection type.

To indicate whether an initial sync should create the account on the
remote (push) or on local (pull).

* Add local server draft implementations.

* Move storage into LocalAccount.

Prepare to support accounts that are uninitialized with no storage.

* Prepare for optional storage.

* Remove obsolete struct.

* Update sync(pull) logic to check if local account exists.

* Local server impl for fetch_account().

* Support import_account_events().

To initialize an account from patches loaded using the sync protocol
CreateSet; used so that linked accounts can be created during a sync
operation.

* Fix conditional files statements.

* Tidy obsolete imports.

* Local server update_account() is forbidden for now.

* Implement sync_account() on local server.

* Prepare LinkedAccount to call sync().

* Improve local sync test spec.

* Implement events_diff() on local server.

* Use shared server helpers.

* Update dependencies.

* Fix wasm builds.

* Implement Merge for NetworkAccount.

* Update generated types.

* Rename types.

* Transfer status code as u16.

* Update type aliases.

* Remove Send requirement for wasm32.

* Fix missing StreamExt import.

* Use shared NetworkError type.

* Set default serde for local transport types.

* Serialize and deserialize headers as a list.

So that are not converted to the Map type in Javascript land otherwise
they cannot cross the postMessage() boundary without special handling.

* Tweak Debug implementations for LocalTransport types.

No need to print the entire body bytes, just the length is enough.

* Improve internal server error handling.

* Move error variant.

Support converting from generic boxed error in sos_protocol as we need
it in the wasm bindings.

* Remove variant.

* Initialize client storage if the account looks usable.

* Tidy dependencies.

* Debug statements in change_set().

* Remove tracing statements.

* fix: Replace tokio imports and spawn_blocking function with async/await syntax

* Update vfs memory file to use futures.

Avoid spawn_blocking() and JoinHandle for wasm support.

* Debug encoding error.

* Bump patch version of vfs crate.

* Update vfs tests, debug encoding issue.

* Add AsyncSeek bound and workaround for wasm bug.

Seek to the end before applying records so the identity bytes are not
overwritten.

* Simplify format_identity_bytes impl.

* Improve logs when importing account events.

* Add read_folders() to client storage.

* Always use same socket name for debug/release builds.

* Ensure folders are loaded in ClientStorage::empty().

* Rename client storage constructors.

* Explore xz compression for local server.

* Switch to zstd.

* Switch to zlib compression due to wasm compiler failure.

See gyscos/zstd-rs#302

Keep the zstd code for now as ultimately we would like to switch once
that bug is fixed.

* Move constants.

* Add local server endpoint for listing accounts.

* Ignore ipc tests for now.

* Remove obsolete variants.

* Refactor to prefer LocalRequest and LocalResponse.

Deletes the now obsolete IpcRequest and IpcResponse types and removes the
logic for delegating from a service to an app.

The new design is that app integrations must sync account data using the
local HTTP server and the standard sync protocol and either store the
data in-memory or cache elsewhere on disc. Users must authenticate to
account(s) in the app integration to give permission for the app to
access the data.

* Remove obsolete protobuf encodings.

* Move client and server modules.

* Refactor and simplify types and generics.

* Delete obsolete ipc tests.

* Tidy dependencies in ipc crate.

* WIP: refactor to HTTP transport.

* Update server to expect HTTP on named pipe.

* Tidy server options.

* Convert client to HTTP transport.

* Remove obsolete codec module.

* Remove all protobuf code from the ipc crate.

* Rename module.

* Tidy ipc dependencies.

* Rename type.

* Fix compression logic.

* Rename types, improve error conversion.

* Remove check from native bridge.

* Tidy features and modules.

* Tidy debug statements.

* Bump patch version.

* Fix error variant.

* Tidy error variants.

* Bump patch version.

* Tidy modules and docs.

* Tidy dependencies.

* Bump versions.

* Update doc comments.

* Update features for docs.rs.

* Update error.

* Improve feature handling.

* Fix feature flags.

* Update type definitions.

* Add initial native bridge test specs.

* Improve native bridge client.

* Update comment.

* Add HttpMessage trait.

* Use NativeBridgeServer.

* Improve client implementation.

So that the LocalSocketStream is re-used between requests rather than
re-connecting to the named pipe each time. Reconnecting was a regression
introduced when we switched to hyper for the transport.

* Prepare logic to handle chunks in native bridge.

* Fix feature dependency.

* Support intercept routes in native bridge server.

So we can mock large responses and therefore test the chunking logic.

* Add test for chunks in native bridge.

Use macro to forbid println! macro.

* Use struct fields to make typeshare happy.

* Tidy debug statements.

* Update doc comments.

* Update types.

* Improve request id handling.

* Update typedefs.

* WIP exploring using duplex stream in native bridge.

* In-memory HTTP server.

Update the native bridge to use the new in-memory server.

We should remove the named pipe code at some point in the future.

* Remove compression support.

* Tidy obsolete code.

* Update features.

* Remove obsolete status route.

* Remove obsolete status route.

* Update open url handling.

* Fix open URL handling.

* Tidy routing in native bridge.

* Tidy imports.

* Add and use ErrorExt trait.

* Add keyring routes.

* Remove obsolete routes.

Now that we are not syncing via wasm the routes used for the local sync
protocol are no longer relevant and have been removed.

* Restore list_accounts() route.

* Support listing authenticated state.

Do not include auto date header.

* Use POST request body.

* Improve native bridge server error replies.

* Add sign out routes.

* Support listing folders.

* Add search functions to ipc service.

* Improve account listing in ipc service.

* Restoring clipboard logic, move to new crate.

* Improve clipboard handling.

* Update route path.

* Split IPC web service routes.

* Support read secret route.

* Update type definitions.

* Add copy_clipboard() to the Account trait.

* Support JSON path queries in copy_clipboard().

* Update type definitions.

* Tweak type name.

* Remove obsolete QualifiedPath.

* Tidy type definitions.

* Write secret variants typescript types by hand.

So we can avoid the tags in the serialized JSON.

If we use tags then there is a duplication between the meta data type
and the secret encoding.

By not using tags we can't use typeshare to generate the typescript
types so do it by hand instead.

* Update types.

* Declare Secret union type.

* More forgiving in JSON Value to string conversion.

For copy to clipboard using JSON path selector.

* Declare FileContent typescript types manually.

Avoid tagged enum representation.

* Tidy type definitions.

* Fix type of authenticated list.

* Revert sort change.

* Add redact() to Secret.

* Redact in native bridge web service.

* Update vcard4.

* Update dependencies.

* Update security framework for macos.

* Update to latest vcard4.

* Improve copy to clipboard API.

Support multiple paths and concatenate multiple matches delimited by a
newline.

* Bump logos and plist dependencies.

* Update vcard dependency.

* Update vcard.

* Expose /avatar route in ipc web service.

To load an avatar for a contact.

* Fix route definition.

* Improve error responses in ipc web service.

* Improve error reporting in json().

* Bump vcard.

* Sketch platform authenticator crate.

* Add basic local_auth module support.

* Derive default for unsupported AuthenticateOptions.

* Update strings for windows.

* Initial work on unified keyring password manager.

* Improve sign in logic in IPC web service.

* Initial interface for platform keyring access.

* Add error conversion.

* Update doc comments.

* Update default SERVICE_NAME.

* Use different service namespace for debug builds.

* Use sign_in() in sos-platform-authenticator.

* Tidy obsolete functions.

* Remove interprocess and named pipe support from ipc crate.

* Map status code in IPC web service.

* Rename function.

* Rename feature flag.

* Improve IPC test specs.

* Remove obsolete, unused test spec.

* Add test spec to IPC tests.

* Improve IPC test specs.

* Remove obsolete constants, prepare to restore test spec.

* Prepare to move LinkedAccount to separate crate.

* Move networking client code to the protocol crate.

* Add web crate with LinkedAccount.

* Add Clipboard placeholders for mobile platforms.

* Update for newer typed-generational-arena.

Removes the dependency on the unmaintained derivative crate.

* Refactor preferences handling.

Avoid global static.

* Support global preferences.

Closes #618.

* Add test spec for global preferences.

* Tidy preference test specs.

* Support format modifiers when copying to clipboard.

* Handle time zone adjustment.

When formatting dates in copy_clipboard().

* Rename native-bridge to extension-helper.

* Tweak comment.

* Bump dependencies.

* Build test executable ahead of time.

So that the executable is ready when the tests run.

* Explore new advisory file lock logic.

* Trying async-fd-lock as part of vfs.

As the previous approach fails on Windows.

* Remove obsolete module.

The async-fd-lock code in vfs appears to be working on Windows.

* Add test spec for preferences concurrent write.

* Improve preferences concurrency test spec.

We need to spawn processes to actually test the advisory file locking,
otherwise the Mutex protects the writes and we don't correctly test the
file lock implementation.

* Fix make task.

* Rename task.

* Add task to check wasm32-unknown-unknown compatibility.

* Use read_exclusive() and write_exclusive() for system messages.

* Fix assertion on linux.

The clipboard does not error on Linux it just returns an empty string
instead.

* Use advisory file locks for audit log file.

* Improve advisory lock module.

Use noop for Android and iOS as well as wasm32-unknown-unknown.

* Update dependencies.

* Use advisory locks in VaultWriter.

* Tidy test handling in vfs crate.

* Prefer write_exclusive().

* Remove FileLock and file-guard dependency.

* Update export_archive_file().

* Do not call set_len() via advisory lock.

* Do not create global preferences in constructor.

This breaks on Android as data_dir() is called before the application
has set the data directory which casues a "could not locate home directory" errror when the app starts on Android.

See saveoursecrets/gui#836.

* Harden wait logic in websocket test specs.

* Disable file lock in audit file to test on windows.

* Restore audit log locks.

* Disable advisory locks for windows at the moment.

* Restore some tests for windows.

* Do not use path to fork of security-framework.

* Be specific about the version we need for security-framework.

* Bump versions in workflow.

* Use explicit security-framework version.

* Prepare file system watcher.

And send a RESET_CONTENT HTTP message over stdout in the extension
helper.

* Support opening debug app when SOS_DEBUG_APP.

The variable must be set at compile time and debug_assertions must be
enabled.

* Change watch directory, update dependencies.

* Bump tokio-rustls-acme.

* Rename types.

* Refine watchers to only watch folder event files.

* Detect and send account/folder events in RESET_CONTENT.

* Rename type and update typegen definitions.

* Add logic to update search index, move type to module.

* Reload event log tree after patch for change records.

So that RESET_CONTENT notifications only reflect the latest changes
correctly.

* Update type definitions.

Write AccountEvent and WriteEvent by hand due to the typeshare
limitations.

* Update type definitions.

* Update types.

* Reload vault data when updating search index.

So that the extension when responding to reset content events loads the
up to date secret information.

* Fix type definition.

When serde sees a tuple enum variant with two entries but the last one
is set to skip then it encodes as an array of one item, however in the
case of WriteEvent::DeleteSecret it's a single item and the containing
array is skipped. I suspect because it is serialized using the new type
logic.

* Fix bug with vault reloading.

Need to reload the vault before updating the search index when
responding to external changes in the extension helper.

* Call load_folders() when accounts events change.

Needs work to debug why the channel is being closed prematurely.

* Improve handling as folders are created and deleted.

* Fix logic for watching for folder create.

Watch the identity event log for changes rather than the account events
because changes to the account events would fire earlier than changes
to the identity folder making it hard to reconcile when the new folder
password would have been saved in the identity folder.

By watching the identity folder event log instead we know that we can
reload the identity folder to access the new folder password which is
required to make watching work well when new secrets are created within
a new folder.

* Remove obsolete feature from sdk.

* Bump patch version.

* Update deny.toml to ignore unmaintained instant crate.

The notify-types crate has already been updated to use web-time but we
need a new release on crates.io to be able to remove the dependency on
instant from the dependency tree.

Author: tmpfs

.github/workflows/publish.yml

@@ -68,7 +68,7 @@ jobs:
 
       # Install
       - name: Install release tools
-        uses: jaxxstorm/action-install-gh-release@v1.11.0
+        uses: jaxxstorm/action-install-gh-release@v1.14.0
         with:
           token: ${{ secrets.GH_RELEASE_DOWNLOAD }}
           repo: saveoursecrets/release-tools

.github/workflows/release.yml

@@ -47,7 +47,7 @@ jobs:
       - name: Check cosign install
         run: cosign version
       - name: Install release tools
-        uses: jaxxstorm/action-install-gh-release@v1.11.0
+        uses: jaxxstorm/action-install-gh-release@v1.14.0
         with:
           token: ${{ secrets.GH_RELEASE_DOWNLOAD }}
           repo: saveoursecrets/release-tools

.gitignore

@@ -16,3 +16,4 @@ server.lock
 tests/server_public_key.txt
 geiger-report.txt
 .aider*
+node_modules