feat(compartment-mapper): Host module exits #2422

kriskowal · 2024-08-20T23:19:18Z

Description

This change closes the remaining gap between Endo as a Zip archive bundler and support for “exits” to host modules from those bundles at import-time.

This begins with a test to demonstrate the latent support for conditional exits to host modules through importLocation. The test fixture has a library that exports the host implementation of itself if the bundler specifies the exit (endo:lib) as a condition. This causes the module to be omitted from bundles and relies on the importer to provide the implementation. There remained a gap for a round-trip through a bundle.

The subsequent changes fix a bottleneck for exit modules in the compartment-mapper. The LavaMoat policy-enforcement runtime is limited to virtual module sources, which constrained support for other kinds of module descriptor. This change opens that up so arbitrary module descriptors pass-through the attenuating adapter if no policy is in effect for that edge. We can return to explore attenuation of other kinds of module-descriptor.

At this point, all exits have to be explicitly marked with an importHook that returns a module descriptor for the named exit module specifier. We then add a feature to the bundler that implicitly recognizes any module specifier that starts with a URI-scheme prefix is an exit, for convenience on the bundler side. This will obviate the need for an additional command-line flag in bundle-source in the common case.

Then, we trivially thread the importHook through importBundle options.

Security Considerations

Host provided modules must be hardened and pure, to avoid being useful as a side-channel or mitm attack surface between guests.

Scaling Considerations

This should allow the creation of smaller bundles.

Documentation Considerations

Any module that implements this feature should document the condition that enables it for bundling and importing.

Testing Considerations

Just a test.

Compatibility Considerations

None.

Upgrade Considerations

None.

dckc

I don't think I get it yet.

dckc · 2024-08-21T02:38:13Z

packages/compartment-mapper/test/fixtures-conditional-host-exports/node_modules/app/index.js

@@ -0,0 +1 @@
+export { feature } from 'lib';


why is app under node_modules?

Ah, this would mostly work at the directory containing node_modules, but the compartment mapper issues a warning if the package.json name does not match the parent directory name.

dckc · 2024-08-21T02:41:52Z

packages/compartment-mapper/test/conditional-host-exports.test.js

+  const { namespace } = await importLocation(readPowers, fixture, {
+    conditions: new Set(['endo:lib']),


This causes the module to be omitted from bundles ...

I'm trying to make such a bundle but losing:

~/projects/endo/packages/compartment-mapper$ ../../node_modules/.bin/bundle-source -C endo:lib ./test/fixtures-conditional-host-exports/node_modules/app/index.js (TypeError#1) TypeError#1: Failed to load module "./index.js" in package "file:///home/connolly/projects/endo/packages/compartment-mapper/test/fixtures-conditional-host-exports/node_modules/app/" (1 underlying failures: Cannot find external module "endo:lib" in package file:///home/connolly/projects/endo/packages/compartment-mapper/test/fixtures-conditional-host-exports/node_modules/lib/ at throwAggregateError (packages/ses/src/module-load.js:546:11) at load (packages/ses/src/module-load.js:594:3) at async digestFromMap (packages/compartment-mapper/src/archive-lite.js:364:3) at async makeAndHashArchiveFromMap (packages/compartment-mapper/src/archive-lite.js:415:52) at async bundleZipBase64 (packages/bundle-source/src/zip-base64.js:83:29) at async main (packages/bundle-source/src/main.js:107:20)

I’ve got some more wrangling to do to make this work through the makeBundle to parseBundle, and a tiny amount of work to thread through from bundleSource to importBundle after that. Retreating to draft.

This should work now. bundle-source -C endo:lib -T produced a bundle with a transparent exit import for me.

confirmed:

$ cd packages/compartment-mapper $ mkdir -p exit $ mkdir -p no-exit $ ../../node_modules/.bin/bundle-source -T ./test/fixtures-conditional-host-exports/node_modules/app/index.js | jq -r .endoZipBase64| base64 -d | bsdtar -C no-exit -xvf- x compartment-map.json x app/index.js x lib/index.js $ ../../node_modules/.bin/bundle-source -C endo:lib -T ./test/fixtures-conditional-host-exports/node_modules/app/index.js | jq -r .endoZipBase64| base64 -d | bsdtar -C exit -xvf- x compartment-map.json x app/index.js x lib/endo.js $ cat exit/lib/endo.js export { feature } from 'endo:lib'; $ cat no-exit/lib/index.js export const feature = 'ponyfill';

POC: Eliding @endo/patterns etc. from a contract bundle

ooh! works on orca.contract.js from dapp-orchestration-basics too!

Contract bundle in its original form is around 2.1M:

$ cd ~/projects/dapp-orchestration-basics/contract $ ~/projects/endo/node_modules/.bin/bundle-source -T src/orca.contract.js | jq -r .endoZipBase64| base64 -d >no-exit.zip && ls -sh no-exit.zip 2.1M no-exit.zip

Then I tweaked @endo/patterns, @endo/far, and @endo/eventual-send something like:

$ git diff . diff --git a/packages/patterns/package.json b/packages/patterns/package.json index b2b9866ea..1a149a52f 100644 --- a/packages/patterns/package.json +++ b/packages/patterns/package.json @@ -18,7 +18,10 @@ "main": "./index.js", "module": "./index.js", "exports": { - ".": "./index.js", + ".": { + "endo:1": "./endo-eventual-send-exit.js", + "default": "./index.js" + }, "./package.json": "./package.json" },

And tweaked the dapp dependencies:

$ ls -l ../node_modules/@endo/{far,patterns,eventual-send} lrwxrwxrwx 1 connolly connolly 51 Aug 26 21:32 ../node_modules/@endo/eventual-send -> /home/connolly/projects/endo/packages/eventual-send lrwxrwxrwx 1 connolly connolly 41 Aug 26 21:25 ../node_modules/@endo/far -> /home/connolly/projects/endo/packages/far lrwxrwxrwx 1 connolly connolly 47 Aug 26 21:37 ../node_modules/@endo/patterns -> /home/connolly/projects/endo/packages/patterns/

Contract bundle shrinks to 17.M:

$ ~/projects/endo/node_modules/.bin/bundle-source -C endo:1 -T src/orca.contract.js | jq -r .endoZipBase64| base64 -d >exit.zip && ls -sh exit.zip 1.7M exit.zip

erights · 2024-08-27T02:45:05Z

packages/compartment-mapper/NEWS.md

+- Archives will now automatically treat any module specifier that appears
+  to start with a URL scheme (like `node:fs`) and these "exits" will no
+  longer need to be marked in the `"modules"` option.


Grammar? I'm having a hard time parsing this sentence.

This should be more clear now.

dckc

Yay! This definitely establishes that we can shrink bundles!

It's more than just a test, though. I'm not confident about approving some of the non-test material.

dckc · 2024-08-27T03:03:24Z

packages/compartment-mapper/src/import-archive-lite.js

-              ),
-              specifier: moduleSpecifier,
-            };
+            return attenuateModuleHook(


The scope of this PR has expanded beyond a test: now, right? Seems like a feat: or a chore: in preparation for a feat:.

dckc · 2024-08-27T03:09:58Z

packages/compartment-mapper/src/link.js

@@ -60,6 +60,8 @@ const has = (object, key) => apply(hasOwnProperty, object, [key]);
 */
 const extensionImpliesLanguage = extension => extension !== 'js';

+const urlish = /^[a-z][a-z0-9+\-.]*:/;


This syntax is subject to an external design constraint; let's please cite it:

Suggested change

const urlish = /^[a-z][a-z0-9+\-.]*:/;

const urlish = /^[a-z][a-z0-9+\-.]*:/; // cf. section 4.3 of RFC 3986 Scheme URI Generic Syntax

optionally include https://www.rfc-editor.org/rfc/rfc3986#section-4.3

dckc · 2024-08-27T03:15:18Z

packages/compartment-mapper/src/policy.js

- * @param {import('ses').ThirdPartyStaticModuleInterface} originalModuleRecord - reference to the exit module
+ * @param {import('ses').VirtualModuleSource | import('ses').StrictModuleDescriptor} moduleDescriptor - reference to the exit module
 * @param {import('./types.js').PackagePolicy} policy - local compartment policy
 * @param {import('./types.js').DeferredAttenuatorsProvider} attenuators - a key-value where attenuations can be found
- * @returns {Promise<import('ses').ThirdPartyStaticModuleInterface>} - the attenuated module
+ * @returns {Promise<import('ses').ModuleDescriptor>} - the attenuated module descriptor
 */
 export const attenuateModuleHook = async (


To what extent is this an externally visible API change?

I don't see any corresponding changes in tests, which suggests it's not an external API change.
But that could be just a lack of test coverage. Or maybe it's only tested indirectly.

As part of convergence with XS, VirtualModuleSource is an alias for ThirdPartyStaticModuleInterface and ModuleDescriptor is a new type that captures all of the new and old types importHooks may return (and the range of types that an importHook adapter is expected to handle). The change here allows the full range of module descriptors to pass through this adapter when a LavaMoat policy is absent, which allows us to surface the whole module descriptor range in an exit module importHook. This was a bottleneck before.

So, the input range expands from VirtualModuleSource to also accept StrictModuleDescriptor, the forward-compatible subset of ModuleDescriptor. I did not feel the need to create a new obligation for backward compatibility past the next SES major version and wasn’t able to get TypeScript to play along with reliably differentiating a ModuleExportsNamespace from a VirtualModuleSource. In SES, we use a brand check for that.

This catches us up with what I expected to be working behavior, so is more like a bugfix than a feature, so I didn’t capture this in the notes.

Was this API in any previous release?

dckc · 2024-08-27T03:19:05Z

packages/ses/NEWS.md

+- Exports a `StrictModuleDescriptor` type that consists only of the
+  `NamespaceModuleDescriptor` and `SourceModuleDescriptor` that are mutually
+  supported by SES and XS.


Yup. PR is clearly more than just test:.

Thanks. The left of the PR description has now caught up with the right of the PR description. The tags on individual commits also capture the conventions.

…tion

…through importHook in absence of a policy Clarifies that policies with module attenuators only work for exit importHooks that provide virtual module sources.

…ule specifiers Like `node:fs`.

…xit and condition

dckc

looks good.

I'm not confident that I know all of the invariants that should be maintained. This review presumes they're covered by tests. If you're not confident about that, please get review from someone who is more familiar with the detailed design of this code.

kriskowal · 2024-08-27T19:38:53Z

Summoning @naugtur since this touches policy enforcement code with delicate invariants.

naugtur · 2024-08-27T19:44:40Z

I'll need some time to dig through this. I'll see if I can go through it before the meeting tomorrow.

naugtur · 2024-08-29T20:32:30Z

packages/compartment-mapper/src/policy.js

+}
+
+/**
+ * Attenuates a module


It'd be nice to have descriptions differ between the two functions

naugtur · 2024-08-29T20:35:16Z

packages/compartment-mapper/src/link.js

+          exports: [],
+          execute() {
+            throw new Error(
+              'Cannot import an application loaded strictly for analysis',


I'll give it another chance later, but I don't understand this error message.

naugtur · 2024-08-29T20:39:41Z

packages/compartment-mapper/src/policy.js

+}) {
+  await null;
+  if ('source' in moduleDescriptor) {
+    const { source: moduleSource } = moduleDescriptor;


const moduleSource = normalizeModuleDescriptorOrSource(moduleDescriptor);

and we don't need all the duplication

kriskowal mentioned this pull request Aug 20, 2024

Externalize Eventual Send in Endo Bundles #1625

Open

kriskowal requested a review from dckc August 20, 2024 23:54

dckc reviewed Aug 21, 2024

View reviewed changes

kriskowal marked this pull request as draft August 21, 2024 19:56

kriskowal force-pushed the kriskowal-ponyfill-host-module branch from 5262c7d to 03c08a3 Compare August 26, 2024 23:59

kriskowal changed the title ~~test(compartment-mapper): Demonstration of host module ponyfill condi…~~ test(compartment-mapper): Host module exits Aug 27, 2024

kriskowal force-pushed the kriskowal-ponyfill-host-module branch from 03c08a3 to 9d1811a Compare August 27, 2024 00:12

kriskowal marked this pull request as ready for review August 27, 2024 00:22

kriskowal requested a review from dckc August 27, 2024 00:22

kriskowal force-pushed the kriskowal-ponyfill-host-module branch from 9d1811a to 754f1b3 Compare August 27, 2024 00:23

erights reviewed Aug 27, 2024

View reviewed changes

dckc reviewed Aug 27, 2024

View reviewed changes

kriskowal force-pushed the kriskowal-ponyfill-host-module branch from 754f1b3 to c7481af Compare August 27, 2024 17:15

kriskowal changed the title ~~test(compartment-mapper): Host module exits~~ feat(compartment-mapper): Host module exits Aug 27, 2024

kriskowal added 8 commits August 27, 2024 10:35

feat(ses): StrictModuleDescriptor type

b1a65c1

test(compartment-mapper): Demonstration of host module ponyfill condi…

8e7b263

…tion

feat(compartment-mapper): Allow arbitrary module descriptors to pass …

c8c6e41

…through importHook in absence of a policy Clarifies that policies with module attenuators only work for exit importHooks that provide virtual module sources.

feat(compartment-mapper): Implicitly exit for URL-scheme-prefixed mod…

a24eaac

…ule specifiers Like `node:fs`.

chore(import-bundle): Fix lint scripts for Yarn 4.

4234ea4

feat(import-bundle): Thread importHook for exits

404b3a3

docs(import-bundle): Release notes for importHook

c0c041f

test(import-hook): Round-trip bundleSource and importBundle with an e…

1bf012f

…xit and condition

kriskowal force-pushed the kriskowal-ponyfill-host-module branch from c7481af to 1bf012f Compare August 27, 2024 17:44

kriskowal requested a review from erights August 27, 2024 17:44

dckc approved these changes Aug 27, 2024

View reviewed changes

kriskowal requested a review from naugtur August 27, 2024 19:38

boneskull self-requested a review August 28, 2024 18:45

naugtur reviewed Aug 29, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(compartment-mapper): Host module exits #2422

feat(compartment-mapper): Host module exits #2422

kriskowal commented Aug 20, 2024 •

edited

Loading

dckc left a comment

dckc Aug 21, 2024

kriskowal Aug 21, 2024

dckc Aug 21, 2024

kriskowal Aug 21, 2024

kriskowal Aug 27, 2024

dckc Aug 27, 2024

dckc Aug 27, 2024

erights Aug 27, 2024

kriskowal Aug 27, 2024

dckc left a comment

dckc Aug 27, 2024

dckc Aug 27, 2024

dckc Aug 27, 2024

kriskowal Aug 27, 2024

dckc Aug 27, 2024

dckc Aug 27, 2024

kriskowal Aug 27, 2024

dckc left a comment

kriskowal commented Aug 27, 2024

naugtur commented Aug 27, 2024

naugtur Aug 29, 2024

naugtur Aug 29, 2024

naugtur Aug 29, 2024

		const { namespace } = await importLocation(readPowers, fixture, {
		conditions: new Set(['endo:lib']),

	const urlish = /^[a-z][a-z0-9+\-.]*:/;
	const urlish = /^[a-z][a-z0-9+\-.]*:/; // cf. section 4.3 of RFC 3986 Scheme URI Generic Syntax

feat(compartment-mapper): Host module exits #2422

Are you sure you want to change the base?

feat(compartment-mapper): Host module exits #2422

Conversation

kriskowal commented Aug 20, 2024 • edited Loading

Description

Security Considerations

Scaling Considerations

Documentation Considerations

Testing Considerations

Compatibility Considerations

Upgrade Considerations

dckc left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

POC: Eliding @endo/patterns etc. from a contract bundle

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dckc left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dckc left a comment

Choose a reason for hiding this comment

kriskowal commented Aug 27, 2024

naugtur commented Aug 27, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kriskowal commented Aug 20, 2024 •

edited

Loading