Skip to content

Commit 643272e

Browse files
lukastaegertTrickyPi
lukastaegert
and
TrickyPi
committedSep 29, 2023
[v4.0] Switch parser to SWC and introduce native/WASM code (rollup#5073)
* Add native compilation for local NodeJS * Link instead of copying for faster dev cycles for now * Parse AST * Convert first trivial AST to a buffer * Use SWC for parsing * Extend AST conversion * Make AST more similar * Fix line number issue by creating a new sourcemap (and thus compiler) for each run * Collect timings * Add code_length to struct * Refine parsing * Extend parsing * Extend AST: ImportDefaultSpecifier, LiteralBoolean, LiteralNull, ExportDefaultExpression * Extend AST: ImportNamespaceSpecifier, ExportAll * Extend AST: BinaryExpression, ArrayPattern, ObjectPattern, AssignmentPatternProperty, ArrayLiteral, ImportExpression * Extend AST: ConditionalExpression * Extend AST: FunctionDeclaration, ClassDeclaration, ClassBody, ReturnStatement * Extend AST: ObjectLiteral, KeyValueProperty * Extend AST: ShorthandProperty * Extend AST: GetterProperty, AssignmentExpression, NewExpression, FunctionExpression * Extend AST: ThrowStatement * Extend AST: ExportDefaultDeclaration * Extend AST: AssignmentPattern, AwaitExpression, BreakStatement Start sorting AST nodes * Extend AST: TryStatement, CatchClause, ChainExpression * Extend AST: ClassExpression, ContinueStatement, DebuggerStatement, DoWhileStatement, EmptyStatement * Extend AST: ExportNamedDeclaration, ForInStatement, ForOfStatement, ForStatement * Extend AST: IfStatement * Extend AST: Import attributes * Extend AST: Literal<RegExp>, Literal<BigInt> * Extend AST: LogicalExpression * Extend AST: MetaProperty * Extend AST: Various Property types * Extend AST: Progress on classes * Extend AST: MethodDefinition, PropertyDefinition, PrivateName, ThisExpression * Extend AST: StaticBlock, Super * Extend AST: RestElement, SequenceExpression, SwitchCase, SwitchStatement * Extend AST: TaggedTemplateExpression, TemplateElement, TemplateLiteral, UnaryExpression, UpdateExpression, YieldExpression * Extend AST: Properties in object patterns * Finishing Fixes * More fixes * Run cargo fmt * Handle directives * Minor fixes * Unicode support * Fix optional chain expressions * Adapt tests * Do not run acorn anymore * Update lockfile * Minor fixes * Move to rust folder * Separate Rust Node bindings to allow adding WASM bindings in another workspace * Make Napi build closer to how rs.napi works * Fix path issues * Disable browser build for now * Add native package directories * Refine runWithEcho * Try initial steps with Github flow * Trigger change * Temporarily add yarn lockfile until we figured out if we get it to work with npm * Use nightly toolchain * Use default locations for Napi files to make things easier * Adapt workflow * Skip regular tests for now * Attempt to fix broken workflows * Attempt to fix broken workflows * Attempt to fix broken workflows * Test MacOS/WIndows * Fix bootstrap build * Skip tests * Rename workflow * Add additional tests * Use zig differently * Try to fix musl build * Skip musl again for now * Add publish to workflow * 4.0.0-0 * Remove yarn lock again * Fix coverage job * Fix artefact handling * Revert "4.0.0-0" This reverts commit 734806f. * 4.0.0-0 * Do not include default triples twice * Fix native npm packages * 4.0.0-1 * Add missing additional tests except browser tests * Try to fix publish and tests * 4.0.0-2 * Switch to faster utf-16 conversion * 4.0.0-3 * Fix positioning algorithm when manually searching the code * 4.0.0-4 * feat: Add WASM browser build (rollup#5077) * feat: add wasm browser build * move wasm binding into a separate cargo workspace * use imports replacing * set the targetEnv option of wasm to browser in browser build * add the wasm build to build command * fix lint error and ci error * add more comments to silence the linter * big change * trigger change * run browser tests * trigger change * tweak wasm build on CI * Increase build timeout * Use shared string constants * Extract fixed strings into constants * Remove comment * Get rid of dbg! calls * Add lockfile hash to cargo cache * Use if let over match in some cases * Return the buffer of the syntax error in parse_ast * Initial annotation support * Put annotation types into string table * Remove invalid annotations * Support nested calls and new expressions * Improve tree-shaking for annotations * Adapt test * Properly handle line-breaks, commas etc. * Mark nested pure annotations as pure * Remove sourcemap comments * Handle function side effect annotatinos * Remove old comment-handling code * Increase timeout for browser tests * Only skip the tests that still fail * Run coverage again on CI * Get the buffer of pos and message from the Syntax error * Handle the lint errors from SWC * Reenable tests about parse errors and adjust some relevant code * Emit native.js to native.cjs * Add cjs extension for native importee when bundling for ESM * Change native importee with replace plugin and emit native.cjs in napi build * Silence the linter for importing native * Use node:path to resolve native binding files * Add Node WASM files to Native packages for StackBlitz and similar use cases * Unignore *.d.ts in the wasm dir * 4.0.0-5 * Fix copy-wasm-node.js * 4.0.0-6 * Remove .gitignore in wasm-node directory * 4.0.0-7 * Include .cjs files when publishing rollup to npm * 4.0.0-8 * Get readString function at runtime * eslint: ignore wasm-node and set node extensions of import/resolver * 4.0.0-9 * Change the required node to >=14.18.0 as before * 4.0.0-10 * Prepare to fix ESM build Still requires bug-fix on Rollup side for relative external dependencies outside the ouput directory * Update Rollup * Enable ESM tests * Re-enable another test * Remove CJS eslint configuration * Fix extension * Fix test * Fix entension for native import * Support for publishing a completely separate package @rollup/wasm-node * 4.0.0-11 * Fix publish for wasm node package * Add AST verification to function tests * Only use plugin arrays in form and function tests * Fix spans in function tests * Verify AST in form tests as well * Try to publish @rollup/browser and fix publish-wasm-node-package.js * 4.0.0-12 * Tweak publish scripts * Fix importing wasm file in browser * Parse code as unknown module type for greater compatibility * Remove acorn options * Tweak getReadStringFunction * Fix browser tests * 4.0.0-13 * Remove polyfills that are no longer needed in browser tests * Convert to new import attributes AST format * Rename assertions to attributes * Deprecate externalImportAssertions in favor of externalImportAttributes * Update SWC version * Remove max-call-stack test SWC is not capable of handling it and we cannot fix it * Improve coverage * re-enable repl-artefacts workflow * Fix test * Preload wasm file in docs * docs: add functions to get full path of url * Delete the build plugin handleImportMetaUrl * Make 'npm install github/branch' work * Verify there is a valid changelog entry before releasing * Create release notes and comments from CI Minor change to maybe trigger a CI run * Fix RegExp * 4.0.0-14 * Minor changes for a test PR (rollup#5139) * Fix RegExp use * 4.0.0-15 * Do not rely on current branch to find the PR * 4.0.0-16 * Fix how to determine git commit range * 4.0.0-17 * Make sure we fetch all history on publish * 4.0.0-18 * Add proper permissions * 4.0.0-19 * Use Double quotation marks instead of Single quotation marks if concurrently scripts with flags * Remove "engines" from native packages * Update CONTRIBUTING.md --------- Co-authored-by: XiaoPi <[email protected]>
1 parent 4576ef3 commit 643272e

File tree

342 files changed

+11987
-3503
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

342 files changed

+11987
-3503
lines changed
 

‎.eslintrc.js

+4-2
Original file line numberDiff line numberDiff line change
@@ -21,11 +21,13 @@ module.exports = {
2121
'coverage',
2222
'_tmp',
2323
'cache',
24-
'wasm',
24+
'native*.*',
2525
'/test/*/samples/**/*.*',
2626
'!/test/*/samples/**/_config.js',
2727
'!/test/*/samples/**/rollup.config.js',
28-
'!.vitepress'
28+
'!.vitepress',
29+
'/wasm/',
30+
'/wasm-node/'
2931
],
3032
overrides: [
3133
{

‎.github/workflows/build-and-tests.yml

+348
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,348 @@
1+
name: CI
2+
env:
3+
DEBUG: napi:*
4+
APP_NAME: rollup
5+
MACOSX_DEPLOYMENT_TARGET: '10.13'
6+
7+
on:
8+
push:
9+
branches:
10+
- master
11+
tags:
12+
- 'v*.*.**'
13+
pull_request:
14+
types:
15+
- synchronize
16+
- opened
17+
- reopened
18+
merge_group:
19+
20+
permissions:
21+
contents: read
22+
23+
jobs:
24+
lint:
25+
name: Lint and Audit
26+
runs-on: ubuntu-latest
27+
timeout-minutes: 15
28+
steps:
29+
- name: Checkout Commit
30+
uses: actions/checkout@v4
31+
- name: Setup Node
32+
uses: actions/setup-node@v3
33+
with:
34+
node-version: 20
35+
- name: Cache Node Modules
36+
id: cache-node-modules
37+
uses: actions/cache@v3
38+
with:
39+
path: node_modules
40+
key: node-modules-${{ runner.os }}-${{ hashFiles('package-lock.json') }}
41+
- name: Install dependencies
42+
if: steps.cache-node-modules.outputs.cache-hit != 'true'
43+
run: npm ci --ignore-scripts
44+
- name: Lint
45+
run: npm run ci:lint
46+
- name: Vulnerabilities
47+
run: npm audit
48+
- name: Optional Dependencies
49+
run: npm run test:package
50+
- name: CLI Docs
51+
run: npm run test:options
52+
53+
build:
54+
strategy:
55+
fail-fast: false
56+
matrix:
57+
settings:
58+
- host: windows-latest
59+
target: x86_64-pc-windows-msvc
60+
name: wasm
61+
is-wasm-build: true
62+
build: npm run build:wasm
63+
path: |
64+
wasm/
65+
.empty
66+
- host: windows-latest
67+
target: x86_64-pc-windows-msvc
68+
name: wasm-node
69+
is-wasm-build: true
70+
build: npm run build:wasm:node
71+
path: |
72+
wasm-node/
73+
.empty
74+
- host: macos-latest
75+
target: x86_64-apple-darwin
76+
build: |
77+
npm run build:napi -- --release
78+
strip -x *.node
79+
- host: windows-latest
80+
build: npm run build:napi -- --release
81+
target: x86_64-pc-windows-msvc
82+
- host: windows-latest
83+
build: |
84+
npm run build:napi -- --release --target i686-pc-windows-msvc
85+
target: i686-pc-windows-msvc
86+
- host: ubuntu-latest
87+
target: x86_64-unknown-linux-gnu
88+
docker: ghcr.io/napi-rs/napi-rs/nodejs-rust:lts-debian
89+
build: |-
90+
set -e &&
91+
rustup default nightly &&
92+
npm run build:napi -- --release --target x86_64-unknown-linux-gnu &&
93+
strip *.node
94+
- host: ubuntu-latest
95+
target: x86_64-unknown-linux-musl
96+
docker: ghcr.io/napi-rs/napi-rs/nodejs-rust:lts-alpine
97+
build: |-
98+
set -e &&
99+
rustup default nightly &&
100+
npm run build:napi -- --release && strip *.node
101+
- host: macos-latest
102+
target: aarch64-apple-darwin
103+
build: |
104+
npm run build:napi -- --release --target aarch64-apple-darwin
105+
strip -x *.node
106+
- host: ubuntu-latest
107+
target: aarch64-unknown-linux-gnu
108+
docker: ghcr.io/napi-rs/napi-rs/nodejs-rust:lts-debian-aarch64
109+
build: |-
110+
set -e &&
111+
rustup default nightly &&
112+
rustup target add aarch64-unknown-linux-gnu &&
113+
npm run build:napi -- --release --target aarch64-unknown-linux-gnu &&
114+
aarch64-unknown-linux-gnu-strip *.node
115+
- host: ubuntu-latest
116+
target: armv7-unknown-linux-gnueabihf
117+
zig: true
118+
setup: |
119+
sudo apt-get update
120+
sudo apt-get install gcc-arm-linux-gnueabihf -y
121+
build: |
122+
npm run build:napi -- --release --target armv7-unknown-linux-gnueabihf
123+
arm-linux-gnueabihf-strip *.node
124+
- host: ubuntu-latest
125+
target: aarch64-linux-android
126+
build: |
127+
npm run build:napi -- --release --target aarch64-linux-android
128+
${ANDROID_NDK_LATEST_HOME}/toolchains/llvm/prebuilt/linux-x86_64/bin/llvm-strip *.node
129+
- host: ubuntu-latest
130+
target: armv7-linux-androideabi
131+
build: |
132+
npm run build:napi -- --release --target armv7-linux-androideabi
133+
${ANDROID_NDK_LATEST_HOME}/toolchains/llvm/prebuilt/linux-x86_64/bin/llvm-strip *.node
134+
# TODO SWC this does not work yet, also add back in package.json
135+
# - host: ubuntu-latest
136+
# target: aarch64-unknown-linux-musl
137+
# docker: ghcr.io/napi-rs/napi-rs/nodejs-rust:lts-alpine
138+
# build: |-
139+
# set -e &&
140+
# rustup default nightly &&
141+
# rustup target add aarch64-unknown-linux-musl &&
142+
# npm run build:napi -- --release --target aarch64-unknown-linux-musl &&
143+
# /aarch64-linux-musl-cross/bin/aarch64-linux-musl-strip *.node
144+
- host: windows-latest
145+
target: aarch64-pc-windows-msvc
146+
build: npm run build:napi -- --release --target aarch64-pc-windows-msvc
147+
name: Build ${{ matrix.settings.name || matrix.settings.target }}
148+
runs-on: ${{ matrix.settings.host }}
149+
timeout-minutes: 30
150+
steps:
151+
- name: Checkout Commit
152+
uses: actions/checkout@v4
153+
- name: Setup Node
154+
uses: actions/setup-node@v3
155+
if: ${{ !matrix.settings.docker }}
156+
with:
157+
node-version: 20
158+
- name: Install Toolchain
159+
uses: dtolnay/rust-toolchain@stable
160+
if: ${{ !matrix.settings.docker }}
161+
with:
162+
toolchain: nightly
163+
targets: ${{ matrix.settings.target }}
164+
- name: Cache cargo
165+
uses: actions/cache@v3
166+
with:
167+
path: |
168+
~/.cargo/registry/index/
169+
~/.cargo/registry/cache/
170+
~/.cargo/git/db/
171+
.cargo-cache
172+
rust/target/
173+
key: ${{ matrix.settings.name || matrix.settings.target }}-cargo-${{ matrix.settings.host }}-${{ hashFiles('rust/Cargo.lock') }}
174+
restore-keys: ${{ matrix.settings.name || matrix.settings.target }}-cargo-${{ matrix.settings.host }}
175+
- uses: goto-bus-stop/setup-zig@v2
176+
if: ${{ matrix.settings.zig }}
177+
with:
178+
version: 0.10.1
179+
- name: Setup toolchain
180+
run: ${{ matrix.settings.setup }}
181+
if: ${{ matrix.settings.setup }}
182+
shell: bash
183+
- name: Cache Node Modules
184+
id: cache-node-modules
185+
uses: actions/cache@v3
186+
with:
187+
path: node_modules
188+
key: node-modules-${{ runner.os }}-${{ hashFiles('package-lock.json') }}
189+
- name: Install dependencies
190+
if: steps.cache-node-modules.outputs.cache-hit != 'true'
191+
run: npm ci --ignore-scripts
192+
- name: Build in docker
193+
uses: addnab/docker-run-action@v3
194+
if: ${{ matrix.settings.docker }}
195+
with:
196+
image: ${{ matrix.settings.docker }}
197+
options: '--user 0:0 -v ${{ github.workspace }}/.cargo-cache/git/db:/usr/local/cargo/git/db -v ${{ github.workspace }}/.cargo/registry/cache:/usr/local/cargo/registry/cache -v ${{ github.workspace }}/.cargo/registry/index:/usr/local/cargo/registry/index -v ${{ github.workspace }}:/build -w /build'
198+
run: ${{ matrix.settings.build }}
199+
- name: Build
200+
run: ${{ matrix.settings.build }}
201+
if: ${{ !matrix.settings.docker }}
202+
shell: bash
203+
- name: Upload wasm artifact
204+
uses: actions/upload-artifact@v3
205+
with:
206+
name: bindings-${{ matrix.settings.name }}
207+
path: ${{ matrix.settings.path }}
208+
if-no-files-found: error
209+
if: ${{ matrix.settings.is-wasm-build }}
210+
- name: Upload napi artifact
211+
uses: actions/upload-artifact@v3
212+
with:
213+
name: bindings-${{ matrix.settings.target }}
214+
path: ${{ env.APP_NAME }}.*.node
215+
if-no-files-found: error
216+
if: ${{ !matrix.settings.is-wasm-build }}
217+
218+
test:
219+
name: Test${{ matrix.additionalName || '' }} Node ${{ matrix.node }} (${{ matrix.settings.target }})
220+
needs:
221+
- build
222+
strategy:
223+
fail-fast: false
224+
matrix:
225+
settings:
226+
- host: ubuntu-latest
227+
target: x86_64-unknown-linux-gnu
228+
- host: windows-latest
229+
target: x86_64-pc-windows-msvc
230+
- host: macos-latest
231+
target: x86_64-apple-darwin
232+
node:
233+
- '18.0.0'
234+
- '20'
235+
include:
236+
- settings:
237+
host: ubuntu-latest
238+
target: x86_64-unknown-linux-gnu
239+
node: '20'
240+
command: 'ci:coverage'
241+
additionalName: ' with Coverage'
242+
coverage: true
243+
- settings:
244+
host: ubuntu-latest
245+
target: x86_64-unknown-linux-gnu
246+
node: '18.0.0'
247+
command: 'ci:test:all'
248+
additionalName: ' with Additional Tests'
249+
runs-on: ${{ matrix.settings.host }}
250+
timeout-minutes: 15
251+
steps:
252+
- name: Checkout Commit
253+
uses: actions/checkout@v4
254+
- name: Setup Node
255+
uses: actions/setup-node@v3
256+
with:
257+
node-version: ${{ matrix.node }}
258+
check-latest: true
259+
- name: Cache Node Modules
260+
id: cache-node-modules
261+
uses: actions/cache@v3
262+
with:
263+
path: node_modules
264+
key: node-modules-${{ runner.os }}-${{ hashFiles('package-lock.json') }}
265+
- name: Install dependencies
266+
if: steps.cache-node-modules.outputs.cache-hit != 'true'
267+
run: npm ci --ignore-scripts
268+
- name: Download napi artifacts
269+
uses: actions/download-artifact@v3
270+
with:
271+
name: bindings-${{ matrix.settings.target }}
272+
path: .
273+
- name: Download wasm artifacts
274+
uses: actions/download-artifact@v3
275+
with:
276+
name: bindings-wasm
277+
- name: Build and test Rollup
278+
run: npm run ${{ matrix.command || 'ci:test:only' }}
279+
env:
280+
CI: true
281+
- name: Upload coverage
282+
uses: codecov/codecov-action@v3
283+
if: matrix.coverage
284+
with:
285+
# It appears adding the token may result in the coverage comment not being added
286+
# token: ${{ secrets.CODECOV_TOKEN }}
287+
commit_parent: ${{ github.event.pull_request.head.sha }}
288+
289+
publish:
290+
permissions:
291+
pull-requests: write # for writing comments
292+
name: Publish
293+
runs-on: ubuntu-latest
294+
timeout-minutes: 15
295+
needs:
296+
- test
297+
- lint
298+
# TODO SWC once v4 is live, this should be changed
299+
if: startsWith(github.ref_name, 'v4')
300+
steps:
301+
- name: Checkout Commit
302+
uses: actions/checkout@v4
303+
with:
304+
# Necessary to find the commits included in the release
305+
fetch-depth: 0
306+
- name: Setup Node
307+
uses: actions/setup-node@v3
308+
with:
309+
node-version: ${{ matrix.node }}
310+
check-latest: true
311+
- name: Cache Node Modules
312+
id: cache-node-modules
313+
uses: actions/cache@v3
314+
with:
315+
path: node_modules
316+
key: node-modules-${{ runner.os }}-${{ hashFiles('package-lock.json') }}
317+
- name: Install dependencies
318+
if: steps.cache-node-modules.outputs.cache-hit != 'true'
319+
run: npm ci --ignore-scripts
320+
- name: Download all artifacts
321+
uses: actions/download-artifact@v3
322+
with:
323+
path: artifacts
324+
- name: Copy wasm files
325+
run: cp -R artifacts/bindings-wasm/wasm .
326+
- name: Build JavaScript
327+
run: npm run build:cjs && cp artifacts/*/*.node dist/ && npm run build:bootstrap
328+
- name: Move artifacts
329+
run: npm run ci:artifacts
330+
- name: Publish
331+
run: |
332+
if git --no-pager log -1 --pretty=%B | grep "^[0-9]\+\.[0-9]\+\.[0-9]\+$";
333+
then
334+
echo "Publishing regular release"
335+
echo "//registry.npmjs.org/:_authToken=$NPM_TOKEN" >> ~/.npmrc
336+
npm publish --access public
337+
elif git --no-pager log -1 --pretty=%B | grep "^[0-9]\+\.[0-9]\+\.[0-9]\+-[0-9]\+$";
338+
then
339+
echo "Publishing beta release"
340+
echo "//registry.npmjs.org/:_authToken=$NPM_TOKEN" >> ~/.npmrc
341+
npm publish --tag beta --access public
342+
else
343+
echo "Not a release, skipping publish"
344+
fi
345+
env:
346+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
347+
NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
348+
ROLLUP_RELEASE: 'releasing'

0 commit comments

Comments
 (0)
Please sign in to comment.