Skip to content

Commit e1e55d3

Browse files
achingbrainachingbrain
committed
fix!: close streams gracefully
- Refactors `.close`, `closeRead` and `.closeWrite` methods on the `Stream` interface to be async - The `Connection` interface now has `.close` and `.abort` methods - `.close` on `Stream`s and `Connection`s wait for the internal message queues to empty before closing - `.abort` on `Stream`s and `Connection`s close the underlying stream immediately and discards any unsent data - `@chainsafe/libp2p-yamux` now uses the `AbstractStream` class from `@libp2p/interface` the same as `@libp2p/mplex` and `@libp2p/webrtc` Follow-up PRs will be necessary to `@chainsafe/libp2p-yamux`, `@chainsafe/libp2p-gossipsub` and `@chainsafe/libp2p-noise` though they will not block the release as their code is temporarily added to this repo to let CI run. Fixes #1793 Fixes #656 BREAKING CHANGE: the `.close`, `closeRead` and `closeWrite` methods on the `Stream` interface are now asynchronous
1 parent e9cafd3 commit e1e55d3

File tree

132 files changed

+4336
-2207
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

132 files changed

+4336
-2207
lines changed

‎.github/workflows/main.yml

+2-1
Original file line numberDiff line numberDiff line change
@@ -186,7 +186,7 @@ jobs:
186186
- uses: actions/checkout@v3
187187
- uses: ipfs/aegir/actions/cache-node-modules@master
188188
- name: Build images
189-
run: (cd interop && make)
189+
run: (cd interop && make -j 4)
190190
- name: Save package-lock.json as artifact
191191
uses: actions/upload-artifact@v2
192192
with:
@@ -197,6 +197,7 @@ jobs:
197197
- uses: libp2p/test-plans/.github/actions/run-interop-ping-test@master
198198
with:
199199
test-filter: js-libp2p-head
200+
test-ignore: nim
200201
extra-versions: ${{ github.workspace }}/interop/node-version.json ${{ github.workspace }}/interop/chromium-version.json ${{ github.workspace }}/interop/firefox-version.json
201202
s3-cache-bucket: ${{ vars.S3_LIBP2P_BUILD_CACHE_BUCKET_NAME }}
202203
s3-access-key-id: ${{ vars.S3_LIBP2P_BUILD_CACHE_AWS_ACCESS_KEY_ID }}

‎doc/METRICS.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,7 @@ const node = await createLibp2p({
6464
To define component metrics first get a reference to the metrics object:
6565

6666
```ts
67-
import type { Metrics } from '@libp2p/interface-metrics'
67+
import type { Metrics } from '@libp2p/interface/metrics'
6868

6969
interface MyClassComponents {
7070
metrics: Metrics

‎interop/BrowserDockerfile

+2
Original file line numberDiff line numberDiff line change
@@ -7,5 +7,7 @@ WORKDIR /app/interop
77
RUN npx playwright install
88
ARG BROWSER=chromium # Options: chromium, firefox, webkit
99
ENV BROWSER=$BROWSER
10+
# disable colored output and CLI animation from test runners
11+
ENV CI true
1012

1113
ENTRYPOINT npm run test:interop:multidim -- --build false --types false -t browser -- --browser $BROWSER

‎interop/Dockerfile

+3
Original file line numberDiff line numberDiff line change
@@ -8,4 +8,7 @@ COPY ./interop ./interop
88

99
WORKDIR /app/interop
1010

11+
# disable colored output and CLI animation from test runners
12+
ENV CI true
13+
1114
ENTRYPOINT [ "npm", "run", "test:interop:multidim", "--", "--build", "false", "--types", "false", "-t", "node" ]

‎interop/README.md

+76-3
Original file line numberDiff line numberDiff line change
@@ -9,14 +9,87 @@
99
1010
## Table of contents <!-- omit in toc -->
1111

12-
- [Install](#install)
12+
- [Usage](#usage)
13+
- [Build js-libp2p](#build-js-libp2p)
14+
- [node.js](#nodejs)
15+
- [Browsers](#browsers)
16+
- [Build another libp2p implementation](#build-another-libp2p-implementation)
17+
- [Running Redis](#running-redis)
18+
- [Start libp2p](#start-libp2p)
19+
- [Start another libp2p implementation](#start-another-libp2p-implementation)
1320
- [License](#license)
1421
- [Contribution](#contribution)
1522

16-
## Install
23+
## Usage
24+
25+
The multidim interop tests use random high ports for listeners. Since you need to know which port will be listened on ahead of time to `EXPOSE` a port in a Docker image to the host machine, this means everything has to be run in Docker.
26+
27+
### Build js-libp2p
28+
29+
This must be repeated every time you make a change to the js-libp2p source code.
30+
31+
#### node.js
32+
33+
```console
34+
$ npm run build
35+
$ docker build . -f ./interop/Dockerfile -t js-libp2p-node
36+
```
37+
38+
#### Browsers
39+
40+
```console
41+
$ npm run build
42+
$ docker build . -f ./interop/BrowserDockerfile -t js-libp2p-browsers
43+
```
44+
45+
### Build another libp2p implementation
46+
47+
1. Clone the test-plans repo somewhere
48+
```console
49+
$ git clone https://github.com/libp2p/test-plans.git
50+
```
51+
2. (Optional) If you are running an M1 Mac you may need to override the build platform.
52+
- Edit `/multidim-interop/dockerBuildWrapper.sh`
53+
- Add `--platform linux/arm64/v8` to the `docker buildx build` command
54+
```
55+
docker buildx build \
56+
--platform linux/arm64/v8 \ <-- add this line
57+
--load \
58+
-t $IMAGE_NAME $CACHING_OPTIONS "$@"
59+
```
60+
3. (Optional) Enable some sort of debug output
61+
- nim-libp2p
62+
- edit `/multidim-interop/impl/nim/$VERSION/Dockerfile`
63+
- Change `-d:chronicles_log_level=WARN` to `-d:chronicles_log_level=DEBUG`
64+
- rust-libp2p
65+
- When starting the docker container add `-e RUST_LOG=debug`
66+
- go-libp2p
67+
- When starting the docker container add `-e GOLOG_LOG_LEVEL=debug`
68+
4. Build the version you want to test against
69+
```console
70+
$ cd impl/$IMPL/$VERSION
71+
$ make
72+
...
73+
```
74+
75+
### Running Redis
76+
77+
Redis is used to allow inter-container communication, exchanging listen addresses etc. It must be started as a Docker container:
78+
79+
```console
80+
$ docker run --name redis --rm -p 6379:6379 redis:7-alpine
81+
```
82+
83+
### Start libp2p
84+
85+
```console
86+
$ docker run -e transport=tcp -e muxer=yamux -e security=noise -e is_dialer=true -e redis_addr=redis:6379 --link redis:redis js-libp2p-node
87+
```
88+
89+
### Start another libp2p implementation
1790

1891
```console
19-
$ npm i multidim-interop
92+
$ docker run -e transport=tcp -e muxer=yamux -e security=noise -e is_dialer=false -e redis_addr=redis:6379 --link redis:redis nim-v1.0
2093
```
2194

2295
## License

‎interop/package.json

+1
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@
3535
"scripts": {
3636
"start": "node index.js",
3737
"build": "aegir build",
38+
"lint": "aegir lint",
3839
"test:interop:multidim": "aegir test"
3940
},
4041
"dependencies": {

‎interop/src/index.ts

+2-2
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
1-
console.log("Everything is defined in the test folder")
1+
// Everything is defined in the test folder
22

3-
export { }
3+
export { }

‎interop/test/ping.spec.ts

+14-4
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,10 @@ let node: Libp2p<{ ping: PingService, identify: IdentifyService }>
2828
const isDialer: boolean = process.env.is_dialer === 'true'
2929
const timeoutSecs: string = process.env.test_timeout_secs ?? '180'
3030

31-
describe('ping test', () => {
31+
describe('ping test', function () {
32+
// make the default timeout longer than the listener timeout
33+
this.timeout((parseInt(timeoutSecs) * 1000) + 30000)
34+
3235
// eslint-disable-next-line complexity
3336
beforeEach(async () => {
3437
// Setup libp2p node
@@ -39,6 +42,9 @@ describe('ping test', () => {
3942

4043
const options: Libp2pOptions<{ ping: PingService, identify: IdentifyService }> = {
4144
start: true,
45+
connectionManager: {
46+
minConnections: 0
47+
},
4248
connectionGater: {
4349
denyDialMultiaddr: async () => false
4450
},
@@ -208,14 +214,18 @@ describe('ping test', () => {
208214
// eslint-disable-next-line complexity
209215
(isDialer ? it : it.skip)('should dial and ping', async () => {
210216
try {
211-
let otherMa: string = (await redisProxy(['BLPOP', 'listenerAddr', timeoutSecs]).catch(err => { throw new Error(`Failed to wait for listener: ${err}`) }))[1]
217+
let otherMaStr: string = (await redisProxy(['BLPOP', 'listenerAddr', timeoutSecs]).catch(err => { throw new Error(`Failed to wait for listener: ${err}`) }))[1]
212218
// Hack until these are merged:
213219
// - https://github.com/multiformats/js-multiaddr-to-uri/pull/120
214-
otherMa = otherMa.replace('/tls/ws', '/wss')
220+
otherMaStr = otherMaStr.replace('/tls/ws', '/wss')
221+
222+
const otherMa = multiaddr(otherMaStr)
215223

216224
console.error(`node ${node.peerId.toString()} pings: ${otherMa}`)
217225
const handshakeStartInstant = Date.now()
218-
await node.dial(multiaddr(otherMa))
226+
227+
await node.dial(otherMa)
228+
219229
const pingRTT = await node.services.ping.ping(multiaddr(otherMa))
220230
const handshakePlusOneRTT = Date.now() - handshakeStartInstant
221231
console.log(JSON.stringify({

‎packages/connection-encryption-noise/package.json

+99
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
{
2+
"name": "@chainsafe/libp2p-noise",
3+
"version": "12.0.1",
4+
"author": "ChainSafe <[email protected]>",
5+
"license": "Apache-2.0 OR MIT",
6+
"homepage": "https://github.com/ChainSafe/js-libp2p-noise#readme",
7+
"repository": {
8+
"type": "git",
9+
"url": "git+https://github.com/ChainSafe/js-libp2p-noise.git"
10+
},
11+
"bugs": {
12+
"url": "https://github.com/ChainSafe/js-libp2p-noise/issues"
13+
},
14+
"keywords": [
15+
"crypto",
16+
"libp2p",
17+
"noise"
18+
],
19+
"engines": {
20+
"node": ">=16.0.0",
21+
"npm": ">=7.0.0"
22+
},
23+
"type": "module",
24+
"types": "./dist/src/index.d.ts",
25+
"files": [
26+
"src",
27+
"dist",
28+
"!dist/test",
29+
"!**/*.tsbuildinfo"
30+
],
31+
"exports": {
32+
".": {
33+
"types": "./dist/src/index.d.ts",
34+
"import": "./dist/src/index.js"
35+
}
36+
},
37+
"eslintConfig": {
38+
"extends": "ipfs",
39+
"parserOptions": {
40+
"sourceType": "module"
41+
},
42+
"rules": {
43+
"@typescript-eslint/no-unused-vars": "error",
44+
"@typescript-eslint/explicit-function-return-type": "warn",
45+
"@typescript-eslint/strict-boolean-expressions": "off"
46+
},
47+
"ignorePatterns": [
48+
"src/proto/payload.js",
49+
"src/proto/payload.d.ts",
50+
"test/fixtures/node-globals.js"
51+
]
52+
},
53+
"scripts": {
54+
"bench": "node benchmarks/benchmark.js",
55+
"clean": "aegir clean",
56+
"dep-check": "aegir dep-check",
57+
"build": "aegir build",
58+
"lint": "aegir lint",
59+
"lint:fix": "aegir lint --fix",
60+
"test": "aegir test",
61+
"test:node": "aegir test -t node",
62+
"test:browser": "aegir test -t browser -t webworker",
63+
"test:electron-main": "aegir test -t electron-main",
64+
"docs": "aegir docs",
65+
"proto:gen": "protons ./src/proto/payload.proto",
66+
"prepublish": "npm run build"
67+
},
68+
"dependencies": {
69+
"@libp2p/crypto": "^1.0.11",
70+
"@libp2p/interface": "~0.0.1",
71+
"@libp2p/logger": "^2.1.1",
72+
"@libp2p/peer-id": "^2.0.0",
73+
"@stablelib/chacha20poly1305": "^1.0.1",
74+
"@noble/hashes": "^1.3.0",
75+
"@stablelib/x25519": "^1.0.3",
76+
"it-length-prefixed": "^9.0.1",
77+
"it-length-prefixed-stream": "^1.0.0",
78+
"it-byte-stream": "^1.0.0",
79+
"it-pair": "^2.0.2",
80+
"it-pipe": "^3.0.1",
81+
"it-stream-types": "^2.0.1",
82+
"protons-runtime": "^5.0.0",
83+
"uint8arraylist": "^2.3.2",
84+
"uint8arrays": "^4.0.2"
85+
},
86+
"devDependencies": {
87+
"@libp2p/interface-compliance-tests": "^3.0.0",
88+
"@libp2p/peer-id-factory": "^2.0.0",
89+
"@types/sinon": "^10.0.14",
90+
"aegir": "^39.0.5",
91+
"iso-random-stream": "^2.0.2",
92+
"protons": "^7.0.0",
93+
"sinon": "^15.0.0"
94+
},
95+
"browser": {
96+
"./dist/src/alloc-unsafe.js": "./dist/src/alloc-unsafe-browser.js",
97+
"util": false
98+
}
99+
}

‎packages/connection-encryption-noise/src/@types/basic.ts

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
export type bytes = Uint8Array
2+
export type bytes32 = Uint8Array
3+
export type bytes16 = Uint8Array
4+
5+
export type uint64 = number

‎packages/connection-encryption-noise/src/@types/handshake-interface.ts

+12
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
import type { bytes } from './basic.js'
2+
import type { NoiseSession } from './handshake.js'
3+
import type { NoiseExtensions } from '../proto/payload.js'
4+
import type { PeerId } from '@libp2p/interface/peer-id'
5+
6+
export interface IHandshake {
7+
session: NoiseSession
8+
remotePeer: PeerId
9+
remoteExtensions: NoiseExtensions
10+
encrypt: (plaintext: bytes, session: NoiseSession) => bytes
11+
decrypt: (ciphertext: bytes, session: NoiseSession, dst?: Uint8Array) => { plaintext: bytes, valid: boolean }
12+
}

‎packages/connection-encryption-noise/src/@types/handshake.ts

+48
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
import type { bytes, bytes32, uint64 } from './basic.js'
2+
import type { KeyPair } from './libp2p.js'
3+
import type { Nonce } from '../nonce.js'
4+
5+
export type Hkdf = [bytes, bytes, bytes]
6+
7+
export interface MessageBuffer {
8+
ne: bytes32
9+
ns: bytes
10+
ciphertext: bytes
11+
}
12+
13+
export interface CipherState {
14+
k: bytes32
15+
// For performance reasons, the nonce is represented as a Nonce object
16+
// The nonce is treated as a uint64, even though the underlying `number` only has 52 safely-available bits.
17+
n: Nonce
18+
}
19+
20+
export interface SymmetricState {
21+
cs: CipherState
22+
ck: bytes32 // chaining key
23+
h: bytes32 // handshake hash
24+
}
25+
26+
export interface HandshakeState {
27+
ss: SymmetricState
28+
s: KeyPair
29+
e?: KeyPair
30+
rs: bytes32
31+
re: bytes32
32+
psk: bytes32
33+
}
34+
35+
export interface NoiseSession {
36+
hs: HandshakeState
37+
h?: bytes32
38+
cs1?: CipherState
39+
cs2?: CipherState
40+
mc: uint64
41+
i: boolean
42+
}
43+
44+
export interface INoisePayload {
45+
identityKey: bytes
46+
identitySig: bytes
47+
data: bytes
48+
}

‎packages/connection-encryption-noise/src/@types/libp2p.ts

+10
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
import type { bytes32 } from './basic.js'
2+
import type { NoiseExtensions } from '../proto/payload.js'
3+
import type { ConnectionEncrypter } from '@libp2p/interface/connection-encrypter'
4+
5+
export interface KeyPair {
6+
publicKey: bytes32
7+
privateKey: bytes32
8+
}
9+
10+
export interface INoiseConnection extends ConnectionEncrypter<NoiseExtensions> {}

‎packages/connection-encryption-noise/src/constants.ts

+4
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
export const NOISE_MSG_MAX_LENGTH_BYTES = 65535
2+
export const NOISE_MSG_MAX_LENGTH_BYTES_WITHOUT_TAG = NOISE_MSG_MAX_LENGTH_BYTES - 16
3+
4+
export const DUMP_SESSION_KEYS = Boolean(globalThis.process?.env?.DUMP_SESSION_KEYS)

‎packages/connection-encryption-noise/src/crypto.ts

+16
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
import type { bytes32, bytes } from './@types/basic.js'
2+
import type { Hkdf } from './@types/handshake.js'
3+
import type { KeyPair } from './@types/libp2p.js'
4+
5+
export interface ICryptoInterface {
6+
hashSHA256: (data: Uint8Array) => Uint8Array
7+
8+
getHKDF: (ck: bytes32, ikm: Uint8Array) => Hkdf
9+
10+
generateX25519KeyPair: () => KeyPair
11+
generateX25519KeyPairFromSeed: (seed: Uint8Array) => KeyPair
12+
generateX25519SharedKey: (privateKey: Uint8Array, publicKey: Uint8Array) => Uint8Array
13+
14+
chaCha20Poly1305Encrypt: (plaintext: Uint8Array, nonce: Uint8Array, ad: Uint8Array, k: bytes32) => bytes
15+
chaCha20Poly1305Decrypt: (ciphertext: Uint8Array, nonce: Uint8Array, ad: Uint8Array, k: bytes32, dst?: Uint8Array) => bytes | null
16+
}

0 commit comments

Comments
 (0)