Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: Golang Http Tcp Bridge #36667

Merged
merged 74 commits into from
Jan 27, 2025
Merged

Feature: Golang Http Tcp Bridge #36667

merged 74 commits into from
Jan 27, 2025

Conversation

duxin40
Copy link
Contributor

@duxin40 duxin40 commented Oct 17, 2024

As mentioned in the proposal: #35749 , this PR is to support using Golang to extend TCP upstream proxy, to make changes to connections and data messages in the http2tcp situation of envoy.
(this PR does)

Here is my thought about Golang extension function points:

  1. Support encoding message processing for upstream TCP requests (route and cluster have been determined, and targeted message processing can be performed for route and cluster)
  2. Support the handling of conn connection status during the encoding stage of upstream TCP requests (for example, by setting end_stream=false to avoid envoy semi connected status)
  3. Support decoding message processing and aggregation for upstream TCP response in onUpstreamData.(for example, by setting end_stream=true to indicate that the message is encapsulated and can be passed to downstream)
  4. Aggregate the tcp messages received multiple times by onUpstreamData of tcp response.
  5. Support obtaining route and cluster information, which can be referenced for targeted processing in the above stages.

What we can do with this Golang extension:

With this golang extension, developers can quickly get started with envoy and use golang to implement http2tcp such as http2dubbo、http2rpc.

Commit Message: support Golang TCP Upstream Extension for http2tcp on Envoy
Additional Description:
Risk Level:
Testing:
Docs Changes:
Release Notes:
Platform Specific Features:
[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Fixes commit #PR or SHA]
[Optional Deprecated:]
[Optional API Considerations:]

duxin40 and others added 4 commits October 17, 2024 11:24
Copy link

As a reminder, PRs marked as draft will not be automatically assigned reviewers,
or be handled by maintainer-oncall triage.

Please mark your PR as ready when you want it to be reviewed!

🐱

Caused by: #36667 was opened by duxin40.

see: more, trace.

@duxin40 duxin40 marked this pull request as ready for review October 17, 2024 08:12
duxin40 added 2 commits October 17, 2024 16:20
Signed-off-by: duxin40 <[email protected]>
Signed-off-by: duxin40 <[email protected]>
@duxin40
Copy link
Contributor Author

duxin40 commented Oct 17, 2024

I will add unit tests for these codes and fix errors related to unit tests.

@duxin40
Copy link
Contributor Author

duxin40 commented Oct 17, 2024

@doujiang24 could you take a look and check if the direction looks good? thanks!

@tyxia
Copy link
Member

tyxia commented Oct 18, 2024

/assign @doujiang24

Per discussion in #35749

Copy link

@duxin40 cannot be assigned to this issue.

🐱

Caused by: a #36667 (comment) was created by @tyxia.

see: more, trace.

Copy link
Member

@doujiang24 doujiang24 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better add a simple example that use Golang to implement http => TCP protocol, to show how it could be for users for easier understanding.
It do not need to be runable, just a demo.

source/common/router/router.cc Outdated Show resolved Hide resolved
@duxin40
Copy link
Contributor Author

duxin40 commented Oct 21, 2024

Better add a simple example that use Golang to implement http => TCP protocol, to show how it could be for users for easier understanding. It do not need to be runable, just a demo.

Done. in the dir: contrib/upstreams/http/tcp/test_data/http2dubbo-examples
image

duxin40 added 3 commits October 21, 2024 21:05
Signed-off-by: duxin40 <[email protected]>
Signed-off-by: duxin40 <[email protected]>
@mattklein123 mattklein123 self-assigned this Oct 21, 2024
}

void TcpUpstream::encodeTrailers(const Envoy::Http::RequestTrailerMap&) {
Buffer::OwnedImpl data;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto, we'd better golang convert trailers to data buffer, but we can just add a TODO for it now, it's not a common use case.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

got it, will finish it in the near future.

…om response header in envoyGoOnUpstreamData

Signed-off-by: duxin40 <[email protected]>
Signed-off-by: duxin40 <[email protected]>
Copy link

Docs for this Pull Request will be rendered here:

https://storage.googleapis.com/envoy-pr/36667/docs/index.html

The docs are (re-)rendered each time the CI envoy-presubmit (precheck docs) job completes.

🐱

Caused by: a #36667 (comment) was created by @duxin40.

see: more, trace.

duxin40 added 2 commits January 19, 2025 21:21
Signed-off-by: duxin40 <[email protected]>
Signed-off-by: duxin40 <[email protected]>
Copy link

CC @envoyproxy/api-shepherds: Your approval is needed for changes made to (api/envoy/|docs/root/api-docs/).
envoyproxy/api-shepherds assignee is @mattklein123
CC @envoyproxy/api-watchers: FYI only for changes made to (api/envoy/|docs/root/api-docs/).

🐱

Caused by: #36667 was synchronize by duxin40.

see: more, trace.

Signed-off-by: duxin40 <[email protected]>
Signed-off-by: duxin40 <[email protected]>
Copy link
Member

@phlax phlax left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for working on this @duxin40

@@ -179,3 +179,73 @@ updates:
schedule:
interval: daily
time: "06:00"

- package-ecosystem: "gomod"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it this still required? afaict this uses the same deps that have been merged, surely we can use an existing go.mod?

docs/root/api-v3/config/contrib/http_tcp_bridge/golang.rst Outdated Show resolved Hide resolved
docs/root/configuration/http/tcp_bridge/golang.rst Outdated Show resolved Hide resolved
docs/root/configuration/http/tcp_bridge/golang.rst Outdated Show resolved Hide resolved
docs/root/configuration/http/tcp_bridge/golang.rst Outdated Show resolved Hide resolved
docs/root/configuration/http/tcp_bridge/golang.rst Outdated Show resolved Hide resolved
docs/root/configuration/http/tcp_bridge/golang.rst Outdated Show resolved Hide resolved
duxin40 and others added 3 commits January 21, 2025 22:25
Signed-off-by: duxin40 <[email protected]>
Signed-off-by: duxin40 <[email protected]>
@duxin40
Copy link
Contributor Author

duxin40 commented Jan 22, 2025

/retest

@mattklein123
Copy link
Member

I don't have time to review this giant PR. Will merge once @doujiang24 and @phlax give the ok.

/wait-any

Signed-off-by: duxin40 <[email protected]>
@duxin40
Copy link
Contributor Author

duxin40 commented Jan 27, 2025

Please run the tests in the asan mode,

Get it, here is the success result for asan mode:

vscode@docker-desktop:/workspaces/envoy$ bash -x ci/do_ci.sh asan
+ set -e
+ export SRCDIR=/workspaces/envoy
+ SRCDIR=/workspaces/envoy
+ export ENVOY_SRCDIR=/workspaces/envoy
+ ENVOY_SRCDIR=/workspaces/envoy
+++ dirname ci/do_ci.sh
++ realpath ci
+ CURRENT_SCRIPT_DIR=/workspaces/envoy/ci
+ . /workspaces/envoy/ci/build_setup.sh
++ set -e
++ [[ -n '' ]]
++++ dirname /workspaces/envoy/ci/build_setup.sh
+++ realpath /workspaces/envoy/ci
++ CURRENT_SCRIPT_DIR=/workspaces/envoy/ci
++ export PPROF_PATH=/thirdparty_build/bin/pprof
++ PPROF_PATH=/thirdparty_build/bin/pprof
++ [[ -z '' ]]
++ [[ linux-gnu == darwin* ]]
+++ grep -c '^processor' /proc/cpuinfo
++ NUM_CPUS=14
++ '[' -z /workspaces/envoy ']'
++ '[' -z '' ']'
++ export ENVOY_BUILD_TARGET=//source/exe:envoy-static
++ ENVOY_BUILD_TARGET=//source/exe:envoy-static
++ '[' -z '' ']'
++ export ENVOY_BUILD_DEBUG_INFORMATION=//source/exe:envoy-static.dwp
++ ENVOY_BUILD_DEBUG_INFORMATION=//source/exe:envoy-static.dwp
++ '[' -z '' ']'
++ export ENVOY_CONTRIB_BUILD_TARGET=//contrib/exe:envoy-static
++ ENVOY_CONTRIB_BUILD_TARGET=//contrib/exe:envoy-static
++ '[' -z '' ']'
++ export ENVOY_CONTRIB_BUILD_DEBUG_INFORMATION=//contrib/exe:envoy-static.dwp
++ ENVOY_CONTRIB_BUILD_DEBUG_INFORMATION=//contrib/exe:envoy-static.dwp
++ '[' -z '' ']'
+++ uname -m
++ ENVOY_BUILD_ARCH=aarch64
++ export ENVOY_BUILD_ARCH
++ read -ra BAZEL_BUILD_EXTRA_OPTIONS
++ read -ra BAZEL_EXTRA_TEST_OPTIONS
++ read -ra BAZEL_STARTUP_EXTRA_OPTIONS
++ read -ra BAZEL_OPTIONS
++ echo ENVOY_SRCDIR=/workspaces/envoy
ENVOY_SRCDIR=/workspaces/envoy
++ echo ENVOY_BUILD_TARGET=//source/exe:envoy-static
ENVOY_BUILD_TARGET=//source/exe:envoy-static
++ echo ENVOY_BUILD_ARCH=aarch64
ENVOY_BUILD_ARCH=aarch64
++ [[ -z /build ]]
++ [[ ! -d /build ]]
++ export BUILD_DIR
++ export ENVOY_TEST_TMPDIR=/build/tmp
++ ENVOY_TEST_TMPDIR=/build/tmp
++ export LLVM_ROOT=/opt/llvm
++ LLVM_ROOT=/opt/llvm
++ export PATH=/opt/llvm/bin:/opt/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/build/.local/bin:/usr/local/go/bin
++ PATH=/opt/llvm/bin:/opt/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/build/.local/bin:/usr/local/go/bin
++ [[ -f /etc/redhat-release ]]
++ cleanup
++ rm -rf '/workspaces/envoy/bazel-*' clang.bazelrc
++ trap cleanup EXIT
+++ which bazel
++ _bazel=/usr/local/bin/bazel
++ BAZEL_STARTUP_OPTIONS=("${BAZEL_STARTUP_EXTRA_OPTIONS[@]}" "--output_user_root=${BUILD_DIR}/bazel_root" "--output_base=${BUILD_DIR}/bazel_root/base")
++ export _bazel
++ export -f bazel
++ BAZEL_GLOBAL_OPTIONS=("--repository_cache=${BUILD_DIR}/repository_cache" "--experimental_repository_cache_hardlinks")
++ BAZEL_BUILD_OPTIONS=("${BAZEL_OPTIONS[@]}" "${BAZEL_GLOBAL_OPTIONS[@]}" "--verbose_failures" "--experimental_generate_json_trace_profile" "${BAZEL_BUILD_EXTRA_OPTIONS[@]}" "${BAZEL_EXTRA_TEST_OPTIONS[@]}")
++ [[ aarch64 == \a\a\r\c\h\6\4 ]]
++ BAZEL_BUILD_OPTIONS+=("--test_env=HEAPCHECK=")
++ [[ -z '' ]]
++ BAZEL_BUILD_OPTIONS+=("--test_tmpdir=${ENVOY_TEST_TMPDIR}")
++ echo 'Setting test_tmpdir to /build/tmp.'
Setting test_tmpdir to /build/tmp.
++ BAZEL_STARTUP_OPTION_LIST='--output_user_root=/build/bazel_root --output_base=/build/bazel_root/base'
++ BAZEL_BUILD_OPTION_LIST='--repository_cache=/build/repository_cache --experimental_repository_cache_hardlinks --verbose_failures --experimental_generate_json_trace_profile --test_env=HEAPCHECK= --test_tmpdir=/build/tmp'
++ BAZEL_GLOBAL_OPTION_LIST='--repository_cache=/build/repository_cache --experimental_repository_cache_hardlinks'
++ export BAZEL_STARTUP_OPTION_LIST
++ export BAZEL_BUILD_OPTION_LIST
++ export BAZEL_GLOBAL_OPTION_LIST
++ [[ -z '' ]]
++ [[ -e /opt/llvm ]]
++ /workspaces/envoy/ci/../bazel/setup_clang.sh /opt/llvm
++ [[ '' == \1 ]]
++ [[ aarch64 == \x\8\6\_\6\4 ]]
++ ENVOY_BUILD_DIR=/build/envoy/arm64
++ export ENVOY_BUILD_DIR
++ mkdir -p /build/envoy/arm64
++ export ENVOY_DELIVERY_DIR=/build/envoy/arm64/source/exe
++ ENVOY_DELIVERY_DIR=/build/envoy/arm64/source/exe
++ mkdir -p /build/envoy/arm64/source/exe
++ export ENVOY_COVERAGE_ARTIFACT=/build/envoy/arm64/generated/coverage.tar.zst
++ ENVOY_COVERAGE_ARTIFACT=/build/envoy/arm64/generated/coverage.tar.zst
++ export ENVOY_FUZZ_COVERAGE_ARTIFACT=/build/envoy/arm64/generated/fuzz_coverage.tar.zst
++ ENVOY_FUZZ_COVERAGE_ARTIFACT=/build/envoy/arm64/generated/fuzz_coverage.tar.zst
++ export ENVOY_FAILED_TEST_LOGS=/build/envoy/arm64/generated/failed-testlogs
++ ENVOY_FAILED_TEST_LOGS=/build/envoy/arm64/generated/failed-testlogs
++ mkdir -p /build/envoy/arm64/generated/failed-testlogs
++ export ENVOY_BUILD_PROFILE=/build/envoy/arm64/generated/build-profile
++ ENVOY_BUILD_PROFILE=/build/envoy/arm64/generated/build-profile
++ mkdir -p /build/envoy/arm64/generated/build-profile
++ export NO_BUILD_SETUP=1
++ NO_BUILD_SETUP=1
+ echo 'building using 14 CPUs'
building using 14 CPUs
+ echo 'building for aarch64'
building for aarch64
+ cd /workspaces/envoy
+ [[ aarch64 == \x\8\6\_\6\4 ]]
+ [[ aarch64 == \a\a\r\c\h\6\4 ]]
+ BUILD_ARCH_DIR=/linux/arm64
+ CI_TARGET=asan
+ shift
+ [[ asan =~ bazel.* ]]
+ [[ 0 -ge 1 ]]
+ COVERAGE_TEST_TARGETS=("//test/...")
+ [[ asan == \r\e\l\e\a\s\e ]]
+ [[ asan == \r\e\l\e\a\s\e\.\t\e\s\t\_\o\n\l\y ]]
+ [[ asan == \m\s\a\n ]]
+ TEST_TARGETS=("${COVERAGE_TEST_TARGETS[@]}" "@com_github_google_quiche//:ci_tests")
+ case $CI_TARGET in
+ setup_clang_toolchain
+ CONFIG_PARTS=()
+ [[ -n '' ]]
+ CONFIG_PARTS+=("clang")
+ ENVOY_STDLIB=libc++
+ [[ libc++ == \l\i\b\c\+\+ ]]
+ CONFIG_PARTS+=("libc++")
++ IFS=-
++ echo clang-libc++
+ CONFIG=clang-libc++
+ BAZEL_BUILD_OPTIONS+=("--config=${CONFIG}")
+ BAZEL_BUILD_OPTION_LIST='--repository_cache=/build/repository_cache --experimental_repository_cache_hardlinks --verbose_failures --experimental_generate_json_trace_profile --test_env=HEAPCHECK= --test_tmpdir=/build/tmp --config=clang-libc++'
+ export BAZEL_BUILD_OPTION_LIST
+ echo 'clang toolchain with libc++ configured: clang-libc++'
clang toolchain with libc++ configured: clang-libc++
+ [[ -n '' ]]
+ ASAN_CONFIG=--config=clang-asan
+ BAZEL_BUILD_OPTIONS+=(-c dbg "${ASAN_CONFIG}" "--build_tests_only" "--remote_download_minimal")
+ echo 'bazel ASAN/UBSAN debug build with tests'
bazel ASAN/UBSAN debug build with tests
+ echo 'Building and testing envoy tests //test/... @com_github_google_quiche//:ci_tests'
Building and testing envoy tests //test/... @com_github_google_quiche//:ci_tests
+ bazel_with_collection test --repository_cache=/build/repository_cache --experimental_repository_cache_hardlinks --verbose_failures --experimental_generate_json_trace_profile --test_env=HEAPCHECK= --test_tmpdir=/build/tmp --config=clang-libc++ -c dbg --config=clang-asan --build_tests_only --remote_download_minimal //contrib/golang/upstreams/http/tcp/test:golang_bridge_integration_test
+ local failed_logs
+ declare -r BAZEL_OUTPUT=/workspaces/envoy/bazel.output.txt
+ bazel test --repository_cache=/build/repository_cache --experimental_repository_cache_hardlinks --verbose_failures --experimental_generate_json_trace_profile --test_env=HEAPCHECK= --test_tmpdir=/build/tmp --config=clang-libc++ -c dbg --config=clang-asan --build_tests_only --remote_download_minimal //contrib/golang/upstreams/http/tcp/test:golang_bridge_integration_test
+ local startup_options
+ read -ra startup_options
+ tee /workspaces/envoy/bazel.output.txt
+ /usr/local/bin/bazel --output_user_root=/build/bazel_root --output_base=/build/bazel_root/base test --repository_cache=/build/repository_cache --experimental_repository_cache_hardlinks --verbose_failures --experimental_generate_json_trace_profile --test_env=HEAPCHECK= --test_tmpdir=/build/tmp --config=clang-libc++ -c dbg --config=clang-asan --build_tests_only --remote_download_minimal //contrib/golang/upstreams/http/tcp/test:golang_bridge_integration_test
WARNING: The following configs were expanded more than once: [clang]. For repeatable flags, repeats are counted twice and may lead to unexpected behavior.
Computing main repo mapping:
WARNING: The following configs were expanded more than once: [clang]. For repeatable flags, repeats are counted twice and may lead to unexpected behavior.
Loading:
Loading: 0 packages loaded
WARNING: Build option --run_under has changed, discarding analysis cache (this can be expensive, see https://bazel.build/advanced/performance/iteration-speed).
Analyzing: target //contrib/golang/upstreams/http/tcp/test:golang_bridge_integration_test (0 packages loaded, 0 targets configured)
Analyzing: target //contrib/golang/upstreams/http/tcp/test:golang_bridge_integration_test (0 packages loaded, 0 targets configured)

INFO: Analyzed target //contrib/golang/upstreams/http/tcp/test:golang_bridge_integration_test (0 packages loaded, 31025 targets configured).
[7,437 / 7,467] [Prepa] Testing //contrib/golang/upstreams/http/tcp/test:golang_bridge_integration_test (shard 1 of 30) ... (8 actions, 0 running)
[7,437 / 7,467] Testing //contrib/golang/upstreams/http/tcp/test:golang_bridge_integration_test (shard 12 of 30); 1s processwrapper-sandbox ... (13 actions, 8 running)
[7,442 / 7,467] [Sched] Testing //contrib/golang/upstreams/http/tcp/test:golang_bridge_integration_test (shard 1 of 30) ... (13 actions, 6 running)
[7,449 / 7,467] Testing //contrib/golang/upstreams/http/tcp/test:golang_bridge_integration_test (shard 13 of 30); 1s processwrapper-sandbox ... (13 actions, 6 running)
[7,457 / 7,467] [Sched] Testing //contrib/golang/upstreams/http/tcp/test:golang_bridge_integration_test (shard 26 of 30) ... (10 actions, 5 running)
[7,462 / 7,467] Testing //contrib/golang/upstreams/http/tcp/test:golang_bridge_integration_test (shard 26 of 30); 0s processwrapper-sandbox ... (5 actions running)
INFO: Found 1 test target...
Target //contrib/golang/upstreams/http/tcp/test:golang_bridge_integration_test up-to-date:
  bazel-bin/contrib/golang/upstreams/http/tcp/test/golang_bridge_integration_test
INFO: Elapsed time: 7.867s, Critical Path: 3.66s
INFO: 34 processes: 4 internal, 30 processwrapper-sandbox.
INFO: Build completed successfully, 34 total actions

Executed 1 out of 1 test: 1 test passes.
+ declare BAZEL_STATUS=0
+ '[' 0 '!=' 0 ']'
+ collect_build_profile test
+ local output_base
+ declare -g build_profile_count=1
++ bazel info --repository_cache=/build/repository_cache --experimental_repository_cache_hardlinks --verbose_failures --experimental_generate_json_trace_profile --test_env=HEAPCHECK= --test_tmpdir=/build/tmp --config=clang-libc++ -c dbg --config=clang-asan --build_tests_only --remote_download_minimal output_base
++ local startup_options
++ read -ra startup_options
++ /usr/local/bin/bazel --output_user_root=/build/bazel_root --output_base=/build/bazel_root/base info --repository_cache=/build/repository_cache --experimental_repository_cache_hardlinks --verbose_failures --experimental_generate_json_trace_profile --test_env=HEAPCHECK= --test_tmpdir=/build/tmp --config=clang-libc++ -c dbg --config=clang-asan --build_tests_only --remote_download_minimal output_base
WARNING: The following configs were expanded more than once: [clang]. For repeatable flags, repeats are counted twice and may lead to unexpected behavior.
+ output_base=/build/bazel_root/base
+ mv -f /build/bazel_root/base/command.profile.gz /build/envoy/arm64/generated/build-profile/1-test.profile.gz
+ (( build_profile_count++ ))
+ cleanup
+ rm -rf /workspaces/envoy/bazel-bin /workspaces/envoy/bazel-envoy /workspaces/envoy/bazel-out /workspaces/envoy/bazel-testlogs clang.bazelrc

Copy link
Member

@doujiang24 doujiang24 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cool, lgtm, it's good enogh to merge, thanks @duxin40
please take another look @phlax , thanks~

@phlax
Copy link
Member

phlax commented Jan 27, 2025

@duxin40 if you can merge main i will land this

@phlax phlax enabled auto-merge (squash) January 27, 2025 10:21
@duxin40
Copy link
Contributor Author

duxin40 commented Jan 27, 2025

/retest

@phlax phlax merged commit b034c57 into envoyproxy:main Jan 27, 2025
24 checks passed
@duxin40
Copy link
Contributor Author

duxin40 commented Jan 27, 2025

@doujiang24 @phlax thanks so much for your patience and review for the gaint PR in the past three months !!!

bazmurphy pushed a commit to bazmurphy/envoy that referenced this pull request Jan 29, 2025
As mentioned in the proposal:
envoyproxy#35749 , this PR is to support
using Golang to extend TCP upstream proxy, to make changes to
connections and data messages in the http2tcp situation of envoy.
(this PR does)

### Here is my thought about  Golang extension function points:
1. Support encoding message processing for upstream TCP requests (route
and cluster have been determined, and targeted message processing can be
performed for route and cluster)
2. Support the handling of conn connection status during the encoding
stage of upstream TCP requests (for example, by setting end_stream=false
to avoid envoy semi connected status)
3. Support decoding message processing and aggregation for upstream TCP
response in onUpstreamData.(for example, by setting end_stream=true to
indicate that the message is encapsulated and can be passed to
downstream)
4. Aggregate the tcp messages received multiple times by onUpstreamData
of tcp response.
5. Support obtaining route and cluster information, which can be
referenced for targeted processing in the above stages.

### What we can do with this  Golang extension:
With this golang extension, developers can quickly get started with
envoy and use golang to implement http2tcp such as http2dubbo、http2rpc.


Signed-off-by: duxin40 <[email protected]>
Signed-off-by: duxin40 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants