test(sample): Add E2E tests for app start transactions #4619
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…rst processed root span
…rst processed root span
…to add-e2e-tests-to-sample
krystofwoldrich
changed the title
Android (new) Performance metrics 🚀
|
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| d16beca+dirty | 389.49 ms | 423.86 ms | 34.37 ms |
| abb7058+dirty | 320.78 ms | 324.08 ms | 3.30 ms |
| 2dc8f22+dirty | 373.17 ms | 400.42 ms | 27.25 ms |
| cdf2bdf+dirty | 391.69 ms | 461.14 ms | 69.45 ms |
| 686b3bc+dirty | 363.48 ms | 356.17 ms | -7.31 ms |
| 1c9e040+dirty | 372.30 ms | 417.80 ms | 45.50 ms |
| 13f280b+dirty | 376.13 ms | 406.41 ms | 30.28 ms |
| 13b68d9+dirty | 372.54 ms | 388.85 ms | 16.31 ms |
| 4a6664f+dirty | 357.02 ms | 394.91 ms | 37.89 ms |
| 5fe8312+dirty | 389.98 ms | 421.46 ms | 31.48 ms |
App size
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| d16beca+dirty | 7.15 MiB | 8.37 MiB | 1.22 MiB |
| abb7058+dirty | 7.15 MiB | 8.10 MiB | 980.40 KiB |
| 2dc8f22+dirty | 7.15 MiB | 8.38 MiB | 1.23 MiB |
| cdf2bdf+dirty | 7.15 MiB | 8.38 MiB | 1.23 MiB |
| 686b3bc+dirty | 7.15 MiB | 8.38 MiB | 1.23 MiB |
| 1c9e040+dirty | 7.15 MiB | 8.38 MiB | 1.23 MiB |
| 13f280b+dirty | 7.15 MiB | 8.35 MiB | 1.20 MiB |
| 13b68d9+dirty | 7.15 MiB | 8.38 MiB | 1.23 MiB |
| 4a6664f+dirty | 7.15 MiB | 8.22 MiB | 1.07 MiB |
| 5fe8312+dirty | 7.15 MiB | 8.38 MiB | 1.23 MiB |
Previous results on branch: add-detox-test
Startup times
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| 61ddd2e+dirty | 416.74 ms | 409.22 ms | -7.51 ms |
| d993785+dirty | 359.24 ms | 360.76 ms | 1.51 ms |
| 576acc7+dirty | 417.75 ms | 416.39 ms | -1.36 ms |
| ec9549b+dirty | 388.98 ms | 383.76 ms | -5.22 ms |
| 922b103+dirty | 387.53 ms | 391.51 ms | 3.98 ms |
App size
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| 61ddd2e+dirty | 7.15 MiB | 8.39 MiB | 1.23 MiB |
| d993785+dirty | 7.15 MiB | 8.39 MiB | 1.23 MiB |
| 576acc7+dirty | 7.15 MiB | 8.39 MiB | 1.23 MiB |
| ec9549b+dirty | 7.15 MiB | 8.39 MiB | 1.23 MiB |
| 922b103+dirty | 7.15 MiB | 8.39 MiB | 1.23 MiB |
iOS (legacy) Performance metrics 🚀
|
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| 484813b+dirty | 1222.45 ms | 1220.79 ms | -1.66 ms |
| 34aba08+dirty | 1276.78 ms | 1308.52 ms | 31.74 ms |
| 8bda0cc+dirty | 1221.90 ms | 1208.11 ms | -13.79 ms |
| ad6c299+dirty | 1244.76 ms | 1260.10 ms | 15.34 ms |
| 60d7316+dirty | 1219.13 ms | 1221.74 ms | 2.61 ms |
| d997097+dirty | 1231.90 ms | 1236.80 ms | 4.90 ms |
| 0d3e677+dirty | 1214.39 ms | 1225.70 ms | 11.31 ms |
| 7143e6f+dirty | 1237.04 ms | 1233.88 ms | -3.16 ms |
| 7e87527+dirty | 1209.85 ms | 1216.31 ms | 6.46 ms |
| c639edf+dirty | 1236.18 ms | 1235.04 ms | -1.14 ms |
App size
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| 484813b+dirty | 2.36 MiB | 3.08 MiB | 734.18 KiB |
| 34aba08+dirty | 2.36 MiB | 2.85 MiB | 495.32 KiB |
| 8bda0cc+dirty | 2.63 MiB | 3.70 MiB | 1.06 MiB |
| ad6c299+dirty | 2.36 MiB | 2.84 MiB | 488.85 KiB |
| 60d7316+dirty | 2.63 MiB | 3.74 MiB | 1.11 MiB |
| d997097+dirty | 2.63 MiB | 3.70 MiB | 1.06 MiB |
| 0d3e677+dirty | 2.36 MiB | 3.10 MiB | 753.12 KiB |
| 7143e6f+dirty | 2.63 MiB | 3.70 MiB | 1.06 MiB |
| 7e87527+dirty | 2.63 MiB | 3.69 MiB | 1.05 MiB |
| c639edf+dirty | 2.36 MiB | 3.08 MiB | 736.63 KiB |
Previous results on branch: add-detox-test
Startup times
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| d993785+dirty | 1226.10 ms | 1232.39 ms | 6.29 ms |
| 61ddd2e+dirty | 1235.82 ms | 1233.81 ms | -2.00 ms |
| 576acc7+dirty | 1213.31 ms | 1217.08 ms | 3.78 ms |
| 922b103+dirty | 1233.41 ms | 1226.48 ms | -6.93 ms |
| ec9549b+dirty | 1233.27 ms | 1237.41 ms | 4.14 ms |
App size
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| d993785+dirty | 2.63 MiB | 3.75 MiB | 1.12 MiB |
| 61ddd2e+dirty | 2.63 MiB | 3.75 MiB | 1.12 MiB |
| 576acc7+dirty | 2.63 MiB | 3.75 MiB | 1.12 MiB |
| 922b103+dirty | 2.63 MiB | 3.75 MiB | 1.12 MiB |
| ec9549b+dirty | 2.63 MiB | 3.75 MiB | 1.12 MiB |
iOS (new) Performance metrics 🚀
|
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| 484813b+dirty | 1225.07 ms | 1221.00 ms | -4.07 ms |
| 34aba08+dirty | 1268.58 ms | 1276.80 ms | 8.22 ms |
| 8bda0cc+dirty | 1217.90 ms | 1223.02 ms | 5.12 ms |
| ad6c299+dirty | 1248.50 ms | 1248.88 ms | 0.38 ms |
| 60d7316+dirty | 1224.33 ms | 1231.29 ms | 6.96 ms |
| d997097+dirty | 1223.29 ms | 1229.76 ms | 6.46 ms |
| 0d3e677+dirty | 1239.02 ms | 1241.22 ms | 2.20 ms |
| 7143e6f+dirty | 1228.29 ms | 1229.02 ms | 0.73 ms |
| 7e87527+dirty | 1223.83 ms | 1222.90 ms | -0.94 ms |
| c639edf+dirty | 1223.63 ms | 1227.98 ms | 4.35 ms |
App size
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| 484813b+dirty | 2.92 MiB | 3.64 MiB | 740.56 KiB |
| 34aba08+dirty | 2.92 MiB | 3.41 MiB | 499.03 KiB |
| 8bda0cc+dirty | 3.19 MiB | 4.26 MiB | 1.08 MiB |
| ad6c299+dirty | 2.92 MiB | 3.40 MiB | 494.12 KiB |
| 60d7316+dirty | 3.19 MiB | 4.30 MiB | 1.12 MiB |
| d997097+dirty | 3.19 MiB | 4.26 MiB | 1.08 MiB |
| 0d3e677+dirty | 2.92 MiB | 3.66 MiB | 758.42 KiB |
| 7143e6f+dirty | 3.19 MiB | 4.26 MiB | 1.08 MiB |
| 7e87527+dirty | 3.19 MiB | 4.25 MiB | 1.06 MiB |
| c639edf+dirty | 2.92 MiB | 3.64 MiB | 742.55 KiB |
Previous results on branch: add-detox-test
Startup times
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| d993785+dirty | 1210.45 ms | 1227.92 ms | 17.46 ms |
| 61ddd2e+dirty | 1232.92 ms | 1230.49 ms | -2.43 ms |
| 576acc7+dirty | 1221.35 ms | 1234.94 ms | 13.59 ms |
| 922b103+dirty | 1218.81 ms | 1211.89 ms | -6.93 ms |
| ec9549b+dirty | 1216.31 ms | 1221.02 ms | 4.71 ms |
App size
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| d993785+dirty | 3.19 MiB | 4.32 MiB | 1.13 MiB |
| 61ddd2e+dirty | 3.19 MiB | 4.32 MiB | 1.13 MiB |
| 576acc7+dirty | 3.19 MiB | 4.32 MiB | 1.13 MiB |
| 922b103+dirty | 3.19 MiB | 4.32 MiB | 1.13 MiB |
| ec9549b+dirty | 3.19 MiB | 4.32 MiB | 1.13 MiB |
Base automatically changed from
fix-attach-app-start-to-the-first-started-root-span
to
main
March 4, 2025 17:57
Android (legacy) Performance metrics 🚀
|
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| 8e80789 | 430.76 ms | 431.45 ms | 0.69 ms |
| cdf2bdf | 448.42 ms | 443.42 ms | -5.00 ms |
| d053743 | 429.96 ms | 444.20 ms | 14.24 ms |
| 946a600 | 384.53 ms | 366.65 ms | -17.88 ms |
| 5a22220 | 412.38 ms | 447.35 ms | 34.97 ms |
| c639edf | 466.48 ms | 489.57 ms | 23.09 ms |
| 5f03ae9 | 444.88 ms | 448.89 ms | 4.01 ms |
| ae7b03d | 428.82 ms | 412.33 ms | -16.49 ms |
| fe13591 | 478.92 ms | 480.84 ms | 1.92 ms |
| 60d7316 | 418.46 ms | 414.12 ms | -4.34 ms |
App size
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| 8e80789 | 17.74 MiB | 20.10 MiB | 2.36 MiB |
| cdf2bdf | 17.74 MiB | 20.10 MiB | 2.36 MiB |
| d053743 | 17.75 MiB | 20.12 MiB | 2.37 MiB |
| 946a600 | 17.74 MiB | 20.09 MiB | 2.35 MiB |
| 5a22220 | 17.73 MiB | 19.93 MiB | 2.20 MiB |
| c639edf | 17.74 MiB | 20.08 MiB | 2.34 MiB |
| 5f03ae9 | 17.75 MiB | 20.11 MiB | 2.36 MiB |
| ae7b03d | 17.75 MiB | 20.11 MiB | 2.37 MiB |
| fe13591 | 17.74 MiB | 20.07 MiB | 2.34 MiB |
| 60d7316 | 17.75 MiB | 20.11 MiB | 2.36 MiB |
Previous results on branch: add-detox-test
Startup times
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| 922b103 | 441.90 ms | 436.64 ms | -5.25 ms |
| d993785 | 399.15 ms | 415.98 ms | 16.83 ms |
| ec9549b | 397.13 ms | 417.94 ms | 20.81 ms |
| 576acc7 | 400.42 ms | 427.40 ms | 26.99 ms |
| 61ddd2e | 384.94 ms | 420.98 ms | 36.04 ms |
App size
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| 922b103 | 17.75 MiB | 20.12 MiB | 2.37 MiB |
| d993785 | 17.75 MiB | 20.12 MiB | 2.37 MiB |
| ec9549b | 17.75 MiB | 20.12 MiB | 2.37 MiB |
| 576acc7 | 17.75 MiB | 20.12 MiB | 2.37 MiB |
| 61ddd2e | 17.75 MiB | 20.12 MiB | 2.37 MiB |
krystofwoldrich
commented
Mar 5, 2025
krystofwoldrich
commented
Mar 5, 2025
krystofwoldrich
commented
Mar 5, 2025
krystofwoldrich
commented
Mar 5, 2025
krystofwoldrich
commented
Mar 5, 2025
krystofwoldrich
commented
Mar 5, 2025
samples/react-native/android/app/src/main/res/xml/network_security_config.xml
Outdated
Show resolved
Hide resolved
krystofwoldrich
commented
Mar 5, 2025
|
I reverted the network config removal commit, although locally it worked. In CI, the configuration is needed. |
antonis
approved these changes
Mar 7, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
📢 Type of change
📜 Description
This PR adds E2E tests for the sample application.
The tests are the same as introduced in #4535 and #4584
I switched from Detox to Maestro in combination with Jest because Detox uses AndroidTest.apk which starts ContentProviders in the background which breaks our app start logic which checks for background starts.
Since app starts are main goal of adding these tests we need to use tool which uses Android UIAutomator which is closer to how actual users operate the app.
At the moment the android test checks for
warmstart. This will get updated after the fix from getsentry/sentry-java#4216 is released.💚 How did you test it?
ci and running locally
📝 Checklist
sendDefaultPIIis enabled🔮 Next steps
#skip-changelog