Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PoC Syncing nimbus EL from Portal network #3124

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from
Draft

PoC Syncing nimbus EL from Portal network #3124

wants to merge 2 commits into from

Conversation

kdeme
Copy link
Contributor

@kdeme kdeme commented Mar 6, 2025 •
edited
Loading

Improved version of #2910 where we no longer download and process blocks per 8192 but instead have a more streaming approach of downloading and processing blocks which should allow for more consistent bandwidth usage (well, not fully as it still relies on synchronous block processing).

kdeme added 2 commits March 6, 2025 13:55
@kdeme
PoC Syncing nimbus EL from Portal network
d870f14 
To be heavily cleaned up, very quick version to make a streaming
version work instead of the previous version which would do batches
of 8192.
@kdeme
Bit of cleanup
ec65b67
@kdeme kdeme mentioned this pull request Mar 6, 2025
Closed
@kdeme
Copy link
Contributor Author

kdeme commented Mar 6, 2025

Some stats:

First 100k blocks, including processing:

INF 2025-03-06 14:47:50.017+01:00 Start downloading blocks                   startBlock=1
INF 2025-03-06 14:48:13.682+01:00 Total blocks downloaded                    totalDownloadCount=11998 totalFailureCount=5 avgBps=507.0015666385596 failureRate=0.0004167361226871145
INF 2025-03-06 14:48:13.846+01:00 Imported blocks                            blockNumber=8193 slot=1 blocks=8192 txs=0 mgas=0.000 bps=343.8 tps=0.000 mgps=0.000 avgBps=343.8 avgTps=0.000 avgMGps=0.000 elapsed=23s827ms900us
INF 2025-03-06 14:48:22.380+01:00 Total blocks downloaded                    totalDownloadCount=20274 totalFailureCount=5 avgBps=626.4536791151613 failureRate=0.0002466212883496103
INF 2025-03-06 14:48:22.538+01:00 Imported blocks                            blockNumber=16385 slot=1 blocks=16384 txs=0 mgas=0.000 bps=942.3 tps=0.000 mgps=0.000 avgBps=503.8 avgTps=0.000 avgMGps=0.000 elapsed=32s521ms721us
INF 2025-03-06 14:48:31.219+01:00 Total blocks downloaded                    totalDownloadCount=28759 totalFailureCount=5 avgBps=697.9870803740861 failureRate=0.0001738586181717028
INF 2025-03-06 14:48:31.363+01:00 Imported blocks                            blockNumber=24577 slot=1 blocks=24576 txs=0 mgas=0.000 bps=928.3 tps=0.000 mgps=0.000 avgBps=594.4 avgTps=0.000 avgMGps=0.000 elapsed=41s346ms648us
INF 2025-03-06 14:48:41.201+01:00 Total blocks downloaded                    totalDownloadCount=37393 totalFailureCount=6 avgBps=730.546825714513 failureRate=0.0001604578397026181
INF 2025-03-06 14:48:41.361+01:00 Imported blocks                            blockNumber=32769 slot=1 blocks=32768 txs=0 mgas=0.000 bps=819.4 tps=0.000 mgps=0.000 avgBps=638.2 avgTps=0.000 avgMGps=0.000 elapsed=51s343ms990us
INF 2025-03-06 14:48:50.014+01:00 History network status                     topics="portal_hist" routingTableNodes=407
INF 2025-03-06 14:48:50.014+01:00 Portal node status                         radiusPercentage=0% radius=1ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff dbSize=561kb
INF 2025-03-06 14:48:50.140+01:00 Total blocks downloaded                    totalDownloadCount=45691 totalFailureCount=6 avgBps=759.9510863925805 failureRate=0.0001313168895406097
INF 2025-03-06 14:48:50.293+01:00 Imported blocks                            blockNumber=40961 slot=1 blocks=40960 txs=0 mgas=0.000 bps=917.1 tps=0.000 mgps=0.000 avgBps=679.5 avgTps=0.000 avgMGps=0.000 elapsed=1m276ms643us
INF 2025-03-06 14:48:56.368+01:00 Total blocks downloaded                    totalDownloadCount=52759 totalFailureCount=6 avgBps=795.138439490008 failureRate=0.0001137246725677136
INF 2025-03-06 14:48:56.509+01:00 Imported blocks                            blockNumber=49153 slot=1 blocks=49152 txs=1514 mgas=33.07 bps=1317 tps=243.6 mgps=5.320 avgBps=739.2 avgTps=22.77 avgMGps=0.497 elapsed=1m6s492ms
INF 2025-03-06 14:49:18.015+01:00 Total blocks downloaded                    totalDownloadCount=60689 totalFailureCount=6 avgBps=689.6635116720527 failureRate=9.886470365305081e-05
INF 2025-03-06 14:49:18.414+01:00 Imported blocks                            blockNumber=57345 slot=1 blocks=57344 txs=5430 mgas=440.9 bps=374.0 tps=178.8 mgps=18.62 avgBps=648.7 avgTps=61.43 avgMGps=4.988 elapsed=1m28s396ms
INF 2025-03-06 14:49:25.703+01:00 Total blocks downloaded                    totalDownloadCount=68940 totalFailureCount=6 avgBps=720.4819989583567 failureRate=8.703220191470844e-05
INF 2025-03-06 14:49:25.779+01:00 Imported blocks                            blockNumber=65537 slot=1 blocks=65536 txs=8106 mgas=530.9 bps=1112 tps=363.4 mgps=12.23 avgBps=684.4 avgTps=84.65 avgMGps=5.544 elapsed=1m35s761ms
INF 2025-03-06 14:49:47.867+01:00 Total blocks downloaded                    totalDownloadCount=77194 totalFailureCount=9 avgBps=655.0178946590144 failureRate=0.0001165893722310024
INF 2025-03-06 14:49:48.204+01:00 Imported blocks                            blockNumber=73729 slot=1 blocks=73728 txs=13927 mgas=728.7 bps=365.3 tps=259.6 mgps=8.818 avgBps=623.8 avgTps=117.8 avgMGps=6.165 elapsed=1m58s187ms
INF 2025-03-06 14:49:50.014+01:00 History network status                     topics="portal_hist" routingTableNodes=415
INF 2025-03-06 14:49:50.014+01:00 Portal node status                         radiusPercentage=0% radius=1ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff dbSize=966kb
INF 2025-03-06 14:49:57.474+01:00 Total blocks downloaded                    totalDownloadCount=85481 totalFailureCount=10 avgBps=670.6652372985329 failureRate=0.0001169850610077093
INF 2025-03-06 14:49:57.794+01:00 Imported blocks                            blockNumber=81921 slot=1 blocks=81920 txs=18131 mgas=860.0 bps=854.2 tps=438.3 mgps=13.69 avgBps=641.1 avgTps=141.9 avgMGps=6.730 elapsed=2m7s777ms
INF 2025-03-06 14:50:06.297+01:00 Total blocks downloaded                    totalDownloadCount=93205 totalFailureCount=11 avgBps=683.9220158590113 failureRate=0.0001180194195590365
INF 2025-03-06 14:50:06.406+01:00 Imported blocks                            blockNumber=90113 slot=1 blocks=90112 txs=22720 mgas=972.2 bps=951.3 tps=532.9 mgps=13.04 avgBps=660.7 avgTps=166.6 avgMGps=7.128 elapsed=2m16s389ms
INF 2025-03-06 14:50:15.037+01:00 Total blocks downloaded                    totalDownloadCount=101588 totalFailureCount=11 avgBps=700.5134511154489 failureRate=0.0001082805055715242
INF 2025-03-06 14:50:15.093+01:00 Imported blocks                            blockNumber=98305 slot=1 blocks=98304 txs=26047 mgas=1056 bps=943.1 tps=383.0 mgps=9.746 avgBps=677.6 avgTps=179.5 avgMGps=7.285 elapsed=2m25s75ms

at ~13.8M range, without block processing however

INF 2025-03-06 15:08:33.633+01:00 Start downloading blocks                   startBlock=13845733
INF 2025-03-06 15:12:24.818+01:00 Total blocks downloaded                    totalDownloadCount=9816 totalFailureCount=310 avgBps=42.45971882044603 failureRate=0.03158109209453953
INF 2025-03-06 15:16:00.488+01:00 Total blocks downloaded                    totalDownloadCount=18116 totalFailureCount=679 avgBps=40.54118905104965 failureRate=0.03748068006182381
INF 2025-03-06 15:19:50.366+01:00 Total blocks downloaded                    totalDownloadCount=26616 totalFailureCount=1046 avgBps=39.33017786263833 failureRate=0.03929966937180644

With block processing it drops to 12-16bps

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@kdeme