Can we reproduce performance result by single machine?

[eecheng87]

Hi, all

Recently, I try to give a primitive performance evaluation by comparing the throughput of native Redis and Redis with dragonfly. I found I can't reproduce the performance improvement described in the README. I run both memtier_benchmark and server in the same host, here is how I launch dragonfly:

./dragonfly --port 8888

Memtier_benchmark is configured with:

./memtier_benchmark -p 8888 --protocol=redis --clients=16 --threads=16 --ratio=1:9 --distinct-client-seed -d 256 --test-time=10 --hide-histogram --pipeline=30

Performance result:

Writing results to stdout
[RUN #1] Preparing benchmark client...
[RUN #1] Launching threads now...
[RUN #1 101%,  10 secs]  0 threads:     1084560 ops,  107956 (avg:  107808) ops/sec, 7.08MB/sec (avg: 6.95MB/sec), 71.25 (avg: 71.17) msec latencyy

16        Threads
16        Connections per thread
10        Seconds


ALL STATS
============================================================================================================================
Type         Ops/sec     Hits/sec   Misses/sec    Avg. Latency     p50 Latency     p99 Latency   p99.9 Latency       KB/sec
----------------------------------------------------------------------------------------------------------------------------
Sets        10783.62          ---          ---        71.17169        61.43900       239.61500       485.37500      3200.22
Gets        97052.55       526.97     96525.58        71.17169        61.43900       239.61500       485.37500      3913.85
Waits           0.00          ---          ---             ---             ---             ---             ---          ---
Totals     107836.17       526.97     96525.58        71.17169        61.43900       239.61500       485.37500      7114.07

Then, I test native Redis (default configuration) with same memtier_benchmark configuration, the result is:

Writing results to stdout
[RUN #1] Preparing benchmark client...
[RUN #1] Launching threads now...
[RUN #1 100%,  10 secs]  0 threads:     3752040 ops,  390469 (avg:  375020) ops/sec, 28.02MB/sec (avg: 26.90MB/sec), 19.66 (avg: 20.45) msec latency

16        Threads
16        Connections per thread
10        Seconds


ALL STATS
============================================================================================================================
Type         Ops/sec     Hits/sec   Misses/sec    Avg. Latency     p50 Latency     p99 Latency   p99.9 Latency       KB/sec
----------------------------------------------------------------------------------------------------------------------------
Sets        37511.11          ---          ---        20.44633        18.17500        38.65500        58.36700     11132.06
Gets       337599.98     12906.10    324693.87        20.44633        18.17500        38.65500        58.36700     16415.19
Waits           0.00          ---          ---             ---             ---             ---             ---          ---
Totals     375111.08     12906.10    324693.87        20.44633        18.17500        38.65500        58.36700     27547.24

I guess I missing some detail to lead the degradation after using dragonfly, but I totally have no idea. Any suggestion will be appreciated. Thanks in advance!

I'm using Redis 6.9.241 (1765d601/1) and memtier_benchmark 1.4.0. Both of them are running on Marvell ThunderX2 CN9975 powered R281-T91 Arm server.

[romange]

@eecheng87 how can I get an access for a 224 core ARM64 machine? :)

Anyway, try running:
./dragonfly --alsologtostderr --proactor_threads=16 --port=8888

and send here the console output or attach /tmp/dragonfly.INFO after the memtier run.

[eecheng87]

Hi, here is the output:

eecheng@arm-server ~/d/build-opt (main)> ./dragonfly --alsologtostderr --proactor_threads=16 --port=8888
I20221016 19:26:32.500291 301462 init.cc:56] ./dragonfly running in opt mode.
I20221016 19:26:32.500926 301462 dfly_main.cc:271] Starting dragonfly df-dev-0000000
I20221016 19:26:32.501008 301462 dfly_main.cc:294] maxmemory has not been specified. Deciding myself....
I20221016 19:26:32.501216 301462 dfly_main.cc:299] Found 122.83GiB available memory. Setting maxmemory to 98.27GiB
I20221016 19:26:32.532685 301477 proactor.cc:418] IORing with 1024 entries, allocated 102720 bytes, cq_entries is 2048
I20221016 19:26:32.557375 301462 proactor_pool.cc:66] Running 16 io threads
I20221016 19:26:32.583112 301462 server_family.cc:383] Data directory is "/home/eecheng/dragonfly/build-opt"
I20221016 19:26:32.583348 301462 server_family.cc:125] Checking "/home/eecheng/dragonfly/build-opt/dump"
I20221016 19:26:32.583577 301465 listener_interface.cc:87] sock[52] AcceptServer - listening on port 8888
I20221016 19:26:50.347461 301476 fiber_sched_algo.cc:112] Took 53 ms since it woke and till became active 4051780000:
I20221016 19:26:50.428754 301472 fiber_sched_algo.cc:112] Took 61 ms since it woke and till became active 4127277472:DispatchFiber
I20221016 19:26:50.428756 301470 fiber_sched_algo.cc:112] Took 62 ms since it woke and till became active 2592161808:ioloop
I20221016 19:26:50.428822 301472 fiber_sched_algo.cc:112] Took 61 ms since it woke and till became active 4127244608:shard_queue9
I20221016 19:26:50.428922 301470 fiber_sched_algo.cc:112] Took 62 ms since it woke and till became active 4035002784:DispatchFiber
I20221016 19:26:50.428929 301478 fiber_sched_algo.cc:112] Took 134 ms since it woke and till became active 4076912992:
I20221016 19:26:50.482551 301469 fiber_sched_algo.cc:112] Took 53 ms since it woke and till became active 4018241984:DflyConnection
I20221016 19:26:50.482623 301473 fiber_sched_algo.cc:112] Took 53 ms since it woke and till became active 2541830160:ioloop
I20221016 19:26:50.482681 301472 fiber_sched_algo.cc:112] Took 53 ms since it woke and till became active 4127277472:DispatchFiber
I20221016 19:26:50.428862 301477 fiber_sched_algo.cc:112] Took 61 ms since it woke and till became active 4135666080:
I20221016 19:26:50.484376 301464 fiber_sched_algo.cc:112] Took 55 ms since it woke and till became active 4118905280:DflyConnection
I20221016 19:26:50.484436 301469 fiber_sched_algo.cc:112] Took 55 ms since it woke and till became active 2600550416:ioloop
I20221016 19:26:50.482532 301476 fiber_sched_algo.cc:112] Took 115 ms since it woke and till became active 4051763616:DflyConnection
I20221016 19:26:50.428773 301475 fiber_sched_algo.cc:112] Took 30 ms since it woke and till became active 4102095264:DflyConnection
I20221016 19:26:50.484319 301467 fiber_sched_algo.cc:112] Took 55 ms since it woke and till became active 4026581312:shard_queue4
I20221016 19:26:50.485011 301476 fiber_sched_algo.cc:112] Took 56 ms since it woke and till became active 4051796416:DflyConnection
I20221016 19:26:50.484838 301477 fiber_sched_algo.cc:112] Took 117 ms since it woke and till became active 4135682496:
I20221016 19:26:50.485077 301476 fiber_sched_algo.cc:112] Took 56 ms since it woke and till became active 4051812800:
I20221016 19:26:50.485122 301477 fiber_sched_algo.cc:112] Took 118 ms since it woke and till became active 2550218768:ioloop
I20221016 19:26:50.485050 301467 fiber_sched_algo.cc:112] Took 56 ms since it woke and till became active 4026614176:DispatchFiber
I20221016 19:26:50.484987 301475 fiber_sched_algo.cc:112] Took 86 ms since it woke and till became active 2533441552:ioloop
I20221016 19:26:50.484840 301469 fiber_sched_algo.cc:112] Took 55 ms since it woke and till became active 4018225568:DispatchFiber
I20221016 19:26:50.485266 301477 fiber_sched_algo.cc:112] Took 118 ms since it woke and till became active 4135633216:shard_queue14
I20221016 19:26:50.484392 301470 fiber_sched_algo.cc:112] Took 55 ms since it woke and till became active 4035019200:DflyConnection
I20221016 19:26:50.485311 301475 fiber_sched_algo.cc:112] Took 86 ms since it woke and till became active 4102078784:shard_queue12
I20221016 19:26:50.485471 301470 fiber_sched_algo.cc:112] Took 56 ms since it woke and till became active 4035035584:
I20221016 19:26:50.485503 301475 fiber_sched_algo.cc:112] Took 56 ms since it woke and till became active 4102128064:DflyConnection
I20221016 19:26:51.462329 301474 fiber_sched_algo.cc:112] Took 41 ms since it woke and till became active 4010213856:DflyConnection
I20221016 19:26:51.462476 301474 fiber_sched_algo.cc:112] Took 41 ms since it woke and till became active 4010230240:
I20221016 19:26:51.462599 301463 fiber_sched_algo.cc:112] Took 42 ms since it woke and till became active 4060578272:DflyConnection
I20221016 19:26:51.462704 301463 fiber_sched_algo.cc:112] Took 42 ms since it woke and till became active 4060135744:shard_queue0
I20221016 19:26:51.463116 301465 fiber_sched_algo.cc:112] Took 213 ms since it woke and till became active 4068901344:DflyConnection
I20221016 19:26:51.463176 301465 fiber_sched_algo.cc:112] Took 213 ms since it woke and till became active 2525052944:ioloop

BTW, 224 is the number of logic cores, it actually has 28 physical cores per node.

[romange]

There is definitely a problem with how Dragonfly interacts with this hardware. Based on the output, it seems there is a CPU starvation in Dragonfly threads. Unfortunately, I can not tell why it happens without being able to access this hardware. I have never used a system that has 8 logical cpus on a single physical core.

[eecheng87]

Thanks for your reply! Let me find another environment to test it.

[romange]

If your company is considering using Dragonfly, you may check out our Early Access Program https://dragonflydb.io/platform/
where we can provide necessary consultation on how to run Dragonfly to better use your hardware system.