Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test(serialize): improved benchmarking #142

Open
wants to merge 34 commits into
base: main
Choose a base branch
from
Open

test(serialize): improved benchmarking #142

wants to merge 34 commits into from

Conversation

zsilbi
Copy link
Collaborator

@zsilbi zsilbi commented Mar 4, 2025

I created a small util to automatically download specific versions from GitHub to compare with each other and the current version.

import { bench } from "vitest";
import { getVersions } from "./fixtures/utils/versions";

// Options for v1
const hashOptions = { unorderedArrays: true, unorderedSets: true };
const versions = await getVersions(["v1.1.5", "v2.0.4", "v2.0.11"]);
for (const version of versions) { 
  bench(version.name, () => {
    version.serialize("hello world", hashOptions); 
  });
}

Added custom benchmarks.
Added the same benchmarks for bun using mitata.
Added bench:bun command to run the bun benchmark.

@pi0
Copy link
Member

pi0 commented Mar 5, 2025

Tip: we can use "ohash-v1": "npm:ohash@^1" in devDependencies to have same effect.

@zsilbi
Copy link
Collaborator Author

zsilbi commented Mar 5, 2025

Tip: we can use "ohash-v1": "npm:ohash@^1" in devDependencies to have same effect.

This allows me to use any branch name, commit hash or version tag to benchmark with the current dev version (and against each other).
If I only wanted to bench dev with v1 that would have been an easier solution of course 👍🏻

For example:

export const benchConfig: BenchConfig = {
  versions: [
    "v2.0.11",
    "7a52c4ac82c33396c2e2ea90e2896ccf3a03256b",
    "main",
  ],
  ...
}
benchmark                   avg (min … max) p75 / p99    (min … top 1%)
------------------------------------------- -------------------------------
• count:1, size:small
------------------------------------------- -------------------------------
ohash @ v2.0.11              497.56 ns/iter 477.08 ns ▆█
                    (457.46 ns … 905.65 ns) 841.28 ns ██
                    (  0.00  b … 726.00  b)  12.63  b ██▃▁▁▁▁▁▁▁▁▁▁▁▁▁▂▁▁▂▁

ohash @ 7a52c4a              453.85 ns/iter 431.23 ns █▅
                    (415.98 ns … 825.99 ns) 787.21 ns ██
                    (  0.00  b …   1.03 kb)  31.17  b ██▂▂▁▁▁▁▁▁▁▁▂▁▂▁▁▁▂▁▁

ohash @ main                 475.14 ns/iter 448.85 ns 
                      (431.98 ns … 1.01 µs) 851.39 ns ██
                    (  0.00  b …   1.29 kb)  36.44  b ██▁▁▁▂▁▁▁▁▁▁▂▂▁▁▁▁▁▁▁

ohash @ dev                  406.66 ns/iter 386.36 ns █▃
                    (370.59 ns … 962.38 ns) 769.31 ns ██
                    (  0.00  b …   1.10 kb)  29.73  b ██▂▁▁▁▁▁▁▁▁▁▁▂▁▁▁▁▁▁▁

summary
  ohash @ dev
   1.12x faster than ohash @ 7a52c4a
   1.17x faster than ohash @ main
   1.22x faster than ohash @ v2.0.11

@zsilbi
Copy link
Collaborator Author

zsilbi commented Mar 9, 2025

I decided to remove vitest benchmarks for serialize and use only mitata for both Bun and node for easier maintainabilty.
Added jiti as dev. dep. to use mitata with node.
Added the option to use arguments to turn on specific benchmarks (--presets (default) / --combined / --custom / --full).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants