Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use LAMBKIN for performance benchmarking #163

Closed
2 of 3 tasks
hidmic opened this issue Apr 18, 2023 · 5 comments
Closed
2 of 3 tasks

Use LAMBKIN for performance benchmarking #163

hidmic opened this issue Apr 18, 2023 · 5 comments
Assignees
Labels
enhancement New feature or request

Comments

@hidmic
Copy link
Collaborator

hidmic commented Apr 18, 2023

Description

Once we are past #85, we will need extensive performance benchmarking to show beluga_amcl is indeed as good if not better than nav2_amcl. While we could extend #117 to suite all our needs, LAMBKIN may already provide the necessary infrastructure for it -- particularly once Ekumen-OS/lambkin#35 lands.

Definition of done

  • Define benchmarks, w.r.t.:
    • Input data
    • Particle filter setup
    • Performance metrics
  • Implement benchmarks using LAMBKIN
    • LAMBKIN may require extension
  • Publish benchmark reports
@hidmic hidmic added the enhancement New feature or request label Apr 18, 2023
@nahueespinosa
Copy link
Member

Currently Beluga has been tested on developer laptops. There is no evidence to indicate that it would not work on low-spec platforms, but, as @hidmic pointed out, we should consider running benchmarks in different hardware platforms:

  • Raspberry Pi 4B
  • Jetson Nano

@hidmic
Copy link
Collaborator Author

hidmic commented Sep 6, 2023

A substantial amount of work was pursued by @serraramiro1 already. I'll push it past the finish line.

@hidmic
Copy link
Collaborator Author

hidmic commented Jul 10, 2024

We have identified a number of additional datasets we can use for this:

In addition to the Magazino dataset, the MARS dataset, the TUM dataset, and the Rawseeds dataset. CC @glpuga.

@glpuga
Copy link
Collaborator

glpuga commented Jul 16, 2024

@hidmic
Copy link
Collaborator Author

hidmic commented Oct 24, 2024

This has materialized in Ekumen-OS/lambkin#93, which will be relocated to this repository with #415 once it consolidates. Pending issues are LAMBKIN specific and we can track them there. Closing as done.

@hidmic hidmic closed this as completed Oct 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants