Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Docker PBF download with hashing #1841

Draft
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

MichaelsJP
Copy link
Member

Pull Request Checklist

  • 1. I have rebased the latest version of the main branch into my feature branch and all conflicts
    have been resolved.
  • 2. I have added information about the change/addition to functionality to the CHANGELOG.md file under the
    [Unreleased] heading.
  • 3. I have documented my code using JDocs tags.
  • 4. I have removed unnecessary commented out code, imports and System.out.println statements.
  • 5. I have written JUnit tests for any new methods/classes and ensured that they pass.
  • 6. I have created API tests for any new functionality exposed to the API.
  • 7. If changes/additions are made to the ors-config.json file, I have added these to the ors config documentation
    along with a short description of what it is for, and documented this in the Pull Request (below).
  • 8. I have built graphs with my code of the Heidelberg.osm.gz file and run the api-tests with all test passing
  • 9. I have referenced the Issue Number in the Pull Request (if the changes were from an issue).
  • 10. For new features or changes involving building of graphs, I have tested on a larger dataset
    (at least Germany), and the graphs build without problems (i.e. no out-of-memory errors).
  • 11. For new features or changes involving the graphbuilding process (i.e. changing encoders, updating the
    importer etc.), I have generated longer distance routes for the affected profiles with different options
    (avoid features, max weight etc.) and compared these with the routes of the same parameters and start/end
    points generated from the current live ORS.
    If there are differences then the reasoning for these MUST be documented in the pull request.
  • 12. I have written in the Pull Request information about the changes made including their intended usage
    and why the change was needed.
  • 13. For changes touching the API documentation, I have tested that the API playground renders correctly.

Fixes # .

Information about the changes

  • Key functionality added:
  • Reason for change:

Examples and reasons for differences between live ORS routes, and those generated from this pull request

Required changes to ors config (if applicable)

@MichaelsJP MichaelsJP changed the title Feat/docker download pbf feat: Docker PBF download with hashing Aug 28, 2024
@MichaelsJP MichaelsJP mentioned this pull request Aug 28, 2024
13 tasks
@MichaelsJP
Copy link
Member Author

MichaelsJP commented Aug 28, 2024

@meetkaushal What do you think about this implementation?
If you like, you could test it out and provide feedback in here.

Copy link

sonarcloud bot commented Aug 29, 2024

@meetkaushal
Copy link

meetkaushal commented Oct 12, 2024

Hello,

The update works when I tried locally on my laptop. However, my main issue still remains...
I am trying to run it as a service on a cloud provider render.com and they only support Dockerfile. Deploying on their lowest compute tier fails because on the first build, ors is trying to rebuild graphs and it takes a ton of resources.

Is it possible to build graphs on local docker on my laptop then SSH the graph folder to the cloud provider once ors is built there?

@MichaelsJP
Copy link
Member Author

MichaelsJP commented Oct 21, 2024

Right. I just checked the system specs for the free tier: 512 MB RAM and 0.1 CPU.
That is really not much, and we don't really have a lot of experience of running ORS with such low specs. This will involve a bit of trying out from your side.


Is it possible to build graphs on local docker on my laptop then SSH the graph folder to the cloud provider once ors is built there?

Yes, of course, you can copy graphs around how you like and also upload them with SCP or rsync to your remote machine.

Another workaround for the resource problem:
Depending on the size of your input pbf, you can also try to activate MMAP for the graph build in the render.com container. Since ors is then caching on the drive, this will take more time to build.

Here is the documentation: https://giscience.github.io/openrouteservice/run-instance/configuration/ors/engine/#ors-engine

In any case, I would advise you to switch to MMAP. Building or just running. You don't really have a choice with 512 MB of RAM.

Switch graphs_data_access to MMAP and set XMX correctly before you start. Don't make your full RAM available but leave the system some.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants