Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: config migration and graph repo client #1889

Draft
wants to merge 6 commits into
base: main
Choose a base branch
from
Draft

feat: config migration and graph repo client #1889

wants to merge 6 commits into from

Conversation

MichaelsJP
Copy link
Member

Pull Request Checklist

  • 1. I have rebased the latest version of the main branch into my feature branch and all conflicts
    have been resolved.
  • 2. I have added information about the change/addition to functionality to the CHANGELOG.md file under the
    [Unreleased] heading.
  • 3. I have documented my code using JDocs tags.
  • 4. I have removed unnecessary commented out code, imports and System.out.println statements.
  • 5. I have written JUnit tests for any new methods/classes and ensured that they pass.
  • 6. I have created API tests for any new functionality exposed to the API.
  • 7. If changes/additions are made to the ors-config.json file, I have added these to the ors config documentation
    along with a short description of what it is for, and documented this in the Pull Request (below).
  • 8. I have built graphs with my code of the Heidelberg.osm.gz file and run the api-tests with all test passing
  • 9. I have referenced the Issue Number in the Pull Request (if the changes were from an issue).
  • 10. For new features or changes involving building of graphs, I have tested on a larger dataset
    (at least Germany), and the graphs build without problems (i.e. no out-of-memory errors).
  • 11. For new features or changes involving the graphbuilding process (i.e. changing encoders, updating the
    importer etc.), I have generated longer distance routes for the affected profiles with different options
    (avoid features, max weight etc.) and compared these with the routes of the same parameters and start/end
    points generated from the current live ORS.
    If there are differences then the reasoning for these MUST be documented in the pull request.
  • 12. I have written in the Pull Request information about the changes made including their intended usage
    and why the change was needed.
  • 13. For changes touching the API documentation, I have tested that the API playground renders correctly.

Fixes # .

Information about the changes

  • Key functionality added:
  • Reason for change:

Examples and reasons for differences between live ORS routes, and those generated from this pull request

Required changes to ors config (if applicable)

takb and others added 4 commits November 8, 2024 17:30
@takb @MichaelsJP
refactor(config)!: Refactor the config classes into a separate struct…
6035301 
…ure into ors-engine

- The config is now entirely encapsulated in a clear class hierarchy that can be imported by any module needed. Ors-api imports those classes and is able to use them directly with the externalized spring-boot config loader functionality.
- The config classes are using jackson to deserialize yaml files. Since its the basic jackson logic, theoretically ors should understand all possible file formats jackson can handle (yml,json etc.). But we assume yml only.
- All formerly bare string accesses to config parameters are now clear function and method accesses from the config classes.
- The config class engine has a new parameter graphManagement for a new graph repo client which will be added in subsequent commits
- The config classes are currently not designed for serialization except the graph management configs.
- Some obvious generic getter/setter scenarios have now been implemented or replaced with the corresponding lombok imports.
- ProfileProperties and ProfileDefaultProperties have been merged. ProfileProperties has new parameters `build`, `repo`, and `service`. Except repo the other two include no new parameters, just moved ones. Repo is new and is part of the coming graph management feature.

Co-authored-by: jhaeu <[email protected]>
Co-authored-by: Julian Psotta <[email protected]>

BREAKING CHANGE: Structural changes in the configuration make this change incompatible with earlier versions and configurations.
@jhaeu @takb @MichaelsJP
feat(graph-repo-client): Download new routing graphs from a remote re…
35d6067 
…pository

- The feature allows ors to download graphs from a pre-defined remote repository. The ors client looks for specific things like graph version, the used profiles, area of interest etc. to decide which graph should be downloaded.
- To differentiate incompatible graphs, these changes are accommodated by a manually defined graph-version. Once a version is different, the graphs are incompatible.
- While building new graphs, the folder of the graphs will include YAML files that contain the settings the graphs were built with. This ensures that graphs cannot be loaded with different graph building relevant configs.
- Downloaded graphs will always contain these files.
- Graph download triggers can be configure by schedule as well as its activation.

Co-authored-by: Takara Baumbach <[email protected]>
Co-authored-by: Julian Psotta <[email protected]>
@MichaelsJP @jhaeu @takb
test(integration-tests): Replace the old bash based integration tests…
45750a3 
… with tests based on junit and TestContainers

The focus of these tests is not to check for correct API tests but for the correct interaction with external tools and dependencies:

- Config environment tests
- Config file tests
- Config lookup tests
- GeoTools temp folder creation test
- GraphRepoTests that mock an external graph repo with different scenarios

The new junit based integration tests are executed like any other junit tests. Under the hood they use test containers to spawn containers with docker to provide isolated and clear defined environments.

All tests run in the same way for all current relevant execution scenarios, maven/spring-boot, war/tomcat, java/jar and test for the same expected behavior.

All tests are executed in parallel. The max parallel tests are configured by `junit-platform.properties`. Before each parallel execution the container layers are built with a single thread to ensure that parallel tests reuse the same image layers for the containers. This leads to a massively reduced build time in general. Graphs are pre-built by the same layer caching process before the individual test run and shared among tests that request shared graphs.
Some tests require new graphs on the fly.

Co-authored-by: jhaeu <[email protected]>
Co-authored-by: Takara Baumbach <[email protected]>
@jhaeu @takb @MichaelsJP
docs: Documentation for the config class migration and graph repo client
52e87ae 
The documentation now includes information for the new configuration of ors. The configuration is now differently structured and thus content moved a lot.

Additionally, the graph download functionality is described.

Co-authored-by: Takara Baumbach <[email protected]>
@MichaelsJP MichaelsJP changed the title Feat/graph repo client feat: graph repo client Nov 8, 2024
@MichaelsJP MichaelsJP changed the title feat: graph repo client feat: config migration and graph repo client Nov 8, 2024
@github-actions github-actions bot added feature and removed feature labels Nov 8, 2024
This was referenced Nov 8, 2024
Closed
Closed
@MichaelsJP MichaelsJP linked an issue Nov 8, 2024 that may be closed by this pull request
Adapt workflows #1892
Open
@MichaelsJP MichaelsJP removed a link to an issue Nov 8, 2024
Adapt workflows #1892
Open
jhaeu and others added 2 commits November 12, 2024 15:11
@jhaeu
test: compare dates instead of date.toString()
9f4a76e 
the latter is flaky because the format depends on local locale, time zone etc.
@MichaelsJP
test: compare dates instead of date.toString() (#1896)
b9439d1
@sonarcloud SonarCloud
Copy link

sonarcloud bot commented Nov 12, 2024

Quality Gate Failed Quality Gate failed

Failed conditions
E Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

@sfendrich
Copy link
Contributor

The commit message title of commit 6035301 is irritating. A refactoring is not supposed to be a breaking change.

@TheGreatRefrigerator
Copy link
Contributor

The commit message title of commit 6035301 is irritating. A refactoring is not supposed to be a breaking change.

The ! indicates a breaking change & the body describes the thing that is breaking:

BREAKING CHANGE: Structural changes in the configuration make this change incompatible with earlier versions and configurations.

according to the specification we are currently using, that's how a breaking change should be committed and it can be on any commit type:
https://www.conventionalcommits.org/en/v1.0.0/#examples

Or do you think the actual title confusing?
refactor the config classes into a separate structure into ors-engine
Would you suggest a different one?

@sfendrich
Copy link
Contributor

No, I think that in general "refactor" and "BREAKING CHANGE" are incompatible by their definition. I wonder whether there are rare exceptions to this?
As the commit changes the behavior it is either a fix or a feat.

@TheGreatRefrigerator
Copy link
Contributor

ah due to refactoring being defined by "not altering existing functionality" 😓. Yeah, agreed.
How about:
feat(config)!: Move the config classes into a separate structure into ors-engine

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@MichaelsJP @sfendrich @TheGreatRefrigerator @takb @jhaeu