feat(Project): New configuration make project and releases relationship more flexible for dependencies

[hoangnt2]

How to enable this feature

To use this function, please:

1. Build the source code and deploy.

2. Add config enable.flexible.project.release.relationship=true (/etc/sw360/sw360.properties) to enable the feature.

The following changes will work when enable.flexible.project.release.relationship=true only.

3. Use the migration script (056_migrate_project_dependency_network.py) we provided to mograte the database.

Before you run the script, please change two places in the script:

(1) Line 30: DRY_RUN = True -> DRY_RUN = False

(2) Line 32: COUCHSERVER = 'http://localhost:5984/' -> COUCHSERVER = 'http://admin:password@localhost:5984/'

admin and password should be your username and password for CouchDB.

The important information

1. This feature has two modes - "ON" and "OFF". If the user doesn't enable this feature, everything will be the same as before (at the source code level).

2. For the "ON" mode, We add three new APIs and modify an old API to help users modify the dependency network of a project. But other old APIs will work as before. Users may not notice the change if they only use the old APIs. So all clients or tools working with sw360 by APIs will work as before as well. For example, our new API 3.3.36 (Creating a project with dependencies network) will allow the user to create a new project with a dependency network, but the user can still use the old API 3.3.20 (Creating a project) to register a project with some releases (i.e. direct dependencies) linked to itself.

3. For the "OFF" mode, we add a new field "releaseRelationNetwork" in the database to store the dependency network. But we still keep the old filed "releaseIdToUsage" which stores the "Linked Release". What's more, we always sync these two fields. So when the user switches back to "ON" mode, no data will be lost.

1. Introduction

This manual aims to introduce the Dependency Network feature. Refer issue: #1538
The dependency network feature is a new function to make the dependency management of a project more flexible by allowing the users to customize the dependency graphs of their projects.
To help the reviewers test and review this new feature more quickly and easily, Toshiba makes this manual.

2. How to test?

This feature modify the GUI of the “Linked Releases And Projects” on the “project edits” page.
Now the “Linked Releases” table could show all dependencies of a project (both direct and transitive ones). Users can modify these dependencies as well.

2.1. The changes of edit project GUI

In this section, we will introduce the changes in GUI behaviors. We modified or added 5 sub-functions below:

a. Modify the “Add Releases” button: This button will add a direct dependency (release) in the dependency graph of this project.

b. A new icon button to add a dependency (release) to another dependency (release) in the dependency graph. Note that this dependency added is seen as the transitive dependency of this project.

c. A new icon button to load the default dependency graph of a dependency (release) by importing the dependency information stored on the component page. Note that this button will load all dependencies (both direct and transitive ones) of the corresponding dependency (release).

d. The combo box allows the user to modify the version of a dependency.

e. The “Check Dependency Network” button will compare and show the different dependency information which is not consistent with the default one stored on the component page by highlighting them. The inconsistency usually happens after users modified the dependency graph or imported an old project.

2.2. The changes of view detail project GUI

The license Clearing tab will work the same as before. But it will show the data from the dependency network we modified on the edit page.

2.3 Rest API changes

New Rest APIs

a. 3.3.35. Get a single project with dependencies network

The response will include the dependencyNetwork field(It will show the dependency network of project (direct and indirect releases)):

{
  "name" : "Emerald Web",
  "dependencyNetwork": [
      {
          "releaseId": "9efc5766cd0c41d4a40547b99f5b91ac",
          "releaseLink": [
              {
                  "releaseId": "3bed97a1c7ac4c32846ef4be985b648c",
                  "releaseLink": [
                      {
                          "releaseId": "6a8250852362462095c57535294039e4",
                          "releaseLink": [],
                          "releaseRelationship": "TO_BE_REPLACED",
                          "mainlineState": "PHASEOUT",
                          "comment": "Test Comment",
                          "createOn": "2023-05-15",
                          "createBy": "[email protected]"
                      }
                  ],
                  "releaseRelationship": "INTERNAL_USE",
                  "mainlineState": "OPEN",
                  "comment": "Test Comment",
                  "createOn": "2023-05-15",
                  "createBy": "[email protected]"
              }
          ],
          "releaseRelationship": "STATICALLY_LINKED",
          "mainlineState": "MAINLINE",
          "comment": "Test Comment",
          "createOn": "2023-05-15",
          "createBy": "[email protected]"
      },
      {
          "releaseId": "f1d860e7576a44798ee3daff57a3a886",
          "releaseLink": [],
          "releaseRelationship": "OPTIONAL",
          "mainlineState": "OPEN",
          "comment": "Test Comment",
          "createOn": "2023-05-15",
          "createBy": "[email protected]"
      }
  ]
}

b. 3.3.36. Creating a project with dependencies network

If the dependencyNetwork field is included in the request body, a dependency network will be registered for the project.

• Simple example request (modify releaseIds to the existing release ids in sw360):

{
    "name": "TestProject1",
    "dependencyNetwork": [
        {
            "releaseId": "9efc5766cd0c41d4a40547b99f5b91ac",
            "releaseLink": [
                {
                    "releaseId": "3bed97a1c7ac4c32846ef4be985b648c",
                    "releaseLink": [
                    ],
                    "releaseRelationship": "INTERNAL_USE",
                    "mainlineState": "OPEN",
                    "comment": "Test Comment",
                    "createOn": "2023-05-15",
                    "createBy": "[email protected]"
                }
            ],
            "releaseRelationship": "STATICALLY_LINKED",
            "mainlineState": "MAINLINE",
            "comment": "Test Comment",
            "createOn": "2023-05-15",
            "createBy": "[email protected]"
        },
        {
            "releaseId": "f1d860e7576a44798ee3daff57a3a886",
            "releaseLink": [],
            "releaseRelationship": "OPTIONAL",
            "mainlineState": "OPEN",
            "comment": "Test Comment",
            "createOn": "2023-05-15",
            "createBy": "[email protected]"
        }
    ]
}

c. 3.3.37. Update a project with dependencies network

Same request body as "Creating a project with dependencies network".

Modify Rest APIs

a. 3.5.18. Delete a release

Cannot delete release if it is existing in the dependency network of any project

2.4 The change in database and migrate data

• Created new field in project to save the dependency network: releaseRelationNetwork
• Create 056_migrate_project_dependency_network.py to migrate data from releaseIdToUsage to releaseRelationNetwork

2.5 Test this feature with component.visibility.restriction.enabled option (refer #1357)

• Case 1: component.visibility.restriction.enabled=true
• Case 2: component.visibility.restriction.enabled=false

3. How to review?

This chapter introduces which file is modified to achieve the corresponding function.

3.1. Rest API

Table 1: RestAPI functions

Rest API function	File Path	Function name
3.3.34. Get a single project with dependencies network	rest/resource-server/src/main/java/org/eclipse/sw360/rest/resourceserver/project/ProjectController.java	getProjectWithNetwork()
3.3.35. Creating a project with dependencies network	rest/resource-server/src/main/java/org/eclipse/sw360/rest/resourceserver/project/ProjectController.java	createProjectWithNetwork()
3.3.36. Update a project with dependencies network	rest/resource-server/src/main/java/org/eclipse/sw360/rest/resourceserver/project/ProjectController.java	patchProjectWithNetwork()
3.5.18. Delete a release	rest/resource-server/src/main/java/org/eclipse/sw360/rest/resourceserver/release/Sw360ReleaseService.java	deleteRelease()
3.3.20. Creating a project	rest/resource-server/src/main/java/org/eclipse/sw360/rest/resourceserver/release/Sw360ReleaseService.java	createProject()
3.3.22. Update a project	rest/resource-server/src/main/java/org/eclipse/sw360/rest/resourceserver/release/Sw360ReleaseService.java	patchProject()

3.2. GUI

Table 2: UI functions

GUI function	File Path
Project edit page (Dependency network tab)	- frontend/sw360-portlet/src/main/resources/META-INF/resources/html/projects/edit.jsp - frontend/sw360-portlet/src/main/resources/META-INF/resources/html/utils/includes/editLinkedReleasesInNetwork.jspf - frontend/sw360-portlet/src/main/java/org/eclipse/sw360/portal/portlets/LinkedReleasesAndProjectsAwarePortlet.java - frontend/sw360-portlet/src/main/java/org/eclipse/sw360/portal/portlets/projects/ProjectPortlet.java
Project detail page (Dependency network tab)	- frontend/sw360-portlet/src/main/resources/META-INF/resources/html/projects/includes/detailOverview.jspf - frontend/sw360-portlet/src/main/resources/META-INF/resources/html/projects/includes/projects/dependencyNetwork.jsp - frontend/sw360-portlet/src/main/java/org/eclipse/sw360/portal/portlets/projects/ProjectPortlet.java
- Components detail page: + Used By Project table	frontend/sw360-portlet/src/main/java/org/eclipse/sw360/portal/portlets/components/ComponentPortlet.java
- Releases detail page: + Used By Project table	frontend/sw360-portlet/src/main/java/org/eclipse/sw360/portal/portlets/components/ComponentPortlet.java

Reference documents:
PR.for.Dependency.network.feature.docx

A.New.Dependency.Management.Function.for.SW360_qiu.pdf

[ag4ums]

@hoangnt2 @KoukiHama "Dependency Network" tab replace existing "License Clearing” tab it was not porposed in this way, can we please discuss this.
from the observations :

• it will break all existing clients.
• not seeing an option to view only the direct dependencies separately.

[KoukiHama]

@shi9qiu @tienlee
Please follow Anupam comments.

[tienlee]

Thanks for your comments. I will discuss with Qiu and give feedback you tomorrow.

[KoukiHama]

@shi9qiu any update?

[shi9qiu]

Hi @ag4ums ,

Thank you for your comments.

I have discussed this with Tien-san, please allow me to discuss more about this issue here.

We proposed this function with two modes - ON and OFF. Changes only happen when the user enables this function. If not, everything will be the same as before (at the source code level).

I would like to begin with the second concern - "not seeing an option to view only the direct dependencies separately". It's easy for us to develop a new feature to show the direct dependencies only based on the developed one. For example, hide the transitive dependencies when the user clicks a button. So this would not be a problem.

But we would like to discuss more on the first concern - "it will break all existing clients". Actually, besides the four APIs we modified and the dependency part in GUI, all other APIs and functions can be used as before. To achieve this, we add a new field (releaseRelationNetwork) to store the dependency network (direct dependencies and transitive dependencies) in the project and keep it in sync with the releaseIdToUsage field. The releaseIdToUsage field is not replaced by the new one and still be there. That's also the reason why other functions will not be affected. Besides, We also create a migration script to convert the data from the releaseIdToUsage field to the releaseRelationNetwork field. It will help users to switch between two modes.

I think maybe we use a bad name "Dependency Network" for this new tag. It looks like we drop the "License Clearing" tab and the function in this tab. But actually, the two tabs are almost the same. All functions are kept as before. The only difference is the new tab can show and edit the transitive dependency. So maybe we can still use the name "License Clearing", and the user may even not notice the change.

I don't know if my reply answers your question and resolves your concerns. Maybe we misunderstand what you mean by "existing clients". But we are so glad to hear more from you and waiting to solve anything you are worrying about.

Thank you!

[ag4ums]

@shi9qiu

We proposed this function with two modes - ON and OFF. Changes only happen when the user enables this function. If not, everything will be the same as before (at the source code level).
ON/OFF mode is good, so user can choose.

I would like to begin with the second concern - "not seeing an option to view only the direct dependencies separately". It's easy for us to develop a new feature to show the direct dependencies only based on the developed one. For example, hide the transitive dependencies when the user clicks a button. So this would not be a problem.
I think this is an important option to keep UI consistent.

But we would like to discuss more on the first concern - "it will break all existing clients". Actually, besides the four APIs we modified and the dependency part in GUI, all other APIs and functions can be used as before. To achieve this, we add a new field (releaseRelationNetwork) to store the dependency network (direct dependencies and transitive dependencies) in the project and keep it in sync with the releaseIdToUsage field. The releaseIdToUsage field is not replaced by the new one and still be there. That's also the reason why other functions will not be affected. Besides, We also create a migration script to convert the data from the releaseIdToUsage field to the releaseRelationNetwork field. It will help users to switch between two modes.
let me try to explain it, I tried to used ort analysis to create project and component, yet to use other rest client/s like capycli, and found that the projects and releases are getting created but releases are not linked to projects, as a result they don’t appear in the UI - Dependency Network tab. guessing other clients will also show the same behavior. tried to migrate the old project using the script but for migrated projects 'Dependency Network' tab is empty. (guessing migration issue)

I think maybe we use a bad name "Dependency Network" for this new tag. It looks like we drop the "License Clearing" tab and the function in this tab. But actually, the two tabs are almost the same. All functions are kept as before. The only difference is the new tab can show and edit the transitive dependency. So maybe we can still use the name "License Clearing", and the user may even not notice the change.
on the naming I think 'License clearing' is more meaning full here as this tab primarily helps with the 'license clearing' activity, Depenendy Netwok' naming somehow feels incomplete as this is does not give you a complete network of the dependencies but only dependencies for a project.... but thats not my major concern

So, coming back to the first point that enable this option works well, that’s good, so user can choose to enable it but I am afraid once user migrated to this new option to use dependency network functionality will they be able to switchback.... though not sure if the switching back is required, but what worries me.... if this feature is breaking the existing rest-client/s, and if someone do changes at the client side to support 'Dependency Network', that should not break the current flow(as I see the otherway its breaking now) .
Let me know your opinion... how we can resolve this.

[shi9qiu]

Hi @ag4ums ,

Thank you very much for your kind explanation. I understand your concern.

Please give me some time, I would like to investigate this issue and discuss with @tienlee to consider how to resolve this. I will share our idea with you as soon as we get it. Thanks!

[shi9qiu]

Hi @ag4ums ,

I have discussed these issues with @tienlee and @hoangnt2 , and got some ideas. Please let me share them with you here.

1. tried to migrate the old project using the script but for migrated projects 'Dependency Network' tab is empty. (guessing migration issue)
   For the bug of the migration script (the migrated projects' 'Dependency Network' tab is empty), please set "DRY_RUN" as "False" in line 30 (DRY_RUN = False) and try it again. For the DRY_RUN mode, the dependency network will not be registered. We will also try to test this script more.

2. I think this is an important option to keep UI consistent.
   About how to show the direct dependencies only, now we can do it by clicking the "Collapse all" to show direct dependency only at the "License Clearing" tab. And by default, when the user goes to the "License Clearing" tab, only direct releases are shown. But if this can't satisfy your need, we will add a sub-function (maybe a button) to switch between "showing all dependencies" and "showing direct dependencies only". Please let me know if this way is better.

3. let me try to explain it, I tried to used ort analysis to create project and component, yet to use other rest client/s like capycli, and found that the projects and releases are getting created but releases are not linked to projects, as a result they don’t appear in the UI - Dependency Network tab. guessing other clients will also show the same behavior.
   We have investigated some clients and found they are using the API to collaborate with SW360. We updated four APIs and changed the body of the request. That's the reason why the clients can't work as before.
   To resolve this issue, we plan to keep the four APIs the same as before so all clients (also the users) can still use these APIs to register or update the project. We will add some new APIs to modify the dependency network. This will ensure the clients can work as before. If you like this idea, we can begin to implement this at once.

4. I am afraid once user migrated to this new option to use dependency network functionality will they be able to switchback.... though not sure if the switching back is required, but what worries me.... if this feature is breaking the existing rest-client/s, and if someone do changes at the client side to support 'Dependency Network', that should not break the current flow(as I see the otherway its breaking now) .
   About how to switch back, the only thing the user needs to do is switch the mode in the setting file. That's because the feature always syncs the new added field for dependency network (releaseRelationNetwork) and the field before (releaseIdToUsage) in DB. We don't drop the "releaseIdToUsage" field so user don't need to do any migration. But if user want to switch to "ON" again, he still need to run the migration script again (we will test the script in this case again).

What do you think about these ideas? If you have any questions or concerns, please feel free to contact us anytime.

Thank you!

[ag4ums]

@shi9qiu, @tienlee, @hoangnt2 if you take care of the 3rd point about the API then it should be fine.

[shi9qiu]

Hi, @ag4ums

We have finished a new version and pushed the source code to this PR. This new version is developed based on what we discussed above. Could you help to review it again?

The biggest change of this new version is that we don't change any existing APIs so they can be used as before for the "ON" mode. Instead, We added 3 new APIs related to the dependency network. We also updated the description of this PR to explain this.

If you have any questions or concerns, please feel free to contact us anytime!

[KoukiHama]

we plan to add some test cases for specific options in sw360 properties in 1st Comment for clarify more specifications and movements.

@tienlee @hoangnt2

[hoangnt2]

@GMishx, thanks for your comments. I updated source code following your comments

[smrutis1]

Kindly look into some minor comments

[hoangnt2]

@smrutis1, I updated source code following your comments, please help me to check them

[KoukiHama]

Thank you @smrutis1 for review codes
Now we can merge this PR.