From 921260d888a1768e09b3e83747b32ae0e50be0c7 Mon Sep 17 00:00:00 2001 From: Qing Tomlinson Date: Tue, 6 Feb 2024 14:49:48 -0800 Subject: [PATCH 1/3] Add documentation --- docs/add-new-harvester.md | 66 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+) create mode 100644 docs/add-new-harvester.md diff --git a/docs/add-new-harvester.md b/docs/add-new-harvester.md new file mode 100644 index 00000000..8174beec --- /dev/null +++ b/docs/add-new-harvester.md @@ -0,0 +1,66 @@ +# Add a new harvest source + +This is a document in progress detailing the steps necessary to add a new harvest source. + +## Community Engagement +When adding a new harvest source, the points to be considered are located at https://github.com/clearlydefined/clearlydefined/blob/master/docs/adding-sources.md#adding-a-new-harvest-source. Please include these considerations and document them in your GitHub issue, similar to https://github.com/clearlydefined/service/issues/882. + +## Crawler +Need to implement: +- xxFetch, which is responsible for downloading package and registry information. +- xxExtract, which is responsible for creating a document based on the fetched information and queuing up scan tools and invoking source discovery. +- To enable the queuing of scan tools, config/map.js needs to be updated. + +Example commit: +``` +Commit: ea1618de0de4663b4d9aeecc9cdbc392edb2feba [ea1618d] +Parents: 968422b174 +Author: Nell Shamrell nells@microsoft.com +Date: October 27, 2021 4:16:41 PM +Committer: Nell Shamrell +adds support for fetching and extracting go packages to the crawler +``` + +## Service + +1. ClearlyDescribedSummarizer + +Example commit: +``` +Commit: 5e8b305f108b8cc9bd18c35ad5c626f71c081ef2 [5e8b305] +Parents: 6906969865 +Author: Nell Shamrell nells@microsoft.com +Date: August 3, 2021 2:55:44 PM +Committer: Nell Shamrell +Commit Date: October 27, 2021 3:56:37 PM +adds in code and test for determining urls for a go package +``` +2. /origin endpoint (for ui query) + +Example commit: +``` +Commit: 8c057670781451cfd7ca22337cecacae2124ac85 [8c05767] +Parents: 542c02763d +Author: Nell Shamrell nells@microsoft.com +Date: October 28, 2021 4:17:40 PM +Committer: Nell Shamrell +adds ability to get go package revisions through the service API +``` +3. update validation schemas + +Example commit: +``` +Commit: 21e11c45b97c06170f436db498c534ff079443d3 [21e11c4] +Parents: 90c8414909 +Author: Nell Shamrell nells@microsoft.com +Date: July 29, 2021 3:17:35 PM +Committer: Nell Shamrell +Commit Date: October 27, 2021 3:55:33 PM +adds go as a type in schemas +``` +## Documentation +Adaptation to reflect the new harvest source in the following documents: +- service/README.md +- service/swagger.yaml +- service/docs/determining-declared-license.md +- clearlydefined/docs/adding-sources.md From cebe3e6566b465054a6e208087f09b5aede48892 Mon Sep 17 00:00:00 2001 From: Qing Tomlinson Date: Tue, 6 Feb 2024 15:18:51 -0800 Subject: [PATCH 2/3] Add config files to .gitignore --- .gitignore | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/.gitignore b/.gitignore index cdb0099d..dd736add 100644 --- a/.gitignore +++ b/.gitignore @@ -6,3 +6,9 @@ coverage/ .nyc_output *~ \#*# + +# dotenv environment variables file +.env +env.json + +*.env.list From 137f37b2653abd8fe82f81e5af7249b3820da51e Mon Sep 17 00:00:00 2001 From: Qing Tomlinson Date: Wed, 28 Feb 2024 06:51:22 -0800 Subject: [PATCH 3/3] Update docs/add-new-harvester.md Co-authored-by: E. Lynette Rayle --- docs/add-new-harvester.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/add-new-harvester.md b/docs/add-new-harvester.md index 8174beec..7fc083c7 100644 --- a/docs/add-new-harvester.md +++ b/docs/add-new-harvester.md @@ -3,7 +3,7 @@ This is a document in progress detailing the steps necessary to add a new harvest source. ## Community Engagement -When adding a new harvest source, the points to be considered are located at https://github.com/clearlydefined/clearlydefined/blob/master/docs/adding-sources.md#adding-a-new-harvest-source. Please include these considerations and document them in your GitHub issue, similar to https://github.com/clearlydefined/service/issues/882. +When adding a new harvest source, the points to be considered are located at https://docs.clearlydefined.io/adding-sources#adding-a-new-harvest-source. Please include these considerations and document them in your GitHub issue, similar to https://github.com/clearlydefined/service/issues/882. ## Crawler Need to implement: