Check what packages will be dropped from package sets

[thomashoneyman]

The current package set contains packages that will not be included in the new registry, and therefore won't be included in the package sets produced by the registry (since the package sets only include packages from the registry).

An example: purescript-parsing will be dropped because of #249, unless its LICENSE file gets fixed. It would be good to understand what other packages will be dropped from the package sets unless we fix something about them -- especially packages within the core, contrib, web, and node organizations (like purescript-parsing).

To handle this, we would need to:

1. Get a list of packages and package versions from the latest current package set
2. Run the legacy import tool to produce a bower-exclusions.json file
3. Cross-reference the bower-exclusions against the package set to find package versions found in both places

The list of package versions that exist in both the package set and the bower exclusions represents packages currently provided by the set but which will no longer be provided going forward. This is also the list of packages that we really ought to fix, if we're going to manually fix any packages, to provide a smooth upgrade experience for users.

[maxdeviant]

Still cleaning up the implementation a bit, but here's the initial list of packages that will currently be dropped from the package set:

aff-bus v5.0.0 will be dropped from the package set due to: manifestError
errors v4.1.0 will be dropped from the package set due to: manifestError
foreign-generic v11.0.0 will be dropped from the package set due to: manifestError
free v6.0.1 will be dropped from the package set due to: manifestError
geometry-plane v1.0.1 will be dropped from the package set due to: manifestError
gl-matrix v2.0.1 will be dropped from the package set due to: manifestError
grid-reactors v0.0.1 will be dropped from the package set due to: manifestError
homogeneous v0.3.0 will be dropped from the package set due to: manifestError
node-he v0.2.0 will be dropped from the package set due to: manifestError
node-sqlite3 v6.0.0 will be dropped from the package set due to: manifestError
parsing v7.0.0 will be dropped from the package set due to: manifestError
parsing-dataview v2.0.1 will be dropped from the package set due to: manifestError
parsing-repetition v0.0.6 will be dropped from the package set due to: manifestError
prettier v0.3.0 will be dropped from the package set due to: manifestError
ps-cst v1.2.0 will be dropped from the package set due to: manifestError
supply v0.1.0 will be dropped from the package set due to: manifestError
typelevel-lists v2.0.1 will be dropped from the package set due to: manifestError
unicode v5.0.0 will be dropped from the package set due to: manifestError

(I'll expand the error messages to make it clearer what error would result in them being dropped)

[maxdeviant]

Cleaned up the script (purescript/registry#266) and here is the more detailed output:

aff-bus v5.0.0 will be dropped from the package set due to:
	bad license:
		Invalid SPDX identifier: NOASSERTION
		Did you mean BSD-Protection?

errors v4.1.0 will be dropped from the package set due to:
	bad license:
		Invalid SPDX identifier: NOASSERTION
		Did you mean BSD-Protection?

foreign-generic v11.0.0 will be dropped from the package set due to:
	bad license:
		Invalid SPDX identifier: NOASSERTION
		Did you mean BSD-Protection?

free v6.1.0 will be dropped from the package set due to:
	bad license:
		Invalid SPDX identifier: NOASSERTION
		Did you mean BSD-Protection?

geometry-plane v1.0.1 will be dropped from the package set due to:
	missing license

gl-matrix v2.0.1 will be dropped from the package set due to:
	bad license:
		Invalid SPDX identifier: NOASSERTION
		Did you mean BSD-Protection?

grid-reactors v2.1.0 will be dropped from the package set due to:
	bad dependency versions:
		Dependency: canvas-action
		Failed bounds: artemisSystem/purescript-canvas-action#43de19ee369d1ff9fe7eff1e583b828809fd9e36

homogeneous v0.3.0 will be dropped from the package set due to:
	bad license:
		Invalid SPDX identifier: NOASSERTION
		Did you mean BSD-Protection?

node-he v0.2.0 will be dropped from the package set due to:
	bad dependency versions:
		Dependency: test-unit
		Failed bounds: justinwoo/purescript-test-unit#compiler/0.12

node-sqlite3 v6.0.0 will be dropped from the package set due to:
	bad dependency versions:
		Dependency: test-unit
		Failed bounds: justinwoo/purescript-test-unit#compiler/0.12
		Dependency: node-fs-aff
		Failed bounds: justinwoo/purescript-node-fs-aff#compiler/0.12
		Dependency: simple-json
		Failed bounds: #compiler/0.12

parsing v7.0.0 will be dropped from the package set due to:
	bad license:
		Invalid SPDX identifier: NOASSERTION
		Did you mean BSD-Protection?

parsing-dataview v2.0.1 will be dropped from the package set due to:
	bad license:
		Invalid SPDX identifier: NOASSERTION
		Did you mean BSD-Protection?

parsing-repetition v0.0.6 will be dropped from the package set due to:
	bad dependency versions:
		Dependency: parsing-expect
		Failed bounds: markfarrell/purescript-parsing-expect

prettier v0.3.0 will be dropped from the package set due to:
	bad dependency versions:
		Dependency: spec
		Failed bounds: justinwoo/purescript-spec#compiler/0.12

ps-cst v1.2.0 will be dropped from the package set due to:
	bad license:
		Invalid SPDX identifier: NOASSERTION
		Did you mean BSD-Protection?

supply v0.1.0 will be dropped from the package set due to:
	bad license:
		Invalid SPDX identifier: NOASSERTION
		Did you mean BSD-Protection?

typelevel-lists v2.0.1 will be dropped from the package set due to:
	bad license:
		Invalid SPDX identifier: NOASSERTION
		Did you mean BSD-Protection?

unicode v5.0.0 will be dropped from the package set due to:
	bad license:
		Invalid SPDX identifier: NOASSERTION
		Did you mean BSD-Protection?

[maxdeviant]

I think there's some additional noise in the output due to #267.

I put a temporary fix in place and was able to narrow down the list further:

geometry-plane v1.0.1 will be dropped from the package set due to:
	missing license

grid-reactors v2.1.0 will be dropped from the package set due to:
	bad dependency versions:
		Dependency: canvas-action
		Failed bounds: artemisSystem/purescript-canvas-action#43de19ee369d1ff9fe7eff1e583b828809fd9e36

node-he v0.2.0 will be dropped from the package set due to:
	bad dependency versions:
		Dependency: test-unit
		Failed bounds: justinwoo/purescript-test-unit#compiler/0.12

node-sqlite3 v6.0.0 will be dropped from the package set due to:
	bad dependency versions:
		Dependency: test-unit
		Failed bounds: justinwoo/purescript-test-unit#compiler/0.12
		Dependency: node-fs-aff
		Failed bounds: justinwoo/purescript-node-fs-aff#compiler/0.12
		Dependency: simple-json
		Failed bounds: #compiler/0.12

parsing-repetition v0.0.6 will be dropped from the package set due to:
	bad dependency versions:
		Dependency: parsing-expect
		Failed bounds: markfarrell/purescript-parsing-expect

prettier v0.3.0 will be dropped from the package set due to:
	bad dependency versions:
		Dependency: spec
		Failed bounds: justinwoo/purescript-spec#compiler/0.12

[maxdeviant]

Nevermind, I see now that we're treating the NOASSERTION issue as valid, so it would appear my original (longer) list is correct 😅

[thomashoneyman]

Only 18 packages? That's pretty good! We should fix anything from core, contrib, web, and node (ie. free, aff-bus, parsing, unicode) and release patch versions with the fixed licenses. That way some packages may be dropped out of the current package set, but only a handful and there's a clear path forward to fix them. We could open issues on all of them if we want to, but it's not our responsibility to fix them all.

[thomashoneyman]

I'm a little confused by the errors license failure; we should be rewriting Apache-2 to Apache-2.0 as part of the legacy import tool, and the LICENSE file itself looks about right to me.

[maxdeviant]

I'm a little confused by the errors license failure; we should be rewriting Apache-2 to Apache-2.0 as part of the legacy import tool, and the LICENSE file itself looks about right to me.

The issue AFAICT is that the presence of a single bad license flags the package as excluded due to a bad license.

So in the case of errors, the package.json file contains a license value of Apache-2, which is being detected as a NOASSERTION by licensee. This NOASSERTION is then present in the list along with the other valid versions of the license, but still results in the package being flagged.

[maxdeviant]

I'm a little confused by the errors license failure; we should be rewriting Apache-2 to Apache-2.0 as part of the legacy import tool, and the LICENSE file itself looks about right to me.

The issue AFAICT is that the presence of a single bad license flags the package as excluded due to a bad license.

So in the case of errors, the package.json file contains a license value of Apache-2, which is being detected as a NOASSERTION by licensee. This NOASSERTION is then present in the list along with the other valid versions of the license, but still results in the package being flagged.

supply is another instance where the license in the package.json is not a valid SPDX identifier.

This is a case that we could fix with rewrite rules, but we'd have to rewrite how we're doing license detection to either:

1. Apply the rewrites to the file before we run licensee against it (which is a slightly more impure approach, as we'd be modifying the files from upstream)
2. Keep track of the where each detected license is coming from so that we can potentially override it later (e.g., licensee picks up a bad license value from package.json, so we fall back to reading it manually with a rewrite rule applied)

[maxdeviant]

I've gone through and opened PRs to fix the issues that would result in the packages being dropped from the package set.

Here is the list, corresponding PRs, and status:

• aff-bus
  • Fix: Update LICENSE purescript-contrib/purescript-aff-bus#29
  • Package Set Update: Update aff-bus to v5.0.1 package-sets#1025
• errors
  • Fix: Fix license identifier in package.json passy/purescript-errors#48
  • Package Set Update: [PENDING]
• foreign-generic
  • Fix: Split LICENSE files paf31/purescript-foreign-generic#72
  • Package Set Update: [PENDING]
• free
  • Fix: Split LICENSE files purescript-free#119
  • Package Set Update: Update free to v6.2.0 package-sets#1034
• geometry-plane
  • Fix: Licensing Ebmtranceboy/purescript-geometry-plane#1
  • Package Set Update: license added package-sets#1038
• gl-matrix
  • Fix: Split licenses dirkz/purescript-gl-matrix#1
  • Package Set Update: Update gl-matrix to v2.1.0 package-sets#1037
• grid-reactors
  • Fix: Use version of canvas-action from package set Eugleo/purescript-grid-reactors#2
  • Package Set Update: Update grid-reactors to v3.0.0 package-sets#1033
• homogeneous
  • Fix: Split licenses paluh/purescript-homogeneous#4
  • Package Set Update: [PENDING]
• node-he
  • Fix: Upgrade purescript-test-unit to v16.0.0 justinwoo/purescript-node-he#4
  • Package Set Update: Update node-he to v0.3.0 package-sets#1028
• node-sqlite3
  • Fix: Update dependencies justinwoo/purescript-node-sqlite3#16
  • Package Set Update: Update node-sqlite3 to v7.0.0 package-sets#1029
• parsing
  • Fix: Split licenses purescript-contrib/purescript-parsing#125
  • Package Set Update: Update parsing to v7.0.1 package-sets#1030
• parsing-dataview
  • Fix: Split licenses rowtype-yoga/purescript-parsing-dataview#11
  • Package Set Update: Update parsing-dataview to v2.1.0 package-sets#1040
• parsing-repetition
  • Fix: Update dependencies markfarrell/purescript-parsing-repetition#2
  • Package Set Update: [PENDING]
• prettier
  • Fix: Update PureScript dependencies epicallan/purescript-prettier#6
  • Package Set Update: [PENDING]
• ps-cst
  • Fix: Fix license identifiers purescript-codegen/purescript-ps-cst#26
  • Package Set Update: [PENDING]
• supply
  • Fix: Fix license identifier in package.json ajnsit/purescript-supply#2
  • Package Set Update: Bump purescript-supply version package-sets#1036
• typelevel-lists
  • Fix: Fix license identifier in package.json purefunctor/purescript-typelevel-lists#15
  • Package Set Update: Update typelevel-lists to v2.1.0 package-sets#1032
• unicode
  • Fix: Split licenses purescript-contrib/purescript-unicode#37
  • Package Set Update: Update unicode to v5.0.1 package-sets#1031

[jamesdbrock]

I merged your changes and published parsing-dataview v2.1.0, thank you.

[f-f]

@thomashoneyman are those license splits still useful in light of purescript/registry#251 (comment)?

[thomashoneyman]

I think the license splits should no longer disallow a package (we did a "best effort" to scan your repo, and if we missed a license because you had it in a technically-not-valid combined LICENSE file, then we can pull your repo out of the registry if you want).

We should update our license parsing code to be more lenient in these cases, too, as I think it will reject packages that come up with a NOASSERTION for the LICENSE file.

[thomashoneyman]

I have closed this because it was meant to help us prioritize packages from the package set to manually fix so that they can stay in the registry, but we've found only purescript-prettier will be dropped from the new registry at this point. We can't fix the package, so I don't think we need to continue working on package sets packages from here on out.