Slash encoding in qualifiers

[aloubyansky]

Continuing the discussion that started in package-url/packageurl-python#121 (comment)

It appears different implementations under https://github.com/package-url implement encoding of slashes (among other things) differently. It'd be good to clarify whether some of them should be fixed to have a consistent encoding across different implementations.

For example, the Python implementation

from packageurl import PackageURL
purl = PackageURL.from_string("pkg:maven/org.apache.james/[email protected]?repository_url=https%3A%2F%2Fmaven.repository.redhat.com%2Fga%2F&type=jar")
print(purl.to_string())

results in

pkg:maven/org.apache.james/[email protected]?repository_url=https://maven.repository.redhat.com/ga/&type=jar

While the Java one

//DEPS com.github.package-url:packageurl-java:1.5.0

import java.util.TreeMap;
import com.github.packageurl.PackageURL;

public class purl {
    public static void main(String[] args) throws Exception {

        final TreeMap<String, String> qualifiers = new TreeMap<>();
        qualifiers.put("type", "jar");
        qualifiers.put("repository_url", "https://maven.repository.redhat.com/ga/");
        var purl = new PackageURL(PackageURL.StandardTypes.MAVEN,
                    "org.apache.james",
                    "apache-mime4j-storage",
                    "0.8.9.redhat-00001",
                    qualifiers, null);
        System.out.println(purl);
    }
}

results in

pkg:maven/org.apache.james/[email protected]?repository_url=https%3A%2F%2Fmaven.repository.redhat.com%2Fga%2F&type=jar

Would it be worth it creating an issue to align implementations of the encoding rules? Thanks.

FYI @tichavskym

[matt-phylum]

The official test suite makes it clear that slashes are expected not to be escaped.

For example (ironically for Java packages, even though the Java implementation does escape them):

purl-spec/test-suite-data.json

Lines 110 to 121 in 3465898

	{
	"description": "maven often uses qualifiers",
	"purl": "pkg:Maven/org.apache.xmlgraphics/[email protected]?classifier=sources&repositorY_url=repo.spring.io/release",
	"canonical_purl": "pkg:maven/org.apache.xmlgraphics/[email protected]?classifier=sources&repository_url=repo.spring.io/release",
	"type": "maven",
	"namespace": "org.apache.xmlgraphics",
	"name": "batik-anim",
	"version": "1.9.1",
	"qualifiers": {"classifier": "sources", "repository_url": "repo.spring.io/release"},
	"subpath": null,
	"is_invalid": false
	},

But I have noticed the following reasons for implementations to fail the official test suite:

• They don't use the official test suite.
• They use an old copy of the test suite.
• They "correct" the test suite.
• They use the test suite, but they skip tests for unsupported package types. This makes sense because some of the tests are testing type-specific rules. If your implementation doesn't support mlflow name normalization, you're not going to pass the mlflow tests, so just skip them all. Unfortunately, the test suite doesn't do a very good job of intentionally covering the core aspects such as escaping rules, so implementations that skip package types might be skipping tests that are relevant for other package types.
• They don't implement the test suite correctly. Implementations need to verify that parsing the provided PURL produces the component values, formatting the component values produces the canonical PURL, and finally that parsing the canonical PURL produces the same component values.

I made https://github.com/phylum-dev/purl-survey to be able to compare implementations and it can run the same test suite consistently for all implementations. I entered GitHub issues for some egregious implementation errors, but there are still serious problems which are not covered by the test suite and can cause PURLs to change meanings.

[aloubyansky]

@stevespringett wdyt? Worth creating an issue to fix it in https://github.com/package-url/packageurl-java? Thanks!

[aloubyansky]

I created package-url/packageurl-java#122

[stevespringett]

The Java version was using a really old test suite. However, updating it causes a lot of issues. It turns out that the official test suite now includes purl type specific tests, which is not suppose to be in there.

The test suite was intended to be used for testing the purl spec itself, not the individual purl types. I recommend we separate out the purl type tests into their own test suite. This will help as we version the purl spec independent of the purl types themselves, which may end up having their own registry in the future.

So, in order to fix this one issue affecting the core spec, I now need to fix a boatload of other issues, all of which are purl type specific, if the Java library is going to use the updated test suite.