[Master issue] Prepare eXist 5.3.0 stable release

[joewiz]

• Memory/Resource leak in XQuery Context Variables of Library Modules* - Cleanup all module contexts #3897
• Maps* - a follow-on PR just for 5.3.0 to Switch map:merge from use-last to use-first as per XQuery spec #3738 (@adamretter please replace "3738" here with the new, split PR when it's ready)
• Security* - [BUG] Guest users can create collections hidden from dba users #3505
• Security - Finish and release* https://github.com/eXist-db/templating
• Security - core+Saxon - add option to limit RFI in fn:doc etc* - [feature] Allow external HTTP requests to be limited to authenticated users or blocked altogether #3927
• Security - core* - Apache XML-RPC CVE-2019-17570 / CVE-2016-5002 - Apache XML-RPC CVE-2016-5002 #3063 Update Apache XML-RPC #3934 cannot be merged because it introduces a breaking change
• file:sync prune param* - added optional 'prune' param to file:sync(...) #3084 still needs XQSuite tests
• Review/merge PR updating crypto lib for compatibility with 5.3.0 - next version, compatible with eXist 5.3.0 expath-crypto-module#37
• Security: Update core apps to resolve security issues identified by dependabot
  • dashboard* - https://github.com/eXist-db/dashboard/security/dependabot
  • [ ] docs* - https://github.com/eXist-db/documentation/security/dependabot - these are build warnings, not security issues with read-only running app
  • eXide* - https://github.com/eXist-db/eXide/security/dependabot
  • monex* - https://github.com/eXist-db/monex/security/dependabot
• Update core apps to use templating v1.0.0:
  • doc - Switch to the new templating module, remove shared-resources dependency documentation#626
  • fundocs - Switch to the new templating module function-documentation#41
  • monex - [feat] switch to new templating library monex#153
• Update core apps to remove all dependencies on shared-resources:
  • doc - Switch to the new templating module, remove shared-resources dependency documentation#626
  • eXide - Drop dependency on shared-resources eXide#309
  • fundocs - remove dependency on shared resources and cleanup function-documentation#42
  • markdown - Remove shared-resources dependency exist-markdown#7
  • monex - remove dependency on shared resources package monex#155
• Publish the updated core apps to public-repo and GitHub Releases
  • doc
  • eXide
  • fundocs
  • markdown
  • monex
• Update release process for GitHub Releases* - Fix release process since migration from BinTray to GitHub Releases #3845

Finally:

• Publish the release according to https://github.com/eXist-db/exist/blob/develop/exist-versioning-release.md#preparing-a-product-release
• Update exist-db.org with eXist 5.3.0 and updated core apps

Open Issue

• push 5.3.0 to master
• Go to GitHub and move all issues and PRs which are still open for the release milestone to the next release milestone and close the milestones 5.3.0 and 5.2.1
• merge open PR and release crypto-lib 6.0.0
• update homebrew
• publish updated “howto release” document [doc] update versioning and release guide #3962
• Github-release-plugin: fix link to release notes and improve layout
• Wikipedia page: Lars asked someone from outside the community to update it. This might take a moment.

Note: Tasks marked with * are drawn from @adamretter's list from Slack.

[joewiz]

Now updated with entries for reviewing and merging the crypto lib package, and publishing it to the public-repo.

[line-o]

Crypto lib needs to wait for 5.3.0 to be released - so technically we should mention this ticket over in the crypto lib PR and not the other way around. Do you agree @joewiz ?

[line-o]

I think Security - core+Saxon - XXE to RFI in fn:doc etc* is already addressed here
[bugfix] Change XML processing defaults for v6.0.0 by dizzzz · Pull Request #3836 · eXist-db/exist
So the question is wether to accept this for security reasons even if it might affect users of the database.

[joewiz]

@line-o Ah, I didn't realize the crypto lib's new version had to come after the release of eXist 5.3.0. Since the crypto lib's release is an important aspect of the eXist 5.3.0 release, I think it's worth keeping a checkbox here. How about if I move that to-do item to the "Finally" section?

[joewiz]

@line-o Regarding the briefly worded "core+saxon" item, I agree that #3836 addresses the XXE issue, but it doesn't address the RFI issue. That topic came up only in the Community Call on May 10:

Functions that can be used to perform external HTTP requests, e.g.: doc(). May need to institute a whitelist/blacklist, or disable external HTTP requests by default. Could be a configuration option—off by default in 5.3.0 but on by default in 6.0.0.

• doc, doc-available, json-doc, unparsed-text, unparsed-text-lines
• EXPath HTTP client
• xinclude
• transform:transform

AR: Suggests we solve these before the forthcoming release.

This deserves its own issue, but the idea is that eXist allows guest users to trigger HTTP requests for remote files (RFI), and the configuration option envisioned here would add a condition to all functions like doc(), limiting the ability to perform HTTP requests. Perhaps the time is right for me to open an issue for further discussion of this?

Update: Issue added: #3927.

[line-o]

@joewiz re: crypto lib checkbox in Finally is good.

[line-o]

@joewiz regarding the introduction of new configuration option to address RFI for guest users: That deserves its separate issue, yes.

[line-o]

CVE-2019-17570 addresses malicious XML-RPC servers. In our scenario this is existdb itself which we ultimately trust as it is under our control, correct?

[adamretter]

CVE-2019-17570

I fixed this on in FusionDB previously. You basically have to host patched versions of the Apache XML-RPC Jars

[line-o]

Is Apache XML-RPC used as a client in exist-db or as a server?

[line-o]

I would like to drop #3738 from the list as it introduces a breaking change (order of application of map:merge).

[joewiz]

Indeed, #3738 is labelled as bound for 6.0.0. Ok to remove it from this list, @adamretter?

[joewiz]

I've split the "Update core apps to use templating v1.0.0 and remove all dependencies on shared-resources" into two separate checklist sets. This way we can track which apps have met each threshold.

[adamretter]

Is Apache XML-RPC used as a client in exist-db or as a server?

Both of course! eXist-db provides an XML-RPC API, and the XML:DB API also is implemented atop XML-RPC. The Java Admin Client is an XML-RPC client as are some of the functions in the XQuery XMLDB Extension module

[adamretter]

Indeed, #3738 is labelled as bound for 6.0.0. Ok to remove it from this list, @adamretter?

@joewiz No, it needs to be split into two parts. The parts for 5.3.0 and the parts for 6.0.0. I will try and do that over the weekend if I can find the time.

[joewiz]

We may want to pause on merging the PRs involving updating the core apps to use the new templating module until #3918 is resolved. The lib:parse-params templating function is affected by this issue.

[joewiz]

Thanks to Wolfgang’s release of templating v1.0.2, which sidesteps the performance issue in #3918, the work on migrating core apps to templating and toward removing shared-resources from them can continue.

These PRs are ready for review & merge so far:

• Switch to the new templating module, remove shared-resources dependency documentation#626
• Switch to the new templating module function-documentation#41
• Remove shared-resources dependency exist-markdown#7

[joewiz]

FYI, someone checked doc and fundocs as having had their dependencies on shared-resources removed, but this is incorrect. The PRs above only switched to the new templating library; they did not fully remove shared-resources dependencies.

[line-o]

Oh, sorry! That was me.

[duncdrum]

@joewiz Docs have had their non templating dependency on shared resources removed about 2years ago, unless someone added them back in or I missed something

[joewiz]

@duncdrum Yes, you were right! It was still listed as a dependency in pom.xml and xar-assembly.xml and a couple of references lingered, but there were no substantive dependencies, so it was easy to pull those out. Thanks!

[joewiz]

The first app without shared-resources from the list above is now published - markdown v0.7.0 - to GitHub Releases and public-repo.

Status of the other core apps being tracked in this master issue:

• documentation has an approved PR #626 removing shared-resources, which passes tests locally but fails in the CI environment for unknown reasons. Attempts at resolving CI issues are in progress in #638. The repo also has 5 open security issues, but because of dependencies on old libraries (see #602 about one of these), we aren't able to advance to the newer versions that resolve the security issues. If these PRs' issues can't be resolved, we will have to release with the continued dependency on shared-resources, since at least the master branch is passing CI.
• eXide has a WIP PR #309 that comprehensively removes shared-resources. The only remaining problem is a regression preventing users from renaming resources.
• fundocs has a merged PR #41 that switches to templating, but so far we have no PR removing the remaining references to shared-resources—namely, jquery and eXist CSS.
• monex has a merged PR #153 that switches to templating, but so far we have no PR removing the remaining references to shared-resources—namely, ace, bootstrap, jquery, eXist CSS, eXist icon, eXist favicon.

[joewiz]

As promised, I added an issue to track the feature to prevent eXist from making external HTTP requests: #3927. I've added the link to the master description above too.

[line-o]

@joewiz I'll add eXist-db/monex#155 to monex

[line-o]

next one on the menu is fundocs: any takers?
eXist-db/function-documentation#42

[line-o]

New versions of monex and eXide are available in the public repository.

[line-o]

All core apps are now released and available in the public package repository.
The PR to update the bundled libraries and apps will be opened tomorrow.

[line-o]

When #3939 is applied shared-resources and markdown are no longer bundled with existdb

[joewiz]

I just unchecked "Update exist-db.org with eXist 5.3.0 and updated core apps" as it appears exist-db.org is still running 5.1.1.

[joewiz]

All incomplete tasks have been accounted for in the new master issue. Work on these should continue there. #3968