Dependency management in this repo / decommision of old software versions used

[TheRealBro]

php/v7.4/Dockerfile.amd64

Line 15 in b10c068

echo "deb https://deb.nodesource.com/node_8.x bionic main" | tee /etc/apt/sources.list.d/node.list && \

It seems we use an old node version in all of those images (beginning with php 7.0 - before we used node 6 than we switched to 8 2 years ago) - this is a big mess - as this node version is no longer supported by nodejs

See here: https://en.wikipedia.org/wiki/Node.js#Releases

php/v7.4/Dockerfile.amd64

Line 17 in b10c068

echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list && \

Furthermore we use an old version of yarn (called yarn classic now) which is still supported, but regarding their webpage shall be decommissioned at any point in future and they "kindly" ask to migrate to yarn2.

See here: https://yarnpkg.com/getting-started/migration#why-should-you-migrate

How to address those issues? Where do we have dependencies from this container in our builds, that use node 8 (explicitly) and don't work with e.g. node 14 (LTS)?
Furthermore where is yarn used and can we migrate the yarn files to yarn2?

We should generally think of dropping every image not using php 7.4 or newer as all other PHP versions are outdated. Regarding this, we would get rid of Ubuntu 16.04 (outdated) and Ubuntu 18.04 (soon outdated) as well and could move to the new base image 20.04(LTS) everywhere.

Furthermore, we tried (should try) to stop using ondrej-repos for php, as they are fully managed by a single-point-of-failure and lead to several real-world-problems in the past, like ondrej starting to build his own version of openssl inside the php repo. That lead to a big mindboggling on our instances - and we should not implement the same in our docker images.

php/v7.4/Dockerfile.amd64

Line 20 in b10c068

LC_ALL=en_US.UTF-8 add-apt-repository ppa:ondrej/php && \

I know this is a big issue - there might be an outcome of multiple PRs but I guess this is a general maintenance issue here - not just a single versioning problem. We should start a workforce to address those problems.

I don't know, but to me it looks like we should have a monthly maintenance round where things like this are addressed, issues opened and tasks are generated and distributed, to get rid of those issues. This affects most of the other repos in owncloud-ci as well - where versioning seems to be a bit off.

(this topic was brought up by @mmattel during depend-a-bot checks for our repos)

[xoxys]

Thanks for writing down. Thats already known and not a sysadmin task. The engineering team and the users of the image need to think about a workflow, what they really need and a maintenance workflow.

I had a discussion with @micbar already and we agreed mixing php and nodejs CI image is a bad idea. Dependency builds should use dedicated ci images for php and node (we have up2date nodejs ci images).

[phil-davis]

Regarding PHP:

• anything before 7.2 is not used by any current oC10 core and apps testing or release process
• there are update tests that still run from very old PHP. They install old oC8 and oC9 versions using a PHP supported by those. Then the next step replaces the code with current oC10 code and runs the upgrade with a new PHP version. If we remove some old PHP versions completely, then we will not be able to install oC8/oC9 in CI
• PHP 7.2 support should be dropped "real soon now" Drop support for PHP 7.2 owncloud/core#39134 Drop support for PHP 7.2 owncloud/core#38697
• PHP 7.3 still has security support to Dec 2021 so we will need to keep that into early 2022

I agree that we should concentrate on PHP 7.4 and use that for all routine CI.

[phil-davis]

we agreed mixing php and nodejs CI image is a bad idea

Agree - steps that build JavaScript stuff and that run JS unit tests... should have separate owncloud-ci docker images, so that we can easily choose the JS dependencies, rather than having them "accidentally" locked to the PHP images.

[phil-davis]

Now I see that there is already https://github.com/owncloud-ci/nodejs - why are we not using those docker images for steps that do JavaScript and nodejs stufff?

[xoxys]

IIRC because it was not possible to build the javascript part in a standalone CI step and isolated from the php deps in core (?) or some apps, maybe this is fixed already, dont remember the details.

[phil-davis]

I will try adjusting some core JS-related CI to use those images and see what explodes...

[mmattel]

FYI, we are using the image with Node LTS14 in docs and have very good experiences with.

[phil-davis]

The 4 PRs linked above get us to the point that current oC10 core CI uses nodejs from owncloudci/nodejs:14 and does not use the nodejs 8 that is baked into the owncloudci/php images.

Next I will sort out the same for the oC10 app repos.

[phil-davis]

This has been sorted out in all oC10 app repos now. The starlark code in owncloud/activity#1002 has been replicated to the other repos.

Do we purposely remove node 8 from this image?
(and see what other CI breaks and fix it on-the-fly)
Or?

[micbar]

yes, let us try it.

[phil-davis]

yes, let us try it.

PR #138 is ready for review.

[phil-davis]

We had to explicitly install python v2, it "went away" when we dropped node, npm, yarn.
See PR #139 - the discussion is there.

[phil-davis]

We are not seeing any problems with the latest owncloud-ci/php images. All CI that I am aware of no longer relies on having JS-related node, yarn etc embedded in the PHP image.

[phil-davis]

The remaining items in the original post seem to be:

1. drop old PHP images:

my comment - we are about to drop PHP 7.2 support with the upcoming 10.9.0 oC10 release. And "recent" oC10 versions have supported PHP 7.3 and 7.4 for "a long time now" (I can dig around to get the exact time if needed). PHP 7.3 security supported ends in the next month or so. I suspect that after 10.9.0 is out, we will discuss when to drop PHP 7.3. And we can easily run regular CI all with PHP 7.4.

For things like the upgrade-testing, there are old upgrade paths from oC9 and oC8 that start by installing, for example, oC8 using some old PHP version (supported by oC8). Then the next step switches to a current PHP version and upgrades to oC10 and runs tests. This reflects what could still happen at a real install - they can do a combination of updating PHP and ownCloud one after the other and get from oC8 to oC10. IMO the upgrade-testing pipelines are the only things that still regularly use old PHP images.

2. remove old Ubuntu

my comment: Ubuntu - IMO 16.04 can go. 18.04 still has LTS support - what do we need to test against that?

3. stop using ondrej-repos for php

my comment: I use those all the time for development - they give a regularly updated PHP and easy to switch PHP versions "on-the-fly". Is it easy to fetch the different PHP versions (at current patch releases) some other way?

[DeepDiver1975]

3. stop using ondrej-repos for php

I see no reason to do so. ondrej-repos are the standard over years on how to get different versions of php into different versions of ubuntu and debian repos.

[phil-davis]

remove old Ubuntu
my comment: Ubuntu - IMO 16.04 can go. 18.04 still has LTS support - what do we need to test against that?

There is no longer any mention of Ubuntu 16.04, that is gone.

Ubuntu 18.04 is used in latest, v7.0, v7.1, v7.2, v7.3 - we can just leave those as-they-are until we delete them some day.

latest installs whatever PHP comes by default with owncloud/ubuntu:18.04-amd64 - as discussed above, we can delete latest because it is not useful for CI anyway. See PR #159

[TheRealBro]

Ondrej repos do very weird things sometimes, e.g. they shipped several other packages (not just php) in the past, that they needed to update, because new php versions needed newer versions of libs as dependencies. Unfortunately, that overwrote system packages with newer versions, that were later fixed by Ubuntu LTS packages with security updates. So there is a security risk, that is coming from those repos and we actually run into it in the past, because system detected packages as "newer than from system repos" and did not install LTS security updates... so honestly. Get rid of the Ondrej repos and make our CI work with the default repos based on LTS support and security fixes.

Furthermore, in upcoming communication, we say that we stay with PHP 7.4 because we rely on Ubuntu 20.04 LTS support and they will fix security issues in the future in PHP 7.4 whenever they come up in their security team. All this communications is worthless, if we use Ondrej repos in our containers or in our documentation.

[DeepDiver1975]

We are talking here about images purely used for ci. These are not the production images.