Support Nexus repository manager for RubyGems

[JrobT]

What would you like Renovate to be able to do?

Nexus repository manager doesn't support the '/api/v1/' pattern for RubyGem discovery, however, bundler does support installing Gems when the Nexus URL is used as the source in the Gemfile.

When Renovate finds a Gemfile with a source of a source of Nexus the logs show:

DEBUG: Datasource connection error (repository=example)
       "datasource": "rubygems",
       "packageName": "example",
       "url": "https://private/repository/example/api/v1/gems/example.json",
       "errCode": "ERR_NON_2XX_3XX_RESPONSE"
DEBUG: Failed to look up dependency ***

The above URL would result in a 400.

If you have any ideas on how this should be implemented, please tell us here.

The source used for bundler is just a URL link to the Gem group rather than an API endpoint. Should maybe take that into account.

Is this a feature you are interested in implementing yourself?

No

[rarkins]

Nexus docs are here: help.sonatype.com/repomanager2/ruby%2C-rubygems-and-gem-repositories

Would be useful if anyone is able to create a reproduction of this problem using the free Nexus registry.

[github-actions]

Hi there,

Help us by making a minimal reproduction repository.

Before we can start work on your issue we first need to know exactly what's causing the current behavior. A minimal reproduction helps us with this.

To get started, please read our guide on creating a minimal reproduction to understand what is needed.

We may close the issue if you (or someone else) have not provided a minimal reproduction within two weeks. If you need more time, or are stuck, please ask for help or more time in a comment.

Good luck,

The Renovate team

[rarkins]

I'm reaching out to the Bundler team indirectly to see if we can get any advice on this. Good chance the proprietary/private registries all implemented the same way

[rarkins]

Related: #6787 #6845

[JrobT]

@rarkins I believe this is a functioning reproduction repo? https://github.com/JrobT/renovate-minimal-reproduction-nexus-rubygems.

Let me know if there's anything I can do.

[rarkins]

Can you provide instructions on how to run nexus locally to reproduce?

[JrobT]

Can you provide instructions on how to run nexus locally to reproduce?

@rarkins In readme

[hasanwhitesource]

We can add a flow to check for a nexus source in the Gemfile and then for example in this case we can do a get request to http://localhost:8081/service/rest/v1/search?repository=gems&name=rubocop&sort=version to fetch versions sorted from highest version.
source : https://help.sonatype.com/repomanager3/integrations/rest-and-integration-api
@rarkins @viceice what do you think?

[rarkins]

I don't think we can reliably know a registry is Nexus from the Gemfile, e.g.

# frozen_string_literal: true

source "http://localhost:8081/repository/gems/"

gem "rubocop", "1.28.1"
[Terms](https://docs.github.com/en/github/site-policy/github-terms-of-service)

Therefore it may require some type of fallback/probe within the rubygems datasource.

[viceice]

does nexus return a header we can use to identify? so make a head request to registry url, check header, cache info for further requests.

[hasanwhitesource]

@viceice Yes in the header there is useful information that we can use to implement this.

[JrobT]

Thanks for your work on this.

[JrobT]

When could I expect the work on this to be released?

[inverse]

@JrobT looks like the PR got closed that was implementing this.

[rarkins]

@shea-mcdonough is there a problem other than that it tries a URL which isn't successful?

[viceice]

I'm pretty sure nexus sends a header to detect this, so we can simply skip the second json fetch?

[zharinov]

I tend to leave it as is, maybe someday it will support more endpoints

[viceice]

I tend to leave it as is, maybe someday it will support more endpoints

doing so causes nexus to be not usable for renovate and rubygems datasource, so it's not an option. I don't think nexus will support that API soon.

[zharinov]

Why not usable? It successfully creates PRs, just checked this yesterday

[viceice]

Why not usable? It successfully creates PRs, just checked this yesterday

so we catch that 404 on the api and continue what we've found on versions endpoint? it's only a cosmetic log issue?

[zharinov]

Yes, currently it's just one optional request for metadata

[zharinov]

Please, feel free to reopen this issue if needed

[shea-mcdonough]

Renovate didn't create any PRs for the Nexus gems. It seems like it tries the old API when looking up each of the gems and I see a bunch of these messages on the dependency dashboard

Failed to look up rubygems package XXXX

I figured we could rule out it being an issue with how we have the credentials configured since we see it hit the correct API and return a 200 but then it tries using the old API and fails

[zharinov]

@shea-mcdonough Please, could you tell the version of Renovate being used? Is this self-hosted or Github App one?

[shea-mcdonough]

@zharinov We are using the self-hosted Renovate

renovatebot/[email protected]

[zharinov]

Thanks, okay, let's try one more PR, I hope it could fix it

[renovate-release]

🎉 This issue has been resolved in version 36.1.15 🎉

The release is available on:

• GitHub release
• 36.1.15

Your semantic-release bot 📦🚀

[shea-mcdonough]

👋 We tested again today and are still having issues.
"renovateVersion": "36.4.0"

It looks like renovate is able to see that there is a new version of the gems but fails to update the gem

"stack": "HTTPError: Response code 400 (Bad Request)\n    at Request.<anonymous> (/opt/containerbase/tools/renovate/36.4.0/node_modules/got/dist/source/as-promise/index.js:118:42)\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)",
         "options": {
           "headers": {
             "user-agent": "RenovateBot/36.4.0 (https://github.com/renovatebot/renovate)",
             "accept": "application/json",
             "authorization": "***********",
             "accept-encoding": "gzip, deflate, br"
           },
           "url": "https://NEXUS_URL/api/v1/versions/aws-sdk-ssm.json",

We did some manual testing against http://rubygems.org/. The following works:

curl https://rubygems.org/info/capybara

The following does NOT:

curl https://rubygems.org/api/v1/capybara.json

[shea-mcdonough]

👋 @zharinov Any update on when the fix will be released?

[zharinov]

Working on it...

[renovate-release]

🎉 This issue has been resolved in version 36.23.1 🎉

The release is available on:

• GitHub release
• 36.23.1

Your semantic-release bot 📦🚀

[shea-mcdonough]

@zharinov 👋 Thanks for pushing up the change! I now see renovate try api/v1/versions/{package}.json which gets a 400 and then it tries /info/{package} and gets a 200 so it looks like the change is working.

The current issue is it didn't actually create the pull requests to upgrade the gems from Nexus. I see this output in the logs:

DEBUG: Aborting Bundler artifacts due to previous failed attempt

Is there a way to work around that?

[zharinov]

@shea-mcdonough

Inside PR, check retry box, it should help:

• If you want to rebase/retry this PR, check this box

[inverse]

Tested with a dry-run and saw it produce some PR logs :)

Thanks!

[shea-mcdonough]

🎉 This is working for us now. Thanks for fixing the Nexus support and answering our questions!