Bintray shuts down

[AlekSi]

https://jfrog.com/blog/into-the-sunset-bintray-jcenter-gocenter-and-chartcenter/

Will it affect Homebrew's bottles?

[SMillerDev]

Yes, definitely. We're currently in talks with JFrog to figure this out. But if you have a couple of terabytes of storage laying around somewhere with unlimited bandwidth ideas are welcome 😄

[gromgit]

You can stash your bottles anywhere, as long as the root_url value is correctly set. That's how the Homebrew GitHub Releases mechanism works.

[SMillerDev]

You also don't need to use the same host as homebrew-core does for taps. If you like internet archive and homebrew supports it, go nuts.

[MikeMcQuaid]

Might I suggest that any solution you setup has custom homebrew DNS for the endpoint. That way if you need to move again you have more control over the situation.

It's very easy for us to push DNS changes to users so: no need to add complexity/another point of failure here.

Someone should email Apple!

We are in contact with Apple.

Per Homebrew/brew#10677, it looks like Homebrew is heading toward storing bottles at the Internet Archive (archive.org).

We investigated this but we are not taking this direction. I'll post a separate answer on what we're doing.

[Uzume]

I am assuming these are going to be a problems for the Linuxbrew people:

• https://github.com/Homebrew/linuxbrew-core/search?q=bintray.com

There also seem to be many references to Bintray in the APIs:

• https://github.com/Homebrew/formulae.brew.sh/search?q=bintray.com

An example of which can be seen here:

• https://formulae.brew.sh/api/formula/berkeley-db.json

Are the plans to address these types of issues?

[MikeMcQuaid]

Homebrew/homebrew-core#74335

[MikeMcQuaid]

I can confirm that we've migrated Homebrew's binary packages to GitHub Packages: https://github.com/orgs/Homebrew/packages

This will be the default download location as-of the next Homebrew stable release: Homebrew/brew#11070

[darkRaspberry]

Linuxbrew packages are moved?

[SMillerDev]

Yes

[darkRaspberry]

Thanks i love it.

[shrinidhi111]

Hi team!

Before I would download the bottles/binaries directly from bintray for some testing... Is that not possible now?

I am only able to see docker pulls but how do I download the binaries themselves?

[MikeMcQuaid]

@shrinidhi111 No longer possible, they have all been deleted by Bintray.

[rawlins]

Following up to some discussion here (sorry to switch locations but I realized an open discussion topic was probably better than a closed PR):

• there's currently a brownout at bintray "to warn users about May 1st as sunset date for Bintray".
• Anyone using a 10.15 runner (i.e. macos-latest) with the default settings will be getting a version of homebrew that is apparently not updated to use the new download location (Homebrew 3.0.10 looks like). In the actions output the error is not very informative about what's going on, users will just see curl: (22) The requested URL returned error: 502 Bad Gateway. The URL itself just gives Forbidden!.
• It's easy to fix by updating homebrew, just was not at all apparent to me (as a casual actions user) that this was the solution without a bunch of digging into the above. (What we did in particular was explicitly use the setup homebrew action rather than relying on the implicit setup.)
• Unless github updates their default macos image by May 1 this will probably come back again. Edit: in macOS-10.15 agent uses Homebrew 3.0.10 which uses Bintray actions/runner-images#3165 an actions staff member indicated that macos-latest would probably update to the new version of homebrew mid-next-week.

Hope this is helpful in saving other people some time!

[lukeredpath]

We're struggling with this problem with our Circle CI pipelines at the moment and I'm not sure what's going wrong.

We were alerted to this issue by the Bintry brown-out - we were not routinely auto-updating Homebrew (we disable auto-update when we install) and I don't know how often Circle CI update their images. As a temporary fix I tried re-enabling auto-update and I can see in our build logs that it is running a brew update but it still fails to install the package xclogparser, with the following output:

brew install xclogparser
Updating Homebrew...
==> Auto-updated Homebrew!
Updated 3 taps (homebrew/cask-versions, homebrew/core and adoptopenjdk/openjdk).
==> New Formulae
as-tree                    kubergrunt                 python-tabulate
bas55                      latino                     [email protected]
brook                      leaf-proxy                 qt-libiodbc
cadence-workflow           libmd                      qt-mariadb
cherrytree                 libpipeline                qt-mysql
clusterctl                 libprelude                 qt-percona-server
coin3d                     libunwind                  qt-postgresql
crackpkcs                  linux-pam                  qt-unixodbc
crispy-doom                lttng-ust                  rdkit
curlie                     mapcidr                    rosa-cli
dasel                      moto                       rtl_433
datalad                    multitime                  saml2aws
delve                      mx                         smu
djl-serving                name-that-hash             snowpack
dua-cli                    net-tools                  spaceship
enkits                     numactl                    sqlancer
enzyme                     obfs4proxy                 systemd
epr                        open-adventure             threemux
geph4                      openmama                   timg
ghc@9                      openmodelica               tomcat@9
glibc                      oras                       truffle
go-boring                  orgalorg                   tz
[email protected]                    pandoc-plot                virtualenv
gopass-jsonapi             pcalc                      waffle
haruhi-dl                  pcp                        wllvm
haskell-language-server    phpbrew                    xcprojectlint
iconsur                    projectm                   xray
influxdb@1                 pure                       yubikey-agent
kertish-dfs                pyqt-3d                    zlib-ng
kickstart                  pyqt-builder               zsh-vi-mode
klee                       pyqt-networkauth
kotlin-language-server     pyside@2
==> Updated Formulae
Updated 5105 formulae.
==> Renamed Formulae
[email protected] -> [email protected]
ht-rust -> xh
kde-extra-cmake-modules -> extra-cmake-modules
kde-karchive -> karchive
kde-kdoctools -> kdoctools
kde-ki18n -> ki18n
kde-threadweaver -> threadweaver
libsasl2 -> cyrus-sasl
minizip2 -> minizip-ng
[email protected] -> [email protected]
pyqt5 -> pyqt@5
qt5 -> qt@5
==> Deleted Formulae
atlassian-cli              avian                      geant4
==> New Casks
ableton-live-intro10       adoptopenjdk16-openj9      dnagedcom-beta
ableton-live-lite10        adoptopenjdk16-openj9-jre  local-beta
ableton-live-standard10    arduino-beta               netnewswire-beta
ableton-live-suite10       clashx-pro                 olympus
adoptopenjdk16             dash5                      qq-beta
adoptopenjdk16-jre         deadbeef-nightly           zulu15
==> Updated Casks
Updated 74 casks.
==> Deleted Casks
hex-fiend-beta

==> Downloading https://homebrew.bintray.com/xclogparser-0.2.26.catalina.bottle.
##O#- #                                                                       
curl: (22) The requested URL returned error: 403 Forbidden
Trying a mirror...
==> Downloading https://ghcr.io/v2/homebrew/core/xclogparser-0.2.26.catalina.bot
==> Downloading from https://github.com/-/v2/packages/container/package/homebrew
#=#=#                                                                         
curl: (22) The requested URL returned error: 404 
Error: Failed to download resource "xclogparser"
Download failed: https://ghcr.io/v2/homebrew/core/xclogparser-0.2.26.catalina.bottle.tar.gz

Exited with code exit status 1
CircleCI received exit code 1

If I SSH into the Circle CI worker after it fails I can see it has updated to brew 3.1.1. I can install this package on my local machine using brew 3.1.1.

Does anybody know how to fix this?

[lukeredpath]

So it looks like an explicit brew update did the trick, I'm not sure why that is any different. Sadly it adds nearly 5 minutes overhead to our builds so its not a sustainable solution, we may need to raise a support ticket with Circle CI for this.

[mikezs]

Just in case anyone's having more problems getting an update to properly be forced, I'm using this script:

cd $(brew --repo); git fetch; git reset --hard origin/master
cd "$(brew --repo)/Library/Taps/homebrew/homebrew-core"; git pull https://github.com/Homebrew/homebrew-core
cd "$(brew --repo)/Library/Taps/homebrew/homebrew-cask"; git pull https://github.com/Homebrew/homebrew-cask

[carlocab]

You can also try brew update-reset. (Make sure to stash any changes you've made locally that you want to keep.)

[maltfield]

In-case anyone else ended-up in this thread trying to figure out how to download files from GitHub Packages. This no longer works:

wget https://homebrew.bintray.com/bottles/wget-1.20.3_2.catalina.bottle.tar.gz

Unfortunately, doing this equivalent in GitHub Packages is extremely difficult. I spent a few weeks figuring it out, and I wrote a long article that describes how to download brew bottles from GitHub Packages here

• https://tech.michaelaltfield.net/2024/09/03/container-download-curl-wget/

[MikeMcQuaid]

Where should I open a bug report for this missing documentation?

It's not a bug so: do not open a bug report. You have requested it here. That is sufficient. We have and will consider your request.

[maltfield]

@MikeMcQuaid is this how you treat contributors?

I currently avoid using homebrew because of these security issues. I would like to use homebrew, but I need to be able to work with your team to address these issues before I can even begin to consider using brew (actually I do use it in my CI pipelines, but only with the 3TOFU method that I documented and published at the top of this thread).

What you're saying doesn't make sense. I've reported over 30 bugs to various software projects on GitHub related to release signing. Most of that software I don't use because I need the signing as a prereq to being able to use the software 😕

How the heck are we supposed to contribute bugs for prereqs that are not met??

[MikeMcQuaid]

@MikeMcQuaid is this how you treat contributors?

Homebrew's definition of "contributors" involves people who have submitted a pull request to one of our repositories. I've been working on open source for 20 years, on Homebrew for 15 years and spent 10 years at GitHub so my views here are not uninformed.

I would like to use homebrew, but I need to be able to work with your team to address these issues

The best person to address your issues using Homebrew is you rather than us. Millions of other people are able to use Homebrew without us making the changes you want us to. Making the changes you want us to takes time away from addressing the other issues. We don't keep issues claiming to be "bugs" around open on our issues tracker when we don't have significant other numbers of people who also need the same thing.

What you're saying doesn't make sense.

It makes sense to me and the other folks that work on Homebrew. In our space: if you think we don't make sense and you do: that's something on you to address, not us.

Most of that software I don't use because I need the signing as a prereq to being able to use the software 😕

My prediction is that not all of that software will make the changes that you "need" because they provide their software as-is and their maintainers owe you nothing.

How the heck are we supposed to contribute bugs for prereqs that are not met??

We don't think you are.

Again, please reread our Code of Conduct and ensure that all future communication is in keeping with it: Homebrew/.github@master/CODE_OF_CONDUCT.md.

Your communication in this thread should be improved in future comments.

[maltfield]

Right, so, back to what I said at the top: Brew is aware of these issues. You're just de-prioritizing security.

Sounds like you have some serious leadership issues 🤦 I wish the rest of the team the best of luck

[MikeMcQuaid]

Right, so, back to what I said at the top: Brew is aware of these issues. You're just de-prioritizing security.

The only "issue" I see here is that we don't have documentation for something that you see as a prerequisite for using Homebrew that others don't.

Note: we have many other issues that do not warrant being included in the issue tracker. That's how we use our issue tracker. It's different to other projects.

We are not de-prioritising security. We have recently had and addressed issues raised by a professional security audit (our second). We have a security team made up with two people who professionally work in security (which you do not seem to).

Sounds like you have some serious leadership issues 🤦 I wish the rest of the team the best of luck

This is an ad-hominem attack. You've been pointed to our Code of Conduct twice and don't seem to be acknowledging that, expressing gratitude for the time that's been put into explaining things to you or apologising when you've been showed to be incorrect or your communication has been suboptimal.

I will be more explicit in the home it warrants a response: if you continue to double-down on this behaviour and refusing to apologise or adjust: you will be blocked from the Homebrew organisation. This is your first, and depending on the response may be your last, formal warning.