Release 21.12.0 #1193

chadwhitacre · 2021-12-07T22:36:16Z

22.1.0

This release carries more risk than usual because of #796. I'm making a ticket here to remember some things to pay attention to.

during the sentry release, make sure that the status provider check lines up name-wise with the check as named in GCB (see this thread)
prep milestones
add something in the changelog about log4j vulnerability (CVE-2021-44228) - not vulnerable #1196 (statement that we are not vulnerable)

The text was updated successfully, but these errors were encountered:

chadwhitacre · 2021-12-15T00:24:29Z

The changelog for tomorrow is actually going to be tricky because we don't have a PR related to #1196, and craft's auto mechanism assumes PRs/commits for every changelog entry. I read craft code and tested out putting an "Unreleased" section but that simply gets updated to the new version without actually listing all of the new material.

chadwhitacre · 2021-12-15T00:30:29Z

What I am going to try is manually modifying the CHANGELOG.md on the release/21.12.0 branch after the prepare action is run but before ... oh wait, CalVer releases are auto-accepted. I guess we need a release blocker? Nope, that prevents event the action-prepare-release from completing.

Okay, plan C: Make a dummy PR related to the milestone in order to work with the existing changelog infra.

chadwhitacre · 2021-12-15T00:39:10Z

prep milestones

Seeing these to account for:

https://github.com/getsentry/self-hosted/milestone/8

Remove link to forum (Deprecate forum in favor of GitHub Issues #1151) (Remove link to forum #1160)
Tweak issue forms (Tweak issue forms #1159)
Sentry styleguide, I did this on the other (Sentry styleguide, I did this on the other #1167)

https://github.com/getsentry/self-hosted/milestone/9

Rename onpremise to self-hosted (Rename onpremise to self-hosted #1169)

https://github.com/getsentry/self-hosted/milestone/10

Add support for custom DotEnv file (Add support for custom DotEnv file #1113)

Skip

Bump nginx:1.21.4-alpine (Bump nginx:1.21.4-alpine #1156)
Unfrobulate "cleanup" in integration test (Unfrobulate "cleanup" in integration test #1170)

Merp

Revert "Rename onpremise to self-hosted" (5495fe2)
Rename onpremise to self-hosted (9ad05d8)

chadwhitacre · 2021-12-15T00:42:33Z

Bloop. Can't retroactively skip commits (re: garbage 5495fe2 9ad05d8).

chadwhitacre · 2021-12-15T01:07:29Z

$ craft prepare --no-push 21.12.0
$ head -n45 CHANGELOG.md

Changelog

21.12.0

Support Docker Compose v2 (ongoing)

Self-hosted Sentry mostly works with Docker Compose v2 (in addition to v1 >= 1.28.0). There is one more bug we are trying to squash.

By: @chadwhitacre (#1179)

Prevent Component Drift

When a user runs the install.sh script, they get the latest version of the Sentry, Snuba, Relay and Symbolicator projects. However there is no guarantee they have pulled the latest self-hosted version first, and running an old one may cause problems. To mitigate this, we now perform a check during installation that the user is on the latest commit if they are on the master branch. You can disable this check with --skip-commit-check.

By: @chadwhitacre (#1191), @aminvakil (#1186)

React to log4shell

Self-hosted Sentry is not vulnerable to the log4shell vulnerability.

By: @chadwhitacre (#1203)

Forum → Issues

In the interest of reducing sources of truth and restarting the fire of the self-hosted Sentry community, we deprecated the Discourse forum in favor of GitHub Issues.

By: @chadwhitacre (#1167, #1160, #1159)

Rename onpremise to self-hosted (ongoing)

In the beginning we used the term "on-premise" and over time we introduced the term "self-hosted." In an effort to regain some consistency for both branding and developer mental overhead purposes we are standardizing on the term "self-hosted." This release includes a fair portion of the work in code towards this, hopefully a future release will include the remainder. The effect of this as a self-hosted user will be that you will see orphaned blah blah blah you need to clean those up floo flah how

By: @chadwhitacre (#1169)

Add support for custom DotEnv file

There are several ways to configure self-hosted Sentry and one of them is the .env file. In this release we add support for a .env.custom file that is git-ignored to make it easier for you to override keys configured this way with custom values. Thanks to @Sebi94nbg for the contribution!

By: @Sebi94nbg (#1113)

Various fixes & improvements

Revert "Rename onpremise to self-hosted" (5495fe2) by @chadwhitacre
Rename onpremise to self-hosted (9ad05d8) by @chadwhitacre

chadwhitacre · 2021-12-15T01:12:59Z

Gosh shouldn't we link to the changelogs for all of the components, as well? I'm thinking of #1131.

chadwhitacre · 2021-12-15T01:21:05Z

Milestones ready.

BYK · 2021-12-15T07:17:41Z

@chadwhitacre you could have blocked the release, prepare locally, edit the Changelog in the release branch, and then finalize the release.

I think we should document this process and potentially find a way to easily edit the changelog before it goes out.

The issue is automation and manual editing are things are at odds so the best I can think of it is having a mechanism like you offered: allow having an unreleased section which gets prepended (or appended?) to the auto-generated changelog.

Happy to dive into Craft code after a while if this sounds like a good idea.

chadwhitacre · 2021-12-15T17:33:59Z

Sentry failure:

Error:  Validation Failed: ***"resource":"Release","code":"custom","field":"body","message":"body is too long (maximum is 125000 characters)"***

Presuming this has to do with the changelog.

chadwhitacre · 2021-12-15T17:38:24Z

Snuba and Relay are out, fwiw.

chadwhitacre · 2021-12-15T17:43:08Z

https://github.com/getsentry/sentry/blob/releases/21.12.0/CHANGES

chadwhitacre · 2021-12-15T17:59:12Z

Here's the failing call:

      const created = await this.github.repos.createRelease(
        createReleaseParams
      );

Here's the payload:

    const createReleaseParams = {
      draft: false,
      name: tag,
      owner: this.githubConfig.owner,
      prerelease: isPreview,
      repo: this.githubConfig.repo,
      tag_name: tag,
      target_commitish: revision,
      ...changes,
    };

...changes 🧐

chadwhitacre · 2021-12-15T18:24:41Z

https://github.com/getsentry/sentry/blob/releases/21.12.0/CHANGES#L32-L1701

Blech, forgot about getsentry/craft#327 tho. :(

chadwhitacre · 2021-12-15T18:58:42Z

Okay, I did a little poking at getsentry/craft#327. I'm not going to try to engineer a fix for that atm.

Also, I have a review on getsentry/craft#335, but I don't think I'm going to try to use that for this release, in the interest of minimizing variability.

I think my plan at this point is to:

Manually modify the changelog on the release/21.12.0 branch in sentry. Due to 327 it's not going to get merged back to master anyway. 🙄
Re-accept publish: getsentry/[email protected] publish#700. As I read/understand it craft will pick up with a pre-existing branch in whatever state it's in.
Once sentry is out, remove the release-blocker label from this current issue, and manually run the [prepare] release action in this repo.
Manually make a PR to sentry after the release to update the changelog for the last three releases (including today's).

BYK · 2021-12-15T19:02:42Z

Manually modify the changelog on the release/21.12.0 branch in sentry. Due to 327 it's not going to get merged back to master anyway. 🙄

This is an excellent workaround. You can still use the publish flow for this.

Manually make a PR to sentry after the release to update the changelog for the last three releases (including today's).

Is this because of the wrong merge base?

BYK · 2021-12-15T19:05:11Z

Re-accept publish: getsentry/[email protected] publish#700. As I read/understand it craft will pick up with a pre-existing branch in whatever state it's in.

Yes, it will use the existing branch.

chadwhitacre · 2021-12-15T19:14:10Z

You can still use the publish flow for this.

Meaning "Re-accept [the publish ticket]"?

Is this because of the wrong merge base?

Yes.

See getsentry/self-hosted#1193

chadwhitacre · 2021-12-15T19:35:37Z

https://github.com/getsentry/sentry/runs/4538784692?check_suite_focus=true 🍿

chadwhitacre · 2021-12-15T19:43:01Z

Prepping the repaired CHANGES file ...

https://github.com/getsentry/sentry/blob/21.12.0/CHANGES
https://github.com/getsentry/sentry/blob/21.11.0/CHANGES
https://github.com/getsentry/sentry/blob/21.10.0/CHANGES

chadwhitacre · 2021-12-15T19:50:03Z

CI is green for the CHANGES repair on the branch, I reaccepted the ticket. Technically this will be a different artifact than what we uploaded to PyPI and Docker, but the only delta is in the changelog so meh.

chadwhitacre · 2021-12-15T19:50:18Z

https://github.com/getsentry/publish/runs/4539048239?check_suite_focus=true 🍿

chadwhitacre · 2021-12-15T19:51:38Z

Was about to say ... need to whack the old 21.12.0 tag ...

chadwhitacre · 2021-12-15T19:57:52Z

Here's the original 21.12.0 tag for reference.

chadwhitacre · 2021-12-15T21:52:45Z

It was probably in a Slack thread that is gone now. Reason to do this here in GitHub instead.

chadwhitacre · 2021-12-15T21:56:15Z

Oh gosh, I think I ended up recreating the release object in GitHub manually? And it was a total cluster?

chadwhitacre · 2021-12-15T21:57:14Z

Yeah it was last month, this one has my name on it:

https://github.com/getsentry/sentry/releases/tag/21.11.0

chadwhitacre · 2021-12-15T21:57:48Z

Sweet! Slack thread still exists! 👍

chadwhitacre · 2021-12-15T21:59:36Z

I'm going to start with relay since it's simpler. My plan is to delete the 21.11.0 release and rerun the release via publish repo.

Going to do the same here.

chadwhitacre · 2021-12-15T22:01:34Z

https://github.com/getsentry/publish/runs/4540347242?check_suite_focus=true 🍿

chadwhitacre · 2021-12-15T22:03:16Z

Failed again ...

chadwhitacre · 2021-12-15T22:03:51Z

Forgot to remove the tag.

chadwhitacre · 2021-12-15T22:06:29Z

https://github.com/getsentry/publish/runs/4540398196?check_suite_focus=true 🍿

chadwhitacre · 2021-12-15T22:07:41Z

Made it past GitHub. 👍

chadwhitacre · 2021-12-15T22:08:04Z

Okay! Relay is really out this time.

chadwhitacre · 2021-12-15T22:08:10Z

Back to self-hosted ...

chadwhitacre · 2021-12-15T22:13:35Z

Exfiltrating the steps to recover from "Cannot upload asset" for sentry in 21.11.0 as it was much more complicated:

On to Sentry ...
Tag and release deleted.
Hrm, actually, the publish ticket claims to have already published everything but GitHub.
I'm going to retag 21.11.0 and rerun.
Fail.
😞
Reference "refs/tags/21.11.0" already exists. Does tag "21.11.0" already exist?
If the tag exists it also wants the release to exist.
If the release were still there it would proceed, and delete assets before re-uploading.
Since the tag would be in the same place the assets would end up the same as on the previous run.
So I either recreate the GH release object manually, or remove ...................................
I'm unraveling how SHA gets passed from the publish issue to the craft publish invocation.
I think I need to delete the tag and rerun. I think it pulls the commit from the ... no, it doesn't.
It deploys master every time, afaict.
I'm going to try manually creating a release object for 21.11.0 and rerunning to populate with assets.
I stubbed it out, I will see about a changelog later.
sentry artifacts are up, I'm going to restart onpremise and then work on sentry changelog.
Rerunning test suite for onpremise.
[changelog blah blah]
onpremise suite passed, rerunning publish for onpremise.
onpremise is out, so the release is done, except for the changelog on the sentry release.

Then I discovered getsentry/craft#327.

chadwhitacre · 2021-12-15T22:14:38Z

Oh right #1171 ...

chadwhitacre · 2021-12-15T22:15:10Z

Oh right it was missing relay 21.12.0 ... 😅

chadwhitacre · 2021-12-15T22:16:22Z

https://github.com/getsentry/self-hosted/actions/runs/1584537387 🍿

chadwhitacre · 2021-12-15T22:38:26Z

Green! Re-accepting getsentry/publish#704 ...

chadwhitacre · 2021-12-15T22:41:48Z

Done.

chadwhitacre · 2021-12-15T23:38:07Z

Post-Mortem

This was anecdotally the most difficult self-hosted release since I started a year ago. Issues:

snuba - no issues
sentry-docs
1. https://github.com/getsentry/publish/issues/705
relay
1. silent failure - not sure why
  1. if recurs frequently debug/address
  2. bare minimum: confirm that release actually exists once publish ticket closes
2. intermittent asset upload failure from GitHub
  1. worked around by deleting the tag and release object and re-accepting the issue
  2. case contributing to Tag more carefully in GitHub target craft#336
  3. probably worth retrying ... Improve GitHub asset upload retry algorithm craft#337
sentry
1. Changelog too large, failed to create GitHub release object
  1. Cut a new craft release with Limit the number of leftovers listed craft#335 before next self-hosted release
2. Last step of publish is selecting the wrong merge target craft#327
  1. Make manual PRs to update changelog post-release
3. INC-87 - delete/recreate tag while addressing (i) led to corrupted pip cache in test image for getsentry, resulting in red CI
  1. New issue: Tag more carefully in GitHub target craft#336
self-hosted
1. Friction around changelog due to log4j vulnerability (CVE-2021-44228) - not vulnerable #1196 and garbage commits
  1. add release-blocker to the release ticket (i.e., this one), manually edit changelog on release branch, and proceed
2. Slowed down by CI flakes under "Ensure cleanup crons are working" #1171

General pattern: something something tags something something. I feel like a number of failure modes end up with us having to recreate a tag, and this can actually be catastrophic (inc-87) ... getsentry/craft#336

BYK · 2021-12-16T07:53:22Z

relay
silent failure - not sure why

This looks like some terrible GitHub mishap. I actually got the Relay release email so my guess is either the job or the logs are corrupted. I'd file a support ticket with a link so they can investigate before the logs disappear.

bare minimum: confirm that release actually exists once publish ticket closes

I don't think you need an extra step for this as Craft should crash itself causing the job to fail. If the release is out, self-hosted release would fail anyway.

add release-blocker to the release ticket (i.e., this one), manually edit changelog on release branch, and proceed

I think this process needs to be documented and maybe even streamlined.

chadwhitacre · 2021-12-16T21:39:19Z

I'd file a support ticket

Done. I cc'd you. 👍

self-hosted release would fail anyway

Yeah, that's in fact how I noticed that relay had not succeeded (even though it seemed it had). Hopefully a rare case. Not a big action item here more just a note to self.

I think this process needs to be documented and maybe even streamlined.

getsentry/publish#710 👍

chadwhitacre · 2021-12-29T23:01:49Z

Sharing out GH support thread for posterity:

Hi Chad,

Since you are running the Docker action docker://getsentry/craft:latest, the runner will simply run docker run getsentry/craft:latest and wait for docker to exit. The runner will mark the step succeed as long as the entry process for the step exit with return code 0.

You might want to add some trace and debug for the container to see why it is exiting without properly erroring out. One thing we've noticed is the bad run seems to go on for a long time compared to the good one.

25m vs 2m

Best,

[GH support]

Could this be some system-level container killer scenario? Or a timeout?

@BYK

Even if it is a system-level timeout or reaper, it seems that they are making us responsible for the container taking so long. Leaving this here for future discoverability in case of a repeat, but I don't think we need to take any further action now.

chadwhitacre · 2021-12-29T23:06:22Z

My original post to them for context:

We saw behavior yesterday where a GitHub Action was marked successful but did not run to completion.

Here is the bad run, showing a truncated log:

[debug] [[status-provider/github]] Got status "success" for revision 91e58909b0dfeb3bbc464a12f10a2c1f714ced6b
[info] [[status-provider/github]] Revision 91e58909b0dfeb3bbc464a12f10a2c1f714ced6b has been built successfully.
[debug] [[artifact-provider/github]] Fetching artifact list for revision `91e58909b0dfeb3bbc464a12f10a2c1f714ced6b`.
[info] [[artifact-provider/github]] Fetching Github artifacts for getsentry/relay, revision 91e58909b0dfeb3bbc464a12f10a2c1f714ced6b
[debug] GET /repos/getsentry/relay/actions/artifacts?per_page=100&page=0 - 200 in 286ms
[debug] [[artifact-provider/github]] Requesting archive URL from Github...
[debug] GET /repos/getsentry/relay/actions/artifacts/126993957/zip - 200 in 20299ms
[debug] [[artifact-provider/github]] Downloading ZIP from Github artifacts...

Here is a good run for comparison, showing the continuation of the log:

[debug] [[status-provider/github]] Got status "success" for revision 2bef2ac8e891048e1f048cba1b5b7ec68d8a2192
[info] [[status-provider/github]] Revision 2bef2ac8e891048e1f048cba1b5b7ec68d8a2192 has been built successfully.
[debug] [[artifact-provider/github]] Fetching artifact list for revision `2bef2ac8e891048e1f048cba1b5b7ec68d8a2192`.
[info] [[artifact-provider/github]] Fetching Github artifacts for getsentry/relay, revision 2bef2ac8e891048e1f048cba1b5b7ec68d8a2192
[debug] GET /repos/getsentry/relay/actions/artifacts?per_page=100&page=0 - 200 in 298ms
[debug] [[artifact-provider/github]] Requesting archive URL from Github...
[debug] GET /repos/getsentry/relay/actions/artifacts/114545885/zip - 200 in 20178ms
[debug] [[artifact-provider/github]] Downloading ZIP from Github artifacts...
[info] [[artifact-provider/github]] Finished downloading.
[debug] [[artifact-provider/github]] Extracting "/tmp/craft-1IJdRJnUoFR6o" to "/tmp/craft-ruzZTh"...
[debug] [[artifact-provider/github]] Found 6 artifacts.
[info]  
[info] Available artifacts: 
┌──────────────────────────────┬──────────┬─────────┬─────────────┐
│ File Name                    │ Size     │ Updated │ ContentType │
├──────────────────────────────┼──────────┼─────────┼─────────────┤
│ relay-Darwin-x86_64          │ 30.98 MB │         │             │
├──────────────────────────────┼──────────┼─────────┼─────────────┤
│ relay-Darwin-x86_64-dsym.zip │ 58.00 MB │         │             │
├──────────────────────────────┼──────────┼─────────┼─────────────┤
│ relay-Linux-x86_64           │ 20.68 MB │         │             │
├──────────────────────────────┼──────────┼─────────┼─────────────┤
│ relay-Linux-x86_64-debug.zip │ 77.97 MB │         │             │
├──────────────────────────────┼──────────┼─────────┼─────────────┤
│ relay-Windows-x86_64-pdb.zip │ 35.34 MB │         │             │
├──────────────────────────────┼──────────┼─────────┼─────────────┤
│ relay-Windows-x86_64.exe     │ 18.52 MB │         │             │
└──────────────────────────────┴──────────┴─────────┴─────────────┘

The expected output of the run is a new release entry here. For the bad run, no release appeared on that page ...

... but! A collaborator reported that they did receive the release email for the bad run:

I actually got the Relay release email so my guess is either the job or the logs are corrupted. I'd file a support ticket with a link so they can investigate before the logs disappear.

... and here we are. :-)

This was referenced Dec 7, 2021

Continue self-hosted rename in build-related files getsentry/sentry#30483

Merged

log4j vulnerability (CVE-2021-44228) - not vulnerable #1196

Closed

chadwhitacre mentioned this issue Dec 15, 2021

Self-hosted Sentry is not vulnerable to log4shell #1203

Merged

chadwhitacre added the release-blocker Any issue open with this tag will stop CalVer releases from happening label Dec 15, 2021

chadwhitacre mentioned this issue Dec 15, 2021

Limit the number of leftovers listed getsentry/craft#335

Merged

chadwhitacre added a commit to getsentry/sentry that referenced this issue Dec 15, 2021

Manually trim changelog

5cd2287

See getsentry/self-hosted#1193

chadwhitacre mentioned this issue Dec 15, 2021

publish: getsentry/[email protected] getsentry/publish#700

Closed

4 tasks

This was referenced Dec 15, 2021

Tag more carefully in GitHub target getsentry/craft#336

Closed

Improve GitHub asset upload retry algorithm getsentry/craft#337

Open

chadwhitacre closed this as completed Dec 15, 2021

This was referenced Dec 16, 2021

Remove BYK from sentry-docs releases getsentry/sentry-docs#4506

Closed

Release 22.1.0 #1206

Closed

Start a playbook getsentry/publish#710

Open

github-actions bot locked and limited conversation to collaborators Jan 14, 2022

Release 21.12.0 #1193

Release 21.12.0 #1193

Comments

chadwhitacre commented Dec 7, 2021 • edited Loading

chadwhitacre commented Dec 15, 2021

chadwhitacre commented Dec 15, 2021 • edited Loading

chadwhitacre commented Dec 15, 2021 • edited Loading

chadwhitacre commented Dec 15, 2021

chadwhitacre commented Dec 15, 2021

Changelog

21.12.0

Support Docker Compose v2 (ongoing)

Prevent Component Drift

React to log4shell

Forum → Issues

Rename onpremise to self-hosted (ongoing)

Add support for custom DotEnv file

Various fixes & improvements

chadwhitacre commented Dec 15, 2021

chadwhitacre commented Dec 15, 2021

BYK commented Dec 15, 2021

chadwhitacre commented Dec 15, 2021

chadwhitacre commented Dec 15, 2021

chadwhitacre commented Dec 15, 2021

chadwhitacre commented Dec 15, 2021

chadwhitacre commented Dec 15, 2021

chadwhitacre commented Dec 15, 2021 • edited Loading

BYK commented Dec 15, 2021

BYK commented Dec 15, 2021

chadwhitacre commented Dec 15, 2021

chadwhitacre commented Dec 15, 2021

chadwhitacre commented Dec 15, 2021

chadwhitacre commented Dec 15, 2021

chadwhitacre commented Dec 15, 2021

chadwhitacre commented Dec 15, 2021

chadwhitacre commented Dec 15, 2021

chadwhitacre commented Dec 15, 2021

chadwhitacre commented Dec 15, 2021

chadwhitacre commented Dec 15, 2021

chadwhitacre commented Dec 15, 2021

chadwhitacre commented Dec 15, 2021

chadwhitacre commented Dec 15, 2021

chadwhitacre commented Dec 15, 2021 • edited Loading

chadwhitacre commented Dec 15, 2021

chadwhitacre commented Dec 15, 2021

chadwhitacre commented Dec 15, 2021

chadwhitacre commented Dec 15, 2021

chadwhitacre commented Dec 15, 2021

chadwhitacre commented Dec 15, 2021

chadwhitacre commented Dec 15, 2021

chadwhitacre commented Dec 15, 2021 • edited Loading

chadwhitacre commented Dec 15, 2021

chadwhitacre commented Dec 15, 2021

chadwhitacre commented Dec 15, 2021

chadwhitacre commented Dec 15, 2021 • edited Loading

Post-Mortem

BYK commented Dec 16, 2021

chadwhitacre commented Dec 16, 2021

chadwhitacre commented Dec 29, 2021

chadwhitacre commented Dec 29, 2021

chadwhitacre commented Dec 7, 2021 •

edited

Loading

chadwhitacre commented Dec 15, 2021 •

edited

Loading

chadwhitacre commented Dec 15, 2021 •

edited

Loading

chadwhitacre commented Dec 15, 2021 •

edited

Loading

chadwhitacre commented Dec 15, 2021 •

edited

Loading

chadwhitacre commented Dec 15, 2021 •

edited

Loading

chadwhitacre commented Dec 15, 2021 •

edited

Loading