Duplicate Sentry Reports for Native Client Crashes

[Atul767]

Is there an existing issue for this?

• I have checked for existing issues https://github.com/getsentry/sentry-javascript/issues
• I have reviewed the documentation https://docs.sentry.io/
• I am using the latest SDK release https://github.com/getsentry/sentry-javascript/releases

How do you use Sentry?

Sentry Saas (sentry.io)

Electron SDK Version

4.22.0

Electron Version

27.2.20

What platform are you using?

None

Link to Sentry event

https://jupiter-ct.sentry.io/issues/5977075889/events/d8cb5ff92ddd4fbb9e7bd0380f77df83/?project=1430058 , https://jupiter-ct.sentry.io/issues/5105499169/events/49718a83fe6142b4a727520fb6ce24d8/?project=4506692758077440

Steps to Reproduce

1. Launch the Native Client from the parent application
2. Trigger a crash in the Native Client.
3. Observe the crash being reported in both Sentry projects (one under the parent application, one under the Native Client).

We have observed an issue where the same crash is being reported in two separate Sentry projects:

The parent application's Sentry project (e.g., Jupiter).
The Native Client's Sentry project.
This duplication occurs when the Native Client is launched from the parent application. Both projects log the same crash as separate issues.

Example:
Here are two examples of duplicate issues for the same crash:

Parent Application Sentry Project: [MTKView draw]: Fatal Error: EXC_BREAKPOINT / EXC_ARM_BREAKPOINT / 0x100742e00 — Project: jupiter-ct.
Native Client Sentry Project: EXC_BREAKPOINT: height — Project: ringcentral-video-nc.

Expected Result

• Prevent the parent application from sending Sentry reports for crashes that originate in the Native Client.
• Ensure that Native Client crashes are reported only in the Native Client's Sentry project

Actual Result

This duplication occurs when the Native Client is launched from the parent application. Both projects log the same crash as separate issues.

[timfish]

So the parent app is the Electron app, or the native app? Is the native app using the Sentry Native SDK?

Does this only occur on macOS or all platforms?

[Atul767]

Hi @timfish ,

So the parent app is the Electron app, or the native app?
Answer: The parent app is the Electron app.

Is the native app using the Sentry Native SDK?
Answer: Yes,

Does this only occur on macOS or all platforms?
Answer: Based on the Sentry logs, I am currently seeing this issue on macOS devices. I’m not yet certain about other platforms.

[timfish]

The Sentry Electron SDK uses the Electron crashReporter API to capture native crashes and it states:

Note that if the crash reporter is started in the main process, it will automatically monitor child processes

There's no option to configure it so I don't think there's anything we can do to stop this. You may be able to custom build Electron and disable this but that would be a lot of work.

When starting your native code from Electron, you could set an environment variable that your native uses to disable crash handling?

[DamonYu6]

Hi @timfish, If we call process.crashReporter.addExtraParameter('ignore', 'true') in the child_process and then filter out such crashes in beforeSend hook, would this approach be feasible? Additionally, I’m concerned that even if we manage to filter out these events from being uploaded, would it still impact the parent project’s Crash Free Session Rate?

[timfish]

If we call process.crashReporter.addExtraParameter('ignore', 'true') in the child_process and then filter out such crashes in beforeSend hook, would this approach be feasible?

The beforeSend hook in the Electron main process will not be able to access anything like that set in the child processes. Native crashes are captured in minidumps which are passed through as attachments. Without parsing the minidump, there's no way to know what caused it or to access the child process metadata.

would it still impact the parent project’s Crash Free Session Rate?

Yes, even if you could filter in beforeSend I think these would still impact sessions.

You could switch the crash reporter off in your child process? Are you trying to avoid doing that because then all crashes will then go to the Electron app and you'd like them going to two different Sentry projects depending on the source?

[DamonYu6]

You could switch the crash reporter off in your child process?

No, we cannot disable crash reporting for the child process because the native app is maintained by a separate development team. They have already enabled the Sentry Native SDK within the native app to report crashes to their own project.

Are you trying to avoid doing that because then all crashes will then go to the Electron app and you'd like them going to two different Sentry projects depending on the source?

Since the native app runs as a child process of the Electron app, its crashes are reported by the Sentry Electron SDK to the Electron project's Sentry instance as well, inflating the project's session crash rate — something we want to avoid. Do you have any suggestions?

[timfish]

Both the Electron crashReporter and the Sentry Native SDK use crashpad to capture native crashes and it looks like by default it reports crashes from all child processes.

Do you have any suggestions?

We could add an option to ignore child process crashes from Electron but this would not be 100% reliable. It would have both false positives and false negatives.

The SDK gets events from Electron for renderer and child process crashes and we assume any minidumps found after those events correspond to the event type. If we find any minidumps at startup, we assume they are for the Electron main process because it exits immediately on crash. However, the assumptions we make are not always correct. For example, your event above that was sent from Electron has event.process: browser which means it was found at startup and we assumed it was from the main (browser) process. From what you've said, this was not the case and it came from a child process so this categorisation was wrong.

With Electron, minidumps are written by an external crashpad process and Electron notifies us but another minidump may be written by the time we read the disk. There are race conditions that we can only attempt to mitigate against.

If you look at all the native events in your Electron project that you'd like to filter out, what do they have for the event.process tag? Are they all browser or do they differ?

You could parse the minidumps and determine which process it came from. If I had to do this it would be via a native module using the minidump Rust crate and napi-rs to expose this to JavaScript. The minidump crate is able to access the main_module which should give the name of the executable. If you used this logic with beforeSend it would still impact crash free session stats but we could add another callback to help work around this.

[DamonYu6]

If you look at all the native events in your Electron project that you'd like to filter out, what do they have for the event.process tag? Are they all browser or do they differ?

Although most of the events have event.process marked as "browser," there are still a few events where event.process is marked as "renderer."
https://jupiter-ct.sentry.io/issues/5977075889/events/db3cc9a2f424430cbba863f217bd2c7f/?project=1430058

If you used this logic with beforeSend it would still impact crash free session stats but we could add another callback to help work around this.

Could you explain in detail how to add these callbacks to work around the impact on the CFSR? It seems this is the only way we have currently. Thanks

[timfish]

I've been working on #1049 which should allow us to better determine which process generated the minidump. I need to test this with the Sentry Native SDK but I'm hoping you'll be able to set process_type in the Crashpad metadata which can be picked up here.

With this I can then add an option to the Electron SDK that will allow these to be ignored while not impacting CSFR.

[timfish]

How are you starting the Electron crash reporter?

The event has mechanism: minidump which suggests that you're not using the SDK in it's default configuration.

The beforeSend callback only works if you let the Sentry SDK handle sending minidumps. If you're using the electronMinidumpIntegration or you're manually starting the Electron crashReporter, beforeSend wont be called and there's no way for th eSDK to stop sending.

[Lms24]

@Lms24 , I noticed the status was changed—could you share the reasoning? Want to make sure we're on the same page."

Just to answer this quickly: I asked Tim internally to take another look and removed the label because it's used internally to track which issues need a reply. So no worries, this was not "just removed" :) We take triaging turns within the team and delegate specific issues to the respective SDK owners or experts.

[Atul767]

How are you starting the Electron crash reporter?

@timfish If Sentry is not enabled, crashReporter.start() is used with uploadToServer: false, meaning crash reports are collected locally but not automatically uploaded.

If Sentry is enabled, Sentry.sentryMinidumpIntegration() is used to capture and process crash reports through Sentry.

[timfish]

In my testing with the following setup on macOS, events are not sent when event.process is unknown:

import * as Sentry from '@sentry/electron/main';

Sentry.init({
  dsn: '__DSN__',
  debug: true,
  beforeSend: (event) => {
    if (event.tags['event.process'] === 'unknown') {
      return null;
    }
    return event;
  }
});

Can you create a reproduction where this isn't working?

[Atul767]

In my testing with the following setup on macOS, events are not sent when event.process is unknown:

import * as Sentry from '@sentry/electron/main';

Sentry.init({
dsn: 'DSN',
debug: true,
beforeSend: (event) => {
if (event.tags['event.process'] === 'unknown') {
return null;
}
return event;
}
});
Can you create a reproduction where this isn't working?

@timfish Have you tested this setup using integrations: [Sentry.electronMinidumpIntegration()]?

As you mentioned earlier, if we use electronMinidumpIntegration, then beforeSend won’t be called.

[Atul767]

@timfish Our team provided the following steps to reproduce this issue:

Reproducible Steps:

• If there is only an NC (native client) crash, the event is not uploaded to the Electron Sentry project.
• However, if Electron has already experienced an NC crash, then Any subsequent Electron crash results in both the Electron crash and the previous NC crash being uploaded to the Electron Sentry project.

[timfish]

beforeSend will not be called if you use electronMinidumpIntegration. This is the expected behaviour because we have no control over the Electron uploader.

The beforeSend hook only works when using the default sentryMinidumpIntegration.

[Atul767]

Hi @timfish,

I tested with a demo app, and in case of a native crash, event.process is set to unknown, which works as expected. However, when testing the same scenario in our codebase, event.process is coming as renderer.

native crash sentry Link :
1.https://jupiter-ct.sentry.io/issues/6104469301/events/d2f18b6f4bb342328976a305b680895f/?project=1428302
2.https://jupiter-ct.sentry.io/issues/6236523747/events/56250c7bb6c845d8b8b4084b1a64e0cd/?project=1428302

I'm not sure why this discrepancy is occurring. Do you have any insights on what could be causing this difference?

[timfish]

The event.process value is pulled directly from the minidump metadata that is written by crashpad:

sentry-electron/src/main/integrations/sentry-minidump/minidump-loader.ts

Lines 173 to 180 in 4a7798a

	/**
	* Crashpad includes it's own custom stream in the minidump file that can include metadata. Electron uses this to
	* include details about the app and process that caused the crash.
	*
	* Rather than parse the minidump by reading the header and parsing through all the streams, we can just look for the
	* 'process_type' key and then pick the string that comes after that.
	*/
	function getMinidumpProcessType(buffer: Buffer): string | undefined {

If event.process === 'renderer' then I would be very surprised if the minidump came from any process other than a renderer.

[timfish]

Looking through the stacktraces from the above linked renderer events I see a couple of strange things that make it look like these crashes didn't come from Electron. For example, I can see the bottom of the stack does start in your Electron app, but I see a lot of frames which mention WebKit. Is your app starting Safari/Webkit? If so, which process are you running WebKit from? main, renderer or your own spawned process?

I have no idea how Electrons crashpad decides to classify these as renderer crashes!

[Atul767]

WebKit is running from a separately spawned process, which is launched by Electron. It is not running directly in the Electron main or renderer process.

[timfish]

The stacktrace I saw looked like it was going through both Electron and WebKit!