Dark Current Reference Files Should be Stored as Rates not Differences

[stscijgbot-jp]

Issue JP-3906 was created on JIRA by Michael Regan:

The current method of storing the dark current reference files is to store it as a cube with each plane containing the difference between samples. This was a conservative method that was decided on early in the development of the pipeline.

With our current knowledge it is now clear that this is inefficient in that it requires references files with as many groups as the maximum allowed in an integration. These differences are almost completely constant for most science quality pixels. In addition, to actually measure the differences requires significant calibration time. Cosmic rays break up ramps leading to decreasing samples up the ramp. This requires a large amounts of calibration time to derive values that are redundant.

The cleanest method it to just store the dark current measurement as a rate. Several instruments actually find the rate first and then convert into the required difference format.

It is possible that the early groups in a dark are not linear. MIRI addresses this by having a Reset Anomaly reference file. This is stored as differences that are added to the dark rate values. The number of groups in this file can be adjusted to the read mode but will be significantly smaller than the current dark reference files.

Current NIRSpec darks take up 6 GB. Requiring these files to be downloaded by users wastes a lot of time. Using rate values would reduce the size by a factor of 200.   

[stscijgbot-jp]

Comment by Jane Morrison on JIRA:

If we switch to storing the rates in the dark reference file then for MIRI we will need to go back and derive a reset anomaly correction and use that to correct the N groups in an integration. Currently the MIRI dark contains the reset anomaly effects. 

[stscijgbot-jp]

Comment by David Law on JIRA:

How many groups does the reset anomaly correction go for?

[stscijgbot-jp]

Comment by Jane Morrison on JIRA:

I would estimate about 20 full groups - more groups for subarray data . Maybe the way to create the reset anomaly reference file is  to take current dark and subtract it from the dark rate and the difference is the reset anomaly up to N frames, where N frames can be determined from what is above noise. 

 

[stscijgbot-jp]

Comment by David Law on JIRA:

Could we work out precisely how long we needed in order to ensure that the reset anomaly correction was complete?  I'm imagining if the pipeline code could handle darks of length N groups; up to N it would do a direct subtraction, and above N it would extrapolate in some way.  Perhaps bast on the last group difference?  E.g., assume that the difference between N-1 and N is the same as between N and N+1, N+1 and N+2, etc?

That might be a simple code change that would then punt the latter stage of implementation to the individual instrument teams to deliver new dark files with whatever number of groups they felt were necessary.  And for subarray darks where the files are already pretty small it might just not be worth delivering updated files.

[stscijgbot-jp]

Comment by Kevin Volk on JIRA:

I think it is difficult to determine at what point-if any-the dark ramps are well approximated by using an extrapolation with the ramp slope image in NIRISS, and presumably also in the other NIR instruments using the 5 micron cut-off detectors. The 2.5 micron cut-off detectors are different because the dark signal is too low to detect as I understand things. The read noise per group and the total noise per group continue to change as the number of groups on the ramp is increased, and this is of concern for science data when the signal level is low. For NIRISS darks and ramps in general there are some obvious effects early in the ramp that are not well described by a linear fit to the sample values. These are seen in the mean dark ramps. I do not think that we have a good idea of how much variability there is in these effects in the different dark ramps for any given pixel.

I am not sure what sort of equivalent of the MIRI reset anomaly correction could be defined for NIRISS darks. I also do not know off the top of my head how we can quantify whether this change adversely affects the science data quality.

If we really feel that we can dispense with the group-by-group dark subtraction then the best approach would be to subtract the dark rate image from the science observation rate image, and not do a group-by-group subtraction of the rate image multiplied by the frame time prior to the ramp slope step.

If the issue is just the fact that for NIRSpec the dark reference files are very long, would there be any way to arrange the the user could specify the required length of the dark reference file for their observation, and then have CRDS reduce the dark ramp size to match before returning it? It may be that this is too big a departure from the architecture of CRDS and the pipeline, but if it could be done that would alleviate the problem somewhat.

[stscijgbot-jp]

Comment by David Law on JIRA:

Kevin Volk Interesting; my understanding was that at least some of the NIR instruments were measuring average dark currents, and multiplying that up in order to produce the current cube-style dark frames.  Also tagging Bryan Hilbert and Christian Hayes to bring NIRCam/NIRSpec into this discussion.

[stscijgbot-jp]

Comment by Kevin Volk on JIRA:

Other detectors may have less in the way of effects early in the ramp than is the case for the NIRISS detector, but I doubt that they can entirely escape them. The kTC noise at least will be there for the first frame and affect the frame 2 - frame 1 signal level. We see this fairly clearly when we take single-frame ramps in the SOSS or AMI modes.

[stscijgbot-jp]

Comment by Bryan Hilbert on JIRA:

As Kevin mentioned, for the SW NIRCam detectors, the dark rate is so low that readnoise, persistence, 1/f, etc make it hard to get a good SNR measurement up the ramp. Our dark reference files for those detectors currently are full of zeros in all groups, with the exception of warm/hot pixels.

For the LW detectors, we do provide group-by-group reference files where all the pixels have non-zero values. These are constructed group by group from stacked ramps, rather than extrapolating from rate images. I haven't actually explored how linear the dark signals are up the ramp within these fiies, and whether there are any reset-related effects that might be making things non-linear ealy in the ramps.

[stscijgbot-jp]

Comment by Christian Hayes on JIRA:

From the NIRSpec side, most pixels use projected dark rates, but because we don't use IPC corrections in the pipeline, neighbors of hot pixels often have non-linear "dark" signal that we subtract off with the group-by-group dark counts/differences rather than projected rates.  The hot pixels themselves are flagged, but their neighbors are not currently (and in a number of cases can be recovered in this way), so the group-by-group darks are needed to correct these pixels and avoid them appearing as outliers/warm pixels.

Using projected dark rates without flagging these pixels at the moment would lead to a significant increase in outliers (which we would like to avoid), but if there is a good way to improve this, it would of course be nice to reduce reference file volume.