JP-3682: split outlier detection into separate steps for each mode

.github/pull_request_template.md

@@ -84,7 +84,11 @@ This PR addresses ...
 - ``changes/<PR#>.tweakreg.rst``
 - ``changes/<PR#>.skymatch.rst``
 - ``changes/<PR#>.exp_to_source.rst``
-- ``changes/<PR#>.outlier_detection.rst``
+- ``changes/<PR#>.outlier_detection_coron.rst``
+- ``changes/<PR#>.outlier_detection_ifu.rst``
+- ``changes/<PR#>.outlier_detection_imaging.rst``
+- ``changes/<PR#>.outlier_detection_spec.rst``
+- ``changes/<PR#>.outlier_detection_tso.rst``
 - ``changes/<PR#>.tso_photometry.rst``
 - ``changes/<PR#>.stack_refs.rst``
 - ``changes/<PR#>.align_refs.rst``

changes/8722.outlier_detection_coron.rst

@@ -0,0 +1 @@
+Split outlier detection into multiple steps for different modes

changes/8722.outlier_detection_ifu.rst

@@ -0,0 +1 @@
+Split outlier detection into multiple steps for different modes

changes/8722.outlier_detection_imaging.rst

@@ -0,0 +1 @@
+Split outlier detection into multiple steps for different modes

changes/8722.outlier_detection_spec.rst

@@ -0,0 +1 @@
+Split outlier detection into multiple steps for different modes

changes/8722.outlier_detection_tso.rst

@@ -0,0 +1 @@
+Split outlier detection into multiple steps for different modes

docs/jwst/data_products/science_products.rst

@@ -354,7 +354,7 @@ source.
 Cosmic-Ray flagged data: ``crf`` and ``crfints``
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 Several of the stage 3 pipelines, such as :ref:`calwebb_image3 <calwebb_image3>` and
-:ref:`calwebb_spec3 <calwebb_spec3>`, include the :ref:`outlier detection <outlier_detection_step>`
+:ref:`calwebb_spec3 <calwebb_spec3>`, include an outlier detection
 step, which finds and flags outlier pixel values within calibrated images. The results of this
 process have the identical format and content as the input ``cal`` and ``calints`` products.
 The only difference is that the DQ arrays have been updated to contain CR flags. If the inputs

docs/jwst/outlier_detection/outlier_detection.rst

@@ -0,0 +1,74 @@
+Outlier Detection Transition Guide
+==================================
+
+As of jwst release 1.18, the outlier detection step has been replaced by
+separate steps for the various observing modes. This page serves as a transition
+guide for migrating to use the new steps.
+
+The new steps and the exposure types processed through those steps are:
+
+.. list-table:: Outlier Detection Steps
+   :header-rows: 1
+
+   * - Step
+     - Exposure Types
+   * - `OutlierDetectionCoronStep`
+     - 'MIR_LYOT', 'MIR_4QPM', 'NRC_CORON'
+   * - `OutlierDetectionIFUStep`
+     - 'MIR_MRS', 'NRS_IFU'
+   * - `OutlierDetectionImagingStep`
+     - 'FGS_IMAGE', 'MIR_IMAGE', 'NRC_IMAGE', 'NIS_IMAGE'
+   * - `OutlierDetectionSpecStep`
+     - 'MIR_LRS-FIXEDSLIT', 'NRS_FIXEDSLIT', 'NRS_MSASPEC'
+   * - `OutlierDetectionTSOStep`
+     - 'MIR_LRS-SLITLESS', 'NRC_TSGRISM', 'NIS_SOSS', 'NRS_BRIGHTOBJ', 'NRC_TSIMAGE'
+
+Command-line syntax
+-------------------
+
+To run the individual step on its own, use the following command line syntax:
+
+::
+
+   strun outlier_detection_coron input.fits
+
+instead of the previous syntax:
+
+::
+
+   strun outlier_detection input.fits # NO LONGER WORKS
+
+
+The new steps have the same optional arguments as the original step, but only
+the ones relevant to that observing mode.
+For example, the `OutlierDetectionCoronStep` has the
+`--save_intermediate_results`, `--good_bits`, `--snr`, and `--maskpt` arguments,
+but no longer has the `--kernel` argument, which didn't have any effect
+for coronagraphic observations in the first place.
+
+Running the new steps within a pipeline works the same as before. To set options
+for the new steps within a pipeline, use the `--steps.<step_name>` syntax, e.g.
+
+::
+
+   strun calwebb_image3 --steps.outlier_detection_coron.save_intermediate_results=True input.fits
+
+Note that for `calwebb_spec3`, the outlier detection step is different for IFU data than
+for slit-like data. For IFU data, use `--steps.outlier_detection_ifu`, and for fixed slit
+and MSA data, use `--steps.outlier_detection_spec`.
+
+Python syntax
+-------------
+The new steps can be run in Python using the following syntax:
+
+::
+
+   from jwst.outlier_detection_coron import OutlierDetectionCoronStep
+   OutlierDetectionCoronStep.call(input.fits, save_intermediate_results=True)
+
+The new pipelines can be run in Python using, e.g.,:
+
+::
+
+   from jwst.pipeline import Image3Pipeline
+   Image3Pipeline.call(input.fits, steps={'outlier_detection_coron': {'save_intermediate_results': True}})

docs/jwst/outlier_detection_coron/arguments_coron.rst

@@ -0,0 +1,32 @@
+.. _outlier_detection_coron_step_args:
+
+Step Arguments
+==============
+
+The outlier detection step for coronagraphic data has the following
+optional arguments that control the behavior of the processing.
+For more details about step arguments (including datatypes, possible values
+and defaults) see :py:obj:`jwst.outlier_detection.OutlierDetectionCoronStep.spec`.
+
+``--save_intermediate_results``
+  Specifies whether or not to save any intermediate products created
+  during step processing.
+
+``--good_bits``
+  The DQ bit values from the input image DQ arrays
+  that should be considered 'good'. Any pixel with a DQ value not included
+  in this value (or list of values) will be ignored when resampling and flagged
+  when building the weight mask. See DQ flag :ref:`dq_parameter_specification` for details.
+
+``--snr``
+  The signal-to-noise values to use for bad pixel identification.
+  Since cosmic rays often extend across several pixels, the user
+  must specify two cut-off values for determining whether a pixel should
+  be masked: the first for detecting the primary cosmic ray, and the
+  second (typically lower threshold) for masking lower-level bad pixels
+  adjacent to those found in the first pass.  Valid values are a pair of
+  floating-point values in a single string (for example "5.0 4.0").
+
+``--maskpt``
+  The percent of maximum weight to use as lower-limit for valid data;
+  valid values go from 0.0 to 1.0.

docs/jwst/outlier_detection_coron/index.rst

@@ -0,0 +1,13 @@
+.. _outlier_detection_coron_step:
+
+========================================
+Outlier Detection for Coronagraphic Data
+========================================
+
+.. toctree::
+   :maxdepth: 2
+
+   main.rst
+   arguments_coron.rst
+
+.. automodapi:: jwst.outlier_detection.outlier_detection_coron_step

docs/jwst/outlier_detection/outlier_detection_coron.rst → docs/jwst/outlier_detection_coron/main.rst

@@ -1,14 +1,16 @@
 .. _outlier-detection-coron:
 
-Coronagraphic Data
-==================
+Overview
+========
 
-This module serves as the interface for applying ``outlier_detection`` to coronagraphic
-image observations. A :py:class:`~jwst.datamodels.CubeModel` serves as the basic format
-for all processing performed by this step. This routine performs the following operations:
+This module serves as the interface for detecting outliers in coronagraphic
+image observations, namely those taken in the following modes:
+'MIR_LYOT', 'MIR_4QPM', 'NRC_CORON'.
+
+This routine performs the following operations:
 
 #. Extract parameter settings from input model and merge them with any user-provided values.
-   See :ref:`outlier detection arguments <outlier_detection_step_args>` for the full list
+   See :ref:`arguments <outlier_detection_coron_step_args>` for the full list
    of parameters.
 
 #. Do not attempt resampling; data are assumed to be aligned and have an identical WCS.
@@ -28,3 +30,5 @@ for all processing performed by this step. This routine performs the following o
 
 #. Update DQ arrays with flags and set SCI, ERR, and variance arrays to NaN at the location
    of identified outliers.
+
+``OutlierDetectionCoronStep`` does not use any reference files.

docs/jwst/outlier_detection_ifu/arguments_ifu.rst

@@ -0,0 +1,26 @@
+.. _outlier_detection_ifu_step_args:
+
+Step Arguments
+==============
+
+The outlier detection step for IFU data has the following
+optional arguments that control the behavior of the processing.
+For more details about step arguments (including datatypes, possible values
+and defaults) see :py:obj:`jwst.outlier_detection.OutlierDetectionIFUStep.spec`.
+
+``--save_intermediate_results``
+  Specifies whether or not to save any intermediate products created
+  during step processing.
+
+``--kernel_size``
+  The size of the kernel to use to normalize the pixel differences. The kernel size
+  must only contain odd values. Valid values are a pair of ints in a single string
+  (for example "7 7", the default).
+
+``--threshold_percent``
+  The threshold (in percent) of the normalized minimum pixel difference used to identify bad pixels.
+  Pixels with a normalized minimum difference above this percentage are flagged as outliers.
+
+``--ifu_second_check``
+  Perform a secondary check searching for outliers. This will set outliers
+  where ever the difference array of adjacent pixels is a Nan.

docs/jwst/outlier_detection_ifu/index.rst

@@ -0,0 +1,13 @@
+.. _outlier_detection_ifu_step:
+
+==============================
+Outlier Detection for IFU Data
+==============================
+
+.. toctree::
+   :maxdepth: 2
+
+   main.rst
+   arguments_ifu.rst
+
+.. automodapi:: jwst.outlier_detection.outlier_detection_ifu_step

docs/jwst/outlier_detection/outlier_detection_ifu.rst → docs/jwst/outlier_detection_ifu/main.rst

@@ -1,10 +1,13 @@
 .. _outlier-detection-ifu:
 
-Integral Field Unit (IFU) Data
-==============================
+Overview
+========
 
-This module serves as the interface for applying ``outlier_detection`` to IFU
-observations, like those taken with NIRSpec and MIRI. A :ref:`Stage 3 association <asn-level3-techspecs>`,
+This module serves as the interface for detecting outliers in IFU
+observations, namely those taken in the following modes:
+'MIR_MRS', 'NRS_IFU'.
+
+A :ref:`Stage 3 association <asn-level3-techspecs>`,
 which is loaded into a :py:class:`~jwst.datamodels.ModelContainer` object,
 serves as the basic format for all processing performed by this step.
 
@@ -22,7 +25,7 @@ by the detector PSF.
 This routine performs the following operations:
 
 #. Extract parameter settings for the input ModelContainer and merge them with any user-provided values.
-   See :ref:`outlier detection arguments <outlier_detection_step_args>` for the full list of parameters.
+   See :ref:`arguments <outlier_detection_ifu_step_args>` for the full list of parameters.
 
 #. Loop over cal files, computing nearest-neighbor differences for each pixel
    in the along-dispersion direction.
@@ -46,3 +49,5 @@ This routine performs the following operations:
 
 #. Update DQ arrays with flags and set SCI, ERR, and variance arrays to NaN at the location
    of identified outliers.
+
+``OutlierDetectionIFUStep`` does not use any reference files.