JP-3848 MIRI LRS s_region and resample WCS

[jemorrison]

This PR replaces: #9104

MIRI LRS specwcs reference file PR has been merged:spacetelescope/stdatamodels#393

Resolves JP-3848

Closes #9072

This PR corrects the s_region for MIRI LRS output from assign_wcs and resample. In addition an error was found in the WCS of resampled MIRI LRS data.
In assign_wcs the s_region is determined using the V2,V3 corners of the slit. These values are provided by a new
reference file.
I added https://jira.stsci.edu/browse/JP-3910 to add a unit test for resample_spec to test the new function find_miri_lrs_sregion. I need some time to get this correct set the wcs in the unit test. My first attempts failed so I put it off into another JP ticket to get this PR in for the next delivery

Tasks

• request a review from someone specific, to avoid making the maintainers review every PR
• add a build milestone, i.e. Build 11.3 (use the latest build if not sure)
• Does this PR change user-facing code / API? (if not, label with no-changelog-entry-needed)
  • write news fragment(s) in changes/: echo "changed something" > changes/<PR#>.<changetype>.rst (see below for change types)
  • update or add relevant tests
  • update relevant docstrings and / or docs/ page
  • start a regression test and include a link to the running job ([click here for instructions]
    https://github.com/spacetelescope/RegressionTests/actions/runs/13705291758
    (https://github.com/spacetelescope/RegressionTests/blob/main/docs/running_regression_tests.md))
    • Do truth files need to be updated ("okified")?
      • after the reviewer has approved these changes, run okify_regtests to update the truth files
• if a JIRA ticket exists, make sure it is resolved properly

news fragment change types...

• changes/<PR#>.general.rst: infrastructure or miscellaneous change
• changes/<PR#>.docs.rst
• changes/<PR#>.stpipe.rst
• changes/<PR#>.datamodels.rst
• changes/<PR#>.scripts.rst
• changes/<PR#>.set_telescope_pointing.rst
• changes/<PR#>.pipeline.rst

stage 1

• changes/<PR#>.group_scale.rst
• changes/<PR#>.dq_init.rst
• changes/<PR#>.emicorr.rst
• changes/<PR#>.saturation.rst
• changes/<PR#>.ipc.rst
• changes/<PR#>.firstframe.rst
• changes/<PR#>.lastframe.rst
• changes/<PR#>.reset.rst
• changes/<PR#>.superbias.rst
• changes/<PR#>.refpix.rst
• changes/<PR#>.linearity.rst
• changes/<PR#>.rscd.rst
• changes/<PR#>.persistence.rst
• changes/<PR#>.dark_current.rst
• changes/<PR#>.charge_migration.rst
• changes/<PR#>.jump.rst
• changes/<PR#>.clean_flicker_noise.rst
• changes/<PR#>.ramp_fitting.rst
• changes/<PR#>.gain_scale.rst

stage 2

• changes/<PR#>.assign_wcs.rst
• changes/<PR#>.badpix_selfcal.rst
• changes/<PR#>.msaflagopen.rst
• changes/<PR#>.nsclean.rst
• changes/<PR#>.imprint.rst
• changes/<PR#>.background.rst
• changes/<PR#>.extract_2d.rst
• changes/<PR#>.master_background.rst
• changes/<PR#>.wavecorr.rst
• changes/<PR#>.srctype.rst
• changes/<PR#>.straylight.rst
• changes/<PR#>.wfss_contam.rst
• changes/<PR#>.flatfield.rst
• changes/<PR#>.fringe.rst
• changes/<PR#>.pathloss.rst
• changes/<PR#>.barshadow.rst
• changes/<PR#>.photom.rst
• changes/<PR#>.pixel_replace.rst
• changes/<PR#>.resample_spec.rst
• changes/<PR#>.residual_fringe.rst
• changes/<PR#>.cube_build.rst
• changes/<PR#>.extract_1d.rst
• changes/<PR#>.resample.rst

stage 3

• changes/<PR#>.assign_mtwcs.rst
• changes/<PR#>.mrs_imatch.rst
• changes/<PR#>.tweakreg.rst
• changes/<PR#>.skymatch.rst
• changes/<PR#>.exp_to_source.rst
• changes/<PR#>.outlier_detection.rst
• changes/<PR#>.tso_photometry.rst
• changes/<PR#>.stack_refs.rst
• changes/<PR#>.align_refs.rst
• changes/<PR#>.klip.rst
• changes/<PR#>.spectral_leak.rst
• changes/<PR#>.source_catalog.rst
• changes/<PR#>.combine_1d.rst
• changes/<PR#>.ami.rst

other

• changes/<PR#>.wfs_combine.rst
• changes/<PR#>.white_light.rst
• changes/<PR#>.cube_skymatch.rst
• changes/<PR#>.engdb_tools.rst
• changes/<PR#>.guider_cds.rst

[codecov]

Codecov Report

Attention: Patch coverage is 99.15254% with 1 line in your changes missing coverage. Please review.

Project coverage is 72.56%. Comparing base (f479eb7) to head (8a43968).
Report is 23 commits behind head on main.

Files with missing lines	Patch %	Lines
jwst/resample/resample_spec.py	94.11%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9193      +/-   ##
==========================================
+ Coverage   72.51%   72.56%   +0.05%     
==========================================
  Files         371      371              
  Lines       37092    37165      +73     
==========================================
+ Hits        26896    26970      +74     
+ Misses      10196    10195       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[drlaw1558]

@jemorrison This is looking much better, but a few thoughts:

• The across-slice positioning now looks good, both for S_REGIONs and for individual pixels. The along-slice is still a little weird though; from poking at this I think it's because the rotation that's being applied can change what the 'edge' pixels are slightly between cal and s2d frames (actual SOURCE positions look ok). One thing that I think we should fix though- in the individual s2d files it looks like the right edge of the resampled array is getting cut off; if I remove the ny = int(np.ceil(xstop)) and corresponding IF statement (see comment above) though then this looks better.
• For some reason the pipeline log says again that there are NaNs in the bounding box during the assign_wcs step and that the S_REGIONS are not being updated. As a result I can't tell whether the new S_REGION code there is doing the right thing or if it's simply not changing the S_REGION from the rate files again. I'd thought that the latest pipeline was running this recalculation though, which is what kicked off much of this work in the first place?
• I've got the new reference file ready to go as soon as the datamodels PR is merged.

[jemorrison]

@drlaw1558 are you supplying the new reference file when you run assign_wcs (spec2) in the override method ?

[drlaw1558]

@jemorrison Hm, I thought I had been but it looks like the new ref file wasn't getting picked up properly. With that fixed the assign_wcs indeed runs and results look good.

[drlaw1558]

LG2M now:

[drlaw1558]

Looks good from the science side.

[melanieclarke]

@jemorrison - there are some unit test failures that need attention when the code structure has settled. In particular, it looks like some of the tests I wrote to make sure flux is conserved are failing with the updates to the MIRI WCS.

[emolter]

This looks much cleaner now from an overall code design standpoint - this is exactly the flow I had in mind. I made a few additional small comments, but overall this looks great and I'm happy for it to be merged

[melanieclarke]

Almost there!

Regression test changes look as expected, but this regression test needs a code change for the S_REGION comparison: test_miri_lrs_slit_wcs, in test_miri_lrs_slit_spec2.py.

I spotted a couple typos on reading through, noted below.

[jemorrison]

@melanieclarke I noticed the regression test failure in the test_miri_lrs_slit_wcs, in test_miri_lrs_slit_spec2.py.
But I thought it was a truth file issue. You think it is more than that ?
I have run it local to test it with an update Truth file and it runs.
But if you think there is something I am missing I can remove it and look at it when I do JP-3910

[melanieclarke]

But I thought it was a truth file issue. You think it is more than that ?
I have run it local to test it with an update Truth file and it runs.

Oh, I think I read it wrong - it looked like it was doing a direct assertion test that would not be okify-able from the failed run. If you were able to get it to pass with an updated truth file, then that should be fine.

[melanieclarke]

All good, thanks for the hard work on this one!