Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DM-49274: Stop sky objects appearing in NO_DATA regions #1056

Merged
merged 4 commits into from
Mar 25, 2025
Merged

DM-49274: Stop sky objects appearing in NO_DATA regions #1056

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
4c91db3
Reorder calexpFlags PixelFlags alphabetically
leeskelvin Mar 24, 2025
e56e629
Add PixelFlags_flag_nodata to schemas
leeskelvin Mar 24, 2025
69b127a
Update photoCal bad flags list
leeskelvin Mar 19, 2025
97f4b7c
Add flag_nodata column to v695833-e0-c000.xy.fits
leeskelvin Mar 19, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions python/lsst/pipe/tasks/photoCal.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,7 @@ def setDefaults(self):
self.match.sourceSelection.doFlags = True
self.match.sourceSelection.flags.bad = [
"base_PixelFlags_flag_edge",
"base_PixelFlags_flag_nodata",
"base_PixelFlags_flag_interpolated",
"base_PixelFlags_flag_saturated",
]
Expand Down
11 changes: 6 additions & 5 deletions schemas/ForcedSource.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,15 +85,16 @@ calexpFlags:
- base_LocalWcs_CDMatrix_2_2
- base_LocalBackground_instFlux
- base_LocalBackground_instFluxErr
- base_PixelFlags_flag_bad
- base_PixelFlags_flag_cr
- base_PixelFlags_flag_crCenter
- base_PixelFlags_flag_edge
- base_PixelFlags_flag_interpolated
- base_PixelFlags_flag_saturated
- base_PixelFlags_flag_cr
- base_PixelFlags_flag_bad
- base_PixelFlags_flag_suspect
- base_PixelFlags_flag_interpolatedCenter
- base_PixelFlags_flag_nodata
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please also add base_PixelFlags_flag_nodataCenter and then @jrmullaney won't have to

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest doing this in all the schemas where other flags have "center" variants

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure adding it in this block will work. Aren't these pulling flags from the calexp, rather than the diff? My understanding is that the nodata flag that I want in the forcedSourceTable comes from the diff catalog, not the calexp. See here.
Having said that, if @leeskelvin wants to add that in a block above (as in that link-to PR), then I could delete my PR and have it all done in one go.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is probably true, but I still think we should add it here! I forgot this schema was for the merged forcedSource catalog, and not for the one upstream a step or two before the merge happens, which doesn't care what flavor of exposure it is run on.

Copy link
Contributor Author

@leeskelvin leeskelvin Mar 24, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't mind adding it here, and the 4 other schema files at the same time. Troubled by this though:

I'm not sure adding it in this block will work.

If I add *nodataCenter into all 5 schemas, will something break?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That feels like a "be sure to run ci_*" flavored question 😅

When @jrmullaney said he wasn't sure it would work, I think he meant it might not let him do the downstream selection/filtering on merged forced source tables to the extent we want, not that it would break anything.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added base_PixelFlags_flag_nodataCenter to the five schema YAMLs in this repo. Currently running Jenkins with:

lsst_distrib lsst_ci lsst_middleware lsst_sitcom ci_cpp ci_hsc ci_imsim ci_middleware

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After discussion with @jrmullaney, I've removed this column. The column needed is different than a straight-forward propagation, and requires more thought and testing. @jrmullaney will follow this up on DM-49554.

- base_PixelFlags_flag_saturated
- base_PixelFlags_flag_saturatedCenter
- base_PixelFlags_flag_crCenter
- base_PixelFlags_flag_suspect
- base_PixelFlags_flag_suspectCenter
# Streak flags not yet propagated from compareWarp (DM-46947, DM-43077).
# - base_PixelFlags_flag_streak
Expand Down
1 change: 1 addition & 0 deletions schemas/Object.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -917,6 +917,7 @@ flags:
- base_PixelFlags_flag_inexact_psfCenter
- base_PixelFlags_flag_interpolated
- base_PixelFlags_flag_interpolatedCenter
- base_PixelFlags_flag_nodata
- base_PixelFlags_flag_offimage
- base_PixelFlags_flag_saturated
- base_PixelFlags_flag_saturatedCenter
Expand Down
1 change: 1 addition & 0 deletions schemas/PreSource.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -284,6 +284,7 @@ flags:
- base_PixelFlags_flag_edge
- base_PixelFlags_flag_interpolated
- base_PixelFlags_flag_interpolatedCenter
- base_PixelFlags_flag_nodata
- base_PixelFlags_flag_offimage
- base_PixelFlags_flag_saturated
- base_PixelFlags_flag_saturatedCenter
Expand Down
1 change: 1 addition & 0 deletions schemas/Source.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -370,6 +370,7 @@ flags:
- base_PixelFlags_flag_edge
- base_PixelFlags_flag_interpolated
- base_PixelFlags_flag_interpolatedCenter
- base_PixelFlags_flag_nodata
- base_PixelFlags_flag_offimage
- base_PixelFlags_flag_saturated
- base_PixelFlags_flag_saturatedCenter
Expand Down
1 change: 1 addition & 0 deletions schemas/initial_stars_detector_standardized.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,7 @@ flags:
- base_PixelFlags_flag_edge
- base_PixelFlags_flag_interpolated
- base_PixelFlags_flag_interpolatedCenter
- base_PixelFlags_flag_nodata
- base_PixelFlags_flag_offimage
- base_PixelFlags_flag_saturated
- base_PixelFlags_flag_saturatedCenter
Expand Down
66 changes: 37 additions & 29 deletions tests/data/v695833-e0-c000.xy.fits
View file
Open in desktop

Large diffs are not rendered by default.