Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DM-49175: Adjust configs/warnings for LSSTCam dense PTC #301

Merged
merged 15 commits into from
Apr 3, 2025
Merged

DM-49175: Adjust configs/warnings for LSSTCam dense PTC #301

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
c3e5c01
Add ampwise-averaged combined calibration frame CalibCombineTask outp…
Alex-Broughton Mar 11, 2025
b7d1204
Change CTI solve errors to warnings if no exposures left after clip.
Alex-Broughton Mar 16, 2025
110e5d1
Change mean combined calib output to a config option
Alex-Broughton Mar 16, 2025
e48d3b5
Move submit path to SCRATCH in bps templates
Alex-Broughton Mar 16, 2025
5b6b51d
Add PTC outlier rejection based on maximum residual sigma vs signal g…
Alex-Broughton Mar 18, 2025
b4f7ae3
Lower sigmaCutPtcOutliers
Alex-Broughton Mar 18, 2025
85b7ff5
Add clustering by exposure for all isr task pipelines
Alex-Broughton Mar 19, 2025
b8a5d89
Fix typo in cpBfkSolve pipeline step name
Alex-Broughton Mar 20, 2025
0466ad9
Undo typo fix
Alex-Broughton Mar 20, 2025
8025bb2
Add to user collections
Alex-Broughton Mar 20, 2025
d05be02
Adjust configs for LSSTCam dense PTC
Alex-Broughton Mar 20, 2025
c5ff93a
Change f-strings to %-style
Alex-Broughton Mar 20, 2025
6206bb9
Remove TAG collection from user calibs
Alex-Broughton Mar 20, 2025
4c6cf56
Fix typos move old variable names to proper format
Alex-Broughton Mar 20, 2025
3ddcf84
Add doManualDefects = true to LSSTCam cpDefects pipelines
Alex-Broughton Apr 1, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions bps/templates/bps_bfk.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,18 @@
pipelineYaml: "${CP_PIPE_DIR}/pipelines/${INSTRUMENT}/cpBfk.yaml"

# Necessary to cluster by exposure if there are many input exposures
clusterAlgorithm: lsst.ctrl.bps.quantum_clustering_funcs.dimension_clustering
cluster:
cpBfkIsrByExposure:
pipetasks: cpBfkIsr
dimensions: exposure
cpBfkPtcExtractPairByExposure:
pipetasks: cpBfkPtcExtractPair
dimensions: exposure

project: "${TICKET}"
campaign: "${TICKET}"
submitPath: "${SCRATCH}/submit/{outputRun}"

payload:
payloadName: "${INSTRUMENT}_${TICKET}_bfk"
Expand Down
8 changes: 8 additions & 0 deletions bps/templates/bps_bias.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,15 @@
pipelineYaml: "${CP_PIPE_DIR}/pipelines/${INSTRUMENT}/cpBias.yaml"

# Necessary to cluster by exposure if there are many input exposures
clusterAlgorithm: lsst.ctrl.bps.quantum_clustering_funcs.dimension_clustering
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a two sentence explanation for this? My understanding was that this was needed for the PTC extract step to ensure that things get paired correctly, but I didn't think we needed it for ISR processing as well. Is it as simple as "makes it go faster"?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having a short explanation of the clustering configs would be great, yes! But I do know that Jim Chiang told us "use these when doing any LSSTCam ISR tasks".

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's there because ISR is so fast that HTCondor can't keep up when there are a lot of input exposures, and many quanta running in parallel. While other pipelines won't have as many input exposures as the PTC and linearizer pipelines, this is there to force it to cluster by exposure and specifically "slow it down" (it's not changing the total run time, just moving from many fast quanta to a small number of slower quanta). It also makes bps report faster since it only needs walk through less sub directories in submit/

I'll add a comment to the bps templates

cluster:
cpBiasIsrByExposure:
pipetasks: cpBiasIsr
dimensions: exposure

project: "${TICKET}"
campaign: "${TICKET}"
submitPath: "${SCRATCH}/submit/{outputRun}"

payload:
payloadName: "${INSTRUMENT}_${TICKET}_bias"
Expand Down
10 changes: 9 additions & 1 deletion bps/templates/bps_biasBootstrap.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,16 @@ pipelineYaml: "${CP_PIPE_DIR}/pipelines/${INSTRUMENT}/cpBiasBootstrap.yaml"

project: "${TICKET}"
campaign: "${TICKET}"
submitPath: "${SCRATCH}/submit/{outputRun}"

biasBootstrapCollection: "${USER_CALIB_PREFIX}$INSTRUMENT/calib/$TICKET/$TAG/biasBootstrapGen.$RERUN"
# Necessary to cluster by exposure if there are many input exposures
clusterAlgorithm: lsst.ctrl.bps.quantum_clustering_funcs.dimension_clustering
cluster:
cpBiasBootstrapIsrByExposure:
pipetasks: cpBiasBootstrapIsr
dimensions: exposure

biasBootstrapCollection: "${USER_CALIB_PREFIX}$INSTRUMENT/calib/$TICKET/$TAG/biasBootstrapGen.${RERUN}"
biasBootstrapRun: "{biasBootstrapCollection}/run${BOOTSTRAP_RUN_NUMBER}"

payload:
Expand Down
8 changes: 8 additions & 0 deletions bps/templates/bps_cti.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,14 @@ pipelineYaml: "${CP_PIPE_DIR}/pipelines/${INSTRUMENT}/cpCti.yaml"

project: "${TICKET}"
campaign: "${TICKET}"
submitPath: "${SCRATCH}/submit/{outputRun}"

# Necessary to cluster by exposure if there are many input exposures
clusterAlgorithm: lsst.ctrl.bps.quantum_clustering_funcs.dimension_clustering
cluster:
cpCtiIsrByExposure:
pipetasks: cpCtiIsr
dimensions: exposure

payload:
payloadName: "${INSTRUMENT}_${TICKET}_cti"
Expand Down
8 changes: 8 additions & 0 deletions bps/templates/bps_dark.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,14 @@ pipelineYaml: "${CP_PIPE_DIR}/pipelines/${INSTRUMENT}/cpDark.yaml"

project: "${TICKET}"
campaign: "${TICKET}"
submitPath: "${SCRATCH}/submit/{outputRun}"

# Necessary to cluster by exposure if there are many input exposures
clusterAlgorithm: lsst.ctrl.bps.quantum_clustering_funcs.dimension_clustering
cluster:
cpDarkIsrByExposure:
pipetasks: cpDarkIsr
dimensions: exposure

payload:
payloadName: "${INSTRUMENT}_${TICKET}_dark"
Expand Down
12 changes: 10 additions & 2 deletions bps/templates/bps_darkBootstrap.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,17 @@ pipelineYaml: "${CP_PIPE_DIR}/pipelines/${INSTRUMENT}/cpDarkBootstrap.yaml"

project: "${TICKET}"
campaign: "${TICKET}"
submitPath: "${SCRATCH}/submit/{outputRun}"

darkBootstrapCollection: "${USER_CALIB_PREFIX}$INSTRUMENT/calib/$TICKET/$TAG/darkBootstrapGen.$RERUN"
biasBootstrapRun: "${USER_CALIB_PREFIX}$INSTRUMENT/calib/$TICKET/$TAG/biasBootstrapGen.$RERUN/run${BOOTSTRAP_RUN_NUMBER}"
# Necessary to cluster by exposure if there are many input exposures
clusterAlgorithm: lsst.ctrl.bps.quantum_clustering_funcs.dimension_clustering
cluster:
cpDarkBootstrapIsrByExposure:
pipetasks: cpDarkBootstrapIsr
dimensions: exposure

darkBootstrapCollection: "${USER_CALIB_PREFIX}$INSTRUMENT/calib/$TICKET/$TAG/darkBootstrapGen.${RERUN}"
biasBootstrapRun: "${USER_CALIB_PREFIX}$INSTRUMENT/calib/$TICKET/$TAG/biasBootstrapGen.${RERUN}/run${BOOTSTRAP_RUN_NUMBER}"
darkBootstrapRun: "{darkBootstrapCollection}/run${BOOTSTRAP_RUN_NUMBER}"

payload:
Expand Down
7 changes: 4 additions & 3 deletions bps/templates/bps_defects.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,11 @@ pipelineYaml: "${CP_PIPE_DIR}/pipelines/${INSTRUMENT}/cpDefects.yaml"

project: "${TICKET}"
campaign: "${TICKET}"
submitPath: "${SCRATCH}/submit/{outputRun}"

biasBootstrapRun: "${USER_CALIB_PREFIX}$INSTRUMENT/calib/$TICKET/$TAG/biasBootstrapGen.$RERUN/run${BOOTSTRAP_RUN_NUMBER}"
darkBootstrapRun: "${USER_CALIB_PREFIX}$INSTRUMENT/calib/$TICKET/$TAG/darkBootstrapGen.$RERUN/run${BOOTSTRAP_RUN_NUMBER}"
flatBootstrapRun: "${USER_CALIB_PREFIX}$INSTRUMENT/calib/$TICKET/$TAG/flatBootstrapGen.$RERUN/run${BOOTSTRAP_RUN_NUMBER}"
biasBootstrapRun: "${USER_CALIB_PREFIX}$INSTRUMENT/calib/$TICKET/$TAG/biasBootstrapGen.${RERUN}/run${BOOTSTRAP_RUN_NUMBER}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do any of the $RERUN variables elsewhere need braces as well? I see above in the bps_darkBootstrap.yaml file that we have $RERUN. Similarly in the bps_flatBootstrap.yaml.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that the braces are optional, but I meant to standardize on using it (like the quotes) in which case $TICKET/$TAG should be updated as well (here and below)

darkBootstrapRun: "${USER_CALIB_PREFIX}$INSTRUMENT/calib/$TICKET/$TAG/darkBootstrapGen.${RERUN}/run${BOOTSTRAP_RUN_NUMBER}"
flatBootstrapRun: "${USER_CALIB_PREFIX}$INSTRUMENT/calib/$TICKET/$TAG/flatBootstrapGen.${RERUN}/run${BOOTSTRAP_RUN_NUMBER}"

payload:
payloadName: "${INSTRUMENT}_${TICKET}_defects"
Expand Down
14 changes: 11 additions & 3 deletions bps/templates/bps_flatBootstrap.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,18 @@ pipelineYaml: "${CP_PIPE_DIR}/pipelines/${INSTRUMENT}/cpFlatBootstrap.yaml"

project: "${TICKET}"
campaign: "${TICKET}"
submitPath: "${SCRATCH}/submit/{outputRun}"

flatBootstrapCollection: "${USER_CALIB_PREFIX}$INSTRUMENT/calib/$TICKET/$TAG/flatBootstrapGen.$RERUN"
biasBootstrapRun: "${USER_CALIB_PREFIX}$INSTRUMENT/calib/$TICKET/$TAG/biasBootstrapGen.$RERUN/run${BOOTSTRAP_RUN_NUMBER}"
darkBootstrapRun: "${USER_CALIB_PREFIX}$INSTRUMENT/calib/$TICKET/$TAG/darkBootstrapGen.$RERUN/run${BOOTSTRAP_RUN_NUMBER}"
# Necessary to cluster by exposure if there are many input exposures
clusterAlgorithm: lsst.ctrl.bps.quantum_clustering_funcs.dimension_clustering
cluster:
cpFlatBootstrapIsrByExposure:
pipetasks: cpFlatBootstrapIsr
dimensions: exposure

flatBootstrapCollection: "${USER_CALIB_PREFIX}$INSTRUMENT/calib/$TICKET/$TAG/flatBootstrapGen.${RERUN}"
biasBootstrapRun: "${USER_CALIB_PREFIX}$INSTRUMENT/calib/$TICKET/$TAG/biasBootstrapGen.${RERUN}/run${BOOTSTRAP_RUN_NUMBER}"
darkBootstrapRun: "${USER_CALIB_PREFIX}$INSTRUMENT/calib/$TICKET/$TAG/darkBootstrapGen.${RERUN}/run${BOOTSTRAP_RUN_NUMBER}"
flatBootstrapRun: "{flatBootstrapCollection}/run${BOOTSTRAP_RUN_NUMBER}"

payload:
Expand Down
8 changes: 8 additions & 0 deletions bps/templates/bps_flat_g.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,14 @@ pipelineYaml: "${CP_PIPE_DIR}/pipelines/${INSTRUMENT}/cpFlat.yaml"

project: "${TICKET}"
campaign: "${TICKET}"
submitPath: "${SCRATCH}/submit/{outputRun}"

# Necessary to cluster by exposure if there are many input exposures
clusterAlgorithm: lsst.ctrl.bps.quantum_clustering_funcs.dimension_clustering
cluster:
cpFlatIsrByExposure:
pipetasks: cpFlatIsr
dimensions: exposure

payload:
payloadName: "${INSTRUMENT}_${TICKET}_flat_g"
Expand Down
8 changes: 8 additions & 0 deletions bps/templates/bps_flat_i.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,14 @@ pipelineYaml: "${CP_PIPE_DIR}/pipelines/${INSTRUMENT}/cpFlat.yaml"

project: "${TICKET}"
campaign: "${TICKET}"
submitPath: "${SCRATCH}/submit/{outputRun}"

# Necessary to cluster by exposure if there are many input exposures
clusterAlgorithm: lsst.ctrl.bps.quantum_clustering_funcs.dimension_clustering
cluster:
cpFlatIsrByExposure:
pipetasks: cpFlatIsr
dimensions: exposure

payload:
payloadName: "${INSTRUMENT}_${TICKET}_flat_i"
Expand Down
8 changes: 8 additions & 0 deletions bps/templates/bps_flat_r.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,14 @@ pipelineYaml: "${CP_PIPE_DIR}/pipelines/${INSTRUMENT}/cpFlat.yaml"

project: "${TICKET}"
campaign: "${TICKET}"
submitPath: "${SCRATCH}/submit/{outputRun}"

# Necessary to cluster by exposure if there are many input exposures
clusterAlgorithm: lsst.ctrl.bps.quantum_clustering_funcs.dimension_clustering
cluster:
cpFlatIsrByExposure:
pipetasks: cpFlatIsr
dimensions: exposure

payload:
payloadName: "${INSTRUMENT}_${TICKET}_flat_r"
Expand Down
8 changes: 8 additions & 0 deletions bps/templates/bps_flat_u.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,14 @@ pipelineYaml: "${CP_PIPE_DIR}/pipelines/${INSTRUMENT}/cpFlat.yaml"

project: "${TICKET}"
campaign: "${TICKET}"
submitPath: "${SCRATCH}/submit/{outputRun}"

# Necessary to cluster by exposure if there are many input exposures
clusterAlgorithm: lsst.ctrl.bps.quantum_clustering_funcs.dimension_clustering
cluster:
cpFlatIsrByExposure:
pipetasks: cpFlatIsr
dimensions: exposure

payload:
payloadName: "${INSTRUMENT}_${TICKET}_flat_u"
Expand Down
8 changes: 8 additions & 0 deletions bps/templates/bps_flat_y.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,14 @@ pipelineYaml: "${CP_PIPE_DIR}/pipelines/${INSTRUMENT}/cpFlat.yaml"

project: "${TICKET}"
campaign: "${TICKET}"
submitPath: "${SCRATCH}/submit/{outputRun}"

# Necessary to cluster by exposure if there are many input exposures
clusterAlgorithm: lsst.ctrl.bps.quantum_clustering_funcs.dimension_clustering
cluster:
cpFlatIsrByExposure:
pipetasks: cpFlatIsr
dimensions: exposure

payload:
payloadName: "${INSTRUMENT}_${TICKET}_flat_y"
Expand Down
8 changes: 8 additions & 0 deletions bps/templates/bps_flat_z.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,14 @@ pipelineYaml: "${CP_PIPE_DIR}/pipelines/${INSTRUMENT}/cpFlat.yaml"

project: "${TICKET}"
campaign: "${TICKET}"
submitPath: "${SCRATCH}/submit/{outputRun}"

# Necessary to cluster by exposure if there are many input exposures
clusterAlgorithm: lsst.ctrl.bps.quantum_clustering_funcs.dimension_clustering
cluster:
cpFlatIsrByExposure:
pipetasks: cpFlatIsr
dimensions: exposure

payload:
payloadName: "${INSTRUMENT}_${TICKET}_flat_z"
Expand Down
8 changes: 8 additions & 0 deletions bps/templates/bps_illuminationCorrection.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,14 @@ pipelineYaml: "${CP_PIPE_DIR}/pipelines/${INSTRUMENT}/cpIlluminationCorrection.y

project: "${TICKET}"
campaign: "${TICKET}"
submitPath: "${SCRATCH}/submit/{outputRun}"

# Necessary to cluster by exposure if there are many input exposures
clusterAlgorithm: lsst.ctrl.bps.quantum_clustering_funcs.dimension_clustering
cluster:
cpIlluminationCorrectionIsrByExposure:
pipetasks: cpIlluminationCorrectionIsr
dimensions: exposure

payload:
payloadName: "${INSTRUMENT}_${TICKET}_illuminationCorrection"
Expand Down
11 changes: 11 additions & 0 deletions bps/templates/bps_linearizer.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,17 @@ pipelineYaml: "${CP_PIPE_DIR}/pipelines/${INSTRUMENT}/cpLinearizer.yaml"

project: "${TICKET}"
campaign: "${TICKET}"
submitPath: "${SCRATCH}/submit/{outputRun}"

# Necessary to cluster by exposure if there are many input exposures
clusterAlgorithm: lsst.ctrl.bps.quantum_clustering_funcs.dimension_clustering
cluster:
cpLinearizerIsrByExposure:
pipetasks: cpLinearizerIsr
dimensions: exposure
cpLinearizerPtcExtractPairByExposure:
pipetasks: cpLinearizerPtcExtractPair
dimensions: exposure

payload:
payloadName: "${INSTRUMENT}_${TICKET}_linearizer"
Expand Down
11 changes: 11 additions & 0 deletions bps/templates/bps_ptc.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,17 @@ pipelineYaml: "${CP_PIPE_DIR}/pipelines/${INSTRUMENT}/cpPtc.yaml"

project: "${TICKET}"
campaign: "${TICKET}"
submitPath: "${SCRATCH}/submit/{outputRun}"

# Necessary to cluster by exposure if there are many input exposures
clusterAlgorithm: lsst.ctrl.bps.quantum_clustering_funcs.dimension_clustering
cluster:
cpPtcIsrByExposure:
pipetasks: cpPtcIsr
dimensions: exposure
cpPtcExtractPairByExposure:
pipetasks: cpPtcExtractPair
dimensions: exposure

payload:
payloadName: "${INSTRUMENT}_${TICKET}_ptc"
Expand Down
7 changes: 7 additions & 0 deletions pipelines/LSSTCam/cpBfk.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,10 @@ tasks:
doExtractPhotodiodeData: true
auxiliaryHeaderKeys: ["CCOBCURR", "CCOBFLUX", "TEMP6"]
matchExposuresType: FLUX
cpBfkPtcSolve:
class: lsst.cp.pipe.ptc.PhotonTransferCurveSolveTask
config:
sigmaCutPtcOutliers: 3.0
maxDeltaInitialPtcOutlierFit: 1_000
maxSignalInitialPtcOutlierFit: 30_000
scaleMaxSignalInitialPtcOutlierFit: False
2 changes: 2 additions & 0 deletions pipelines/LSSTCam/cpDefects.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,4 +26,6 @@ tasks:
cpMergeDefects:
class: lsst.cp.pipe.MergeDefectsCombinedTask
config:
connections.inputManualDefects: "manual_defects"
edgesAsDefects: true
doManualDefects: true
7 changes: 7 additions & 0 deletions pipelines/LSSTCam/cpLinearizer.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,13 @@ tasks:
doExtractPhotodiodeData: true
auxiliaryHeaderKeys: ["CCOBCURR", "CCOBFLUX", "TEMP6"]
matchExposuresType: FLUX
cpLinearizerPtcSolve:
class: lsst.cp.pipe.ptc.PhotonTransferCurveSolveTask
config:
sigmaCutPtcOutliers: 3.0
maxDeltaInitialPtcOutlierFit: 1_000
maxSignalInitialPtcOutlierFit: 30_000
scaleMaxSignalInitialPtcOutlierFit: False
cpLinearizerSolve:
class: lsst.cp.pipe.LinearitySolveTask
config:
Expand Down
7 changes: 7 additions & 0 deletions pipelines/LSSTCam/cpPtc.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,10 @@ tasks:
doExtractPhotodiodeData: true
auxiliaryHeaderKeys: ["CCOBCURR", "CCOBFLUX", "TEMP6"]
matchExposuresType: FLUX
cpPtcSolve:
class: lsst.cp.pipe.ptc.PhotonTransferCurveSolveTask
config:
sigmaCutPtcOutliers: 3.0
maxDeltaInitialPtcOutlierFit: 1_000
maxSignalInitialPtcOutlierFit: 30_000
scaleMaxSignalInitialPtcOutlierFit: False
Loading