Add resource handler #178
Merged
Add resource handler #178
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Bleep bloop, I am a robot. This is embarrassing... the cd /path/to/your/repository/
git checkout mmootor-resource-handler
git fetch origin
git merge origin/main
git push origin |
|
Successful NFTest run on an F16 - |
|
Bump here, can you fix the tests and make sure this is working? @Faizal-Eeman |
|
Bleep bloop, I am a robot. Alas, some of the Nextflow configuration tests failed! test/configtest-F16.json@ ["params","proc_resource_params"]
- {"call_gCNV_Delly":{"cpus":"1","memory":"8 GB","retry_strategy":{"memory":{"operand":"2","strategy":"exponential"}}},"call_gSV_Delly":{"cpus":"1","memory":"8 GB","retry_strategy":{"memory":{"operand":"2","strategy":"exponential"}}},"call_gSV_Manta":{"cpus":"6","ext":{"retry_codes":[]},"memory":"8 GB","retry_strategy":{"memory":{"operand":"2","strategy":"exponential"}}},"convert_gCNV_BCF2VCF_BCFtools":{"cpus":"1","memory":"1 GB","retry_strategy":{"memory":{"operand":"2","strategy":"exponential"}}},"convert_gSV_BCF2VCF_BCFtools":{"cpus":"1","memory":"1 GB","retry_strategy":{"memory":{"operand":"2","strategy":"exponential"}}},"run_gCNV_vcf_validator_VCFtools":{"cpus":"1","memory":"1 GB","retry_strategy":{"memory":{"operand":"2","strategy":"exponential"}}},"run_gCNV_vcfstats_RTGTools":{"cpus":"1","memory":"1 GB","retry_strategy":{"memory":{"operand":"2","strategy":"exponential"}}},"run_gSV_vcf_validator_VCFtools":{"cpus":"1","memory":"1 GB","retry_strategy":{"memory":{"operand":"2","strategy":"exponential"}}},"run_gSV_vcfstats_RTGTools":{"cpus":"1","memory":"1 GB","retry_strategy":{"memory":{"operand":"2","strategy":"exponential"}}},"run_validate_PipeVal":{"cpus":"1","memory":"1 GB"}}
@ ["params","base_allocations"]
+ {"call_gCNV_Delly":{"cpus":"1","memory":"8 GB"},"call_gSV_Delly":{"cpus":"1","memory":"8 GB"},"call_gSV_Manta":{"cpus":"6","memory":"8 GB"},"convert_gCNV_BCF2VCF_BCFtools":{"cpus":"1","memory":"1 GB"},"convert_gSV_BCF2VCF_BCFtools":{"cpus":"1","memory":"1 GB"},"run_gCNV_vcf_validator_VCFtools":{"cpus":"1","memory":"1 GB"},"run_gCNV_vcfstats_RTGTools":{"cpus":"1","memory":"1 GB"},"run_gSV_vcf_validator_VCFtools":{"cpus":"1","memory":"1 GB"},"run_gSV_vcfstats_RTGTools":{"cpus":"1","memory":"1 GB"},"run_validate_PipeVal":{"cpus":"1","memory":"1 GB"}}
@ ["params","max_time"]
+ "24d 20h 31m 24s"
@ ["params","min_time"]
+ "1s"
@ ["params","retry_information"]
+ {"call_gCNV_Delly":{"memory":{"operand":"2","strategy":"exponential"}},"call_gSV_Delly":{"memory":{"operand":"2","strategy":"exponential"}},"call_gSV_Manta":{"memory":{"operand":"2","strategy":"exponential"}},"convert_gCNV_BCF2VCF_BCFtools":{"memory":{"operand":"2","strategy":"exponential"}},"convert_gSV_BCF2VCF_BCFtools":{"memory":{"operand":"2","strategy":"exponential"}},"run_gCNV_vcf_validator_VCFtools":{"memory":{"operand":"2","strategy":"exponential"}},"run_gCNV_vcfstats_RTGTools":{"memory":{"operand":"2","strategy":"exponential"}},"run_gSV_vcf_validator_VCFtools":{"memory":{"operand":"2","strategy":"exponential"}},"run_gSV_vcfstats_RTGTools":{"memory":{"operand":"2","strategy":"exponential"}}}
@ ["resource_handler"]
+ {"handle_resources":"closure()","set_resource_limit_params":"closure()","set_retry":"closure()","setup_retry":"closure()"}If the above changes are surprising, stop and determine what happened. If the above changes are expected, there are two ways to fix this:
|
|
/fix-tests |
|
Bleep bloop, I am a robot. I have updated all of the failing tests for you with a015ad1. You must review my work before merging this pull request! |
Wiz Scan Summary
To detect these findings earlier in the dev lifecycle, try using Wiz Code VS Code Extension. |
…179) Bumps the pipeline-submodules group with 2 updates in the / directory: [external/pipeline-Nextflow-module](https://github.com/uclahs-cds/pipeline-Nextflow-module) and [external/pipeline-Nextflow-config](https://github.com/uclahs-cds/pipeline-Nextflow-config). Updates `external/pipeline-Nextflow-module` from `1cfa9fa` to `b1d7a78` - [Commits](uclahs-cds/pipeline-Nextflow-module@1cfa9fa...b1d7a78) Updates `external/pipeline-Nextflow-config` from `a4e99e0` to `07c99b8` - [Release notes](https://github.com/uclahs-cds/pipeline-Nextflow-config/releases) - [Commits](uclahs-cds/pipeline-Nextflow-config@a4e99e0...07c99b8) --- updated-dependencies: - dependency-name: external/pipeline-Nextflow-module dependency-type: direct:production dependency-group: pipeline-submodules - dependency-name: external/pipeline-Nextflow-config dependency-type: direct:production dependency-group: pipeline-submodules ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
|
NFTest log - /hot/software/pipeline/pipeline-call-gSV/Nextflow/development/pre-5.2.0/mmootor-resource-handler/sbatch-nftest-20250305T220929Z.log |
@yashpatel6 This PR is ready. |
yashpatel6
approved these changes
Mar 6, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Add generalized resource handler.
Closes #177
Testing Results
sample: HG003 downsampled
NFTest YAML: /hot/code/mmootor/GitHub/uclahs-cds/pipeline-call-gSV/nftest.yaml
Manta
Delly - gSV & gCNV
Checklist
I have read the code review guidelines and the code review best practice on GitHub check-list.
I have reviewed the Nextflow pipeline standards.
The name of the branch is meaningful and well formatted following the standards, using [AD_username (or 5 letters of AD if AD is too long)]-[brief_description_of_branch].
I have set up or verified the branch protection rule following the github standards before opening this pull request.
I have added my name to the contributors listings in the
manifestblock in thenextflow.configas part of this pull request, am listedalready, or do not wish to be listed. (This acknowledgement is optional.)
I have added the changes included in this pull request to the
CHANGELOG.mdunder the next release version or unreleased, and updated the date.I have updated the version number in the
metadata.yamlandmanifestblock of thenextflow.configfile following semver, or the version number has already been updated. (Leave it unchecked if you are unsure about new version number and discuss it with the infrastructure team in this PR.)I have tested the pipeline on at least one A-mini sample with
run_delly = true,run_manta = true,run_qc = true. Forrun_delly = true, I have tested 'variant_type' set togSV,gCNV, and both. The paths to the test config files and output directories are captured above in the Testing Results section.