Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: beginning of scratch and private object stores #1398

Draft
wants to merge 5 commits into
base: master
Choose a base branch
from
Draft
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 39 additions & 6 deletions templates/galaxy/config/object_store_conf.xml.j2
Original file line number Diff line number Diff line change
Expand Up @@ -231,16 +231,49 @@ The storage consists of 16 Gen6 type A200 storage nodes and 16 Gen5 type X410 no
</backend>

<!--
<backend type="generic_s3" id="s3_netapp01" allow_selection="false" weight="0" store_by="uuid" name="Short term storage for method development">
<description>S3 based objetct storage matained by the compute center of the University of Freiburg.
This storage is purged weekly and so it is only appropriate for short term methods development and such.
The rapid deletion of stored data enables us to provide this storage without a quota.
More information about our storage offerings can be found [here](https://galaxyproject.org/eu/).

Q:
- use_reduced_redundancy - is this even available in Netapp? If so, should we use it for the scratch?
- max_chunk_size of 250 is that ok?
- `s3_netapp01_private` should this be `s3_netapp02_private` or should we use a separate bucket?
- do we want to disable quota for scratch or set a high quota, let's say 10TB?

TODO:
- We need a website where we talk and describe our storage. This link needs to be adopted below "https://galaxyproject.org/eu/"
- adapt the cache path to the new flash storage

<!- - also called Scratch storage - ->
<backend type="generic_s3" id="s3_netapp01" allow_selection="true" weight="0" store_by="uuid" name="Short term storage for method development">
<description>S3-based object storage is maintained by the compute center of the University of Freiburg.
This storage, also called scratch storage, is purged monthly so it is only appropriate for short-term methods development and such.
The rapid deletion of stored data enables us to provide this storage without a quota. This storage is not backed up.
More information about our storage offerings is documented at [https://galaxyproject.org/eu/](https://galaxyproject.org/eu/).
</description>
<quota enabled="false" />
<badges>
<less_stable />
<not_backed_up />
<short_term>The data stored here is purged after a week.</short_term>
<short_term>The data stored here is purged after one month.</short_term>
</badges>
<auth access_key="{{ s3_netapp01.usegalaxy_eu.access_key }}" secret_key="{{ s3_netapp01.usegalaxy_eu.secret_key }}" />
<bucket name="{{ s3_netapp01.usegalaxy_eu.bucket_name }}" use_reduced_redundancy="False" max_chunk_size="250" />
<connection host="s3.bwsfs.uni-freiburg.de" port="443" is_secure="True" conn_path="" multipart="True"/>
<cache path="/data/jwd02f/s3_object_store_cache" size="10000" />
<extra_dir type="job_work" path="/data/jwd02f/main"/>
<extra_dir type="temp" path="/data/jwd02f/tmp"/>
</backend>

<!- - private object store - ->
<backend type="generic_s3" id="s3_netapp01_private" allow_selection="true" weight="0" store_by="uuid" private="true" name="User-private storage with additional safeguards">
<description>S3-based object storage is maintained by the compute center of the University of Freiburg.
Data on this storage cannot be used for public datasets, cannot be shared between users, etc..
All your data in Galaxy is by default only available to you and can not be seen by other users. However, you can always share data, and histories with others.
This storage prevents sharing and provides an additional safeguard for you and your data.
More information about our storage offerings can be found [here](https://galaxyproject.org/eu/).
</description>
<quota source="private_quota" />
<badges>
<more_stable />
</badges>
<auth access_key="{{ s3_netapp01.usegalaxy_eu.access_key }}" secret_key="{{ s3_netapp01.usegalaxy_eu.secret_key }}" />
<bucket name="{{ s3_netapp01.usegalaxy_eu.bucket_name }}" use_reduced_redundancy="False" max_chunk_size="250" />
Expand Down
Loading