Add docker multi-fs, chunker update and documentation update #17
Conversation
| echo "Current archive is full, moving to ${timestamp}_${uuid}." | ||
| mv "${CHUNKER_WORKING_DIR}/current" "${PACKING_QUEUE_DIR}/${timestamp}_${uuid}" | ||
| cur_size=0 | ||
| rm $CHUNKER_WORKING_DIR/FINISH |
| @@ -7,7 +7,17 @@ | |||
| exit 1 | |||
| } | |||
|
|
|||
| mkdir -p /data/incoming /data/chunker-work /data/packing-queue /data/packer-work-in /data/packer-work-out /data/upload-queue /data/uploader-work | |||
| if [ -d "/data2" ] | |||
There was a problem hiding this comment.
use test instead of [] shorthand form
There was a problem hiding this comment.
I don't like having /data and /data2 (optional). If we're going to do this, I would prefer an explicit /fs1 and /fs2 (even if in most cases they will point to the same place. Also reduces confusion as other parts of the factory refer to fs1 and fs2 explicitly)
| @@ -27,6 +27,16 @@ mkdir -p "${PACKING_QUEUE_DIR}" || exit 1 | |||
| mkdir -p "${CHUNKER_WORKING_DIR}/current" || exit 1 | |||
| cur_size=$( du -B1 -s "${CHUNKER_WORKING_DIR}/current" | grep -oE "^[0-9]+" ) | |||
|
|
|||
| if [ -f "$CHUNKER_WORKING_DIR/FINISH" ]; then | |||
There was a problem hiding this comment.
use ${name} instead of $name and use test instead of [] shorthand form
| DISK_LIMIT=75 - Limit in percent the target sets rsync connections to -1 (no new connections accepted) | ||
| DISK_HARD_LIMIT=90 - Limit in percent the target kills rsync to kill any existing connections | ||
| #### target-uploader | ||
| LOAD_BALANCER=s3-lb0 - Define loadbalancer you wish to connect to s3-lb1.us.archive.org becomes s3-lb1 |
There was a problem hiding this comment.
This is one of the times I'd suggest not documenting a feature. IA doesn't like this, we shouldn't be using this, but was added for convenience for large projects.
| @@ -7,7 +7,17 @@ | |||
| exit 1 | |||
| } | |||
|
|
|||
| mkdir -p /data/incoming /data/chunker-work /data/packing-queue /data/packer-work-in /data/packer-work-out /data/upload-queue /data/uploader-work | |||
| if [ -d "/data2" ] | |||
There was a problem hiding this comment.
I don't like having /data and /data2 (optional). If we're going to do this, I would prefer an explicit /fs1 and /fs2 (even if in most cases they will point to the same place. Also reduces confusion as other parts of the factory refer to fs1 and fs2 explicitly)
| Filesystems 1 and 2 do not have to be the same. | ||
|
|
||
| ### Running dual filesystems with docker | ||
| Overall running dual FS with docker is very simple and easily done. The only container that requires access to both file systems is the packer. To make this simple inside the CT filesystem 1 (fs1) is mounted at /data and fs2 is mounted at /data2. |
| FILE_PREFIX="PROJECT_" | ||
| IA_AUTH="no:thanks" | ||
| MEGABYTES_PER_CHUNK="$((25*1024))" | ||
| ZST_DICTIONARY_API="http://trackerproxy.archiveteam.org:25654/dictionary" |
There was a problem hiding this comment.
Not a huge issue, but we need to change this.
Docker will now support multiple file systems
Chunker now supports
touch FINISHto force into packing queue before minimum megawarc size is hitDocumentation refresh