Local copo instance

[jjkoehorst]

I saw COPO during the ENA meeting and wanted to see how the infrastructure and more specifically the backend looks like. This to see if it has the potential to integrate it into our systems as we are still looking for a data management / and in the future submission system.

I created a demo bash script to initialise all the random passwords needed

# Making demo directory and move into directory
mkdir demo && cd demo
# Making folders and corresponding code
mkdir copo_mongo_initdb_root_password && uuidgen | sed 's/-//g' | awk '{system("touch copo_mongo_initdb_root_password/"$1)}'
mkdir copo_mongo_user_password && uuidgen | sed 's/-//g' | awk '{system("touch copo_mongo_user_password/"$1)}'
mkdir copo_postgres_user_password && uuidgen | sed 's/-//g' | awk '{system("touch copo_postgres_user_password/"$1)}'
mkdir copo_web_secret_key && uuidgen | sed 's/-//g' | awk '{system("touch copo_web_secret_key/"$1)}'
mkdir copo_orcid_secret_key && uuidgen | sed 's/-//g' | awk '{system("touch copo_orcid_secret_key/"$1)}'
mkdir copo_figshare_consumer_secret_key && uuidgen | sed 's/-//g' | awk '{system("touch copo_figshare_consumer_secret_key/"$1)}'
mkdir copo_figshare_client_id_key && uuidgen | sed 's/-//g' | awk '{system("touch copo_figshare_client_id_key/"$1)}'
mkdir copo_figshare_client_secret_key && uuidgen | sed 's/-//g' | awk '{system("touch copo_figshare_client_secret_key/"$1)}'
mkdir copo_google_secret_key && uuidgen | sed 's/-//g' | awk '{system("touch copo_google_secret_key/"$1)}'
mkdir copo_twitter_secret_key && uuidgen | sed 's/-//g' | awk '{system("touch copo_twitter_secret_key/"$1)}'
mkdir copo_facebook_secret_key && uuidgen | sed 's/-//g' | awk '{system("touch copo_facebook_secret_key/"$1)}'
mkdir copo_webin_user && uuidgen | sed 's/-//g' | awk '{system("touch copo_webin_user/"$1)}'
mkdir copo_webin_user_password && uuidgen | sed 's/-//g' | awk '{system("touch copo_webin_user_password/"$1)}'

echo "COPO_KEYS=demo/" > ../.env.sh

Then I modified the yaml file as I do not want to build all the packages but just use the ones from the docker hub

for copo-web changed to image: copo/copo-web and image: copo/copo-mongo in copo-mongo.

Then docker-compose up and all seems fine but then copo-postgres crashes continuously...

➜  COPO git:(master) ✗ docker-compose up
Creating network "copo_default" with the default driver
Creating volume "copo_redisdata" with default driver
Creating volume "copo_mongo-data" with default driver
Creating volume "copo_postgres-data" with default driver
Pulling copo-redis (redis:latest)...
latest: Pulling from library/redis
8d691f585fa8: Pull complete
8ccd02d17190: Pull complete
4719eb1815f2: Pull complete
200531706a7d: Pull complete
eed7c26916cf: Pull complete
e1285fcc6a46: Pull complete
Digest: sha256:fe80393a67c7058590ca6b6903f64e35b50fa411b0496f604a85c526fb5bd2d2
Status: Downloaded newer image for redis:latest
Pulling copo-postgres (postgres:9.6)...
9.6: Pulling from library/postgres
80369df48736: Pull complete
b18dd0a6efec: Pull complete
5c20c5b8227d: Pull complete
c5a7f905c8ec: Pull complete
5a3f55930dd8: Pull complete
ffc097878b09: Pull complete
3106d02490d4: Pull complete
88d1fc513b8f: Pull complete
77bcb9a8971f: Pull complete
a1d54d716706: Pull complete
50a08c3d34b8: Pull complete
cda8b5ca36c7: Pull complete
643276ec28f6: Pull complete
f1653f203925: Pull complete
Digest: sha256:3d2856d01428052f1d4f6efa6ef78b15149c69e9574b9dae1488e3bed19abc4b
Status: Downloaded newer image for postgres:9.6
Pulling copo-mongo (copo/copo-mongo:)...
latest: Pulling from copo/copo-mongo
f5cc0ee7a6f6: Pull complete
d99b18c5f0ce: Pull complete
2abe504e5492: Pull complete
010336760fd5: Pull complete
1bebc569a09b: Pull complete
99973eccc29a: Pull complete
3caee68cdf37: Pull complete
2f024b88f543: Pull complete
74f877b8f67d: Pull complete
72dc91cfe502: Pull complete
d610498cfcc7: Pull complete
12630a797fb0: Pull complete
Digest: sha256:b40d9f1ff2ac8ea4a29e439df4649a88f3ed5b8204a819133560db72f7e6eca8
Status: Downloaded newer image for copo/copo-mongo:latest
Pulling copo-web (copo/copo-web:)...
latest: Pulling from copo/copo-web
f2b6b4884fc8: Pull complete
4fb899b4df21: Pull complete
74eaa8be7221: Pull complete
2d6e98fe4040: Pull complete
414666f7554d: Pull complete
5050375187a2: Pull complete
cb093a23dc23: Pull complete
679baa26509f: Pull complete
2b7d476f4eb7: Pull complete
1131e4db121e: Pull complete
d75b96cc3c5b: Pull complete
dd61eac126bc: Pull complete
faa3c7a11029: Pull complete
1d31d6c22452: Pull complete
Digest: sha256:5de2bc4e0e06999723a0aefac291f4fa0f09682dc0d0d058d3ba1983df6704ae
Status: Downloaded newer image for copo/copo-web:latest
Creating copo_copo-mongo    ... done
Creating copo_copo-redis    ... done
Creating copo_copo-postgres ... done
Creating copo_copo-web      ... done
Attaching to copo_copo-redis, copo_copo-mongo, copo_copo-postgres, copo_copo-web
copo-redis_1     | 1:C 31 Oct 2019 07:23:05.673 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
copo-redis_1     | 1:C 31 Oct 2019 07:23:05.673 # Redis version=5.0.6, bits=64, commit=00000000, modified=0, pid=1, just started
copo-redis_1     | 1:C 31 Oct 2019 07:23:05.673 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
copo-web_1       | sending incremental file list
copo-redis_1     | 1:M 31 Oct 2019 07:23:05.682 * Running mode=standalone, port=6379.
copo-redis_1     | 1:M 31 Oct 2019 07:23:05.683 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
copo-redis_1     | 1:M 31 Oct 2019 07:23:05.683 # Server initialized
copo-redis_1     | 1:M 31 Oct 2019 07:23:05.683 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
copo-redis_1     | 1:M 31 Oct 2019 07:23:05.683 * Ready to accept connections
copo_copo-mongo exited with code 1
copo_copo-postgres exited with code 1
copo-web_1       | 
copo-web_1       | sent 36.67K bytes  received 346 bytes  6.73K bytes/sec
copo-web_1       | total size is 132.90M  speedup is 3,590.03
copo_copo-mongo exited with code 1
copo_copo-postgres exited with code 1
copo_copo-postgres exited with code 1
copo_copo-mongo exited with code 1
copo_copo-mongo exited with code 1
copo_copo-postgres exited with code 1
copo_copo-mongo exited with code 1
copo_copo-postgres exited with code 1
copo_copo-mongo exited with code 1
copo_copo-postgres exited with code 1
copo-web_1       | Traceback (most recent call last):
copo-web_1       |   File "manage.py", line 9, in <module>
copo-web_1       |     execute_from_command_line(sys.argv)
copo-web_1       |   File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 371, in execute_from_command_line
copo-web_1       |     utility.execute()
copo-web_1       |   File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 317, in execute
copo-web_1       |     settings.INSTALLED_APPS
copo-web_1       |   File "/usr/local/lib/python3.5/site-packages/django/conf/__init__.py", line 56, in __getattr__
copo-web_1       |     self._setup(name)
copo-web_1       |   File "/usr/local/lib/python3.5/site-packages/django/conf/__init__.py", line 43, in _setup
copo-web_1       |     self._wrapped = Settings(settings_module)
copo-web_1       |   File "/usr/local/lib/python3.5/site-packages/django/conf/__init__.py", line 106, in __init__
copo-web_1       |     mod = importlib.import_module(self.SETTINGS_MODULE)
copo-web_1       |   File "/usr/local/lib/python3.5/importlib/__init__.py", line 126, in import_module
copo-web_1       |     return _bootstrap._gcd_import(name[level:], package, level)
copo-web_1       |   File "<frozen importlib._bootstrap>", line 985, in _gcd_import
copo-web_1       |   File "<frozen importlib._bootstrap>", line 968, in _find_and_load
copo-web_1       |   File "<frozen importlib._bootstrap>", line 957, in _find_and_load_unlocked
copo-web_1       |   File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
copo-web_1       |   File "<frozen importlib._bootstrap_external>", line 697, in exec_module
copo-web_1       |   File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
copo-web_1       |   File "/copo/web/settings/all.py", line 2, in <module>
copo-web_1       |     from .data import *
copo-web_1       |   File "/copo/web/settings/data.py", line 40, in <module>
copo-web_1       |     MONGO_CLIENT.authenticate(MONGO_USER, MONGO_USER_PASSWORD, source='admin')
copo-web_1       |   File "/usr/local/lib/python3.5/site-packages/pymongo/database.py", line 1167, in authenticate
copo-web_1       |     connect=True)
copo-web_1       |   File "/usr/local/lib/python3.5/site-packages/pymongo/mongo_client.py", line 583, in _cache_credentials
copo-web_1       |     writable_preferred_server_selector)
copo-web_1       |   File "/usr/local/lib/python3.5/site-packages/pymongo/topology.py", line 224, in select_server
copo-web_1       |     address))
copo-web_1       |   File "/usr/local/lib/python3.5/site-packages/pymongo/topology.py", line 183, in select_servers
copo-web_1       |     selector, server_timeout, address)
copo-web_1       |   File "/usr/local/lib/python3.5/site-packages/pymongo/topology.py", line 199, in _select_servers_loop
copo-web_1       |     self._error_message(selector))
copo-web_1       | pymongo.errors.ServerSelectionTimeoutError: copo-mongo:27017: [Errno -2] Name or service not known
copo-web_1       | 
copo-web_1       | sent 20 bytes  received 12 bytes  64.00 bytes/sec
copo-web_1       | total size is 0  speedup is 0.00
copo-web_1       | rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1183) [sender=3.1.1]
copo_copo-web exited with code 23
copo_copo-mongo exited with code 1
copo_copo-postgres exited with code 1
copo-web_1       | 
copo-web_1       | sent 20 bytes  received 12 bytes  64.00 bytes/sec
copo-web_1       | total size is 0  speedup is 0.00
copo-web_1       | rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1183) [sender=3.1.1]

[tonietuk]

@jjkoehorst apologies for not getting back to you on this earlier. As a first step to resolving your problem, can you share your modified yaml file and the generated keys for us to try and reproduce the issues encountered. Thanks.