Merge pull request #116 from hubmapconsortium/yuanzhou/versioning

v1.1.1 release - Add versioning, conf file mount, update docs, and deployment script

.gitignore

@@ -17,6 +17,9 @@ scripts/.python-version
 scripts/robot
 server/.openapi-generator
 
+# config file
+server/openapi_server/resources/app.properties
+
 # log files in any path
 **/*.log
 

README.md

@@ -2,41 +2,68 @@
 
 The HuBMAP Ontology API contains a Neo4j graph loaded with the [UMLS](https://www.nlm.nih.gov/research/umls/index.html).  The UMLS allows extensive cross-referencing across many biomedical vocabulary systems.  The UMLS structure will form the basis for the rest of the ontologies added into this system.  The backbone of the UMLS data is the Concept to Concept relationship (aka the UMLS CUI to CUI).  The UMLS Concept provides an anchor for the various vocabularies and ontologies.  The user can "walk" backbone of the graph in a vocabulary agnostic manner and/or branch off into specific voabularies.
 
-There are three pieces/containers here:
-* The neo4j server that contains the onthology which is build with .csv files on startup. The .csv files are not stored in the git repo because some are covered by licenses. The .csv files will be deleted when the server is built to save disk space on the server.
-* Constraints need to be added to the data in the neo4j database, but that can only be done after it is up and running. The neo4j-constraints container will wait for the neo4j server to start to take commands and then run a batch of constraints to 'fixup' the databse making it ready to be used.
-* The server container is a RESTful API server that is created by the 'build-serer.sh' script from the 'ontology-x.x.x.yml' OpenAPI specification file. There is an additional 'server/openapi_server/controllers/neo4j_manager.py' that is used to define neo4j queries that are used by the endpoints in a one-to-one manner. From this YAML file it is also possible to create clients for the server that can be included in programs acessing the serer.
+There are two containers here:
 
-## Local Deployment Instructions
+* The `ontology-neo4j` container that contains the ontology which is build with .csv files on startup. The .csv files are not stored in the git repo because some are covered by licenses. The .csv files will be deleted when the server is built to save disk space on the server.
 
-* Create an 'import' directory under the 'neo4j' that contains the CSV files that should be imported into the neo4j database by the 'neo4j-admin' program in the Docker file.
-* Run the file 'docker-compose build --no-cache' which will create the three containers described above.
-* After the containers are built, run 'docker-compose up' to start all three containers. When the 'neo4j-constraints' container has finished the database will have been build.
+* The `ontology-api` container is a RESTful API server that is created by the 'build-serer.sh' script from the 'ontology-x.x.x.yml' OpenAPI specification file. There is an additional 'server/openapi_server/controllers/neo4j_manager.py' that is used to define neo4j queries that are used by the endpoints in a one-to-one manner. From this YAML file it is also possible to create clients for the server that can be included in programs acessing the serer.
 
-## Use it Instructions
+## Local Development Instructions
 
-* The URL 'http://<<host>>:8080/ui/' will render an interface from which the OPENapi endpoints can be viewed and tested.
+For local development, [HuBMAP Gateway](https://github.com/hubmapconsortium/gateway) is not needed, simply following the below steps:
 
-## Remote Deployment
+* Create an `import` directory under the `neo4j` that contains the CSV files that should be imported into the neo4j database by the 'neo4j-admin' program in the Docker file.
+* Under the project root directory, run command `docker-compose -f docker-compose.localhost.yml build --no-cache` which will create the two container images described above.
+* After the images are built, run `docker-compose -f docker-compose.localhost.yml up` to start the two containers.
 
-### Deploy Neo4j services
 
-On the VM where the `ontology-neo4j` and `ontology-neo4j-constraints` services will be running:
+Once the containers are ready, they can be accessed at:
 
-````
-docker-compose -f docker-compose.deployment.neo4j.yml up -d
-````
+- `ontology-api`: `http://localhost:8080/`
+- `ontology-neo4j`: `http://localhost:7474` and the default username is `neo4j` with password `HappyG0at`
+
+
+## Remote Deployment on DEV and PROD
+
+When deploying the ontology services on the DEV and PROD, we'll use the `ontology-api` docker image that has been pre-built and published to DockerHub: https://hub.docker.com/r/hubmap/ontology-api. And the `ontology-api` will also be running behind the [HuBMAP Gateway](https://github.com/hubmapconsortium/gateway). The `ontology-neo4j` image can not be published to DockerHub, so we'll always need to build it on the deployment VM. 
+
+### Publish ontology-api docker image
+
+First we need to build a released version of the `ontology-api` image locally, specify the `latest` tag as well as the new version tag based on the version number in `VERSION` file, for example:
+
+```
+cd server
+docker build -t hubmap/ontology-api:latest -t hubmap/ontology-api:1.1.1 .
+```
+
+Then publish this image with the the `latest` tag as well as the released version tag to DockerHub:
+
+```
+docker login
+docker push hubmap/ontology-api:latest
+docker push hubmap/ontology-api:1.1.1
+```
 
 ### Deploy ontology-api
 
-First need to configure `ontology-api/server/openapi_server/resources/app.properties` with the correct neo4j connection information, in order to connect to the `ontology-neo4j` container running on another VM.
+First we need to create a new configuration file `ontology-api/server/openapi_server/resources/app.properties` with the correct neo4j connection information, in order to connect to the `ontology-neo4j` container running on another VM.
 
 On the VM where the `ontology-api` service will be running:
 
 ````
-docker-compose -f docker-compose.deployment.api.yml up -d
+Usage: ./ontology-api-docker-deployment.sh [dev|prod] [start|stop|down]
 ````
 
+### Deploy ontology-neo4j
+
+On the VM where the `ontology-neo4j` service will be running, have the CSV files placed under `neo4j/import/current`, then run
+
+````
+docker-compose -f docker-compose.deployment.neo4j.yml up -d
+````
+
+This will create the docker image and spin up the neo4j container as well, the process will take some time.
+
 ## Data persistence
 
 The ontology data is being mounted from the `ontology-neo4j` container to the host VM using named volume: `ontology-neo4j-data`, which is defiened in the `docker-compose.deployment.neo4j.yml`. 

VERSION

@@ -0,0 +1 @@
+1.1.1

docker-compose.deployment.api.yml

@@ -5,7 +5,9 @@ services:
   ontology-api:
     hostname: ontology-api
     container_name: ontology-api
-    build: server
+    # Use the published image from DockerHub instead of local build
+    # Exit with an error message containing err if ONTOLOGY_API_VERSION is unset or empty in the environment
+    image: hubmap/ontology-api:${ONTOLOGY_API_VERSION:?err}
     # Avoid accidentally creating zombie processes
     init: true
     # Specifying a restart policy to avoid downtime
@@ -14,8 +16,10 @@ services:
       - HOST_GID=${HOST_GID:-1000}
       - HOST_UID=${HOST_UID:-1000}
     volumes:
-      # Mount the logging to container
+      # Persist the logging
       - "./server/log:/usr/src/app/log"
+      # Mount the neo4j connection file
+      - "./server/openapi_server/resources/app.properties:/usr/src/app/openapi_server/resources/app.properties"
     networks:
       - gateway_hubmap
 

docker-compose.deployment.neo4j.yml

@@ -5,6 +5,7 @@ services:
   ontology-neo4j:
     hostname: ontology-neo4j
     container_name: ontology-neo4j
+    # Will not publish the neo4j image to DockerHub
     build: neo4j
     environment:
       - NEO4J_USER=${NEO4J_USER:-neo4j}

docker-compose.yml → docker-compose.localhost.yml

@@ -1,8 +1,8 @@
-version: '3.8'
+version: '3.7'
 # https://docs.docker.com/compose/compose-file/compose-file-v3/
 
 services:
-  neo4j:
+  ontology-neo4j:
     container_name: ontology-neo4j
     build: neo4j
     restart: unless-stopped
@@ -29,39 +29,34 @@ services:
     # -XX:MaxRAMPercentage (double) is depending on the max memory limit assigned to the contaienr
     # When the container has > 1G memory, set -XX:MaxRAMPercentage=75.0 is good (doesn't waste too many resources)
     environment:
+      - NEO4J_USER=${NEO4J_USER:-neo4j}
+      # Used for setting the initial neo4j password, defaut to HappyG0at
+      - NEO4J_PASSWORD=${NEO4J_PASSWORD:-HappyG0at}
+      # Allow the JVM to read cgroup limits
+      # -XX:+UseContainerSupport is enabled by default on linux machines,
+      # this feature was introduced in java10 then backported to Java-8u191, the base image comes with OpenJDK(build 1.8.0_232-b09)
+      # -XX:MaxRAMPercentage (double) is depending on the max memory limit assigned to the contaienr
+      # When the container has > 1G memory, set -XX:MaxRAMPercentage=75.0 is good (doesn't waste too many resources)
       - _JAVA_OPTIONS=-XX:+UseContainerSupport -XX:MaxRAMPercentage=75.0
-    networks:
-      - neo4j-network
 
-  neo4j-constraints:
-    container_name: neo4j-constraints
-    build: neo4j-constraints
-    deploy:
-      resources:
-        limits:
-          memory: 256M
-    networks:
-      - neo4j-network
-    depends_on:
-      - neo4j
-
-  onthology:
-    container_name: onthology_server
+  ontology-api:
+    container_name: ontology-api
     build: server
     restart: unless-stopped
+    environment:
+      - HOST_GID=${HOST_GID:-1000}
+      - HOST_UID=${HOST_UID:-1000}
+    volumes:
+      # Persist the logging
+      - "./server/log:/usr/src/app/log"
     deploy:
       resources:
         limits:
           memory: 256M
     ports:
       - 8080:8080
-    networks:
-      - neo4j-network
-    depends_on:
-      - neo4j-constraints
+
 
 volumes:
   neo4j-data:
 
-networks:
-  neo4j-network:

ontology-api-docker-deployment.sh

@@ -0,0 +1,49 @@
+#!/bin/bash
+
+# Print a new line and the banner
+echo
+echo "==================== ONTOLOGY-API ===================="
+
+# This function sets DIR to the directory in which this script itself is found.
+# Thank you https://stackoverflow.com/questions/59895/how-to-get-the-source-directory-of-a-bash-script-from-within-the-script-itself                                                                      
+function get_dir_of_this_script () {
+    SCRIPT_SOURCE="${BASH_SOURCE[0]}"
+    while [ -h "$SCRIPT_SOURCE" ]; do # resolve $SCRIPT_SOURCE until the file is no longer a symlink
+        DIR="$( cd -P "$( dirname "$SCRIPT_SOURCE" )" >/dev/null 2>&1 && pwd )"
+        SCRIPT_SOURCE="$(readlink "$SCRIPT_SOURCE")"
+        [[ $SCRIPT_SOURCE != /* ]] && SCRIPT_SOURCE="$DIR/$SCRIPT_SOURCE" # if $SCRIPT_SOURCE was a relative symlink, we need to resolve it relative to the path where the symlink file was located
+    done
+    DIR="$( cd -P "$( dirname "$SCRIPT_SOURCE" )" >/dev/null 2>&1 && pwd )"
+    echo 'DIR of script:' $DIR
+}
+
+# Set the version environment variable for the docker build
+# Version number is from the VERSION file
+# Also remove newlines and leading/trailing slashes if present in that VERSION file
+function export_version() {
+    export ONTOLOGY_API_VERSION=$(tr -d "\n\r" < VERSION | xargs)
+    echo "ONTOLOGY_API_VERSION: $ONTOLOGY_API_VERSION"
+}
+
+
+if [[ "$1" != "dev" && "$1" != "prod" ]]; then
+    echo "Unknown build environment '$1', specify one of the following: dev|prod"
+else
+    if [[ "$2" != "start" && "$2" != "stop" && "$2" != "down" ]]; then
+        echo "Unknown command '$2', specify one of the following: start|stop|down"
+    else
+        # Show the script dir
+        get_dir_of_this_script
+
+        # Export and show the version
+        export_version
+
+        if [ "$2" = "start" ]; then
+            docker-compose -f docker-compose.deployment.api.yml -p ontology-api up -d
+        elif [ "$2" = "stop" ]; then
+            docker-compose -f docker-compose.deployment.api.yml -p ontology-api stop
+        elif [ "$2" = "down" ]; then
+            docker-compose -f docker-compose.deployment.api.yml -p ontology-api down
+        fi
+    fi
+fi

ontology-openapi3.yaml

@@ -2,7 +2,7 @@ openapi: 3.0.3
 info:
   title: HuBMAP Ontology API
   description: This document describes the HuBMAP Ontology API
-  version: 0.1.0
+  version: 1.1.1
 servers:
   - url: https://ontology-api.dev.hubmapconsortium.org/
 paths:

server/openapi_server/resources/app.properties → server/openapi_server/resources/app.properties.example

@@ -1,4 +1,4 @@
 [neo4j]
-Server = bolt://18.205.215.12:7688
+Server = bolt://localhost:7687
 Username = neo4j
 Password = HappyG0at