Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mongo7 upgrade #140

Open
wants to merge 75 commits into
base: develop
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
75 commits
Select commit Hold shift + click to select a range
6ec3d63
mongo7 upgrade
Xiangs18 Dec 17, 2024
26b7d9e
upgrade pymongo to 4.7.2
Xiangs18 Jan 6, 2025
c5f1294
update python to 3.9.19
Xiangs18 Jan 6, 2025
1e57094
update Makefile and run_tests.sh scripts
Xiangs18 Jan 9, 2025
47ff558
update submodules
Xiangs18 Jan 9, 2025
4e1f8bf
fix gradle not found
Xiangs18 Jan 9, 2025
4e0c2dc
start nms service with docker-compose
Xiangs18 Jan 11, 2025
8095c7f
update nms submodule to the latest
Xiangs18 Jan 11, 2025
6f22c74
install docker-compose in GHA
Xiangs18 Jan 11, 2025
dff0b20
switch catalog mongo7 port to 27018
Xiangs18 Jan 11, 2025
76bff46
fix mongo port
Xiangs18 Jan 11, 2025
0b7cd2a
fix mongo port mapping
Xiangs18 Jan 11, 2025
66f2485
set admin_users
Xiangs18 Jan 11, 2025
85c65af
check admin users
Xiangs18 Jan 13, 2025
9fda21f
shut down NMS service properly
Xiangs18 Jan 13, 2025
91c72f7
fix nms curl
Xiangs18 Jan 13, 2025
dd1d2da
update mock_auth service
Xiangs18 Jan 14, 2025
2e84037
update the commit
Xiangs18 Jan 27, 2025
727753c
Update test.cfg.example
bio-boris Jan 27, 2025
77d161b
Update test.cfg.example
bio-boris Jan 27, 2025
bca78a9
Update core_registration_test.py
bio-boris Jan 27, 2025
178a63c
update message
Xiangs18 Jan 27, 2025
a3b1038
Merge branch 'dev-mongo7_upgrade' of github.com:kbase/catalog into de…
Xiangs18 Jan 27, 2025
5379082
test method_spec_admin_users
Xiangs18 Jan 28, 2025
631b52f
test simple_good_repo old commit
Xiangs18 Jan 28, 2025
b412672
test register hack
Xiangs18 Jan 28, 2025
80be977
remove jars nms submodules
Xiangs18 Jan 28, 2025
7ead4f7
update Dockerfile
Xiangs18 Jan 28, 2025
01f1539
remove kbapi_common submodule
Xiangs18 Jan 29, 2025
35a18c4
fix deprecated mongo auth
Xiangs18 Jan 29, 2025
c228438
remove all submodules in catalog
Xiangs18 Jan 29, 2025
9153235
create MongoClient through passing in params
Xiangs18 Jan 31, 2025
2344d64
add a comment for mock_auth server.py
Xiangs18 Jan 31, 2025
f147120
simplify code
Xiangs18 Jan 31, 2025
7a5b354
adjust params in MongoClient
Xiangs18 Feb 3, 2025
3e3673f
add debug log messages in list_user_favorites func
Xiangs18 Feb 4, 2025
093510e
try reinitiate mongoclient
Xiangs18 Feb 4, 2025
ec3064b
update __init__ in db.py file
Xiangs18 Feb 5, 2025
4aabfb9
fix typo
Xiangs18 Feb 5, 2025
fe87438
add check_db_schema
Xiangs18 Feb 5, 2025
5af0ccd
add boolean flag for schema check
Xiangs18 Feb 5, 2025
93e0033
collection direct call
Xiangs18 Feb 5, 2025
37f083d
remove debugging message
Xiangs18 Feb 5, 2025
d0c2522
display catalog version
Xiangs18 Feb 5, 2025
6b5fcde
fix failed unit test in startup_test.py
Xiangs18 Feb 5, 2025
0143000
trigger lazy load collection
Xiangs18 Feb 5, 2025
d04e2eb
clean up files
Xiangs18 Feb 6, 2025
59c653e
remove comments from Makefile
Xiangs18 Feb 7, 2025
0859dbe
validate Install Docker Compose step in ci
Xiangs18 Feb 8, 2025
dda9eac
use docker compose
Xiangs18 Feb 10, 2025
bbfc2db
map NMS to the local port 27018
Xiangs18 Feb 10, 2025
293953c
Add a clearer comment for the environment variable in test.yml
Xiangs18 Feb 11, 2025
7b4a495
move env to jobs level
Xiangs18 Feb 11, 2025
377f651
add more comments in server.py
Xiangs18 Feb 11, 2025
ae1b224
add more comments in db.py
Xiangs18 Feb 11, 2025
370c65e
remove check_db_schema related functions
Xiangs18 Feb 13, 2025
0015ac0
revert back changes
Xiangs18 Feb 13, 2025
1428c85
move self._db_schema_checked flag outside function
Xiangs18 Feb 13, 2025
d106e96
refactor _create_indexes function
Xiangs18 Feb 14, 2025
64f6fdb
encapsulate create_indexes in the DBIndexes class
Xiangs18 Feb 14, 2025
d6bd556
add comment at the top of server.py file
Xiangs18 Feb 15, 2025
96a72e6
refactor db.py file
Xiangs18 Feb 19, 2025
a4025ad
add flag for mongo client
Xiangs18 Feb 19, 2025
46731b1
update wrapper function
Xiangs18 Feb 19, 2025
9836af9
add initialize_mongo_client function manually
Xiangs18 Feb 19, 2025
1f6024a
add _close_mongo_client function
Xiangs18 Feb 19, 2025
4511078
try run tests without a lock
Xiangs18 Feb 20, 2025
8806f22
revert self.lock changes
Xiangs18 Feb 21, 2025
62bb424
Recreate the database and collection handles after reinitializing the…
Xiangs18 Feb 21, 2025
c919000
revert back tests
Xiangs18 Feb 21, 2025
dc79ddc
remove decorator related code
Xiangs18 Feb 21, 2025
0c7b906
add comment why need to manually close mongoclient
Xiangs18 Feb 21, 2025
9ae4b7b
update comments
Xiangs18 Feb 24, 2025
6c54bf0
refactor the db.py and add comments for docker-compose file
Xiangs18 Feb 25, 2025
26d8910
add a comment for _ensure_mongo_connection function
Xiangs18 Feb 25, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,22 +21,25 @@ jobs:
fail-fast: false
matrix:
include:
- python-version: '3.7'
- python-version: '3.9.19'
mongo-version: '3.6'
# - python-version: '3.7'
# mongo-version: '7.0.4'
- python-version: '3.9.19'
mongo-version: '7.0.4'
services:
mongo:
image: mongo:${{matrix.mongo-version}}
ports:
- 27017:27017
options: --name mongo${{matrix.mongo-version}}

env:
KBASE_CI_TOKEN: ${{ secrets.KBASE_CI_TOKEN }}
# This env var is also used in the catalog test docker compose file for starting NMS
ADMIN_USER: ${{ secrets.KBASE_BOT_USER_CI }}

steps:
- name: Repo checkout
uses: actions/checkout@v4
with:
submodules: recursive

- name: Setup Python
uses: actions/setup-python@v5
Expand All @@ -45,9 +48,6 @@ jobs:

- name: Install dependencies and set up test config
shell: bash
env:
KBASE_CI_TOKEN: ${{ secrets.KBASE_CI_TOKEN }}
ADMIN_USER: ${{ secrets.KBASE_BOT_USER_CI }}

run: |

Expand Down
12 changes: 0 additions & 12 deletions .gitmodules

This file was deleted.

4 changes: 0 additions & 4 deletions DEPENDENCIES

This file was deleted.

23 changes: 18 additions & 5 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,25 +1,38 @@
FROM kbase/sdkbase2:python AS build
FROM python:3.9.19 AS build

# The rsync installation is required for the Makefile
RUN apt-get update && apt-get install -y rsync
RUN mkdir -p /kb/deployment/lib/biokbase

COPY . /tmp/catalog
RUN cd /tmp/catalog && make deploy-service deploy-server-control-scripts

FROM kbase/sdkbase2:python
FROM python:3.9.19
# These ARGs values are passed in via the docker build command
ARG BUILD_DATE
ARG VCS_REF
ARG BRANCH

RUN apt-get update && apt-get install -y wget uwsgi

# install dockerize
WORKDIR /opt
RUN wget -q https://github.com/kbase/dockerize/raw/master/dockerize-linux-amd64-v0.6.1.tar.gz \
&& tar xvzf dockerize-linux-amd64-v0.6.1.tar.gz \
&& rm dockerize-linux-amd64-v0.6.1.tar.gz
RUN mkdir -p /kb/deployment/bin/
RUN ln -s /opt/dockerize /kb/deployment/bin/dockerize

ENV KB_DEPLOYMENT_CONFIG "/kb/deployment/conf/deploy.cfg"

COPY --from=build /kb/deployment/lib/biokbase /kb/deployment/lib/biokbase
COPY --from=build /kb/deployment/services /kb/deployment/services
COPY --from=build /tmp/catalog/deployment/conf /kb/deployment/conf

SHELL ["/bin/bash", "-c"]
WORKDIR /tmp/catalog
RUN pip install --upgrade pip
COPY requirements.txt requirements.txt
RUN source activate root && \
pip install -r requirements.txt
RUN pip install -r requirements.txt

LABEL org.label-schema.build-date=$BUILD_DATE \
org.label-schema.vcs-url="https://github.com/kbase/catalog.git" \
Expand Down
9 changes: 0 additions & 9 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -27,11 +27,6 @@ LIB_DIR = lib

PATH := kb_sdk/bin:$(PATH)

default: init

init:
git submodule init
git submodule update

compile-kb-module:
kb-sdk compile $(SPEC_FILE) \
Expand Down Expand Up @@ -102,10 +97,6 @@ setup-tests:
mkdir -p $(TESTLIB)/biokbase
mkdir -p $(TESTDIR)/nms
rsync -av lib/biokbase/* $(TESTLIB)/biokbase/. --exclude *.bak-*
rsync -av kbapi_common/lib/biokbase/* $(TESTLIB)/biokbase/.
cd narrative_method_store; make; make build-classpath-list;
# rsync -av narrative_method_store/lib/biokbase/* $(TESTLIB)/biokbase/.



test: setup-tests
Expand Down
1 change: 0 additions & 1 deletion jars
Submodule jars deleted from 7a0eda
1 change: 0 additions & 1 deletion kbapi_common
Submodule kbapi_common deleted from 65a6d7
Loading
Loading