Mongo7 upgrade #140

Xiangs18 · 2024-12-17T22:50:16Z

This PR does the following:

Removed all submodules (jars, kbapi_common, nms).
Added the NMS Docker service for local testing.
Fixed a bug in mock_auth/servicer.py.
Updated the catalog Docker build script.
Implemented lazy loading for MongoDB.
Upgraded mongo and Cleaned up files in the repository.

Note: Next PR will add retryWrites and update release notes.

codecov · 2024-12-17T22:54:31Z

Codecov Report

Attention: Patch coverage is 95.81152% with 8 lines in your changes missing coverage. Please review.

Project coverage is 80.95%. Comparing base (dd630ce) to head (26d8910).

Files with missing lines	Patch %	Lines
lib/biokbase/catalog/db.py	95.81%	8 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #140      +/-   ##
===========================================
+ Coverage    80.14%   80.95%   +0.80%     
===========================================
  Files            8        8              
  Lines         2695     2846     +151     
===========================================
+ Hits          2160     2304     +144     
- Misses         535      542       +7

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

test/mock_auth/server.py

…v-mongo7_upgrade

bio-boris · 2025-02-14T16:27:50Z

lib/biokbase/catalog/db.py

+        """Create indexes for the given collection lazily."""
+        collection = self.db[collection_name]
+
+        # Get the indexes for the collection from the DBIndexes class


It seems that all of this code can go into the db_indexes.py code.

You mean line 242?

If you mean the entire thing, it makes more sense to me to separate the steps of getting and creating indexes.

I agree. What I am suggesting is the code that creates the indexes be encapsulated in the DBIndexes class

Does it look better now?

no more elif. Call _create_indexes function in __init__

I think the original way it looked was much better. The way before elif was introduced, in which it declaratively did each function call and is less confusing then the way it is now.

We are reverting to the previous method because lazy Mongo load is no longer needed. It is done.

https://github.com/kbase/catalog/blob/dev-mongo7_upgrade/lib/biokbase/catalog/db.py#L210

bio-boris · 2025-02-14T19:10:40Z

lib/biokbase/catalog/db_indexes.py

@@ -0,0 +1,132 @@
+from pymongo import ASCENDING


It seems a bit strange to create indexes based on a specific collection name being rather than just create them in one go.

Here's one way to do it

from pymongo import ASCENDING import logging logging.basicConfig(level=logging.INFO) logger = logging.getLogger(__name__) class DBIndexes: _INDEX_MAP = { "module_versions": [ {"fields": ["module_name_lc"], "unique": False, "sparse": False}, {"fields": ["git_commit_hash"], "unique": False, "sparse": False}, {"fields": ["module_name_lc", "git_commit_hash"], "unique": True, "sparse": False}, ], "local_functions": [ {"fields": ["function_id"], "unique": False, "sparse": False}, {"fields": ["module_name_lc"], "unique": False, "sparse": False}, {"fields": ["git_commit_hash"], "unique": False, "sparse": False}, {"fields": ["module_name_lc", "function_id", "git_commit_hash"], "unique": True, "sparse": False}, ], "developers": [ {"fields": ["kb_username"], "unique": True, "sparse": False}, ], "build_logs": [ {"fields": ["registration_id"], "unique": True, "sparse": False}, {"fields": ["module_name_lc"], "unique": False, "sparse": False}, {"fields": ["timestamp"], "unique": False, "sparse": False}, {"fields": ["registration"], "unique": False, "sparse": False}, {"fields": ["git_url"], "unique": False, "sparse": False}, {"fields": ["current_versions.release.release_timestamp"], "unique": False, "sparse": False}, ], "favorites": [ {"fields": ["user"], "unique": False, "sparse": False}, {"fields": ["module_name_lc"], "unique": False, "sparse": False}, {"fields": ["id"], "unique": False, "sparse": False}, {"fields": ["user", "id", "module_name_lc"], "unique": True, "sparse": False}, ], "exec_stats_raw": [ {"fields": ["user_id"], "unique": False, "sparse": False}, {"fields": ["app_module_name", "app_id"], "unique": False, "sparse": True}, {"fields": ["func_module_name", "func_name"], "unique": False, "sparse": True}, {"fields": ["creation_time"], "unique": False, "sparse": False}, {"fields": ["finish_time"], "unique": False, "sparse": False}, ], "exec_stats_apps": [ {"fields": ["module_name"], "unique": False, "sparse": True}, {"fields": ["full_app_id", "type", "time_range"], "unique": True, "sparse": False}, {"fields": ["type", "time_range"], "unique": False, "sparse": True}, ], "exec_stats_users": [ {"fields": ["user_id", "type", "time_range"], "unique": True, "sparse": False}, ], "client_groups": [ {"fields": ["module_name_lc", "function_name"], "unique": True, "sparse": False}, ], "volume_mounts": [ {"fields": ["client_group", "module_name_lc", "function_name"], "unique": True, "sparse": False}, ], "secure_config_params": [ {"fields": ["module_name_lc"], "unique": False, "sparse": False}, {"fields": ["module_name_lc", "version", "param_name"], "unique": True, "sparse": False}, ], } @classmethod def get_indexes(cls, collection_name): return cls._INDEX_MAP.get(collection_name, []) @staticmethod def create_indexes(collection, indexes): index_definitions = [ { "key": [(field, ASCENDING) for field in index["fields"]], "unique": index.get("unique", False), "sparse": index.get("sparse", False), } for index in indexes ] collection.create_indexes(index_definitions) logger.info(f"Created indexes: {index_definitions}")

I forget, why did this code even get changed to this instead of keeping the original code?

ec3064b Looks like the code was fine before but a bunch of else ifs were added

It did everything in one go, rather than doing things by a specific collection.

The original Mongo initialization is in init, and it created the collection/index in a single operation. When we switched to lazy Mongo loading, the collection/index was created only once on their first call, which is why there are multiple 'elif' statements.

Now we are switching to #140 (comment)

I think keeping in the "declarative" and straightforward style that it was before would make this code less confusing. In addition, this can be run as a script before the server starts up.

Call _create_indexes function in init

I don't understand what you are saying.

There is no longer a db_indexes.py file. This thread is no longer relevant.

… client

lib/biokbase/catalog/db.py

Dockerfile

Makefile

MrCreosote · 2025-02-21T23:46:23Z

test/docker-compose_nms.yml

+      - method_spec_temp_dir=narrative_method_store_temp
+      - method_spec_mongo_host=mongo:27017
+      - method_spec_mongo_dbname=method_store_repo_db
+      - method_spec_admin_users=${ADMIN_USER}


how do you expect a dev to deal with this when testing locally?

For testing locally, either export the variable directly in the shell, or hardcode an admin user in the YAML file.

How would a local dev know to do that?

added a comment

The first time dev is still going to have a hard time figuring out why the tests don't pass when they run them. Put yourself in their shoes - if you were running the tests for the first time, what would happen if you didn't know about this env var and how easy would it be to debug? How would you want this to work / be documented?

lib/biokbase/catalog/db.py

Xiangs18 · 2025-02-25T20:28:51Z

MrCreosote · 2025-02-25T21:33:42Z

lib/biokbase/catalog/db.py

+        except ConnectionFailure as e:
+            error_msg = "Cannot connect to Mongo server\n"
+            error_msg += "ERROR -- {}:\n{}".format(
+                e, "".join(traceback.format_exception(None, e, e.__traceback__))
+            )
+            raise ValueError(error_msg)


Suggested change

except ConnectionFailure as e:

error_msg = "Cannot connect to Mongo server\n"

error_msg += "ERROR -- {}:\n{}".format(

e, "".join(traceback.format_exception(None, e, e.__traceback__))

)

raise ValueError(error_msg)

except ConnectionFailure as e:

raise ValueError(f"Cannot connect to Mongo server: {e}") from e

MrCreosote · 2025-02-25T21:34:12Z

lib/biokbase/catalog/db.py

+
+    def _initialize_mongo_client(self):
+        """Initialize MongoDB client."""
+        # Use the lock to ensure only one thread initializes the mongo client at a time


I think this comment is in the wrong place

MrCreosote · 2025-02-25T21:34:54Z

lib/biokbase/catalog/db.py

-    def update_db_1_to_2(self):
-        for m in self.modules.find({'release_versions': {'$exists': True}}):
+    def update_db_1_to_2(self, db):
+        modules_collection = db[MongoCatalogDBI._MODULES]
+        for m in modules_collection.find({'release_versions': {'$exists': True}}):
            release_version_list = []


Why can't we delete all these upgrade methods again?

mongo7 upgrade

6ec3d63

Xiangs18 added 16 commits January 6, 2025 10:10

upgrade pymongo to 4.7.2

26b7d9e

update python to 3.9.19

c5f1294

update Makefile and run_tests.sh scripts

1e57094

update submodules

47ff558

fix gradle not found

4e1f8bf

start nms service with docker-compose

4e0c2dc

update nms submodule to the latest

8095c7f

install docker-compose in GHA

6f22c74

switch catalog mongo7 port to 27018

dff0b20

fix mongo port

76bff46

fix mongo port mapping

0b7cd2a

set admin_users

66f2485

check admin users

85c65af

shut down NMS service properly

9fda21f

fix nms curl

91c72f7

update mock_auth service

dd1d2da

bio-boris reviewed Jan 23, 2025

View reviewed changes

test/mock_auth/server.py Show resolved Hide resolved

Xiangs18 and others added 11 commits January 27, 2025 13:00

update the commit

2e84037

Update test.cfg.example

727753c

Update test.cfg.example

77d161b

Update core_registration_test.py

bca78a9

update message

178a63c

Merge branch 'dev-mongo7_upgrade' of github.com:kbase/catalog into de…

a3b1038

…v-mongo7_upgrade

test method_spec_admin_users

5379082

test simple_good_repo old commit

631b52f

test register hack

b412672

remove jars nms submodules

80be977

update Dockerfile

7ead4f7

Xiangs18 added 6 commits February 11, 2025 12:50

Add a clearer comment for the environment variable in test.yml

293953c

move env to jobs level

7b4a495

add more comments in server.py

377f651

add more comments in db.py

ae1b224

remove check_db_schema related functions

370c65e

revert back changes

0015ac0

Xiangs18 mentioned this pull request Feb 13, 2025

AFAIK we're the only ones using the catalog, all these db version changes happened years ago, and so we can just put a note in the release notes that db version upgrades are no longer supported and delete all the upgrade methods #144

Open

Xiangs18 added 2 commits February 13, 2025 10:57

move self._db_schema_checked flag outside function

1428c85

refactor _create_indexes function

d106e96

bio-boris reviewed Feb 14, 2025

View reviewed changes

encapsulate create_indexes in the DBIndexes class

64f6fdb

bio-boris reviewed Feb 14, 2025

View reviewed changes

Xiangs18 added 12 commits February 14, 2025 16:03

add comment at the top of server.py file

d6bd556

refactor db.py file

96a72e6

add flag for mongo client

a4025ad

update wrapper function

46731b1

add initialize_mongo_client function manually

9836af9

add _close_mongo_client function

1f6024a

try run tests without a lock

4511078

revert self.lock changes

8806f22

Recreate the database and collection handles after reinitializing the…

62bb424

… client

revert back tests

c919000

remove decorator related code

dc79ddc

add comment why need to manually close mongoclient

0c7b906

MrCreosote reviewed Feb 21, 2025

View reviewed changes

Xiangs18 added 3 commits February 24, 2025 13:55

update comments

9ae4b7b

refactor the db.py and add comments for docker-compose file

6c54bf0

add a comment for _ensure_mongo_connection function

26d8910

MrCreosote reviewed Feb 25, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mongo7 upgrade #140

Mongo7 upgrade #140

Xiangs18 commented Dec 17, 2024 •

edited

Loading

codecov bot commented Dec 17, 2024 •

edited

Loading

bio-boris Feb 14, 2025

Xiangs18 Feb 14, 2025

Xiangs18 Feb 14, 2025 •

edited

Loading

bio-boris Feb 14, 2025 •

edited

Loading

Xiangs18 Feb 14, 2025

Xiangs18 Feb 21, 2025

bio-boris Feb 21, 2025 •

edited

Loading

Xiangs18 Feb 22, 2025

bio-boris Feb 14, 2025

bio-boris Feb 14, 2025

bio-boris Feb 14, 2025

bio-boris Feb 14, 2025

Xiangs18 Feb 14, 2025 •

edited

Loading

bio-boris Feb 15, 2025

Xiangs18 Feb 21, 2025

bio-boris Feb 21, 2025

Xiangs18 Feb 22, 2025

MrCreosote Feb 21, 2025

Xiangs18 Feb 24, 2025

MrCreosote Feb 24, 2025

Xiangs18 Feb 25, 2025

MrCreosote Feb 25, 2025

Xiangs18 commented Feb 25, 2025

MrCreosote Feb 25, 2025

MrCreosote Feb 25, 2025

MrCreosote Feb 25, 2025

Mongo7 upgrade #140

Are you sure you want to change the base?

Mongo7 upgrade #140

Conversation

Xiangs18 commented Dec 17, 2024 • edited Loading

codecov bot commented Dec 17, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Xiangs18 Feb 14, 2025 • edited Loading

Choose a reason for hiding this comment

bio-boris Feb 14, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bio-boris Feb 21, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Xiangs18 Feb 14, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Xiangs18 commented Feb 25, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Xiangs18 commented Dec 17, 2024 •

edited

Loading

codecov bot commented Dec 17, 2024 •

edited

Loading

Xiangs18 Feb 14, 2025 •

edited

Loading

bio-boris Feb 14, 2025 •

edited

Loading

bio-boris Feb 21, 2025 •

edited

Loading

Xiangs18 Feb 14, 2025 •

edited

Loading