More migration stuff, more endpoints, map categories (#3)

* More migration stuff, more endpoints, map categories - Add a system user for `last_modified_by` in migrations. This user is hidden by default in `CncUser.objects.all()` queries. - Clean up migrations a bit - Make the user management class available in migrations - Make map category slugs auto-generate - Pop fields that non-admins shouldn't see over the API - Start standardizing what our `response.data` is going to look like. I want to avoid the UI needing a `is list or is dict` check for api returns - Fix the pagination class to use limits and offsets - Make pagination return standard api format - Fix and test map category endpoint - Add standard map categories in a migration. Copied from https://github.com/CnCNet/cncnet-yr-client-package/blob/develop/package/INI/MPMaps.ini - Add Kirovy mascot. PR: #3 * More migration stuff, more endpoints, map categories - Make map categories many 2 many. - Actually test the map upload path PR: #3 * More migration stuff, more endpoints, map categories - Initial work to show UI permissions PR: #3 * More migration stuff, more endpoints, map categories - First attempt at a CI and make a docker file for tests PR: #3 * More migration stuff, more endpoints, map categories - First attempt at a CI and make a docker file for tests PR: #3 * More migration stuff, more endpoints, map categories - First attempt at a CI and make a docker file for tests PR: #3 * CI and permissions - Made it unnecessary to have testing api creds in CI. - Added a way to disallow certain boolean settings in prod - Renamed some authentication stuff to be less confusing - UI permissions endpoint and tests * Map Upload WIP - Add library to check file types - Update docker and readme - Update requirements - Add field for incomplete uploads in case someone uploads a map, but never fills out metadata. These maps should be deleted if the user never finalizes the upload. - Make a response for errors - Fix the `is_text` check to account for `InMemoryUploads` being deleted when they are closed, meaning we couldn't use `with open(temp_file, "tr")` to check for text anymore. - Make `ByteSized`. Might move into its own repo at some point. - Fix binary fixture not reading as binary * Map Upload WIP - Fix the map parser service to use django's `UploadedFile` correctly - Start figuring out the serialization and creation of the map model objects TODO: - Map category fetching. - Actually save the map file - Generate a filename - check map hashes and cncnetid. Exclude cncnet id section from hashing - Add the cncnetID - Actually verify saved files in test. - Move everything to a DRF serializer for more safety. * Map Upload WIP - Update readme for running tests - Use `hex` for UUIDs in filenames to remove hyphens - Update naming pattern - Make map version uneditable. Only set version when it is None so we don't overwrite the version with each save call. - Fix files loaded from file fixtures not working with the map parser. The issue was we used `"r"` but a file uploaded via django was `"rb"`. - Got map uploader basically working and writing cncnet ID - Add support for detecting reuploads and marking the original map as the parent.
CnCNet · May 28, 2024 · b2986b2 · b2986b2
1 parent 07d3a47
commit b2986b2
Show file tree

Hide file tree

Showing 49 changed files with 1,383 additions and 211 deletions.
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -0,0 +1,24 @@
+name: CI
+on:
+  pull_request:
+  push: { branches: main }
+
+jobs:
+  test:
+    name: Run PyTest
+    runs-on: ubuntu-latest
+    env:
+      COMPOSE_FILE: docker-compose.yml
+
+    steps:
+    - name: Checkout code
+      uses: actions/checkout@v2
+
+    - name: Make CI env file
+      run: cp ci.env .env
+
+    - name: Build docker images
+      run: docker-compose build
+
+    - name: Run PyTest
+      run: docker-compose run test
diff --git a/Dockerfile b/Dockerfile
@@ -12,9 +12,11 @@ COPY requirements.txt /cncnet-map-api
 COPY requirements-dev.txt /cncnet-map-api
 COPY start.sh /cncnet-map-api
 
+RUN apt-get update && apt-get install -y liblzo2-dev  # Compression library used by westwood.
+RUN apt-get install libmagic1  # File type checking.
 RUN pip install --upgrade pip
-RUN pip install -r ./requirements.txt
-RUN pip install -r ./requirements-dev.txt
+# The cflags are needed to build the lzo library on Apple silicon.
+RUN CFLAGS=-I$(brew --prefix)/include LDFLAGS=-L$(brew --prefix)/lib pip install -r ./requirements-dev.txt
 
 RUN chmod +x /cncnet-map-api/start.sh
 ENTRYPOINT "/cncnet-map-api/start.sh"
diff --git a/README.md b/README.md
@@ -1,44 +1,75 @@
 # CnCNet Map Browser API
 
+
 The backend for the CnCNet map browser.
 
 UI is here https://github.com/CnCNet/cncnet-map-ui
 
 
-# Development
+# Kirovy
+
+
+The mascot for the backend API is Kirovy, by [Direct & Dominate](https://www.youtube.com/@DirectandDominate)
+
+![Kirovy enjoying his job](docs/images/kirovy_direct_and_dominate.png)
 
+# Development
 
 ## Frontend devs
 
-Just set up your environment file and run the full docker compose
+Just set up your environment file and run the full docker compose.
+
+[Example env file](example.env)
 
 ## Backend devs
 
+You can use the docker files if you'd like, but Django + docker is known to have issue attaching
+to debuggers and hitting breakpoints, so here are the native OS instructions.
+
 1. Download and install [pyenv](https://github.com/pyenv/pyenv)
 2. Install [PostgreSQL](https://www.postgresql.org/) for your system. This is required for Django
    - On Mac you can do `brew install postgresql` if you have brew installed.
-3. Checkout the repository
-4. Switch to the repository directory
-5. Setup Python
+3. Install LibMagic for [Python Magic](https://github.com/ahupp/python-magic)
+   - On Mac you can do `brew install libmagic` if you have breq installed.
+4. Checkout the repository
+5. Switch to the repository directory
+6. Setup Python
    - Install Python 3.12 `pyenv install 3.12` or whatever the latest python is.
    - Setup the virtual environments `pyenv virtualenv 3.12 cncnet-map-api`
    - Set the virtual enviornment for the directory `pyenv local cncnet-map-api`
-6. Setup requirements `pip install -r requirements-dev.txt`
-   -  On Apple Silicon you'll need to run
+7. Setup requirements `pip install -r requirements-dev.txt`
+   -  On Apple Silicon you'll need to install lzo with `brew install lzo` then run
       `CFLAGS=-I$(brew --prefix)/include LDFLAGS=-L$(brew --prefix)/lib pip install -r requirements-dev.txt`
       to get `python-lzo` to install. You shouldn't need to include those flags again unless `python-lzo` updates.
-7. Install the pre-commit hooks `pre-commit install`
-8. Setup the environment variables
+8. Install the pre-commit hooks `pre-commit install`
+9. Setup the environment variables
    - Create a `.env` file at the root of the repo
    - Copy the contents of `example.env` to your `.env` file.
    - Fill out the required values in `.env`
    - If the app doesn't run due to a missing required variable, add said variable to `example.env` because the person
    who made the variable forgot to do so.
-9. Run the `db` service in `docker-compose`
-10. Load your `.env` file into your shell, (you can use `./load_env.sh`) then migrate the database `./manage.py migrate`
-11. `./manage.py runserver`
-
-I **strongly** recommend using PyCharm and the `.env` plugin for running the PyTests.
+10. Run the `db` service in `docker-compose`
+11. Load your `.env` file into your shell, (you can use `source load_env.sh && read_env`)
+then migrate the database `./manage.py migrate`
+12. `./manage.py runserver`
 
 You can technically use PyCharm to launch everything via `docker-compose`, but there is some
 weird issue with breakpoints not triggering.
+
+
+## Running tests
+
+I **strongly** recommend using PyCharm and the `.env` plugin for running the PyTests.
+All you need to do is run the database from `docker-compose`, then launch the tests via PyCharm.
+
+**If you want to run the tests via CLI:**
+
+- Make sure your database is running from the docker compose file. `docker-compose start db`
+- Make sure your environment variables are setup and loaded to your shell. See [backend dev setup](#backend-devs)
+- Run `DJANGO_SETTINGS_MODULE="kirovy.settings.testing" pytest tests`
+
+Django should automatically run migrations as part of the test startup.
+
+**Run tests with docker compose:**
+
+- `docker-compose up --build test`
diff --git a/ci.env b/ci.env
@@ -0,0 +1,10 @@
+POSTGRES_DB=cncnet-map-api
+POSTGRES_USER=kane
+POSTGRES_PASSWORD=cidevelopmentpasswordtechnologyofpeace
+POSTGRES_PORT=5432
+POSTGRES_DATA_DIR=/data/db/
+MEDIA_ROOT=/data/cnc_net_files/
+STATIC_ROOT=/data/cnc_net_static/
+POSTGRES_TEST_HOST=db
+SECRET_KEY=";thetechnologyofpeaceforcidevwork6352722!@#$$#@"
+RUN_ENVIRONMENT=ci
diff --git a/docker-compose.yml b/docker-compose.yml
@@ -26,3 +26,17 @@ services:
       - .env
     depends_on:
       - db
+  test:
+    build:
+      context: ./
+      dockerfile: test.DockerFile
+    volumes:
+      - .:/cncnet-map-api
+    env_file:
+      - .env
+    environment:
+      POSTGRES_TEST_HOST: db  # Necessary to connect to docker db. Overrides the .env setting.
+    depends_on:
+      - db
+    command:
+      - pytest
diff --git a/docs/images/kirovy_direct_and_dominate.png b/docs/images/kirovy_direct_and_dominate.png
diff --git a/docs/images/kirovy_square.png b/docs/images/kirovy_square.png
diff --git a/kirovy/authentication.py b/kirovy/authentication.py
@@ -12,22 +12,19 @@
 from kirovy.models import CncUser
 
 
-class CncNetAuthenticator:
+class _CncNetAuthenticator:
     """The class for sending requests to the cncnet Ladder API.
 
     This exists to house logic related to authenticating with CnCNet,
     and to easily monkey-patch CnCNet auth in tests.
     """
 
     @classmethod
-    def authenticate(
+    def authenticate_with_cncnet(
         cls, request: HttpRequest
     ) -> t.Tuple[CncUser, t.Optional[objects.CncnetUserInfo]]:
         """Authenticate a request's JWT with CnCNet.
 
-        Monkeypatch this function in tests to return whichever value you need
-        for testing endpoint permissions.
-
         :param request:
             The request to Kirovy. We will send its header to CnCNet.
         :return:
@@ -61,7 +58,7 @@ def request_user_info(request: HttpRequest) -> objects.CncnetUserInfo:
             Raised if we successfully authenticate with CnCNet, but can't parse the user info.
         """
         cncnet_response = requests.get(
-            constants.cncnet_user_url, headers=request.headers
+            constants.CNCNET_USER_URL, headers=request.headers
         )
         if cncnet_response.status_code != status.HTTP_200_OK:
             raise exceptions.CncNetAuthFailed(
@@ -87,6 +84,10 @@ def authenticate(
         Extracts the JWT from ``request.headers`` then forwards that to CnCNet.
         If the JWT authenticates, then get, or create, the Kirovy user object for the CnCNet user.
 
+
+        If you don't want to deal with token headers in tests, then monkeypatch this function to return
+        whichever value you need for testing endpoint permissions.
+
         :param request:
             The raw request.
         :return:
@@ -101,4 +102,4 @@ def authenticate(
         if len(token) != 2 or token[0].lower() != "bearer":
             raise exceptions.MalformedTokenError()
 
-        return CncNetAuthenticator.authenticate(request)
+        return _CncNetAuthenticator.authenticate_with_cncnet(request)
diff --git a/kirovy/constants/__init__.py b/kirovy/constants/__init__.py
@@ -3,10 +3,14 @@
 
 from django.utils.translation import gettext as _
 
-cnc_prefix = _("Command & Conquer:")
+CNC_PREFIX = _("Command & Conquer:")
 
-cncnet_token_url = "https://ladder.cncnet.org/api/v1/auth/token"
-cncnet_user_url = "https://ladder.cncnet.org/api/v1/user/info"
+CNCNET_TOKEN_URL = "https://ladder.cncnet.org/api/v1/auth/token"
+CNCNET_USER_URL = "https://ladder.cncnet.org/api/v1/user/info"
+
+CNCNET_INI_SECTION = "CnCNet"
+CNCNET_INI_MAP_ID_KEY = "ID"
+CNCNET_INI_PARENT_ID_KEY = "ParentID"
 
 
 class CncnetUserGroup:
@@ -85,6 +89,12 @@ def is_messiah(cls, user_group: str) -> bool:
         return user_group == cls.KANE
 
 
+class MigrationUser:
+    ID = -1
+    USERNAME = "MobileConstructionVehicle_Migrator"
+    GROUP = CncnetUserGroup.USER
+
+
 class GameSlugs(enum.StrEnum):
     """The slugs for each game / total conversion mod.
 

diff --git a/kirovy/migrations/0001_initial.py b/kirovy/migrations/0001_initial.py
@@ -289,10 +289,8 @@ class Migration(migrations.Migration):
         ),
         migrations.AddField(
             model_name="cncmap",
-            name="category",
-            field=models.ForeignKey(
-                on_delete=django.db.models.deletion.PROTECT, to="kirovy.mapcategory"
-            ),
+            name="categories",
+            field=models.ManyToManyField(to="kirovy.mapcategory"),
         ),
         migrations.AddField(
             model_name="cncmap",
@@ -326,4 +324,67 @@ class Migration(migrations.Migration):
                 fields=("cnc_map_id", "version"), name="unique_map_version"
             ),
         ),
+        migrations.AlterModelManagers(
+            name="cncuser",
+            managers=[
+                ("objects", kirovy.models.cnc_user.CncUserManager()),
+            ],
+        ),
+        migrations.AddField(
+            model_name="cncfileextension",
+            name="last_modified_by",
+            field=models.ForeignKey(
+                null=True,
+                on_delete=django.db.models.deletion.SET_NULL,
+                related_name="modified_%(class)s_set",
+                to=settings.AUTH_USER_MODEL,
+            ),
+        ),
+        migrations.AddField(
+            model_name="cncgame",
+            name="last_modified_by",
+            field=models.ForeignKey(
+                null=True,
+                on_delete=django.db.models.deletion.SET_NULL,
+                related_name="modified_%(class)s_set",
+                to=settings.AUTH_USER_MODEL,
+            ),
+        ),
+        migrations.AddField(
+            model_name="cncmap",
+            name="is_banned",
+            field=models.BooleanField(
+                default=False,
+                help_text="If true, this map will be hidden everywhere. Likely due to breaking a rule.",
+            ),
+        ),
+        migrations.AddField(
+            model_name="cncmap",
+            name="legacy_upload_date",
+            field=models.DateTimeField(
+                default=None,
+                help_text="The original upload date for entries imported from the legacy map database.",
+                null=True,
+            ),
+        ),
+        migrations.AddField(
+            model_name="cncmapfile",
+            name="last_modified_by",
+            field=models.ForeignKey(
+                null=True,
+                on_delete=django.db.models.deletion.SET_NULL,
+                related_name="modified_%(class)s_set",
+                to=settings.AUTH_USER_MODEL,
+            ),
+        ),
+        migrations.AddField(
+            model_name="mapcategory",
+            name="last_modified_by",
+            field=models.ForeignKey(
+                null=True,
+                on_delete=django.db.models.deletion.SET_NULL,
+                related_name="modified_%(class)s_set",
+                to=settings.AUTH_USER_MODEL,
+            ),
+        ),
     ]