Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

wrapdb server static v1 api mirror and nginx config update #1772

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
102 changes: 89 additions & 13 deletions nginx/default
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,10 @@
# https://www.nginx.com/resources/wiki/start/
# https://www.nginx.com/resources/wiki/start/topics/tutorials/config_pitfalls/
# https://wiki.debian.org/Nginx/DirectoryStructure
##

# This is the web server config for wrapdb.mesonbuild.com

upstream uwsgi {
server 127.0.0.1:8081;
}

server {
location /v2 {
rewrite ^.*/v2/releases.json https://raw.githubusercontent.com/mesonbuild/wrapdb/master/releases.json permanent;
Expand All @@ -19,7 +16,8 @@ server {
}

# Redirect legacy v1 patch URLs.
# FIXME: This breaks download of sqlite that has not been imported to v2 because it is replaced by sqlite3
# FIXME: This breaks download of sqlite that has not been imported to v2 because it is replaced by sqlite3 (xclaesse)
# NOTE: no longer needed, we now serve v1 patch URLs directly from a static files mirror (in /var/www/v1-static, see below) (Tim, Nov 2024)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it's no longer needed, just remove it :)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OTOH, why not just use that redirect? Files are already hosted on github, we don't need another static mirror, do we?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As per my previous comment, if you want to make sure the files are all there including the old sqlite things I'm all for it, I'm just not planning to do that work, and I also suspect there might be some discussions otherwise surely you'd have just done it back when you added the comment.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we should break things is what I'm saying, so if you want to enable the redirect make sure it won't break anything.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran this test script inside wrapdb-v1-api-mirror repo:

import glob
import requests

for i in glob.glob('**/get_zip', recursive=True):
    parts = i.split('/')
    name = parts[2]
    version = parts[3]
    revision = parts[4]
    url = f'https://github.com/mesonbuild/wrapdb/releases/download/{name}_{version}-{revision}/{name}_{version}-{revision}_patch.zip'
    response = requests.get(url)
    if response.status_code == 200:
        print(url, 'OK')
    else:
        print(url, 'not found')

Missing URLs are:
https://github.com/mesonbuild/wrapdb/releases/download/libjpeg_9c-4/libjpeg_9c-4_patch.zip
https://github.com/mesonbuild/wrapdb/releases/download/libjpeg_9c-2/libjpeg_9c-2_patch.zip
https://github.com/mesonbuild/wrapdb/releases/download/libjpeg_9c-3/libjpeg_9c-3_patch.zip
https://github.com/mesonbuild/wrapdb/releases/download/libjpeg_9c-1/libjpeg_9c-1_patch.zip
https://github.com/mesonbuild/wrapdb/releases/download/libjpeg_9a-4/libjpeg_9a-4_patch.zip
https://github.com/mesonbuild/wrapdb/releases/download/libjpeg_9a-2/libjpeg_9a-2_patch.zip
https://github.com/mesonbuild/wrapdb/releases/download/libjpeg_9a-5/libjpeg_9a-5_patch.zip
https://github.com/mesonbuild/wrapdb/releases/download/sqlite_3310100-1/sqlite_3310100-1_patch.zip
https://github.com/mesonbuild/wrapdb/releases/download/sqlite_3320300-4/sqlite_3320300-4_patch.zip
https://github.com/mesonbuild/wrapdb/releases/download/sqlite_3320300-3/sqlite_3320300-3_patch.zip
https://github.com/mesonbuild/wrapdb/releases/download/sqlite_3320300-5/sqlite_3320300-5_patch.zip
https://github.com/mesonbuild/wrapdb/releases/download/sqlite_3250100-2/sqlite_3250100-2_patch.zip
https://github.com/mesonbuild/wrapdb/releases/download/sqlite_3250100-1/sqlite_3250100-1_patch.zip
https://github.com/mesonbuild/wrapdb/releases/download/sqlite_3230100-1/sqlite_3230100-1_patch.zip
https://github.com/mesonbuild/wrapdb/releases/download/sqlite_3080802-4/sqlite_3080802-4_patch.zip
https://github.com/mesonbuild/wrapdb/releases/download/sqlite_3080802-2/sqlite_3080802-2_patch.zip
https://github.com/mesonbuild/wrapdb/releases/download/sqlite_3080802-5/sqlite_3080802-5_patch.zip
https://github.com/mesonbuild/wrapdb/releases/download/sqlite_3140200-2/sqlite_3140200-2_patch.zip
https://github.com/mesonbuild/wrapdb/releases/download/sqlite_3140200-1/sqlite_3140200-1_patch.zip

IMHO it's fine to break queries and download of wraps, we just need to keep patch zips. Those 2 deps are trivial to update to v2, I don't think it's a big deal to break them, it's been deprecated for years.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you want to break it, break it, just leave me out of it please.

I don't think we should break it if there's no need and there isn't really any need since it work just fine as it is now.

Just leave the v1 alone as-is I'd say. You're now breaking something that works for no good reason other than to make the config a bit tidier.

libjpeg and sqlite are common wraps and breaking wrapdb stuff even if old will leave a bad taste imho.

Anyway, this ticket is a docs update updating the nginx config to the status quo.

Perhaps other changes you want to make should be tracked separately 🙂

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re. "trivial to update" that's beside the point because people will be consuming third party source repositories/branches/tags that they can't easily update without switching to their own modified copies or overwriting the checkouts manually.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I now uploaded missing assets, the redirect URL should be enough.

#location ~ get_zip$ {
# rewrite ^.*/v1/projects/([^/]+)/([^/]+)/([^/]+)/get_zip https://github.com/mesonbuild/wrapdb/releases/download/$1_$2-$3/$1_$2-$3_patch.zip permanent;
#}
Expand All @@ -28,18 +26,96 @@ server {
return 301 https://mesonbuild.com/Wrapdb-projects.html;
}

# Serve v1 files from a static mirror (requires redirects because the
# structure of the mirror can't 1:1 map to the file path in the request URIs,
# as we have to disambiguate files and directories for the same path).
# (Tim, 3 Nov 2024)
#
# See https://github.com/tp-m/wrapdb-v1-api-mirror/
#
location /v1/ {

location / {
try_files $uri @uwsgi;
}
# v1 project list redirect
location = /v1/projects {
return 301 https://wrapdb.mesonbuild.com/v1-static/-/projects;
}

# v1 project info redirect
location /v1/projects/ {
rewrite ^.*/v1/projects/([^/]+)$ https://wrapdb.mesonbuild.com/v1-static/projects/-/$1 permanent;
}

# v1 get_latest redirect to static mirror
location /v1/query/get_latest/ {
rewrite ^.*/v1/query/get_latest/([^/]+)$ https://wrapdb.mesonbuild.com/v1-static/query/get_latest/-/$1 permanent;
}

# v1 get_wrap redirect to static mirror
location ~ /v1/projects/[^/]+/[^/]+/[^/]+/get_wrap$ {
rewrite ^.*/v1/projects/([^/]+)/([^/]+)/([^/]+)/get_wrap$ https://wrapdb.mesonbuild.com/v1-static/projects/$1/$2/$3/-/get_wrap permanent;
#rewrite ^.*/v1/projects/([^/]+)/([^/]+)/([^/]+)/get_wrap$ https://wrapdb.mesonbuild.com/v1-static/projects/$1/$2/$3/$1-$2-$3.wrap permanent;
}

location /static {
root /home/legeana/wrapweb/wrapweb/;
# v1 get_zip redirect to static mirror
location ~ /v1/projects/[^/]+/[^/]+/[^/]+/get_zip$ {
rewrite ^.*/v1/projects/([^/]+)/([^/]+)/([^/]+)/get_zip$ https://wrapdb.mesonbuild.com/v1-static/projects/$1/$2/$3/-/get_zip permanent;
#rewrite ^.*/v1/projects/([^/]+)/([^/]+)/([^/]+)/get_zip$ https://wrapdb.mesonbuild.com/v1-static/projects/$1/$2/$3/$1-$2-$3-wrap.zip permanent;
}
}

location @uwsgi {
include uwsgi_params;
uwsgi_pass uwsgi;
# Make sure v1 files served from static mirror are decorated with the
# right Content-type and where applicable Content-disposition headers.
#
# Data: https://github.com/mesonbuild/wrapdb/pull/1772#issuecomment-2466273083
#
location /v1-static/ {
root /var/www/;

# v1 project list (json)
location = /v1-static/-/projects {
types { }
default_type application/json;
}

# v1 project info (json)
location /v1-static/projects/-/ {
types { }
default_type application/json;
}

# v1 project latest version (json)
location /v1-static/query/get_latest/ {
types { }
default_type application/json;
}

# v1 get_wrap (text)
location ~ /v1-static/projects/[^/]+/[^/]+/[^/]+/-/get_wrap$ {
types { }
default_type "text/plain; charset=utf-8";
}

# v1 wrap file full name alias (text) (bonus addition, not part of the original v1 api)
location ~ /v1-static/projects/[^/]+/[^/]+/[^/]+/[^/]+.wrap$ {
types { }
default_type "text/plain; charset=utf-8";
}

# v1 get_zip (zip)
location ~ /v1-static/projects/[^/]+/[^/]+/[^/]+/-/get_zip$ {
types { }
default_type application/zip;
if ( $request_filename ~ "^.*/projects/([^/]+)/([^/]+)/([^/]+)/-/get_zip$" ){
set $fname $1-$2-$3-wrap.zip;
add_header Content-Disposition 'attachment; filename=$fname';
}
}

# v1 zip file full name alias (zip) (bonus addition, not part of the original v1 api)
location ~ /v1-static/projects/[^/]+/[^/]+/[^/]+/[^/]+.zip$ {
types { }
default_type application/zip;
}
}

server_name wrapdb.mesonbuild.com;
Expand Down
Loading