configuring for image uploads and ML inference - issues with file buffering

[austinben]

Hello,

Ive used this image in the past, and decided to re-visit it for a new task.

I am attempting to create an interface where images are uploaded via a POST request, perform inference with a yolov3 model, and return some JSON at the end.

I have tested the implementation with a local flask server via flask run, and tested with the base python:3.8 image, both of which run successfully, returning results at an acceptable rate.

As soon as I try to use this image, or the tiangolo/meinheld-gunicorn-flask images, requests to the flask server are incredibly slow. Increasing the timeout enough, I find that eventually it works, returning a response almost 5 minutes later, which is unfortunate.

I believe the issue stems from buffering file contents somewhere between nginx and uwsgi, as I see a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000001.

I have not attempted to test concurrent requests yet, as it is struggling to handle a single one already.

I have tried various custom configurations for various parameters from http://nginx.org/en/docs/http/ngx_http_uwsgi_module.html, adding with COPY buffer.conf /etc/nginx/conf.d/ in the Dockerfile, without much success, and was hoping to get some guidance about how to configure this.

The flask endpoint is designed as follows:

@app.route('/detect', methods=['POST'])
def detection():
    try:

        print('Reading image...')
        image = request.files['image'].read()
        nparr = np.frombuffer(image, np.uint8)
        cwt_image = cv2.imdecode(nparr, cv2.IMREAD_COLOR)

        boxes = detection.getBoundingBoxes(cwt_image)

        detections = []
        for box in boxes:
            detections.append(box)

        response = {
            'detections' : detections
        }

    except Exception as e:
        error = 'error: {}'.format(repr(e))
        print(error)
        response = (error, 500, [])

    return response

And I am testing by sending a request to the endpoint as follows:

file = {'image': open('image', 'rb')}
response = requests.post(url=addr, files=file)

My Dockerfile is as follows:

FROM tiangolo/uwsgi-nginx-flask:python3.8 

RUN apt-get update -y

RUN apt-get install ffmpeg libsm6 libxext6 -y

RUN pip install opencv-python

COPY buffer.conf /etc/nginx/conf.d/

ADD app /app

WORKDIR /app

I wonder if it is related to how I am handling the image data? The images are typically around 100-150KB in size.

Any guidance to configuring this (or any other) image properly to be able to handle concurrent requests of this nature would be great. Thanks in advance.

[austinben]

Id just like to add that I cant seem to disable the buffers with my custom configuration settings:

uwsgi_buffering off;
uwsgi_max_temp_file_size 0;

I am not sure if they are being over-written somewhere?

Logs from the container startup:

WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
Checking for script in /app/prestart.sh
Running script /app/prestart.sh
Running inside /app/prestart.sh, you could add migrations to this file, e.g.:

#! /usr/bin/env sh

# Let the DB start
sleep 10;
# Run migrations
alembic upgrade head

/usr/lib/python2.7/dist-packages/supervisor/options.py:461: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security.
  'Supervisord is running as root and it is searching '
2022-04-19 19:12:36,434 CRIT Supervisor is running as root.  Privileges were not dropped because no user is specified in the config file.  If you intend to run as root, you can set user=root in the config file to avoid this message.
2022-04-19 19:12:36,436 INFO Included extra file "/etc/supervisor/conf.d/supervisord.conf" during parsing
2022-04-19 19:12:36,462 INFO RPC interface 'supervisor' initialized
2022-04-19 19:12:36,462 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2022-04-19 19:12:36,463 INFO supervisord started with pid 1
2022-04-19 19:12:37,480 INFO spawned: 'quit_on_failure' with pid 23
2022-04-19 19:12:37,491 INFO spawned: 'nginx' with pid 25
2022-04-19 19:12:37,499 INFO spawned: 'uwsgi' with pid 27
2022-04-19 19:12:37,571 INFO success: nginx entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2022-04-19 19:12:37,571 INFO success: uwsgi entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
[uWSGI] getting INI configuration from /app/uwsgi.ini
[uWSGI] getting INI configuration from /etc/uwsgi/uwsgi.ini

;uWSGI instance configuration
[uwsgi]
ini = /app/uwsgi.ini
module = main
callable = app
processes = 16
cheaper = 2
ini = /etc/uwsgi/uwsgi.ini
socket = /tmp/uwsgi.sock
chown-socket = nginx:nginx
chmod-socket = 664
hook-master-start = unix_signal:15 gracefully_kill_them_all
need-app = true
die-on-term = true
show-config = true
;end of configuration

*** Starting uWSGI 2.0.20 (64bit) on [Tue Apr 19 19:12:37 2022] ***
compiled with version: 8.3.0 on 26 October 2021 17:01:27
os: Linux-5.10.76-linuxkit #1 SMP PREEMPT Mon Nov 8 11:22:26 UTC 2021
nodename: 98d519764447
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 8
current working directory: /app
detected binary path: /usr/local/bin/uwsgi
your memory page size is 4096 bytes
detected max file descriptor number: 1048576
lock engine: pthread robust mutexes
!!! it looks like your kernel does not support pthread robust mutexes !!!
!!! falling back to standard pthread mutexes !!!
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to UNIX address /tmp/uwsgi.sock fd 3
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
Python version: 3.8.12 (default, Oct 13 2021, 09:22:51)  [GCC 8.3.0]
2022/04/19 19:12:37 [emerg] 33#33: io_setup() failed (38: Function not implemented)
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x4000170380
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 1239640 bytes (1210 KB) for 16 cores
*** Operational MODE: preforking ***
2022-04-19 19:12:38,732 INFO success: quit_on_failure entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
WSGI app 0 (mountpoint='') ready in 2 seconds on interpreter 0x4000170380 pid: 27 (default app)
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 27)
spawned uWSGI worker 1 (pid: 45, cores: 1)
spawned uWSGI worker 2 (pid: 47, cores: 1)
running "unix_signal:15 gracefully_kill_them_all" (master-start)...
[pid: 45|app: 0|req: 1/1] 172.17.0.1 () {36 vars in 404 bytes} [Tue Apr 19 19:12:53 2022] GET / => generated 6 bytes in 17 msecs (HTTP/1.1 200) 2 headers in 78 bytes (2 switches on core 0)
172.17.0.1 - - [19/Apr/2022:19:12:53 +0000] "GET / HTTP/1.1" 200 6 "-" "python-requests/2.25.1" "-"
2022/04/19 19:12:53 [warn] 33#33: *3 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000001, client: 172.17.0.1, server: , request: "POST /detection HTTP/1.1", host: "0.0.0.0"

[tiangolo]

I wouldn't know the best way to handle it. And unfortunately I'm not using Flask nor this Docker image for a long while, so I wouldn't have a chance to try and optimize configs for your use case.

I'm biased, but I would try to use FastAPI instead 😅 I think handling Uvicorn alone might be much simpler than this setup with several components. 🤓