Possible Bug

[saifulkhan]

I see the following issue frequently when I start/restart the service in production.

INFO:     Started server process [7204]
2022-01-12 12:28:51.305 | INFO     | asyncio.events:_run:80 - Started server process [7204]
INFO:     Waiting for application startup.
2022-01-12 12:28:51.305 | INFO     | uvicorn.server:startup:97 - Waiting for application startup.
INFO:     Application startup complete.
2022-01-12 12:28:51.305 | INFO     | uvicorn.server:startup:97 - Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:4010 (Press CTRL+C to quit)
2022-01-12 12:28:51.306 | INFO     | uvicorn.server:startup:176 - Uvicorn running on http://127.0.0.1:4010 (Press CTRL+C to quit)
2022-01-12 12:29:15.795 | DEBUG    | urllib3.connectionpool:_get_conn:281 - Starting new HTTPS connection (1): gist.githubusercontent.com:443
2022-01-12 12:29:16.008 | DEBUG    | urllib3.connectionpool:urlopen:703 - https://gist.githubusercontent.com:443 "GET /rampvisdevelopment/a73ff62e4b648541e5fa75ce0a5edae4/raw/example_sensitivity_input.json HTTP/1.1" 200 33754
2022-01-12 12:29:23.690 | DEBUG    | urllib3.connectionpool:_get_conn:281 - Starting new HTTPS connection (1): gist.githubusercontent.com:443
2022-01-12 12:29:23.906 | DEBUG    | urllib3.connectionpool:urlopen:703 - https://gist.githubusercontent.com:443 "GET /rampvisdevelopment/52a5836735eff0d33692cfd06d049197/raw/uncertainty_inventory.json HTTP/1.1" 200 80
2022-01-12 12:29:23.913 | DEBUG    | urllib3.connectionpool:_get_conn:281 - Starting new HTTPS connection (1): gist.githubusercontent.com:443
2022-01-12 12:29:24.119 | DEBUG    | urllib3.connectionpool:urlopen:703 - https://gist.githubusercontent.com:443 "GET /rampvisdevelopment/acb6a1e6e33d0358553d7de09d6232e0/raw/sensitivity_inventory.json HTTP/1.1" 200 557
2022-01-12 12:29:59.205 | ERROR    | app.services.download_service:download_urls:183 - IncompleteRead(209715200 bytes read, 43243700 more expected)
Traceback (most recent call last):

  File "/home/scrc/miniconda3/envs/py-3.9.2/lib/python3.9/threading.py", line 912, in _bootstrap
    self._bootstrap_inner()
    │    └ <function Thread._bootstrap_inner at 0x7f877599b040>
    └ <Thread(Thread-1, started 140219028420352)>
  File "/home/scrc/miniconda3/envs/py-3.9.2/lib/python3.9/threading.py", line 954, in _bootstrap_inner
    self.run()
    │    └ <function Thread.run at 0x7f8775999d30>
    └ <Thread(Thread-1, started 140219028420352)>
  File "/home/scrc/miniconda3/envs/py-3.9.2/lib/python3.9/threading.py", line 892, in run
    self._target(*self._args, **self._kwargs)
    │    │        │    │        │    └ {}
    │    │        │    │        └ <Thread(Thread-1, started 140219028420352)>
    │    │        │    └ ()
    │    │        └ <Thread(Thread-1, started 140219028420352)>
    │    └ <function run_agents at 0x7f8749ec0a60>
    └ <Thread(Thread-1, started 140219028420352)>

  File "./app/controllers/agents/__init__.py", line 52, in run_agents
    download_data()
    └ <function download_data at 0x7f874d017820>

  File "./app/controllers/agents/data_downloader_agent.py", line 35, in download_data
    download_urls(urls, DATA_PATH_LIVE)
    │             │     └ '/home/scrc/rampvis-api/data-api/app/core/../../../data/live'
    │             └ [{'name': 'sobol', 'url': 'https://gist.githubusercontent.com/rampvisdevelopment/a73ff62e4b648541e5fa75ce0a5edae4/raw/example...
    └ <function download_urls at 0x7f8749eb9790>

> File "./app/services/download_service.py", line 159, in download_urls
    df = pd.read_csv(url['url'], encoding='iso-8859-1')
         │  │        └ {'name': 'phe', 'url': 'https://api.coronavirus.data.gov.uk/v2/data?areaType=ltla&metric=vaccinationsAgeDemographics&format=c...
         │  └ <function read_csv at 0x7f87658790d0>
         └ <module 'pandas' from '/home/scrc/rampvis-api/data-api/venv/lib/python3.9/site-packages/pandas/__init__.py'>

  File "/home/scrc/rampvis-api/data-api/venv/lib/python3.9/site-packages/pandas/util/_decorators.py", line 311, in wrapper
    return func(*args, **kwargs)
           │     │       └ {'encoding': 'iso-8859-1'}
           │     └ ('https://api.coronavirus.data.gov.uk/v2/data?areaType=ltla&metric=vaccinationsAgeDemographics&format=csv',)
           └ <function read_csv at 0x7f8765876f70>
  File "/home/scrc/rampvis-api/data-api/venv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 586, in read_csv
    return _read(filepath_or_buffer, kwds)
           │     │                   └ {'delimiter': ',', 'header': 'infer', 'names': None, 'index_col': None, 'usecols': None, 'squeeze': False, 'prefix': None, 'm...
           │     └ 'https://api.coronavirus.data.gov.uk/v2/data?areaType=ltla&metric=vaccinationsAgeDemographics&format=csv'
           └ <function _read at 0x7f8765876e50>
  File "/home/scrc/rampvis-api/data-api/venv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 482, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
             │              │                     └ {'delimiter': ',', 'header': 'infer', 'names': None, 'index_col': None, 'usecols': None, 'squeeze': False, 'prefix': None, 'm...
             │              └ 'https://api.coronavirus.data.gov.uk/v2/data?areaType=ltla&metric=vaccinationsAgeDemographics&format=csv'
             └ <class 'pandas.io.parsers.readers.TextFileReader'>
  File "/home/scrc/rampvis-api/data-api/venv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 811, in __init__
    self._engine = self._make_engine(self.engine)
    │              │    │            │    └ 'c'
    │              │    │            └ <pandas.io.parsers.readers.TextFileReader object at 0x7f8748100fa0>
    │              │    └ <function TextFileReader._make_engine at 0x7f8765879700>
    │              └ <pandas.io.parsers.readers.TextFileReader object at 0x7f8748100fa0>
    └ <pandas.io.parsers.readers.TextFileReader object at 0x7f8748100fa0>
  File "/home/scrc/rampvis-api/data-api/venv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 1040, in _make_engine
    return mapping[engine](self.f, **self.options)  # type: ignore[call-arg]
           │       │       │    │    │    └ {'delimiter': ',', 'escapechar': None, 'quotechar': '"', 'quoting': 0, 'doublequote': True, 'skipinitialspace': False, 'linet...
           │       │       │    │    └ <pandas.io.parsers.readers.TextFileReader object at 0x7f8748100fa0>
           │       │       │    └ 'https://api.coronavirus.data.gov.uk/v2/data?areaType=ltla&metric=vaccinationsAgeDemographics&format=csv'
           │       │       └ <pandas.io.parsers.readers.TextFileReader object at 0x7f8748100fa0>
           │       └ 'c'
           └ {'c': <class 'pandas.io.parsers.c_parser_wrapper.CParserWrapper'>, 'python': <class 'pandas.io.parsers.python_parser.PythonPa...
  File "/home/scrc/rampvis-api/data-api/venv/lib/python3.9/site-packages/pandas/io/parsers/c_parser_wrapper.py", line 51, in __init__
    self._open_handles(src, kwds)
    │    │             │    └ {'delimiter': ',', 'escapechar': None, 'quotechar': '"', 'quoting': 0, 'doublequote': True, 'skipinitialspace': False, 'linet...
    │    │             └ 'https://api.coronavirus.data.gov.uk/v2/data?areaType=ltla&metric=vaccinationsAgeDemographics&format=csv'
    │    └ <function ParserBase._open_handles at 0x7f8765946f70>
    └ <pandas.io.parsers.c_parser_wrapper.CParserWrapper object at 0x7f8748100fd0>
  File "/home/scrc/rampvis-api/data-api/venv/lib/python3.9/site-packages/pandas/io/parsers/base_parser.py", line 222, in _open_handles
    self.handles = get_handle(
    │    │         └ <function get_handle at 0x7f8765d49280>
    │    └ None
    └ <pandas.io.parsers.c_parser_wrapper.CParserWrapper object at 0x7f8748100fd0>
  File "/home/scrc/rampvis-api/data-api/venv/lib/python3.9/site-packages/pandas/io/common.py", line 609, in get_handle
    ioargs = _get_filepath_or_buffer(
             └ <function _get_filepath_or_buffer at 0x7f8765d49040>
  File "/home/scrc/rampvis-api/data-api/venv/lib/python3.9/site-packages/pandas/io/common.py", line 317, in _get_filepath_or_buffer
    reader = BytesIO(req.read())
             │       │   └ <function HTTPResponse.read at 0x7f87741655e0>
             │       └ <http.client.HTTPResponse object at 0x7f873be42a60>
             └ <class '_io.BytesIO'>
  File "/home/scrc/miniconda3/envs/py-3.9.2/lib/python3.9/http/client.py", line 471, in read
    s = self._safe_read(self.length)
        │    │          │    └ 252958900
        │    │          └ <http.client.HTTPResponse object at 0x7f873be42a60>
        │    └ <function HTTPResponse._safe_read at 0x7f87741659d0>
        └ <http.client.HTTPResponse object at 0x7f873be42a60>
  File "/home/scrc/miniconda3/envs/py-3.9.2/lib/python3.9/http/client.py", line 614, in _safe_read
    raise IncompleteRead(data, amt-len(data))
          │              │     │       └ b'areaCode,areaName,areaType,date,age,VaccineRegisterPopulationByVaccinationDate,cumPeopleVaccinatedCompleteByVaccinationDate...
          │              │     └ 252958900
          │              └ b'areaCode,areaName,areaType,date,age,VaccineRegisterPopulationByVaccinationDate,cumPeopleVaccinatedCompleteByVaccinationDate...
          └ <class 'http.client.IncompleteRead'>

http.client.IncompleteRead: IncompleteRead(209715200 bytes read, 43243700 more expected)
2022-01-12 12:30:38.394 | INFO     | app.algorithms.sim_search.cube_functions:cube_master:91 - cube_master will take a few minutes...

[phongvis]

ok, it's the IncompleteRead error again. I'll check the datastreams to directly use the the API from PHE.