Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow connection_pool to be set in RedisSettings #473

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

brian-goo
Copy link

@brian-goo brian-goo commented Jul 28, 2024

@samuelcolvin
Thank you for a great library!
It would be really grateful if this PR could be reviewed or a workaround for this issue provided.
Many thanks.


We are using arq for our production app, and it appears that connection pool throws an error and arq worker crashes if the connection limit(max_connections) is reached.
Not only does worker but the enqueue_job method also throws an error under the same condition.

This seems to be caused by the default ConnectionPool of redis-py.
redis/redis-py#2517

Code to reproduce:

import asyncio
from arq import ArqRedis, create_pool
from arq.connections import RedisSettings

REDIS_SETTINGS = RedisSettings(
    max_connections=1
)

async def foo(ctx) -> None:
    print('ok')

async def main() -> None:
    redis: ArqRedis = await create_pool(REDIS_SETTINGS)
    for _ in range(2):
        await redis.enqueue_job('foo')

class Worker:
    functions = [foo]
    redis_settings = REDIS_SETTINGS

if __name__ == '__main__':
    asyncio.run(main())

By allowing connection_pool to be set in RedisSettings, it seems to work as expected.

import asyncio
from arq import ArqRedis, create_pool
from arq.connections import RedisSettings
from redis.asyncio import BlockingConnectionPool

REDIS_SETTINGS = RedisSettings(
    connection_pool=BlockingConnectionPool(max_connections=1)
)

async def foo(ctx) -> None:
    print("ok")

async def main() -> None:
    redis: ArqRedis = await create_pool(REDIS_SETTINGS)
    for _ in range(10):
        await redis.enqueue_job("foo")

class Worker:
    functions = [foo]
    redis_settings = REDIS_SETTINGS

if __name__ == "__main__":
    asyncio.run(main())

It appears that Worker can take redis_pool as an argument, but I guess we can handle this case more concisely this way for both Worker and Enqueuer, provided it does not break anything else.

@codecov-commenter
Copy link

codecov-commenter commented Jul 28, 2024

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

All modified and coverable lines are covered by tests ✅

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

@@            Coverage Diff             @@
##             main     #473      +/-   ##
==========================================
- Coverage   96.27%   96.01%   -0.27%     
==========================================
  Files          11       11              
  Lines        1074     1079       +5     
  Branches      209      190      -19     
==========================================
+ Hits         1034     1036       +2     
- Misses         19       21       +2     
- Partials       21       22       +1     
Files Coverage Δ
arq/connections.py 87.50% <100.00%> (-2.57%) ⬇️

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3f865a6...6048460. Read the comment docs.

@@ -19,7 +19,7 @@ def test_settings_changed():
settings = RedisSettings(port=123)
assert settings.port == 123
assert (
"RedisSettings(host='localhost', port=123, unix_socket_path=None, database=0, username=None, password=None, "
"RedisSettings(connection_pool=None, host='localhost', port=123, unix_socket_path=None, database=0, username=None, password=None, "
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably worth adding a test where the connectionpool is set

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants