Skip to content

Commit f66ce06

Browse files
committed
Add support for SIGNAL_ENQUEUED.
1 parent 13a377c commit f66ce06

File tree

4 files changed

+54
-21
lines changed

4 files changed

+54
-21
lines changed

docs/signals.rst

+30-5
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ The following signals are implemented by Huey:
1212
* ``SIGNAL_CANCELED``: task was canceled due to a pre-execute hook raising
1313
a :py:class:`CancelExecution` exception.
1414
* ``SIGNAL_COMPLETE``: task has been executed successfully.
15+
* ``SIGNAL_ENQUEUED``: task has been enqueued (**see note**).
1516
* ``SIGNAL_ERROR``: task failed due to an unhandled exception.
1617
* ``SIGNAL_EXECUTING``: task is about to be executed.
1718
* ``SIGNAL_EXPIRED``: task expired.
@@ -33,6 +34,14 @@ The following signals will include additional arguments:
3334
* ``SIGNAL_ERROR``: includes a third argument ``exc``, which is the
3435
``Exception`` that was raised while executing the task.
3536

37+
.. note::
38+
Signals are run within the context of the consumer **except** that the
39+
``SIGNAL_ENQUEUED`` signal will also run within the context of your
40+
application code (since your application code will typically enqueue
41+
tasks). Recall that signal handlers are run sequentially and synchronously,
42+
so be careful about introducing overhead in them -- particularly when they
43+
may be run by the application process.
44+
3645
To register a signal handler, use the :py:meth:`Huey.signal` method:
3746

3847
.. code-block:: python
@@ -88,8 +97,10 @@ Here is a simple example of a task execution we would expect to succeed:
8897
>>> result = add(1, 2)
8998
>>> result.get(blocking=True)
9099
91-
The consumer would send the following signals:
100+
The following signals would be fired:
92101

102+
* ``SIGNAL_ENQUEUED`` - the task has been enqueued (happens in the application
103+
process).
93104
* ``SIGNAL_EXECUTING`` - the task has been dequeued and will be executed.
94105
* ``SIGNAL_COMPLETE`` - the task has finished successfully.
95106

@@ -102,10 +113,13 @@ Here is an example of scheduling a task for execution after a short delay:
102113
103114
The following signals would be sent:
104115

116+
* ``SIGNAL_ENQUEUED`` - the task has been enqueued (happens in the **application**
117+
process).
105118
* ``SIGNAL_SCHEDULED`` - the task is not yet ready to run, so it has been added
106119
to the schedule.
107-
* After 10 seconds, the consumer will run the task and send
108-
the ``SIGNAL_EXECUTING`` signal.
120+
* After 10 seconds, the consumer will re-enqueue the task as it is now ready to
121+
run, sending the ``SIGNAL_ENQUEUED`` (in the **consumer** process!).
122+
* Then the consumer will run the task and send the ``SIGNAL_EXECUTING`` signal.
109123
* ``SIGNAL_COMPLETE``.
110124

111125
Here is an example that may fail, in which case it will be retried
@@ -124,11 +138,14 @@ automatically with a delay of 10 seconds.
124138
Assuming the task failed the first time and succeeded the second time, we would
125139
see the following signals being sent:
126140

141+
* ``SIGNAL_ENQUEUED`` - task has been enqueued.
127142
* ``SIGNAL_EXECUTING`` - the task is being executed.
128143
* ``SIGNAL_ERROR`` - the task raised an unhandled exception.
129144
* ``SIGNAL_RETRYING`` - the task will be retried.
130145
* ``SIGNAL_SCHEDULED`` - the task has been added to the schedule for execution
131146
in ~10 seconds.
147+
* ``SIGNAL_ENQUEUED`` - 10s have elapsed and the task is ready to run and has
148+
been re-enqueued.
132149
* ``SIGNAL_EXECUTING`` - second try running task.
133150
* ``SIGNAL_COMPLETE`` - task succeeded.
134151

@@ -141,6 +158,7 @@ What happens if we revoke the ``add()`` task and then attempt to execute it:
141158
142159
The following signal will be sent:
143160

161+
* ``SIGNAL_ENQUEUED`` - the task has been enqueued for execution.
144162
* ``SIGNAL_REVOKED`` - this is sent before the task enters the "executing"
145163
state. When a task is revoked, no other signals will be sent.
146164

@@ -169,8 +187,9 @@ Performance considerations
169187
--------------------------
170188

171189
Signal handlers are executed **synchronously** by the consumer as it processes
172-
tasks. It is important to use care when implementing signal handlers, as one
173-
slow signal handler can impact the overall responsiveness of the consumer.
190+
tasks (with the exception of ``SIGNAL_ENQUEUED``). It is important to use care
191+
when implementing signal handlers, as one slow signal handler can impact the
192+
overall responsiveness of the consumer.
174193

175194
For example, if you implement a signal handler that posts some data to REST
176195
API, everything might work fine until the REST API goes down or stops being
@@ -183,3 +202,9 @@ handles. Signal handlers are called by the consumer workers, which (depending
183202
on how you are running the consumer) may be separate processes, threads or
184203
greenlets. As a result, care should be taken to ensure proper initialization
185204
and cleanup of any resources you plan to use in signal handlers.
205+
206+
Lastly, take care when implementing ``SIGNAL_ENQUEUED`` handlers, as these may
207+
run in your application-code (e.g. whenever your application enqueues a task),
208+
**or** by the consumer process (e.g. when re-enqueueing a task for retry, or
209+
when enqueueing periodic tasks, when moving a task from the schedule to the
210+
queue, etc).

huey/api.py

+2
Original file line numberDiff line numberDiff line change
@@ -305,6 +305,8 @@ def enqueue(self, task):
305305
if task.expires:
306306
task.resolve_expires(self.utc)
307307

308+
self._emit(S.SIGNAL_ENQUEUED, task)
309+
308310
if self._immediate:
309311
self.execute(task)
310312
else:

huey/signals.py

+2-1
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,8 @@
1010
SIGNAL_RETRYING = 'retrying'
1111
SIGNAL_REVOKED = 'revoked'
1212
SIGNAL_SCHEDULED = 'scheduled'
13-
SIGNAL_INTERRUPTED = "interrupted"
13+
SIGNAL_INTERRUPTED = 'interrupted'
14+
SIGNAL_ENQUEUED = 'enqueued'
1415

1516

1617
class Signal(object):

huey/tests/test_signals.py

+20-15
Original file line numberDiff line numberDiff line change
@@ -23,17 +23,17 @@ def task_a(n):
2323
return n + 1
2424

2525
r = task_a(3)
26-
self.assertSignals([])
26+
self.assertSignals([SIGNAL_ENQUEUED])
2727
self.assertEqual(self.execute_next(), 4)
2828
self.assertSignals([SIGNAL_EXECUTING, SIGNAL_COMPLETE])
2929

3030
r = task_a.schedule((2,), delay=60)
31-
self.assertSignals([])
31+
self.assertSignals([SIGNAL_ENQUEUED])
3232
self.assertTrue(self.execute_next() is None)
3333
self.assertSignals([SIGNAL_SCHEDULED])
3434

3535
r = task_a(None)
36-
self.assertSignals([])
36+
self.assertSignals([SIGNAL_ENQUEUED])
3737
self.assertTrue(self.execute_next() is None)
3838
self.assertSignals([SIGNAL_EXECUTING, SIGNAL_ERROR])
3939

@@ -58,9 +58,10 @@ def task_a(n):
5858
return n + 1
5959

6060
r = task_a(None)
61-
self.assertSignals([])
61+
self.assertSignals([SIGNAL_ENQUEUED])
6262
self.assertTrue(self.execute_next() is None)
63-
self.assertSignals([SIGNAL_EXECUTING, SIGNAL_ERROR, SIGNAL_RETRYING])
63+
self.assertSignals([SIGNAL_EXECUTING, SIGNAL_ERROR, SIGNAL_RETRYING,
64+
SIGNAL_ENQUEUED])
6465
self.assertTrue(self.execute_next() is None)
6566
self.assertSignals([SIGNAL_EXECUTING, SIGNAL_ERROR])
6667

@@ -69,7 +70,7 @@ def task_b(n):
6970
return n + 1
7071

7172
r = task_b(None)
72-
self.assertSignals([])
73+
self.assertSignals([SIGNAL_ENQUEUED])
7374
self.assertTrue(self.execute_next() is None)
7475
self.assertSignals([SIGNAL_EXECUTING, SIGNAL_ERROR, SIGNAL_RETRYING,
7576
SIGNAL_SCHEDULED])
@@ -81,13 +82,14 @@ def task_a(n):
8182

8283
task_a.revoke(revoke_once=True)
8384
r = task_a(2)
84-
self.assertSignals([])
85+
self.assertSignals([SIGNAL_ENQUEUED])
8586
self.assertTrue(self.execute_next() is None)
8687
self.assertSignals([SIGNAL_REVOKED])
8788

8889
r = task_a(3)
8990
self.assertEqual(self.execute_next(), 4)
90-
self.assertSignals([SIGNAL_EXECUTING, SIGNAL_COMPLETE])
91+
self.assertSignals([SIGNAL_ENQUEUED, SIGNAL_EXECUTING,
92+
SIGNAL_COMPLETE])
9193

9294
def test_signals_locked(self):
9395
@self.huey.task()
@@ -96,13 +98,13 @@ def task_a(n):
9698
return n + 1
9799

98100
r = task_a(1)
99-
self.assertSignals([])
101+
self.assertSignals([SIGNAL_ENQUEUED])
100102
self.assertEqual(self.execute_next(), 2)
101103
self.assertSignals([SIGNAL_EXECUTING, SIGNAL_COMPLETE])
102104

103105
with self.huey.lock_task('lock-a'):
104106
r = task_a(2)
105-
self.assertSignals([])
107+
self.assertSignals([SIGNAL_ENQUEUED])
106108
self.assertTrue(self.execute_next() is None)
107109
self.assertSignals([SIGNAL_EXECUTING, SIGNAL_LOCKED])
108110

@@ -114,12 +116,12 @@ def task_a(n):
114116
now = datetime.datetime.now()
115117
expires = now + datetime.timedelta(seconds=15)
116118
r = task_a(2)
117-
self.assertSignals([])
119+
self.assertSignals([SIGNAL_ENQUEUED])
118120
self.assertTrue(self.execute_next(expires) is None)
119121
self.assertSignals([SIGNAL_EXPIRED])
120122

121123
r = task_a(3)
122-
self.assertSignals([])
124+
self.assertSignals([SIGNAL_ENQUEUED])
123125
self.assertTrue(self.execute_next(), 4)
124126
self.assertSignals([SIGNAL_EXECUTING, SIGNAL_COMPLETE])
125127

@@ -138,18 +140,21 @@ def task_a(n):
138140
self.assertEqual(extra_state, [])
139141
self.assertEqual(self.execute_next(), 4)
140142
self.assertEqual(extra_state, [3])
141-
self.assertSignals([SIGNAL_EXECUTING, SIGNAL_COMPLETE])
143+
self.assertSignals([SIGNAL_ENQUEUED, SIGNAL_EXECUTING,
144+
SIGNAL_COMPLETE])
142145

143146
r2 = task_a(1)
144147
self.assertEqual(self.execute_next(), 2)
145148
self.assertEqual(extra_state, [3, 1])
146-
self.assertSignals([SIGNAL_EXECUTING, SIGNAL_COMPLETE])
149+
self.assertSignals([SIGNAL_ENQUEUED, SIGNAL_EXECUTING,
150+
SIGNAL_COMPLETE])
147151

148152
self.huey.disconnect_signal(extra_handler, SIGNAL_EXECUTING)
149153
r3 = task_a(2)
150154
self.assertEqual(self.execute_next(), 3)
151155
self.assertEqual(extra_state, [3, 1])
152-
self.assertSignals([SIGNAL_EXECUTING, SIGNAL_COMPLETE])
156+
self.assertSignals([SIGNAL_ENQUEUED, SIGNAL_EXECUTING,
157+
SIGNAL_COMPLETE])
153158

154159
def test_multi_handlers(self):
155160
state1 = []

0 commit comments

Comments
 (0)