Multi channel selecting queue for worker prioritization

[albertoa]

I am having a hard time figuring out how to implement multi-channels and have specific queues listening on specific channels. All the examples in the tutorials seem to be single channel.

The idea is to have a task system where there are priorities. Workers need to work on tasks with high priorities and go to lower priority tasks as the other queues empty out.

For the discussion let's say we have 3 priorities: 1 - really important, 2 - medium importance, 3 - low importance.

Each worker can do up to 5 things at once.

So, I'm thinking about doing this via 3 queues and then have 3 channels each consume from 1 queue setting the channel qos to 5. This should throttle things across cluster workers, although not optimally since I may end up with 15 messages at the worker at any time.

Any examples on how to do that? (or maybe some other way of accomplishing the same thing)

Thanks,

Alberto

[bakkerthehacker]

The priorities of the messages should not be handled by your node client but instead by the AMQP server, which has the queue and should reorder your messages before they hit the worker. The spec says that AMQP servers MUST implement 2 priorities:

The server MUST implement at least 2 priority levels for basic messages, where priorities 0-4 and 5-9 are treated as two distinct levels.

The server MAY implement up to 10 priority levels.

However, RabbitMQ does not support priorities at all, which I assume you are using. See this list of compliance. I would suggest using the plugin for priority queues to add this support.

Priorities cannot be defined for channels. Channels should be used instead to separate each worker. The master would open the connection and each worker would open a channel.

[albertoa]

I'll take a close look at the plugin, it seems it may do what I need. If
it does then that would be my first choice.

The idea behind having multiple channels within a connection was as
a workaround to rabbitmq not supporting priorities. I am in the
design phase, so I can re-implement the code before it hits production.

Are there any examples on using multiple channels anyway? I know from
some of the posts that it is supported by bramqp, I just don't know how
to do it.

I assume you currently see channels as a way to support the interaction
with rabbitmq without conflicts when parallelizing communications (ie.
having sent a msg. to rabbit and waiting a response, then sending
another unrelated msg.) Even though nodejs is single threaded any
async io to rabbit needs its own channel. That's what you call a worker,
right?

On 1/8/2015 2:34 PM, Grant Bakker wrote:

The priorities of the messages should not be handled by your node
client but instead by the AMQP server, which has the queue and should
reorder your messages before they hit the worker. The spec says that
AMQP servers MUST implement 2 priorities:

The server MUST implement at least 2 priority levels for basic
messages, where priorities 0-4 and 5-9 are treated as two distinct
levels.

The server MAY implement up to 10 priority levels.

However, RabbitMQ does not support priorities at all, which I assume
you are using. See this list of compliance
https://www.rabbitmq.com/specification.html. I would suggest using
the plugin for priority queues
https://github.com/rabbitmq/rabbitmq-priority-queue to add this
support.

Priorities cannot be defined for channels. Channels should be used
instead to separate each worker. The master would open the connection
and each worker would open a channel.

—
Reply to this email directly or view it on GitHub
#34 (comment).

[bakkerthehacker]

Are there any examples on using multiple channels anyway?

None yet. I have been planning some advanced tutorials and this will be part of it. See #36

Even though nodejs is single threaded any async io to rabbit needs its own channel. That's what you call a worker, right?

Multiple processes are supported in node.js by the child process and cluster modules.