stored message count not changing

[edgale]

Hi there,

I am trying to use $SYS/broker/store/messages/count to get insight how a client performs under heavy load.
While testing the client subscribed to a topic can not handle the message amount in time and the messages get droped;

Feb 18 14:36:42 mosquitto[370185]: 1739885650: Outgoing messages are being dropped for client client-id.

While testing I subscribe to $SYS/broker/store/messages/count with mosquitto_sub and there the value does not change and is always 53:

~$ mosquitto_sub -h localhost -p 1884 -t '$SYS/broker/store/messages/count' -v
$SYS/broker/store/messages/count 53

Also this value does not get updated while I am testing or when nothing is running.

Am I doing something wrong? I thought this $SYS topic would show me how many messages are in queue (as per doc):

$SYS/broker/store/messages/count,
$SYS/broker/messages/stored (deprecated)
The number of messages currently held in the message store. This includes retained messages and messages queued for durable clients.

Config:

# global configuration
per_listener_settings true
log_dest syslog
log_dest stdout
log_dest topic
#log_type all
connection_messages true
log_timestamp true

# listener 1
listener 1884 localhost
allow_anonymous true

# listener 2
listener 1883
password_file /etc/mosquitto/passwd

mosquitto version 2.0.20

Thanks for your help.

[Daedaluz]

What type of clients are you using then?

Also; you can check the stored messages with mosquitto_db_dump.

[edgale]

I am using libmosquitto for publishing the messages and rumqttc for subscribing.

I will try mosquitto_db_dump. For this I would need:

persistence true
persistence_location /some/location

right?

Currently I am not using a persistence db

[Daedaluz]

I was more looking for the specifics, like clean session client id etc

[edgale]

Here's the setup for the publishing client:

void Setup() {
  mosquitto_lib_init();
  mosq_ = mosquitto_new(nullptr, true, nullptr);
  mosquitto_loop_start(mosq_);
  mosquitto_connect_async(mosq_, remote.c_str(), port, keepalive);
}

void SendMsg() {
  mosquitto_publish(mosq_, nullptr, topic.c_str(), len, payload, 1, false);
}

• clean session = true
• client id = a random generated one
• qos = 1

For the subscribing client I use this:

let mut mqttoptions = MqttOptions::new("client-id", "localhost", 1884);
mqttoptions.set_keep_alive(Duration::from_secs(60));
mqttoptions.set_clean_session(false);
...
client
  .subscribe(topic, QoS::AtLeastOnce)
  .await?;

• clean session = false
• client id = some hard coded id
• keep alive = 60
• qos = 1

I also tried qos 0 and 2. This resulted in the same outcome.
With mosquitto_db_dump I get the same output while the client is publishing:

$ ./mosquitto_db_dump --stats /var/lib/mosquitto/mosquitto.db
DB_CHUNK_CFG:        1
DB_CHUNK_MSG_STORE:  0
DB_CHUNK_CLIENT_MSG: 0
DB_CHUNK_RETAIN:     0
DB_CHUNK_SUB:        0
DB_CHUNK_CLIENT:     0

[Daedaluz]

I wonder if the store count is only changing when the durable client is unavailable.

DB_CHUNK_CLIENT: 0 suggests no clients stored either.

1: Start subscriber with clean session = false
2: Make subscriptions with qos > 0
3: Stop the subscriber

4: killall -USR1 mosquitto (this will force mosquitto to save the database to disk)
5: mosquitto_db_dump --stats /var/lib/mosquitto/mosquitto.db

6: Start publisher
7: Publish a few messages on subscribed topic with qos > 0
7: Stop publisher

8: Repeat 4 and 5

Now there should be messages in store with one client.

[edgale]

This changed the output of mosquitto_db_dump:

$ mosquitto_db_dump --stats /var/lib/mosquitto/mosquitto.db
DB_CHUNK_CFG:        1
DB_CHUNK_MSG_STORE:  8
DB_CHUNK_CLIENT_MSG: 0
DB_CHUNK_RETAIN:     0
DB_CHUNK_SUB:        15
DB_CHUNK_CLIENT:     2

But the topic $SYS/broker/store/messages/count still reports 53

[ralight]

Using persistence or not doesn't change anything for this value - it's the number of messages stored in memory. You would expect it to increase when retained messages are published to new topics, or when there are messages published to the broker that are yet to be delivered to a subscriber - exactly as has been discussed already.

If there is a subscriber that is not keeping up, and hence messages are being dropped for that client, then you wouldn't expect the count to increase - however it doesn't look like you have changed the max queue length which is much bigger than 53 by default. The other point is that if you're using QoS 0 messages, then messages will only be stored very briefly.

[ralight]

This changed the output of mosquitto_db_dump:

$ mosquitto_db_dump --stats /var/lib/mosquitto/mosquitto.db
DB_CHUNK_CFG:        1
DB_CHUNK_MSG_STORE:  8
DB_CHUNK_CLIENT_MSG: 0
DB_CHUNK_RETAIN:     0
DB_CHUNK_SUB:        15
DB_CHUNK_CLIENT:     2

But the topic $SYS/broker/store/messages/count still reports 53

I would expect your subscriber to have messages queued for it (DB_CHUNK_CLIENT_MSG > 0) with the steps Daedaluz suggested. DB_CHUNK_MSG_STORE is effectively the message as published to the broker, then DB_CHUNK_CLIENT_MSG and DB_CHUNK_RETAIN are references to that message as used when sending to subscribers and as a retained message, respectively. It's a bit odd that DB_CHUNK_MSG_STORE is 8 and the others are 0.

[edgale]

Using persistence or not doesn't change anything for this value - it's the number of messages stored in memory. You would expect it to increase when retained messages are published to new topics, or when there are messages published to the broker that are yet to be delivered to a subscriber - exactly as has been discussed already.

If there is a subscriber that is not keeping up, and hence messages are being dropped for that client, then you wouldn't expect the count to increase - however it doesn't look like you have changed the max queue length which is much bigger than 53 by default. The other point is that if you're using QoS 0 messages, then messages will only be stored very briefly.

But I am right in trying to observe $SYS/broker/store/messages/count to get an idea of the queue size, right? If the subscriber is not keeping up I would assume the count is getting bigger until the max queue length is hit and then the queue/messages are droped.

[ralight]

@edgale Yes you're right. By the way, is this a work thing?

[edgale]

@edgale Yes you're right. By the way, is this a work thing?

Ok. Thanks.
Yes I am currently testing this for work.

[ralight]

In that case, you could try getting one of the free trials at cedalo.com - this gives you a preconfigured instance of mosquitto to use for two weeks. If there's something odd about your setup for some reason then this eliminates that factor.

I'm part of cedalo, we also offer support for mosquitto and extra features for businesses - which is the version that's running on the trials.