Topics not shown when using a wildcard

[kevinwieland]

If there is a given retained topic f.e.:

user/hello/foo/bar

If I do:
mosquitto_sub -v -t 'userA/hello/foo/bar'
This will with 100% give me the output I suggested.

If I do:
mosquitto_sub -v -t '+/hello/foo/bar'
To get the topics from various Users there is a chance the topic with userA is not shown.
Even If I wait 5 Minutes.

Could be possible that half an hour later it's shown, while
mosquitto_sub -v -t 'userA/hello/foo/bar'
always and instantly outputs the topic.

Any ideas how to debug that?

[ralight]

This works fine for me locally, running without any config:

I guess there may be something in your config that is behaving oddly, can you share it?

[kevinwieland]

This is on a standard config, nothing special.
The problem occurs not always which makes it difficult to debug.
In 99% of the tries its working without a problem.
If it's not working while using
mosquitto_sub -v -t '+/hello/foo/bar'
even waiting 5 minutes doesn't help.

[ralight]

Ugh, how annoying! :)

Is there any chance that the message is being published with the MQTT v5 message-expiry-interval property set, and hence it is expiring?

[kevinwieland]

@ralight
No. If I it's not showing up with that command
mosquitto_sub -v -t '+/hello/foo/bar'
I can do a STRG+C and 5 seconds later a:
mosquitto_sub -v -t 'userA/hello/foo/bar'
and I see the message instantly.

[ralight]

Are there any more details you can share that might help? I've successfully run mosquitto_sub -C 1 -v -t +/hello/foo/bar 10,000 times in a row here.

It sounds like it's happening fairly often, at least often enough for you to test. Are you able to provide an exact process to reproduce it? I don't know if you're just running a single client or 10,000 clients, for example. Likewise what version you're on.

These are some other questions that might be useful to have answers for:

How frequently does it occur? What version are you on? What is it installed on (docker, ubuntu, windows, raspberry pi, ...)? Does it do it if you've published to that topic once only, or are you publishing multiple times? Are you publishing to other topics at the same time, or have you reproduced it with just that topic in use? Do any other mqtt clients show the same problem? If you have the problem with the + and then do a STRG+C and repeat it with the +, does it still not work? Really anything you can think of that might help to reproduce it would help, even if it seems a trivial thing.

[kevinwieland]

Are there any more details you can share that might help?

Sure

I don't know if you're just running a single client or 10,000 clients, for example.

In total there are 60k Clients connected. Like 4000 clients do have values in that topics.
From that 4000 clients 50-60 are not always working with wildcard

Are you able to provide an exact process to reproduce it?

For me, I can reproduce this quite on some 'userX' names. Others are working. Maybe the next day another set of users is not working.

How frequently does it occur?

Always, but with different userX names.

What version are you on?
I tried 2.0.11 and 2.0.20

What is it installed on (docker, ubuntu, windows, raspberry pi, ...)?

Ubuntu 22 LTS

Does it do it if you've published to that topic once only, or are you publishing multiple times?

The clients have configured a bridge. They're publishing once and retained. But it could be that - without doing anything - 2 hours later die exact same userX is visible with the mosquitto_sub -v -t '+/hello/foo/bar' command. But it is always instantly visible with the mosquitto_sub -v -t 'userA/hello/foo/bar' command.

Are you publishing to other topics at the same time, or have you reproduced it with just that topic in use?

There is another Topic in the /hello/foo/foo tree which also is not available through the wildcard on certain users, while it is available directly.

Do any other mqtt clients show the same problem?

There are different userX with that problem

If you have the problem with the + and then do a STRG+C and repeat it with the +, does it still not work?

Yes, problem still persists.

If I can give any more information or could test something, let me know :)

[ckrey]

I guess it is a symptom of the problem discussed in #2887 etc. which seems not to be easy to fix

[kevinwieland]

@ckrey
I've tested 2.0.11 and 2.0.20. Seems to be a different issue.

But I tried a bit more and what I've found out:
It doesn't matter which client(machine) I use remotely (Linux / Mac) and if it's in the same data center (10GB Uplink) or over the internet.
When I got a "problem topic" I won't get it through
mosquitto_sub -v -t '+/hello/foo/bar' -h server
from different Clients.
From every client I get the topic with:
mosquitto_sub -v -t 'userA/hello/foo/bar' -h server

When I use the same command directly on the shell of the Mosquito server and not a remote machine
mosquitto_sub -v -t '+/hello/foo/bar' -h server
It's always working and gives me every topic I expect instantly.

But on any client machine I won't get it. Maybe a couple of hours later.

How to debug that any further?

[ralight]

Thanks for the details - it's otherwise very difficult to tell the difference between "I'm running home automation on a Pi at home" and "I've got 60k clients connected to a data centre".

So just to be clear, if you find a problem topic then it works when connecting remotely if you use topic userA/foo/bar, it doesn't work when connection remotely if you use topic +/foo/bar, but if you connect locally using the topic +/foo/bar then it works? That is extremely odd!

I'd still really like to see your config file if possible. If you're not comfortable posting it here then could you send it to [email protected]? What method are you using for topic access control?

[kevinwieland]

but if you connect locally using the topic +/foo/bar then it works? That is extremely odd!

Exactly.

[ralight]

There's no reason I can think of on the mosquitto side why this should be happening. I agree that the config you sent isn't anything unusual. Next time you see this, could you add -d to the mosquitto_sub call? This will print out debug information as well which may show there is something going on at the network level. Do you have a proxy or load balancer in front of the broker?

[kevinwieland]

I can reproduce that.
mosquitto_sub -v -t '+/hello/foo/bar' -h server -d
Client (null) sending CONNECT Client (null) received CONNACK (0) Client (null) sending SUBSCRIBE (Mid: 1, Topic: +/foo/bar, QoS: 0, Options: 0x00) Client (null) received SUBACK Subscribed (mid: 1): 0
For every topic that is found I get:
Client (null) received PUBLISH (d0, q0, r1, m0, 'topic....', ... (xx bytes))

All topics are coming within 2-3 seconds. From a remote machine getting 1958 Messages in my test.

When doing this from the server itself with the same command I'm getting 2212 messages.

When I'm grepping for one of those topics that don't show up on a remote machine with -d attached I get:
Client (null) received PUBLISH (d0, q0, r1, m0, 'topic', .... (50bytes))

Do you have a proxy or load balancer in front of the broker?

No.

[ralight]

Right, I think Christoph nailed it actually.

On the local machine the transfer happens quickly so the queue on the broker doesn't fill up. When you are connecting remotely the speed is lower, so the broker fills up the queue and some messages are dropped. Try setting max_queued_messages 3000 and see if it solves it.

[kevinwieland]

Set this to 5000 for this.
We're getting closer! Thanks!
Remote machine gets 2210 Topics within 10 seconds.
Same command on the mosquitto broker gives me 2212 Topics.
There 2 Topics missing, but they are always the same on different machines.