Shuffleboard data disappears from widgets

[Crossle86]

We have been seeing a problem with Shuffleboard that is still happening on 2023.4.2 (robot and SB). What we see is that after a period of time of successful deploys and test runs, you will approach the DS to run a test and SB widgets will be blank, though they had data moments before. Normally, the widgets always display the last data sent. We have to restart SB to get our widgets to display data. This happens one or two times a day. Haven't observed the data disappear, it's just gone the next time you look at it. Have seen this after DS PC has been closed overnight, but not always. Also observe this after a deploy, but no problem on most deploys. When the widgets are blank, SB indicates Network tables are connected. Been trying to define this issue further without success.

This is a scary proposition for competition.

[Starlight220]

Can you upload Shuffleboard log files from when this happened? Shuffleboard log files are located at ~/Shuffleboard/.

[Crossle86]

I am watching for it do it again, will get the log...hasn't happened in a match but lots of times in the pits and our lab.

[Starlight220]

In multiple logs I've seen there are messages that might indicate problems connecting between live NT sources and the sources in a layout file, but I'm not sure if that's this issue.

[nihonjinrxs]

We've also seen this behavior, and have had a similarly hard time pinning it down.

Might it be possible to put a button on the interface to hard re-sync connections to Network Tables if this happens? If that could be reliable, it could at least be a workaround for competition until we can figure out why it's happening.

If you think this is a good idea, can you point me to where in the code base to look? I'm happy to try to PR this, but it would be my first time working on this repo, so I'd need a bit of guidance to get started, particularly around code base organization and how best to test such a thing.

[Crossle86]

So we continue to have this problem on a regular basis. I have two log files that one of which should cover a failure.

shuffleboard.2023-03-24-18.36.22.log
shuffleboard.2023-03-25-08.28.33.log

[Starlight220]

Yep, there indeed is an informative error in that first log (haven't looked at the second). Something is throwing a ConcurrentModificationException when trying to reconnect a source.

[Crossle86]

Great. Caught those files at a competition so I have not looked at them. Now that I think about what you found, it may well be that this happens after SB loses connection to the robot. I can't say I have seen it after a deploy but I'm sure I see it after connection to robot is lost, though it seems happen some time after the disconnection event. Seems I notice it when returning to SB after robot has been offline and is now back online for operation.

[Starlight220]

@Crossle86 is it specific sources that don't display data, or all sources? The relevant exception in the log points to a specific data source (network_table:///Shuffleboard/Drivetrain/Front Right Module/Current Velocity); I'm not sure how a failure in restoring one source would break all sources.

[Crossle86]

All sources do not display. The source you mention set very early in our start up. It and a number of other sources are created dynamically in code by our swerve code. The rest of the sources we display are in the SB layout file. Note we are using the NT3 support in the ShuffleBoard class.

[Starlight220]

Can you test with the newly-released 2023.4.3?
If this wasn't fixed, reopen the issue.

[EddieDL]

I believe we have the latest version and are experiencing this issue still. Where can I find the Shuffleboard logs exactly and do I just need to get them when it happens?

[Starlight220]

Shuffleboard's logs are located at ~/Shuffleboard/. On Windows this would be C:/Users/<yourusername>/Shuffleboard/.
When the issue happens, close shuffleboard and upload the most recent log to here.

[EddieDL]

It just happened to me and this is the most recent log file.
shuffleboard.2023-03-31-12.20.36.log

Let me know if you need anything more.

[Crossle86]

I'm just back from vacation so I have not observed a failure myself but my students updated the SB to the latest version and told me that they have observed the same failure. I don't have a log file at this time but will catch one the next time it fails.