shard 1 is reporting lower to_sign

[sophoah]

Describe the bug
Harmony staking protocol should account for 32768 blocks every epoch. However, first epoch of HIP32 (leader rotation) the shards only accounted for 32704 in shard 0 and 29910 for shard 1.

To Reproduce
NA

Expected behavior
During normal operation shard 0 and shard 1 should be equal 32768 or a bit above for shard 1 due to the shard 1 taking few more seconds to learn about the new epoch

Screenshots
shard 0 :

shard 1:

Environment (please complete the following information):
NA

Additional context
and to add on to the issue we have two shard 1 validator with different to_sign :

• https://harmony.smartstake.io/val/614
• https://harmony.smartstake.io/val/392

[sophoah]

Checking today, only shard 1 is impacted. I suspect when I last checked it was too near the new epoch and crosslink were still being processed.

[sophoah]

After developing a script to analyze the block production the issue is because shard 1 is slower to produce blocks

Shard 0 result :

Total blocks processed: 32768 between blocks 65017896 and 65050663
Total time: 65978 seconds
block time: 2.01348876953125 (vs 2s)
Total blocks time should be : 65536s
Total time difference: 442 seconds
Missing blocks production number: 221.0
Number of blocks with time difference above 2s: 755

Shard 1 result:

Total blocks processed: 32768 between blocks 67775889 and 67808656
Total time: 71869 seconds
block time: 2.193267822265625 (vs 2s)
Total blocks time should be : 65536s
Total time difference: 6333 seconds
Missing blocks production number: 3166.5
Number of blocks with time difference above 2s: 3778

with a slower block production time that shard 0, it is expected to have less blocks