Use cluster occlusion for better silhouette preservation during simplification

[Hexlord]

I read about Meshclusters and Meshoptimizer, and thought that maybe automatically created lods (through simplification by e.g. meshoptimizer) could benefit from abusing the fact that a single cluster is only viewable from a fixed number of angles (due to occlusion from the other side by own mesh), hence we could estimate how well removing a single vertex preserves the silhouette using two metrics:

1. how much the outer edges of the cluster deviate from Lod0, and
2. how much the inner shared edges of the cluster deviate from Lod0 (imagine a nose side going into the human face cheek, they share an edge, which usually means two normals and high visual noticeability, just like the silhouette from the outer edges of the mesh, which we downgrade to outer edges of the cluster in our estimation for simplicity reasons).

Then we look at our cluster with a removed vertex from a number of viewing angles (except those that are occluded) and pick median deviation factored separately for first and second heurestics (optimal coefficients could be different for types of meshes, e.g. for human face inner shared edges are very important). Then we find a vertex removing which has the lowest cost, and repeat. Then we recluster with the neighbours until a full cluster is formed (or the other things meshoptimizer already does, e.g. we could write out the new lod vertices and recalculate clusters from scratch).

[zeux]

I do have plans to look into estimating self-occlusion for clusters for the culling purposes; the main issue there is that since the calculation needs to be conservative, it's not clear how to do this efficiently.

The existing metrics try to optimize for silhouette preservation without knowledge of occlusion data. I'm not sure how much the occlusion data can help -- when clusters aren't perfectly flat there's already a lot of uncertainty in how the silhouette looks like from different profiles - looking directly at the cluster from the direction opposite to cluster normal means the silhouette contribution of the cluster is close to nil (assuming it's an interior cluster), but looking along the plane orthogonal to the normal means the silhouette contribution is maximized. Maybe this is different for clusters that are deep within a concavity which restricts the viewing angles sufficiently, but this probably doesn't need cluster information - just an "importance" weighting of different vertices - and has the same issue (how do you compute this quickly with a reasonable precision?) and may break under deformation...

Feel free to share any research that explored something like this and I can take a closer look.

[Hexlord]

Good points, if we think of grass or a small rock on the ground, we almost never have extreme angles on its clusters (so viewing angle could be approximated to a constant angle equal to the surface normal / normalized displacement of cluster relative to mesh center)

But if we think of a lamp post shaped like a "1", we do care about extreme angles and must consider different view angles, and like you say angle of zero means the silhoutte collapses into "|", perhaps we should look at each cluster from N evenly distributed positions, but with denser distribution around the surface/cluster normal (like an umbrella, with center at the tip of the normal), and with occlusion data plus meta data of whether the object is small (like a rock that we never look at upclose), we can make it even denser convering into a single view direction that we optimize for.

We probably want to use median (like avg0.5 + med0.5) of the removal cost heurestics from different viewing angles so as to respect the silhouette contribution from vertex even if it comes beneficial for very view angles, but still weigh efficiently towards removing vertices that are barely useful

The existing metrics try to optimize for silhouette preservation without knowledge of occlusion data.
That's awesome! I will take a closer look at the code