Potential ideas for addressing Linux fractional scaling artifacts and fuzzy text

[lbrame]

Currently, Ghostty has some bugs on Linux on fractionally-scaled display outputs on a Wayland compositor. Two notable issues are #1938 and #3412 .

After doing some digging and debugging on my own, I came up with (now-closed) PR #3419 . What this PR did was a quick and dirty experiment of removing the code that disabled Vulkan and prevented the ngl renderer for GTK. I think there are some good arguments for enabling ngl, that I have better discussed in my PR, such as much improved positioning and anti-aliasing with fractional scaling.

However, I later came across #1891 after following the Blame history for the piece of code I had deleted to see where it came from. Apparently, ngl was explicitly disabled to resolve a problem some users experienced with flickering on tab destroy / detach and other operations, as documented by issue #1667.

I am personally not able to reproduce #1667 on my current setup, but I am well aware that these graphical things can go wrong at any layer in the stack, including Vulkan implementation, GPU driver, etc. FWIW, I am using a Ryzen 7 7840HS APU, which includes a RDNA3-based Radeon 780M iGPU, which leverages the amdgpu kernel driver and the radv Vulkan implementation.

I am also aware ngl comes with a slight performance penalty on some systems, though I am unable to reproduce it on mine (but things like small launch time differences are hard to detect on high-end hardware).

The idea

It looks like there are conflicting issues here. On one hand, there are arguments for enabling ngl and improving the experience for users using systems with fractional scaling. On the other, ngl seems to cause performance and graphical issues to users with certain system configurations.

The idea I had was to go back to the drawing board and try coming up with a "toggleable" solution that can be configured at compile time and that is disabled by default. That way:

• Users could more easily make the choice on what set of compromises to take; users running on systems that are not affected by the performance and graphical issues introduced by ngl could choose to have a better HiDPI experience instead
• Linux distributions could decide whether they prioritize launch performance or HiDPI experience
• The default, vanilla ghostty behaviour would not be affected.

Another idea I had to support this one was to informally survey what kind of hardware people who are experiencing these artifacts are using, in an attempt to track down the culprit. This would potentially allow to:

• Add some checks to the GTK app initialization and i.e. serve systems with a buggy Vulkan implementation the opengl backend or something to that effect
• Report the issue upstream to whatever component is responsible for it

[jokeyrhyme]

I'm on an AMD iGPU + AMD dGPU system primarily, and have no Vulkan issues at all

However, I did notice an issue a little while ago on my Intel iGPU + nVidia dGPU laptop where simply enumerating Vulkan devices/drivers seemed to take multiple seconds, which means simple auto-detection code led to a much worse UX

I'm hoping whatever driver issues have since been solved on the nVidia side ...