Enhanced Mouse Navigation: Handles cases where it is not possible to navigate a control that is covered by a container control. #17108
Labels
app/chrome
blocked/needs-external-fix
p2
https://github.com/nvaccess/nvda/blob/master/projectDocs/issues/triage.md#priority
triaged
Has been triaged, issue is waiting for implementation.
Comments
|
CC: @seanbudd, @michaelDCurran, @jcsteh, @josephsl I'm sorry to bother you guys, maybe you can offer some more insight. |
|
I don't think this is going to be possible, though it's very much something that would have to be determined on a case by case basis. In this particular case, if the panel covering the object you want is a Chromium object, we can't fix it because NVDA can only directly ask Chromium what object is at a particular point. If Chromium says the panel is at that point, that's as far as we can go. Even if it were feasible to walk every single object in the tree (which it isn't because it would be ridiculously slow), that wouldn't be sufficient because that tells you nothing about the z-index of the object, points in the rectangle that aren't covered if the object isn't actually rectangular, whether the object is transformed (skewed, rotated, etc.), whether the object is partly scrolled off the screen, etc. Hit testing is complicated enough even inside a web engine with all the internal information available to it. |
|
Hi, Do you know the window class name for the container control i.e. Electron control that is overlaying the controls underneath? Thanks. |
|
Hi,
['name: None',
'role: PANE',
'processID: 6880',
'roleText: None',
'states: ',
'isFocusable: False',
'hasFocus: False',
'Python object: <NVDAObjects.IAccessible.ia2Web.Ia2Web object at 0x05B3DD30>',
"Python class mro: (<class 'NVDAObjects.IAccessible.ia2Web.Ia2Web'>, <class "
"'NVDAObjects.IAccessible.IAccessible'>, <class 'NVDAObjects.window.Window'>, "
"<class 'NVDAObjects.NVDAObject'>, <class "
"'documentBase.TextContainerObject'>, <class 'baseObject.ScriptableObject'>, "
"<class 'baseObject.AutoPropertyObject'>, <class "
"'garbageHandler.TrackedObject'>, <class 'object'>)",
'description: None',
'location: RectLTWH(left=0, top=0, width=1920, height=1032)',
"value: ''",
"TextInfo: <class 'NVDAObjects.NVDAObjectTextInfo'>",
"appModule: AppModule(code, appName='code - insiders', processID=6880)",
"appModule.productName: 'Visual Studio Code - Insider'",
"appModule.productVersion: '1.93.0-insider'",
'appModule.helperLocalBindingHandle: c_long(118321776)',
"appModule.appArchitecture: 'AMD64'",
'windowHandle: 527444',
"windowClassName: 'Chrome_WidgetWin_1'",
'windowControlID: 0',
'windowStyle: 365363200',
'extendedWindowStyle: 256',
'windowThreadID: 8448',
"windowText: 'Visual Studio Code - Insiders'",
"displayText: ''",
'IAccessibleObject: <POINTER(IAccessible2) ptr=0x7109324 at 12f8cd50>',
'IAccessibleChildID: 0',
'IAccessible event parameters: windowHandle=527444, objectID=-4, childID=-280',
'IAccessible accName: None',
'IAccessible accRole: ROLE_SYSTEM_PANE',
'IAccessible accState: (0)',
'IAccessible accDescription: None',
"IAccessible accValue: ''",
'IAccessible2 windowHandle: 527444',
'IAccessible2 uniqueID: -280',
'IAccessible2 role: ROLE_SYSTEM_PANE',
'IAccessible2 states: IA2_STATE_OPAQUE (1024)',
"IAccessible2 attributes: 'class:View;'",
'IAccessible2 relations: ']The windowNlassName is Thanks. |
|
You might have a chance in this case because the HWND for the document and the pane is different. You'd have to override the HWND for the pane and use objectFromPointRedirect to redirect to the correct object inside the document. To get the correct object, you would use docObj.IAccessibleObject.accHitTest(x, y), where docObj is the document object you retrieved somehow. |
|
Hi, I'm sorry I don't know much about the underlying Accessibility API and COM and such, and the underlying NVDAObject. So I may need to explain more.
Why override the pane's windowHandle? Using docObj.windowHandle or obj.windowHandle redirected from docObj?
This seems to get an IAccessibleObject, what do I need to do to turn it into an NVDAObject? If the object is created and returned using NVDAObjects.IAccessible.IAccessible, won't this duplicate the object creation? I've made a few attempts, but they don't seem to be working. import IAccessibleHandler
from NVDAObjects import NVDAObject
import controlTypes
import globalPluginHandler
from NVDAObjects.IAccessible import IAccessible
from NVDAObjects.IAccessible.ia2Web import Ia2Web
from NVDAObjects.IAccessible.chromium import Document
class RedirectDocument(Ia2Web):
def objectFromPointRedirect(self, x: int, y: int):
docObj: Document = self.previous.lastChild
accHandle = IAccessibleHandler.accHitTest(docObj.IAccessibleObject, x, y)
if not accHandle:
return None
(pacc, child) = accHandle
# redirect = docObj.IAccessibleObject.accHitTest(x, y)
return IAccessible(IAccessibleObject=pacc, IAccessibleChildID=child)
class GlobalPlugin(globalPluginHandler.GlobalPlugin):
def chooseNVDAObjectOverlayClasses(self, obj: Ia2Web, clsList: list[NVDAObject]):
if (
obj.windowClassName == "Chrome_WidgetWin_1"
and obj.role == controlTypes.Role.PANE
and obj.childCount == 0
and obj.previous.lastChild.windowClassName == "Chrome_RenderWidgetHostHWND"
):
obj.windowHandle = obj.previous.lastChild.windowHandle
clsList.insert(0, RedirectDocument)Thanks. |
Err, I'm honestly not sure what I meant there. Sorry about that. Just ignore it. 😳 I think I meant to say you need to override the object for the pane.
No. The object you get back from accHitTest is a raw COM object. Passing it to the NVDAObjects.IAccessible.IAccessible constructor wraps it in an appropriate NVDAObject that NVDA can use.
No. You should use CHILDID_SELF. |
Oddly enough I can't seem to create an NVDAObject from an IAccessibleObject! By the way, mouse objects can be weird sometimes. There seems to be a discrepancy with the objects looked up from the object properties mouse == obj
True
mouse is obj
False
obj.devInfo
['name: None',
'role: PANE',
'processID: 5180',
'roleText: None',
'states: ',
'isFocusable: False',
'hasFocus: False',
'Python object: <NVDAObjects.IAccessible.ia2Web.Ia2Web object at 0x010995D0>',
"Python class mro: (<class 'NVDAObjects.IAccessible.ia2Web.Ia2Web'>, <class "
"'NVDAObjects.IAccessible.IAccessible'>, <class 'NVDAObjects.window.Window'>, "
"<class 'NVDAObjects.NVDAObject'>, <class "
"'documentBase.TextContainerObject'>, <class 'baseObject.ScriptableObject'>, "
"<class 'baseObject.AutoPropertyObject'>, <class "
"'garbageHandler.TrackedObject'>, <class 'object'>)",
'description: None',
'location: RectLTWH(left=0, top=0, width=1920, height=1032)',
"value: ''",
"TextInfo: <class 'NVDAObjects.NVDAObjectTextInfo'>",
"appModule: AppModule(code, appName='code - insiders', processID=5180)",
"appModule.productName: 'Visual Studio Code - Insider'",
"appModule.productVersion: '1.93.0-insider'",
'appModule.helperLocalBindingHandle: c_long(124620464)',
"appModule.appArchitecture: 'AMD64'",
'windowHandle: 15206884',
"windowClassName: 'Chrome_WidgetWin_1'",
'windowControlID: 0',
'windowStyle: 365363200',
'extendedWindowStyle: 256',
'windowThreadID: 31212',
"windowText: '欢迎 - Visual Studio Code - Insiders'",
"displayText: ''",
'IAccessibleObject: <POINTER(IAccessible2) ptr=0x7bb700c at 6607120>',
'IAccessibleChildID: 0',
'IAccessible event parameters: windowHandle=15206884, objectID=-4, '
'childID=-530',
'IAccessible accName: None',
'IAccessible accRole: ROLE_SYSTEM_PANE',
'IAccessible accState: (0)',
'IAccessible accDescription: None',
"IAccessible accValue: ''",
'IAccessible2 windowHandle: 15206884',
'IAccessible2 uniqueID: -530',
'IAccessible2 role: ROLE_SYSTEM_PANE',
'IAccessible2 states: IA2_STATE_OPAQUE (1024)',
"IAccessible2 attributes: 'class:View;'",
'IAccessible2 relations: ']
mouse.devInfo
['name: None',
'role: PANE',
'processID: 5180',
'roleText: None',
'states: ',
'isFocusable: False',
'hasFocus: False',
'Python object: <NVDAObjects.IAccessible.ia2Web.Ia2Web object at 0x057E9ED0>',
"Python class mro: (<class 'NVDAObjects.IAccessible.ia2Web.Ia2Web'>, <class "
"'NVDAObjects.IAccessible.IAccessible'>, <class 'NVDAObjects.window.Window'>, "
"<class 'NVDAObjects.NVDAObject'>, <class "
"'documentBase.TextContainerObject'>, <class 'baseObject.ScriptableObject'>, "
"<class 'baseObject.AutoPropertyObject'>, <class "
"'garbageHandler.TrackedObject'>, <class 'object'>)",
'description: None',
'location: RectLTWH(left=-32000, top=-32000, width=0, height=0)',
"value: ''",
"TextInfo: <class 'NVDAObjects.NVDAObjectTextInfo'>",
"appModule: AppModule(code, appName='code - insiders', processID=5180)",
"appModule.productName: 'Visual Studio Code - Insider'",
"appModule.productVersion: '1.93.0-insider'",
'appModule.helperLocalBindingHandle: c_long(124620464)',
"appModule.appArchitecture: 'AMD64'",
'windowHandle: 19333304',
"windowClassName: 'Chrome_RenderWidgetHostHWND'",
'windowControlID: 705200',
'windowStyle: 1177550848',
'extendedWindowStyle: 32',
'windowThreadID: 31212',
"windowText: 'Chrome Legacy Window'",
"displayText: ''",
'IAccessibleObject: <POINTER(IAccessible2) ptr=0x7bb700c at ce247b0>',
'IAccessibleChildID: 0',
'IAccessible event parameters: windowHandle=19333304, objectID=-4, '
'childID=-530',
'IAccessible accName: None',
'IAccessible accRole: ROLE_SYSTEM_PANE',
'IAccessible accState: (0)',
'IAccessible accDescription: None',
"IAccessible accValue: ''",
'IAccessible2 windowHandle: 15206884',
'IAccessible2 uniqueID: -530',
'IAccessible2 role: ROLE_SYSTEM_PANE',
'IAccessible2 states: IA2_STATE_OPAQUE (1024)',
"IAccessible2 attributes: 'class:View;'",
'IAccessible2 relations: ']import controlTypes
import globalPluginHandler
import winUser
from logHandler import log
from NVDAObjects import NVDAObject
from NVDAObjects.IAccessible import IAccessible
from NVDAObjects.IAccessible.chromium import Document
from NVDAObjects.IAccessible.ia2Web import Ia2Web
class RedirectDocument(Ia2Web):
def objectFromPointRedirect(self, x: int, y: int):
docObj: Document = self.previous.lastChild
log.info(
"Developer info for the document object:\n%s" % "\n".join(docObj.devInfo),
)
redirect = docObj.IAccessibleObject.accHitTest(x, y)
log.info(f"IAccessibleObject.accHitTest returned {redirect}")
obj = IAccessible(IAccessibleObject=redirect, IAccessibleChildID=winUser.CHILDID_SELF)
# log.info(
# "Developer info for the redirected object:\n%s" % "\n".join(obj.devInfo),
# )
log.info(f"Redirected object: {obj}")
return obj
class GlobalPlugin(globalPluginHandler.GlobalPlugin):
def chooseNVDAObjectOverlayClasses(self, obj: Ia2Web, clsList: list[NVDAObject]):
if (
obj.windowClassName.startswith("Chrome_")
and obj.role == controlTypes.Role.PANE
and obj.childCount == 0
and obj.previous
and obj.previous.childCount != 0
and obj.previous.lastChild.windowClassName == "Chrome_RenderWidgetHostHWND"
):
clsList.insert(0, RedirectDocument) |
|
|
Ah. I forgot accHitTest returns an IDispatch. You'll need to pass that to IAccessibleHandler.normalizeIAccessible, then pass that return value to the IAccessible NVDAObject constructor. |
|
I actually tried to automate this process using IAccessibleHandler.accHitTest, but there seems to be a problem. I will try calling normalizeIAccessible the way you described. from contextlib import contextmanager
import controlTypes
import globalPluginHandler
import IAccessibleHandler
import winUser
from logHandler import log
from NVDAObjects import NVDAObject
from NVDAObjects.IAccessible import IAccessible
from NVDAObjects.IAccessible.chromium import Document
from NVDAObjects.IAccessible.ia2Web import Ia2Web
@contextmanager
def debugLog():
import config
import logHandler
import logging
curLevel = log.getEffectiveLevel()
config.conf["general"]["loggingLevel"] = "DEBUG"
logHandler.setLogLevelFromConfig()
try:
yield
finally:
config.conf["general"]["loggingLevel"] = logging.getLevelName(curLevel)
logHandler.setLogLevelFromConfig()
class RedirectDocument(Ia2Web):
def objectFromPointRedirect(self, x: int, y: int):
with debugLog():
docObj: Document = self.previous.lastChild
log.info(
"Developer info for the document object:\n%s" % "\n".join(docObj.devInfo),
)
# redirect = docObj.IAccessibleObject.accHitTest(x, y)
redirect = IAccessibleHandler.accHitTest(docObj.IAccessibleObject, x, y)
log.info(f"IAccessibleObject.accHitTest returned {redirect}")
if not redirect:
return None
(pacc, child) = redirect
# obj = IAccessible(IAccessibleObject=redirect, IAccessibleChildID=winUser.CHILDID_SELF)
obj = IAccessible(IAccessibleObject=pacc, IAccessibleChildID=child)
# log.info(
# "Developer info for the redirected object:\n%s" % "\n".join(obj.devInfo),
# )
log.info(f"Redirected object: {obj}")
return obj
class GlobalPlugin(globalPluginHandler.GlobalPlugin):
def chooseNVDAObjectOverlayClasses(self, obj: Ia2Web, clsList: list[NVDAObject]):
if (
obj.windowClassName.startswith("Chrome_")
and obj.role == controlTypes.Role.PANE
and obj.childCount == 0
and obj.previous
and obj.previous.childCount != 0
and obj.previous.lastChild.windowClassName == "Chrome_RenderWidgetHostHWND"
):
clsList.insert(0, RedirectDocument) |
|
That's great! Thank you so much! It worked! from contextlib import contextmanager
import controlTypes
import globalPluginHandler
import IAccessibleHandler
import winUser
from logHandler import log
from NVDAObjects import NVDAObject
from NVDAObjects.IAccessible import IAccessible
from NVDAObjects.IAccessible.chromium import Document
from NVDAObjects.IAccessible.ia2Web import Ia2Web
@contextmanager
def debugLog():
import config
import logHandler
import logging
curLevel = log.getEffectiveLevel()
config.conf["general"]["loggingLevel"] = "DEBUG"
logHandler.setLogLevelFromConfig()
try:
yield
finally:
config.conf["general"]["loggingLevel"] = logging.getLevelName(curLevel)
logHandler.setLogLevelFromConfig()
class RedirectDocument(Ia2Web):
def objectFromPointRedirect(self, x: int, y: int):
with debugLog():
docObj: Document = self.previous.lastChild
log.info(
"Developer info for the document object:\n%s" % "\n".join(docObj.devInfo),
)
redirect = docObj.IAccessibleObject.accHitTest(x, y)
# redirect = IAccessibleHandler.accHitTest(docObj.IAccessibleObject, x, y)
log.info(f"IAccessibleObject.accHitTest returned {redirect}")
if not redirect:
return None
redirect = IAccessibleHandler.normalizeIAccessible(redirect)
# (pacc, child) = redirect
obj = IAccessible(IAccessibleObject=redirect, IAccessibleChildID=winUser.CHILDID_SELF)
# obj = IAccessible(IAccessibleObject=pacc, IAccessibleChildID=child)
# log.info(
# "Developer info for the redirected object:\n%s" % "\n".join(obj.devInfo),
# )
log.info(f"Redirected object: {obj}")
return obj
class GlobalPlugin(globalPluginHandler.GlobalPlugin):
def chooseNVDAObjectOverlayClasses(self, obj: Ia2Web, clsList: list[NVDAObject]):
if (
obj.windowClassName.startswith("Chrome_")
and obj.role == controlTypes.Role.PANE
and obj.childCount == 0
and obj.previous
and obj.previous.childCount != 0
and obj.previous.lastChild.windowClassName == "Chrome_RenderWidgetHostHWND"
):
clsList.insert(0, RedirectDocument) |
|
Yeah, looking at the code, it looks to me like IAccessibleHandler.accHitTest might be broken in this situation. I'm not certain, but it's very, very old code, so it wouldn't surprise me. It's probably just easier to use IAccessible.accHitTest directly for now. |
|
This issue can be fixed in NVDA and I intend to release the above code as an add-on. However, this add-on uses an API that was introduced in 2024.4. Is it necessary to add this fix to the NVDA core? |
|
If this issue gets triaged I would encourage fixing it within NVDA rather than an add-on, depending on the approach. |
That I understand. Then I will release it today. Doubtful though considering the issue is very widespread and it's not clear when Electron will fix it. |
seanbudd
added
p2
|
Closing - we don't think its possible to consistently handle these cases. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is your feature request related to a problem? Please describe.
Some time ago I discovered that mouse tracking doesn't work on most controls in the latest version of VS Code.
This regression was introduced by Electron.
The reason seems to be that an empty panel is covering all the controls, causing the mouse object to focus only on this invalid panel.
Although we can probably wait for Electron to fix this issue.
But, until Electron fixes it, this will affect quite a few applications.
And there are many similar cases in other programmes, such as Windows Terminal.
Describe the solution you'd like
When looking up an object via
objectFromPoint, you can ignore the upper level object and find the lower level object.I'm not sure this is the right direction, the logic of finding objects is complex, especially with all the inheritance and dynamic binding
Describe alternatives you've considered
Additional context
#13506
electron/electron#42945
microsoft/vscode#224704
The text was updated successfully, but these errors were encountered: