Create tool for comparing GUI and UI elements in copies of NVDA (nvac…

…cess#12308)

GUI and UI features for NVDA can become lost through code refactors. Generating screenshots and comparing text across copies of NVDA is an annoying process for developers.

A tool is created using our system tests to generate screenshots and text of all the content in NVDA settings. This can be expanded upon to include other GUI and UI features of NVDA.

.gitignore

@@ -39,6 +39,8 @@ uninstaller/UAC.nsh
 *.pyo
 *.dmp
 tests/unit/nvda.ini
+tests/system/settingsCache/*
+!tests/system/settingsCache/2020.4/*.txt
 source/locale/*/cldr.dic
 .vscode
 .vs

runsettingsdiff.bat

@@ -0,0 +1,2 @@
+@echo off
+call "%~dp0\venvUtils\venvCmd.bat" py -m robot --argumentfile "%~dp0\tests\system\guiDiff.robot" %* "%~dp0\tests\system\robot"

tests/system/guiDiff.robot

@@ -0,0 +1,8 @@
+--loglevel DEBUG
+--outputdir .\testOutput\system
+--xunit systemTests.xml
+--pythonpath .\tests\system\libraries
+--suite NVDASettings
+--variable whichNVDA:source
+--variable currentVersion:source
+--variable cacheFolder:.\tests\system\settingsCache

tests/system/libraries/NvdaLib.py

@@ -55,12 +55,12 @@ def __init__(self):
 			self._runNVDAFilePath = _pJoin(self.repoRoot, "runnvda.bat")
 			self.baseNVDACommandline = self._runNVDAFilePath
 		elif self.whichNVDA == "installed":
-			self._runNVDAFilePath = _pJoin(_expandvars('%PROGRAMFILES%'), 'nvda', 'nvda.exe')
+			self._runNVDAFilePath = self.findInstalledNVDAPath()
 			self.baseNVDACommandline = f'"{str(self._runNVDAFilePath)}"'
 			if self._installFilePath is not None:
 				self.NVDAInstallerCommandline = f'"{str(self._installFilePath)}"'
 		else:
-			raise AssertionError("RobotFramework should be run with argument: '-v whichNVDA [source|installed]'")
+			raise AssertionError("RobotFramework should be run with argument: '-v whichNVDA:[source|installed]'")
 
 		self.profileDir = _pJoin(self.stagingDir, "nvdaProfile")
 		self.logPath = _pJoin(self.profileDir, 'nvda.log')
@@ -69,6 +69,18 @@ def __init__(self):
 			"nvdaTestRunLogs"
 		)
 
+	def findInstalledNVDAPath(self) -> Optional[str]:
+		NVDAFilePath = _pJoin(_expandvars('%PROGRAMFILES%'), 'nvda', 'nvda.exe')
+		legacyNVDAFilePath = _pJoin(_expandvars('%PROGRAMFILES%'), 'NVDA', 'nvda.exe')
+		exeErrorMsg = f"Unable to find installed NVDA exe. Paths tried: {NVDAFilePath}, {legacyNVDAFilePath}"
+		try:
+			opSys.file_should_exist(NVDAFilePath)
+			return NVDAFilePath
+		except AssertionError:
+			# Older versions of NVDA (<=2020.4) install the exe in NVDA\nvda.exe
+			opSys.file_should_exist(legacyNVDAFilePath, exeErrorMsg)
+			return legacyNVDAFilePath
+
 	def ensureInstallerPathsExist(self):
 		fileWarnMsg = f"Unable to run NVDA installer unless path exists. Path given: {self._installFilePath}"
 		opSys.file_should_exist(self._installFilePath, fileWarnMsg)

tests/system/libraries/SystemTestSpy/configManager.py

@@ -51,6 +51,18 @@ def _installSystemTestSpyToScratchPad(repoRoot: str, scratchPadDir: str):
 		],
 		libsDest=spyPackageLibsDir
 	)
+
+	try:
+		opSys.directory_should_exist(_pJoin(spyPackageLibsDir, "xmlrpc"))
+	except AssertionError:
+		# installed copies of NVDA <= 2020.4 don't copy this over
+		_copyPythonLibs(
+			pythonImports=[  # relative to the python path
+				"xmlrpc",
+			],
+			libsDest=spyPackageLibsDir
+		)
+
 	# install the global plugin
 	# Despite duplication, specify full paths for clarity.
 	opSys.copy_file(

tests/system/readme.md

@@ -103,3 +103,29 @@ NVDA is started with the `-c` option to specify this profile directory to be use
 Both Robot Framework and NVDA logs are captured in the `testOutput` directory in the repo root.
 NVDA logs (NVDA log, stdOut, and stdErr for each test) are under the `nvdaTestRunLogs` directory. 
 The log files are named by suite and test name.
+
+### Comparing changes to NVDA Settings
+`.\runsettingsdiff.bat` is a tool used to compare the settings dialog by reading text and generating screenshots for comparison.  The default behaviour is to run using the source code and output to `.\tests\system\settingsCache\source`. 
+
+
+#### Usage
+To check for unreleased changes to the settings dialogs, one can use this tool to compare against two copies of NVDA. 
+
+The following arguments should be used with the script.
+
+Default arguments used are stored  in `.\tests\system\guiDiff.robot`
+
+- `--variable whichNVDA:[installed|source]` to decide where to run NVDA from
+- `--variable cacheFolder:[filePath]` screenshots and text files of each settings panel are generated in `$cacheFolder\$currentVersion`
+- `--variable currentVersion:[nvdaVersion]` where `[nvdaVersion]` is used to name the generated screenshot and cache folder
+- `--variable compareVersion:[nvdaVersion]` using a `$nvdaVersion` that this script has already been run against, run the system tests and fail if there are differences between the read text. This generates a multiline diff. 
+
+#### Example usage to compare settings between NVDA 2020.4 and the current source
+
+1. Install NVDA 2020.4
+1. Run `.\runsettingsdiff.bat -v whichNVDA:installed -v currentVersion:2020.4`
+1. Run `.\runsettingsdiff.bat -v whichNVDA:source -v currentVersion:source -v compareVersion:2020.4`
+   - The test will fail and display a diff of any read changes
+1. Use a diff tool to compare folders:
+   - `diff ./tests/system/settingsCache/2020.4 ./tests/system/settingsCache/source`
+   - [ImageMagick Compare](https://imagemagick.org/script/compare.php) can be used to compare images

tests/system/robot/NVDASettings.py

@@ -0,0 +1,72 @@
+# A part of NonVisual Desktop Access (NVDA)
+# Copyright (C) 2021 NV Access Limited
+# This file may be used under the terms of the GNU General Public License, version 2 or later.
+# For more details see: https://www.gnu.org/licenses/gpl-2.0.html
+
+"""Logic for smoketesting the settings.
+"""
+
+from robot.libraries.BuiltIn import BuiltIn
+# relative import not used for 'systemTestUtils' because the folder is added to the path for 'libraries'
+# imported methods start with underscore (_) so they don't get imported into robot files as keywords
+from SystemTestSpy import (
+	_getLib,
+)
+
+# Imported for type information
+from robot.libraries.Process import Process as _ProcessLib
+
+from AssertsLib import AssertsLib as _AssertsLib
+
+import os
+from typing import Optional
+import NvdaLib as _nvdaLib
+from NvdaLib import NvdaLib as _nvdaRobotLib
+_nvdaProcessAlias = _nvdaRobotLib.nvdaProcessAlias
+
+_builtIn: BuiltIn = BuiltIn()
+_process: _ProcessLib = _getLib("Process")
+_asserts: _AssertsLib = _getLib("AssertsLib")
+
+
+def navigate_to_settings(settingsName):
+	spy = _nvdaLib.getSpyLib()
+	# open settings menu
+	spy.emulateKeyPress("NVDA+n")
+	spy.emulateKeyPress("p")
+	spy.emulateKeyPress("s")
+	spy.emulateKeyPress("leftWindows+upArrow")  # maximise
+	spy.wait_for_speech_to_finish()
+
+	# naviagte to setting
+	for letter in settingsName.lower():
+		spy.emulateKeyPress(letter)
+
+	spy.wait_for_speech_to_finish()
+	spy.reset_all_speech_index()
+
+
+def read_settings(settingsName, cacheFolder, currentVersion, compareVersion: Optional[str] = None):
+	spy = _nvdaLib.getSpyLib()
+	start_speech_index = spy.get_next_speech_index()
+	advancedWarning = "I understand that changing these settings may cause NVDA to function incorrectly."
+
+	# read new setting
+	lastSpeech = ""
+	while "OK" not in lastSpeech:
+		spy.emulateKeyPress("tab")
+		spy.wait_for_speech_to_finish()
+		lastSpeech = spy.get_last_speech()
+		if lastSpeech.split("  ")[0] == advancedWarning:
+			spy.emulateKeyPress("space")
+
+	actualSpeech = spy.get_speech_at_index_until_now(start_speech_index)
+	os.makedirs(f"{cacheFolder}/{currentVersion}", exist_ok=True)
+	with open(f"{cacheFolder}/{currentVersion}/{settingsName}.txt", "w") as f:
+		f.write(actualSpeech)
+
+	if compareVersion:
+		with open(f"{cacheFolder}/{compareVersion}/{settingsName}.txt", "r") as f:
+			compareText = f.read()
+
+		_asserts.strings_match(compareText, actualSpeech)

tests/system/robot/NVDASettings.robot

@@ -0,0 +1,65 @@
+# A part of NonVisual Desktop Access (NVDA)
+# Copyright (C) 2021 NV Access Limited
+# This file may be used under the terms of the GNU General Public License, version 2 or later.
+# For more details see: https://www.gnu.org/licenses/gpl-2.0.html
+*** Settings ***
+Documentation	Smoke test the settings panel, use for checking diffs
+Force Tags	NVDA	smoke test	excluded_from_build
+
+# for start & quit in Test Setup and Test Test Teardown
+Library	NvdaLib.py
+Library	NVDASettings.py
+Library	ScreenCapLibrary
+
+Test Setup	start NVDA	standard-dontShowWelcomeDialog.ini
+Test Teardown	default teardown
+
+*** Keywords ***
+default run read test
+	[Arguments]	${settingsName}
+	navigate to settings	${settingsName}
+	Create Directory	${cacheFolder}/${currentVersion}
+	Set Screenshot Directory	${cacheFolder}/${currentVersion}
+	Take Screenshot	${settingsName}.png
+	read settings	${settingsName}	${cacheFolder}	${currentVersion}	${compareVersion}
+	Take Screenshot	${settingsName}-end.png
+
+default teardown
+	quit NVDA
+
+*** Test Cases ***
+Read General
+	default run read test	General
+
+Read Speech
+	default run read test	Speech
+
+Read Braille
+	default run read test	Braille
+
+Read Vision
+	default run read test	Vision
+
+Read Keyboard
+	default run read test	Keyboard
+
+Read Mouse
+	default run read test	Mouse
+
+Read Review Cursor
+	default run read test	Review Cursor
+
+Read Input Composition
+	default run read test	Input Composition
+
+Read Object Presentation
+	default run read test	Object Presentation
+
+Read Browse Mode
+	default run read test	Browse Mode
+
+Read Document Formatting
+	default run read test	Document Formatting
+
+Read Advanced
+	default run read test	Advanced

tests/system/settingsCache/2020.4/Advanced.txt

@@ -0,0 +1,23 @@
+Advanced  property page  Warning! The following settings are for advanced users. Changing them may cause NVDA to function incorrectly. Please only change these if you know what you are doing or have been specifically instructed by NVDA developers.
+I understand that changing these settings may cause NVDA to function incorrectly.  check box  not checked
+space
+checked
+Restore defaults  button
+NVDA Development  grouping
+Enable loading custom code from Developer Scratchpad directory  check box  checked
+Open developer scratchpad directory  button
+Microsoft UI Automation  grouping
+Enable selective registration for UI Automation events and property changes  check box  not checked  Alt plus s
+Use UI Automation to access Microsoft Word document controls when available  check box  not checked  Alt plus w
+Use UI Automation to access the Windows Console when available  check box  not checked  Alt plus o
+Speak passwords in UIA consoles (may improve performance)  check box  not checked  Alt plus p
+Terminal programs  grouping
+Use the new typed character support in Windows Console when available  check box  checked  Alt plus y
+Speech  grouping
+Attempt to cancel speech for expired focus events:  combo box  Default (No)  collapsed
+Editable Text  grouping
+Caret movement timeout (in ms)  edit  selected 100
+Debug logging  grouping
+Enabled logging categories  list
+hw Io  check box  not checked
+OK  button

tests/system/settingsCache/2020.4/Braille.txt

@@ -0,0 +1,19 @@
+Braille  property page
+Braille display  grouping
+Braille display  edit  read only  multi line  Alt plus d  Automatic
+Change...  button  Alt plus h
+Output table:  combo box  Unified English Braille Code grade 1  collapsed  Alt plus o
+Input table:  combo box  Unified English Braille Code grade 1  collapsed  Alt plus i
+Expand to computer braille for the word at the cursor  check box  checked  Alt plus x
+Show cursor  check box  checked  Alt plus s
+Blink cursor  check box  checked
+Cursor blink rate (ms)  edit  selected 500
+Cursor shape for focus:  combo box  Dots 7 and 8  collapsed  Alt plus f
+Cursor shape for review:  combo box  Dot 8  collapsed  Alt plus r
+Show messages  combo box  Use timeout  collapsed
+Message timeout (sec)  edit  Alt plus t  selected 4
+Tether Braille:  combo box  automatically  collapsed  Alt plus r
+Read by paragraph  check box  not checked  Alt plus p
+Avoid splitting words when possible  check box  checked  Alt plus w
+Focus context presentation:  combo box  Fill display for context changes  collapsed
+OK  button

tests/system/settingsCache/2020.4/Browse Mode.txt

@@ -0,0 +1,13 @@
+Browse Mode  property page
+Maximum number of characters on one line  edit  Alt plus m  selected 100
+Number of lines per page  edit  Alt plus n  selected 25
+Use screen layout (when supported)  check box  checked  Alt plus s
+Enable browse mode on page load  check box  checked  Alt plus e
+Automatic Say All on page load  check box  not checked  Alt plus s
+Include layout tables  check box  not checked  Alt plus a
+Automatic focus mode for focus changes  check box  checked
+Automatic focus mode for caret movement  check box  not checked
+Audio indication of focus and browse modes  check box  not checked
+Trap all non command gestures from reaching the document  check box  checked  Alt plus t
+Automatically set system focus to focusable elements  check box  not checked  Alt plus f
+OK  button

tests/system/settingsCache/2020.4/Document Formatting.txt

@@ -0,0 +1,39 @@
+Document Formatting  property page  The following options control the types of document formatting reported by NVDA.
+Font  grouping
+Font name  check box  not checked  Alt plus f
+Font size  check box  not checked  Alt plus s
+Font attributes  check box  not checked  Alt plus u
+Superscripts and subscripts  check box  not checked  Alt plus p
+Emphasis  check box  not checked  Alt plus m
+Marked (highlighted text)  check box  checked  Alt plus k
+Style  check box  not checked  Alt plus y
+Colors  check box  not checked  Alt plus c
+Document information  grouping
+Notes and comments  check box  checked  Alt plus t
+Editor revisions  check box  checked  Alt plus e
+Spelling errors  check box  checked  Alt plus r
+Pages and spacing  grouping
+Pages  check box  checked  Alt plus p
+Line numbers  check box  not checked  Alt plus n
+Line indentation reporting:  combo box  Off  collapsed  Alt plus i
+Paragraph indentation  check box  not checked  Alt plus p
+Line spacing  check box  not checked  Alt plus l
+Alignment  check box  not checked  Alt plus a
+Table information  grouping
+Tables  check box  checked  Alt plus t
+Row slash column headers  check box  checked  Alt plus e
+Cell coordinates  check box  checked  Alt plus o
+Cell borders:  combo box  Off  collapsed  Alt plus b
+Elements  grouping
+Headings  check box  checked  Alt plus h
+Links  check box  checked  Alt plus k
+Graphics  check box  checked  Alt plus g
+Lists  check box  checked  Alt plus l
+Block quotes  check box  checked  Alt plus q
+Groupings  check box  checked  Alt plus g
+Landmarks and regions  check box  checked  Alt plus d
+Articles  check box  not checked  Alt plus c
+Frames  check box  checked  Alt plus m
+Clickable  check box  checked  Alt plus c
+Report formatting changes after the cursor (can cause a lag)  check box  not checked  Alt plus g
+OK  button

tests/system/settingsCache/2020.4/General.txt

@@ -0,0 +1,12 @@
+General  property page
+NVDA Language (requires restart):  combo box  User default  collapsed  Alt plus l
+Save configuration when exiting NVDA  check box  checked  Alt plus s
+Show exit options when exiting NVDA  check box  checked  Alt plus w
+Play sounds when starting or exiting NVDA  check box  checked  Alt plus p
+Start NVDA after I sign in  check box  not checked  Alt plus a
+Use NVDA during sign in (requires administrator privileges)  check box  not checked
+Use currently saved settings during sign in and on secure screens (requires administrator privileges)  button
+Automatically check for updates to NVDA  check box  not checked  Alt plus u
+Notify for pending update on startup  check box  not checked  Alt plus p
+Allow the NVDA project to gather NVDA usage statistics  check box  not checked
+OK  button

tests/system/settingsCache/2020.4/Input Composition.txt

@@ -0,0 +1,7 @@
+Input Composition  property page
+Automatically report all available candidates  check box  checked  Alt plus c
+Announce selected candidate  check box  checked  Alt plus s
+Always include short character description when announcing candidates  check box  checked  Alt plus d
+Report changes to the reading string  check box  checked  Alt plus r
+Report changes to the composition string  check box  checked  Alt plus c
+OK  button

tests/system/settingsCache/2020.4/Keyboard.txt

@@ -0,0 +1,14 @@
+Keyboard  property page
+Keyboard layout:  combo box  desktop  collapsed  Alt plus k
+Select NVDA Modifier Keys  list
+caps lock  check box  not checked
+Speak typed characters  check box  checked  Alt plus c
+Speak typed words  check box  not checked  Alt plus w
+Speech interrupt for typed characters  check box  checked  Alt plus i
+Speech interrupt for Enter key  check box  checked  Alt plus n
+Allow skim reading in Say All  check box  not checked  Alt plus r
+Beep if typing lowercase letters when caps lock is on  check box  checked  Alt plus b
+Speak command keys  check box  not checked  Alt plus o
+Play sound for spelling errors while typing  check box  checked  Alt plus s
+Handle keys from other applications  check box  checked  Alt plus a
+OK  button

tests/system/settingsCache/2020.4/Mouse.txt

@@ -0,0 +1,9 @@
+Mouse  property page
+Report mouse shape changes  check box  not checked  Alt plus s
+Enable mouse tracking  check box  checked  Alt plus t
+Text unit resolution:  combo box  paragraph  collapsed  Alt plus u
+Report role when mouse enters object  check box  not checked  Alt plus r
+Play audio coordinates when mouse moves  check box  not checked  Alt plus p
+Brightness controls audio coordinates volume  check box  not checked  Alt plus b
+Ignore mouse input from other applications  check box  not checked  Alt plus a
+OK  button