Acacia (for ARIA WG)

# Acacia (for ARIA WG) ### Automated AAM Test Suite in WPT Val + Alice --- #### Goal: Extend WPT infrastructure to allow automated testing for AAM specifications - We want to be able to test platform API mappings directly, as well as testing the internal accessibility tree like `computedRole` and `computedLabel` --- ![Block diagram illustrating how these new tests fit in to existing infrastructure. Full image description available in RFC.](https://notes.igalia.com/uploads/6d015435-18c1-453c-9faa-4ccec976af96.png) Note: Existing code allows testing internal computed role and computed name. This would add the ability to test that accessibility API mappings are correctly implemented. --- [Check out the working prototype!](https://github.com/Igalia/wpt/pull/2/files#diff-4c6cae4648eadb186a175370e04a9e9581664a469c60fdb72d0d20600b28adeb) * Extends `testdriver` infrastructure * This will allow us to use WebDriver commands (like clicks) intermixed with accessibility API queries * You can run the tests by following the instructions in the PR Note: Demo - blockquote and button tests --- Prototype works for linux, mac and windows Accesses API using python bindings, see the code: - [Linux, Atspi](https://github.com/Igalia/wpt/pull/2/files#diff-9247f1aa2fe3d167af87ee04c48bc3ceb244d3de5040fdf97cdb129e05fa47e4) - [Mac, AX API](https://github.com/Igalia/wpt/pull/2/files#diff-a52eb168f8a233c8e81ba15c97691a7ac144e3168a3bfbd2c6c80ece5ab55c58) - [Windows, only MSAA + IA2](tools/wptrunner/wptrunner/executors/executorwindowsaccessibility.py) --- * This is a big enough change to land in WPT that it requires an [RFC](https://github.com/web-platform-tests/rfcs/pull/204) * There are a few [open technical questions](https://github.com/Igalia/rfcs/blob/wpt-for-aams-rfc/rfcs/wpt-for-aams.md#open-questions-on-technical-implementations-and-test-design), to be discussed later in the week: * how to get browser process ID (for querying the API) * how to ensure browsers are running with accessibility enabled for these tests (cf. chrome's `--force-renderer-accessibility` flag) * how to query the accessibility tree at the right moment --- #### Test design What we most want your input on today is how you'd like to author tests, since we are still designing this system --- Test design assumptions: * One test file per "entry" in an AAM e.g. [blockquote in core-aam](https://w3c.github.io/core-aam/#role-map-blockquote) * Currently constrained by [wpt.fyi](https://wpt.fyi/results/wai-aria/role?label=experimental&label=master&aligned) having no concept of platform-specific tests or "not applicable" tests * Modelled on existing testdriver-based tests, with the test logic and the HTML in the same file --- Constraints of [wpt.fyi](https://wpt.fyi/results/wai-aria/role?label=experimental&label=master&aligned) until we have "not applicable": * Results will show up as "pass" on platforms other than the one the test is being run on, even though no assertions are run * [Example test results](https://spectranaut.github.io/examples/wpt/role_blockquote_test.html) for a test run on Linux, which is only applicable for AT-SPI --- Test designs we are considering: 1. "bag of properties" sent from the backend 2. "testable statements" sent to the backend --- ### Bag of properties #### [link to test example](https://github.com/Igalia/wpt/pull/2/files#diff-5e42f56c823622ddd377d3313d4c4c173f5873bca0d207b43178b18fe66406f5) <pre><p id="test" role="blockquote">quotable thing</p> <script> promise_test(async t => { const node = await test_driver.get_platform_accessibility_node('test'); if (node.API == 'atspi') { assert_equals(node.role, 'block quote', 'Atspi role'); } }, 'Atspi: role blockquote'); </script></pre> --- ### Testable statements #### [link to test example](https://github.com/Igalia/wpt/pull/14/files#diff-1482e1c87c453c135673c1bb0fb68a905f10b828bbacc9db938774a29dfccf85) <pre><div id="test" role="blockquote">quotable thing</div> <script> AAMUtils.verifyAPI( 'role=blockquote', 'test', { "Atspi" : [ [ "property", "role", "is", "block quote" ] ], "AXAPI" : [ [ "property", "AXRole", "is", "AXGroup" ], [ "property", "AXSubrole", "is", "<nil>" ] ], "IAccessible2" : [...], "UIA" : [...] } ); </script></pre> --- ## Discussion - How would you want to write tests? Any ideas beyond the two listed here? - (If we have time) AT API experts: any advice on how to make sure we're querying the accessibility tree only once it has actually been built?