Contracts for automated UI testing
I’ve seen people write tests that assert that a button has a ‘disabled’ CSS classname to test if a button is disabled. I have then seen those same tests break because someone changed the styles so that ‘btn-disabled’ is now the classname of choice for styling that state, but of course nothing actually broke, the test failure was a false negative. At the same time the old styles for the ‘disabled’ class have been removed, but some buttons have been left with the old classname – tests still pass but it is (to the user) broken, a false positive. The example test below shows this kind of assert.
assert(page.myButton).hasClassName('.disabled', 'Should be disabled');
The above example is simply the result of a false assumption that the CSS classname represents a contract of state for that button. CSS classnames should never be used as a contract between UI and test, they’re not and should not be coupled to functionality because (more often than not) visual design is changed more than functionality. The only way to test if a button is disabled, is to click it and assert that nothing happened. This of course is just a functional test of the UI, it won’t tell you if the button is hidden or the wrong colour.
click(page.myButton); assert(page.myReceipt).doesnot.containText('Action complete', 'Should not have completed action');
In the snippet above the page elements are probably defined using a CSS or XPATH selector, this implies that there is still a contract between the tests and the UI. Let’s say page.myButton is defined with the following CSS selector
'div.popupbox .footer > button'. By using this selector you are assuming that the button in question will always be inside a div element with a classname of ‘popupbox’, the button will always live inside a footer, the button will always be … a button, not a link, you sure about that? Coupling your tests to markup and styling will lead to incredibly fragile tests, especially in an environment where UI design is constantly iterated upon.
To overcome this fragility, rather than assume a contact, define a contract. Creating a consistant vocabulary for element hooks for testing is quite easy. At Huddle we’ve defined a contract using data attributes in the markup, the CSS3 selector for the disabled button could look like this
'[data-automation=submit-comment-button]', note that the intent of the button is used for naming, not the style. Decorating code like this makes markup a little messier but gives an immediate prompt to a future developer that this page is under test while also giving them the freedom to change styling without having to update and fix tests. Of course you can make up a contract that bests suits you and your team, perhaps web-aria attributes might be better?
But what about contracts for visual testing, couldn’t we just agree that
'[data-automation=disabled]' means that the button is visually disabled? No, it just doesn’t work. You’ve given some protection to the CSS developer that’s re-skinning the UI but you have no guarantee that the styling won’t have broke, nor that clicking the button doesn’t submit the form. You have to simulate the user, it’s the only reliably way to test UI.
In a future post I will discuss how visual regression testing is possible.