The format of the session was that each panellist was provided with a few minutes to present a position statement on accessibility in today’s practice, and this was followed by questions and discussion. In the short time available to them, my fellow panelists each took a specific angle on accessibility:

• Liam McGee (Communis) argued that usability is ‘accessibility for sissies’, and illustrated how he viewed accessibility as a term that covered many objectives of usability and search-engine optimisation.
• Amy Chen (Senior Usability Secialist at Oracle) described how a large technology vendor can adopt accessibility as part of the design, development and implementation of their technology products.
• Rolf Molich (DialogDesign). As a highly respected and highly influential figure in the usability and HCI field, Rolf took on the mantle of devil’s advocate, arguing that the message of accessibility advocates can sometimes be obfuscated, disguising general good practice in usable and user-centred design as complexly-worded accessibility guidelines.

The role of the accessibility specialist in the usability profession

I decided to focus on what I saw, based on my experience over the last 10 years, were the key roles that an accessibility specialist should perform – whether as a member of a web/software development team, or as an advisor to a large organisation procuring and implementing technology to help it perform its day-to-day activities. These were:

1. Technical advisor – someone who understands the principles behind and implementation techniques of accessibility guidelines, and how to evaluate whether they have been successfully met. Someone who is up to speed on the accessibility benefits and shortcomings of relevant programming languages and digital information formats, whether established or emerging.
2. Motivator – someone who generates empathy for the objectives of accessible design, by encouraging others to appreciate the diversity of ways in which people access and use technology, and the impact accessibility (or lack of) can have on them. Someone who shows that accessibility is something that can inspire innovation and spark creativity, rather than constraining what can be done to the mundane and unexciting. (at this point I had to yet again plug Graham Pullin’s excellent book Design meets Disability )
3. Translator – someone who can effectively present accessibility requirements in their appropriate context – from legislative requirements to practical, pragmatic design requirements. Someone who can ensure that accessibility requirements expressed in an invitation to tender or internal policy are achievable, unambiguous and, if met, genuinely lead to more inclusive technology. Organisations who express accessibility requirements in a coherent and appropriate way are more likely to encourage technogy suppliers to meet those needs. I recounted here examples of where I’ve seen poorly expressed accessibility requirements in a technology specification that could not feasibly be met let alone tested.
4. Gerontechnologist – perhaps my most left-field suggestion, I think accessibility specialists should recognise the particular benefits of involving older people in user centred design, for the added-value that they are likely to provide as participants in requirements gathering activities, and evaluators throughout the design lifecycle. Arguably accessibility guidelines focus on the more extreme end of impairment, at the expense of those with less severe, but multiple, impairments. Evaluating with disabled people is important, and rewarding, but recruitment and scheduling can sometimes be difficult. So the unpredictability of the presence of any age-related sensory, dexterity or cognitive impairments make recruitment of older participants for participatory design and usability testing a particularly attractive option, particularly if resources are tight (see Henny Swan’s comments on the value of testing a web browser with older people).

I offered these definitions as a way of helping people decide whether an accessibility specialist was indeed a specialist, or whether this was a role a usability professional could or should take on.

Discussion

The focus of the discussion with the audience was, from my perspective, largely focused on the challenge of selling accessibility, a topic that is always near the top of the discussion charts and which has received much attention in recent blog posts from Gary Barber and Vlad Alexander.

It’s hard to argue against the objective of accessibility, but in a financially driven context, we’re all too aware that accessibility can sometimes be perceived as a luxury, as a lot of effort for a small group of people. Several members of the audience gave examples of how they have found it difficult to persuade others of the value of investing in accessibility considerations.

Of course, one way to counter that argument is to downplay accessibility as a separate objective – most of good practice in accessibility is general good practice in user-centred design; the SEO overlap is also a powerful argument. That’s more difficult to sell when there is obvious additional work to do, like synchronised captioning.

We can also argue that for every group of disabled people who benefit from a particular accessibility intervention, there is another group of ‘situationally disabled’ people who will also benefit at a particular time and place. But how do we provide hard figures for the number of unexpected beneficiaries of accessibility interventions? And isn’t it an awkward dilemma for an accessibility advocate to be faced with: providing hard statistics that help to calculate cost-benefit of an accessibility requirement when that effort may actually lead someone to justify exclusion?

The most powerful tool seems to be more examples of how accessibility is done well, and in particular how it can spark or encourage innovation. There are plenty examples out there of where this has happened, and it was great to hear that WAI is collecting such examples for an addition to their suite of resources arguing for accessibility. We need to be able to show sceptics examples of where innovation in accessible web design can genuinely lead to benefits for end-user and provider, because if we can’t, it makes advocating inclusive design that much more difficult.

What does all this mean for the usability profession? Do we need accessibility specialists, or is this knowledge and skills that all usability professionals who acknowledge human diversity should expect to have? What do you think?

That’s why earlier this year I found myself in the longest queue for the ticket machines outside the Museo del Prado in Madrid, because I was curious to know why nobody seemed to be able to make it work (the answer was confusion caused by two user interfaces – one for selecting tickets, one for the credit card reader positioned below).

Yesterday, I had to send someone a fax. I’d sent the same person a fax last week, and made the common mistake of sending the fax to a telephone number instead of using the fax number (there is a peculiar and specific embarrassment of hearing a disembodied voice try to answer the fax machine’s call while you can do nothing about it). I was so determined not to make the same mistake again…but I did.

I can’t blame what I did on the fancy document-management machine we have in the office that supports printing, copying, scanning and faxing. So I looked at the document I was trying to send, on the second page of which was the fax number.

This was the first problem – the destination fax number was on the document I wanted to send, and being too disorganised to write it down somewhere else, I had to quickly mentally note the number, type it into the machine, and put the document back in the slot ready for scanning.

The second problem is the obvious one. Below the fax number was a phone number. Fax numbers are in an identical format to phone numbers (at least here in the UK), and most commonly will start with the same digits as a phone number: (0nnn) nnnnnn. Chances are the area code and first few digits will be identical for an organisation’s phone and fax number, which means without a distinguishing label it’s impossible to tell whether a number is a fax or phone number.

So when the two are placed close to each other on a document – even when clearly labelled as ‘fax’ or ‘phone’, the chances of entering the wrong number are pretty high – and as I just proved, even when the consequences are known. Just like Derren Brown‘s TV ‘experiments’ where people  press a big red button despite (because of?) being told not to and being shown the consequences.

OK, you’re probably wondering what my point is – we must have known about the problem of fax numbers for years, and aren’t faxes yesterday’s technology anyway? But in 2009, here I am, making the error – twice – so what could have stopped me?

The problem was the presence of two pieces of similar data – one important to the task, one irrelevant. The information design on the document was such that the two very similar numbers were physically close to each other. I didn’t need the phone number to send a fax, so it could have been somewhere else in the document, or not there at all (As it happens, there was also an email address beside the phone number, but I would bet very few people accidentally send faxes to an email address).

It’s a simple information design lesson – think carefully about showing information that may hinder successful completion of a task. If it’s not essential, don’t show it, otherwise position it away from where the user’s focus will likely be.

The paper critiques 5 prominent (i.e. published in prominent academic publications and subsequently cited) studies that compared different Usability Evaluation Methodologies (UEMs). It found that for each study the experimental design casts doubt over the validity of the conclusions made.

In a clear and accessible fashion, the paper:

• outlines the value of UEMs in interface design, and explains the relative merits of empirical UEMs (involving watching users interact with a system) and analytical UEMs (using some pre-defined knowledge to methodically assess the system for potential barrriers) in identifying true barriers and providing the design team with information necessary to fix them;
• reminds us that the value of experiments is in establishing causality (that X causes Y) and generality (that X will cause Y across different circumstances);
• introduces 4 measures of validity that can be applied to an experiment (from Quasi-Experimentation: Design and Analysis issues for field settings; Cook T and Campbell D, 1979);
• uses these measures to identify ‘threats to validity’ that might exist in the design of an experiment;
• treats each UEM comparison as a case study of how validity of the experiment and the results it presents can be questioned;
• offers advice for minimising threats to validity through experimental design and analysis.

What are the four measures of validity? Two concern causality, and two concern generality.

1. Causality issues:
   • statistical conclusion validity – concerning whether real differences do exist between experiment groups. Did the experiment really find differences in the results of using different UEMs? Validity may be affected by the impact of low numbers of participants; lack of appropriate statistical analysis; ‘random heterogeneity’ (or the influence of wildcard participants on results). This is explored in a post on bias in DIY usability testing.
   • internal validity – concerning whether measured differences are causal or correlational. Were these differences definitely due to using different UEMs? Or could some other factor have influenced results? Selection (of participant groups) and setting (conditions under which the experiment was carried out) can influence internal validity.
2. Generality issues:
   • Construct validity: in the words of the authors, “are the experimenters manipulating what they claim to be manipulating?” (this is causal construct validity) and “are they measuring what they claim to be measuring?” (this is effect construct validity)
   • External validity: how valid are claims that results can be generalised across different settings and persons?

There is a fifth validity issue – conclusion validity, where the conclusions are not based on the data generated by the experiment. The authors note the tendency of usability evaluators to include general ‘good advice’ amongst conclusions based on the findings of an experiment, when the data gathered cannot possibly support this advice. If it is accepted as good advice, it should be presented as such, not as the findings of the experiment.

Why is this work important? Well, given that these studies were selected as being of particularly high impact in the community, there is potential for major decisions to have been made relating to using one UEM over another, or for further research to have been conducted, based on unsafe assertions. What’s not clear to me, 11 years on, is just how big the impact has been on usable technology design of the flaws identified in these studies.

But more practically, for all of us who do usability or accessibility testing, this paper reminds us of the difference between analytical evaluation methods and empirical methods. There’s a danger that our eagerness to promote what we believe is best practice may obscure what we actually find out in empirical testing (the “guideline compliance vs designing for humans” argument in another form). Finding participants can be difficult; finding disabled participants for testing is very difficult, so while of course user involvement is still recommended in order to achieve valuable insight, presenting results with due qualifications and caveats is essential.

For a lot of people, this stuff will be nothing new – it’s basic good practice in science. But, like many people who have come into applied science from other areas, I don’t have a background in rigorous experimental design. And while designing major experiments is not something I do often, knowing how to devise and follow a process of generating new knowledge that is reliable and repeatable – such as conducting a usability testing programme of a software application or web application – is certainly wisdom worth having.

Read more:

• Damaged Merchandise – the original paper, and a rejoinder – commenting on feedback the authors received.
• Wikipedia on experimental design.