Don’t break Fix the Web

This week, Fix the Web was launched, and attracted some excellent publicity. It’s an initiative that is a potentially significant step towards empowering people who experience accessibility barriers on line to report them in a way that might increase the chances of someone fixing the problem. One great difficulty in evaluation and repair of the sheer mass of web content we have today is helping people who experience accessibility barriers identify them as such and tell someone who might be able to make a difference.

Not all disabled web users are web developers; it’s unlikely many people will have heard of WCAG, let alone be able to describe a problem with a web page with reference to WCAG success criteria. But Fix the Web gives a voice to someone with a disability to say “hey, I couldn’t add an item to my shopping cart”, and have that problem passed on to volunteer accessibility specialists who can investigate, and back up that real-world problem with a technical description of the underlying code or design defect. And the more people who identify the same problem, the louder the voice.

While Fix the Web may not be able to actually repair the page (though other crowd-sourcing approaches have taken this step, such as IBM’s Social Accessibility project ), it does give web site owners evidence of real customers being unable to use the site for its intended purpose AND informed technical information that might help them fix it.

However, there’s a problem. As pointed out on Twitter today by Eric Eggert (@yatil), Fix the Web ‘s own web site has accessibility issues. This, it’s argued, may lead to a loss of trust by the very people it’s meant to serve – disabled people who think “well, if Fix the Web can’t make their own site accessible, how can they tell other people to fix the problem I report?” This would indeed be a huge problem, if the only way to report barriers was through the site, and some people were prevented from doing so.

But it’s not. The great challenge of Fix the Web is reaching out to people who experience accessibility problems, who have never heard of Fix the Web, never heard of WCAG and who might never have considered reporting barriers. Installing browser toolbars in public access PCs is one approach, encouraging people to email or tweet details of problems is another.

The accessibility of the Fix the Web site should be as good as possible – of course it should, and it needs to be, soon. However, the credibility and viability of the initiative should be based on the quality of the reporting system – the awareness it raises amongst disabled web users, the skills of the volunteer accessibility specialists in following up problems, and the quality of the communication with web site owners.

Casting aspersions on the whole system based on the accessibility of its own site – which is, after all, an intermediary in a wider communication chain – is a waste of energy, particularly by accessibility advocates. Let’s not break Fix the Web before it has a chance to make a difference.

User-centred Design where it’s needed most

On the same day this issue was being discussed on Twitter, I attended a presentation by Simon Judge, an augmentative and alternative communication (AAC) specialist working at Barnsley Hospital in northern England. I knew Simon’s name from various open source accessibility initiatives such as OATSoft and MAAVIS; he was in Dundee as a guest of my colleague Dr Annalu Waller, who leads a research group on AAC.  While a busy day meant I didn’t have a chance to speak to him, it was interesting to hear his work on exploring AAC user needs and preferences (and enjoy some vintage 70s TV – Toby Churchill demonstrating assistive technology on Tomorrow’s World!)

In listening to Simon’s talk about work on the Devices for Dignity project, I was again struck by the relationship of accessibility and communication, and why we haven’t quite got things right yet.

It’s impossible to overestimate the positive impact AAC systems can have on people with severe communication difficulties. Yet, for the importance of such systems,  there have been  long term problems with AAC systems relating to their usability and appropriateness for the person whom they are supposed to be helping.  The problem seems to stem from the medical context in which they’re usually given to people – without real appreciation of the environment or personal contexts in which they might be used. Unlike choosing, say, a mobile phone, AAC users are likely to be limited in their knowledge of what choices are available to them, and normally there will be an intermediary (AAC professional) involved in the selection process. Practical issues such as reliability and durability can reduce trust in a device that ultimately is the difference between being able to communicate and not.

Simon has spent time speaking to AAC users and professionals in order to understand the similarities and differences in their perspectives; generally users are more negative in their views. So clearly there are issues in usability of such technologies. On the surface, the apparent limited use of user-centred design by the AAC and assistive technology field might seem surprising, but traditional UCD techniques can be difficult to apply in the field of AAC design particularly since access to users early in the design stage might be challenging.

This is an ongoing area of study, but work like Simon’s, and that of Annalu, Graham Pullin and colleagues here in Dundee, will hopefully lead to a more user-centred, more context-aware design process in creating genuinely usable and useful assistive technology.

Web accessibility advocates may recognise parallels with screen reading technology too – so be forgiving of people who are not intimately familiar with their AT, and be conservative in your assumptions of the role a user’s AT plays in enhancing the quality of their web interactions.

In the early days of web accessibility, post WCAG 1.0 release, published surveys of the accessibility of large numbers of web sites were relatively rare (I’m distinguishing these from reviews of a single site conducted by or on behalf of the development team, with the specific aim of identifying and repairing barriers present). So whenever a new survey emerged, it usually provided informative data on levels of conformance against WCAG 1.0, which took time achieve any significant impact on the web design industry. The data allowed us to see how particular sectors were faring, and which checkpoints were most frequently not met.

The publicity surrounding a published accessibility survey that presented data showing how poorly sites were dealing with WCAG conformance could also be claimed to raise awareness of web accessibility in general, and more specifically shame the organisations in question into doing something about the barriers present on their site. The former effect probably did take place, although I’d like to see concrete evidence that surveys actually have a positive effect on the organisations whose sites were reviewed.

(Indeed, there have been concerns that surveys may have a negative impact on ‘usable accessibility.’ If the methodology used focuses excessively on a technical measure of accessibility that becomes a highly public ‘official’ ranking of each site’s performance – with rewards for finishing high up a ranking, there would be understandable pressures for site developers to design to satisfy the surveyors and not disabled people.)

Many published surveys have had severe limitations in methodology and scope – frequently conducted using automated tools only, using a subset of WCAG, and often of the Home page only; and very, very rarely have researchers extended their survey to contact each site’s organisation for follow-up data (a notable exception was a study by Ronald Milliman in 2002). Several published surveys have appeared as academic papers in a wide variety of journals (not just computing/HCI). As the topic was initally relatively uncovered in academic literature, an investigation into the accessibility of web sites in a particular sector – be it higher education, government, e-commerce, tourism, or whatever – made for an attractive publication topic. I should know, I wrote a couple! But at least in the early days we had some data to help say ‘ hey, we all need to do a bit better here’.

Over time, we’ve had some very high impact surveys, such as the UK DRC Formal Investigation into web accessibility, published in 2004, and which combined automated testing with manual inspections; evaluations with disabled people and interviews with web developers. Just recently, at the Accessing Higher Ground 2009 conference, Terrill Thompson presented data from a longitudinal study that compared progress towards accessible content between organisations who had received technical support and those who hadn’t.

Additionally, publications have presented accessibility survey data as part of other valuable research activity, for example when testing new methodologies and tools for large scale accessibility evaluation (important in the real world for regulatory bodies as well as organisations with many web pages), and testing how effective particular evaluation methodologies could be in minimising false positives and false negatives (check the proceedings of W4A for papers by researchers such as Markel Vigo, Giorgio Brajnik and Bambang Parmanto).

Putting aside limitations of scope and evaluation methodology, surveys have not been so good at focusing on content providers and the content provision circumstances. This is to some extent understandable, as it’s easier to run an automated tool across multiple sites than identify, contact and gather data from all the content providers for each site.

Some exceptions are mentioned above – to add to these, my thesis focused on the impact of accessibility audits on the recipient organisations, and my research yielded a limited amount of information on how organisations responded. A few surveys have specifically looked perceptions on and attitudes to web accessibility (for example Lazar et al’s 2004 paper on Improving web accessibility: a study of webmaster perceptions and Bloor Research’s 2009 survey), but we still don’t know very much about the organisational reasons as to why accessibility of a particular web site is not as good as it could be.

As time goes by, therefore, the impact of a survey that presents data on web accessibility diminishes, unless it adds something new to our understanding of the problem. In 2009, it’s not enough to simply claim that results are ‘disappointing’, and that web content authors must ‘do better’. WCAG has been here for 10 years, so it’s not as if we have no best practice; and people and resources promoting and supporting web accessibility are easy to find. A failure to acknowledge in a survey report that that this has been a problem for many years does a disservice to everyone who has been advancing the cause of web accessibility (even if it also gives another stark reminder that there’s much work to do).

So if you conduct a survey of web sites and find accessibility barriers, don’t stop at reporting conformance levels. Find out why the barriers exist. Is it a lack of awareness or training amonst the providers of the content you surveyed? Is it a lack of involvement of disabled people in the design process? Is it a lack of resources; of money; a lack of will, or lack of prominence of accessibility in an organisation’s business practice and philosophy? Is it sub-standard authoring tools, content management systems; quality assurance systems? Is there another reason?

And whatever you find out, please share it with us!

The paper critiques 5 prominent (i.e. published in prominent academic publications and subsequently cited) studies that compared different Usability Evaluation Methodologies (UEMs). It found that for each study the experimental design casts doubt over the validity of the conclusions made.

In a clear and accessible fashion, the paper:

• outlines the value of UEMs in interface design, and explains the relative merits of empirical UEMs (involving watching users interact with a system) and analytical UEMs (using some pre-defined knowledge to methodically assess the system for potential barrriers) in identifying true barriers and providing the design team with information necessary to fix them;
• reminds us that the value of experiments is in establishing causality (that X causes Y) and generality (that X will cause Y across different circumstances);
• introduces 4 measures of validity that can be applied to an experiment (from Quasi-Experimentation: Design and Analysis issues for field settings; Cook T and Campbell D, 1979);
• uses these measures to identify ‘threats to validity’ that might exist in the design of an experiment;
• treats each UEM comparison as a case study of how validity of the experiment and the results it presents can be questioned;
• offers advice for minimising threats to validity through experimental design and analysis.

What are the four measures of validity? Two concern causality, and two concern generality.

1. Causality issues:
   • statistical conclusion validity – concerning whether real differences do exist between experiment groups. Did the experiment really find differences in the results of using different UEMs? Validity may be affected by the impact of low numbers of participants; lack of appropriate statistical analysis; ‘random heterogeneity’ (or the influence of wildcard participants on results). This is explored in a post on bias in DIY usability testing.
   • internal validity – concerning whether measured differences are causal or correlational. Were these differences definitely due to using different UEMs? Or could some other factor have influenced results? Selection (of participant groups) and setting (conditions under which the experiment was carried out) can influence internal validity.
2. Generality issues:
   • Construct validity: in the words of the authors, “are the experimenters manipulating what they claim to be manipulating?” (this is causal construct validity) and “are they measuring what they claim to be measuring?” (this is effect construct validity)
   • External validity: how valid are claims that results can be generalised across different settings and persons?

There is a fifth validity issue – conclusion validity, where the conclusions are not based on the data generated by the experiment. The authors note the tendency of usability evaluators to include general ‘good advice’ amongst conclusions based on the findings of an experiment, when the data gathered cannot possibly support this advice. If it is accepted as good advice, it should be presented as such, not as the findings of the experiment.

Why is this work important? Well, given that these studies were selected as being of particularly high impact in the community, there is potential for major decisions to have been made relating to using one UEM over another, or for further research to have been conducted, based on unsafe assertions. What’s not clear to me, 11 years on, is just how big the impact has been on usable technology design of the flaws identified in these studies.

But more practically, for all of us who do usability or accessibility testing, this paper reminds us of the difference between analytical evaluation methods and empirical methods. There’s a danger that our eagerness to promote what we believe is best practice may obscure what we actually find out in empirical testing (the “guideline compliance vs designing for humans” argument in another form). Finding participants can be difficult; finding disabled participants for testing is very difficult, so while of course user involvement is still recommended in order to achieve valuable insight, presenting results with due qualifications and caveats is essential.

For a lot of people, this stuff will be nothing new – it’s basic good practice in science. But, like many people who have come into applied science from other areas, I don’t have a background in rigorous experimental design. And while designing major experiments is not something I do often, knowing how to devise and follow a process of generating new knowledge that is reliable and repeatable – such as conducting a usability testing programme of a software application or web application – is certainly wisdom worth having.

Read more:

• Damaged Merchandise – the original paper, and a rejoinder – commenting on feedback the authors received.
• Wikipedia on experimental design.