The '58 sound

Web accessibility surveys – results are frequently disappointing


A recent exchange on Twitter has motivated me to write about the contribution published surveys on web site accessibility make towards understanding and addressing the problems that hold back web accessibility. I’ve read, and continue to read, many, many papers presenting the results of surveys of web sites, and I think we need surveys to look beyond just the data and instead delve more deeply into why the results are as they are. We’ve gone way beyond the point where a paper simply reporting that a study of x web sites from y sector revealed ‘disappointing’ levels of accessibility provides anything more than a minor contribution. Surveys need to look at process not product.

In the early days of web accessibility, post WCAG 1.0 release, published surveys of the accessibility of large numbers of web sites were relatively rare (I’m distinguishing these from reviews of a single site conducted by or on behalf of the development team, with the specific aim of identifying and repairing barriers present). So whenever a new survey emerged, it usually provided informative data on levels of conformance against WCAG 1.0, which took time achieve any significant impact on the web design industry. The data allowed us to see how particular sectors were faring, and which checkpoints were most frequently not met.

The publicity surrounding a published accessibility survey that presented data showing how poorly sites were dealing with WCAG conformance could also be claimed to raise awareness of web accessibility in general, and more specifically shame the organisations in question into doing something about the barriers present on their site. The former effect probably did take place, although I’d like to see concrete evidence that surveys actually have a positive effect on the organisations whose sites were reviewed.

(Indeed, there have been concerns that surveys may have a negative impact on ‘usable accessibility.’ If the methodology used focuses excessively on a technical measure of accessibility that becomes a highly public ‘official’ ranking of each site’s performance – with rewards for finishing high up a ranking, there would be understandable pressures for site developers to design to satisfy the surveyors and not disabled people.)

Many published surveys have had severe limitations in methodology and scope – frequently conducted using automated tools only, using a subset of WCAG, and often of the Home page only; and very, very rarely have researchers extended their survey to contact each site’s organisation for follow-up data (a notable exception was a study by Ronald Milliman in 2002). Several published surveys have appeared as academic papers in a wide variety of journals (not just computing/HCI). As the topic was initally relatively uncovered in academic literature, an investigation into the accessibility of web sites in a particular sector – be it higher education, government, e-commerce, tourism, or whatever – made for an attractive publication topic. I should know, I wrote a couple! But at least in the early days we had some data to help say ‘ hey, we all need to do a bit better here’.

Over time, we’ve had some very high impact surveys, such as the UK DRC Formal Investigation into web accessibility, published in 2004, and which combined automated testing with manual inspections; evaluations with disabled people and interviews with web developers. Just recently, at the Accessing Higher Ground 2009 conference, Terrill Thompson presented data from a longitudinal study that compared progress towards accessible content between organisations who had received technical support and those who hadn’t.

Additionally, publications have presented accessibility survey data as part of other valuable research activity, for example when testing new methodologies and tools for large scale accessibility evaluation (important in the real world for regulatory bodies as well as organisations with many web pages), and testing how effective particular evaluation methodologies could be in minimising false positives and false negatives (check the proceedings of W4A for papers by researchers such as Markel Vigo, Giorgio Brajnik and Bambang Parmanto).

Putting aside limitations of scope and evaluation methodology, surveys have not been so good at focusing on content providers and the content provision circumstances. This is to some extent understandable, as it’s easier to run an automated tool across multiple sites than identify, contact and gather data from all the content providers for each site.

Some exceptions are mentioned above – to add to these, my thesis focused on the impact of accessibility audits on the recipient organisations, and my research yielded a limited amount of information on how organisations responded. A few surveys have specifically looked perceptions on and attitudes to web accessibility (for example Lazar et al’s 2004 paper on Improving web accessibility: a study of webmaster perceptions and Bloor Research’s 2009 survey), but we still don’t know very much about the organisational reasons as to why accessibility of a particular web site is not as good as it could be.

As time goes by, therefore, the impact of a survey that presents data on web accessibility diminishes, unless it adds something new to our understanding of the problem. In 2009, it’s not enough to simply claim that results are ‘disappointing’, and that web content authors must ‘do better’. WCAG has been here for 10 years, so it’s not as if we have no best practice; and people and resources promoting and supporting web accessibility are easy to find. A failure to acknowledge in a survey report that that this has been a problem for many years does a disservice to everyone who has been advancing the cause of web accessibility (even if it also gives another stark reminder that there’s much work to do).

So if you conduct a survey of web sites and find accessibility barriers, don’t stop at reporting conformance levels. Find out why the barriers exist. Is it a lack of awareness or training amonst the providers of the content you surveyed? Is it a lack of involvement of disabled people in the design process? Is it a lack of resources; of money; a lack of will, or lack of prominence of accessibility in an organisation’s business practice and philosophy? Is it sub-standard authoring tools, content management systems; quality assurance systems? Is there another reason?

And whatever you find out, please share it with us!