There are currently three different options available to you when testing a website for accessibility:
An accessibility audit involves an accessibility expert reviewing your site, highlighting all accessibility issues and providing recommendations for fixing them. The reviewer would typically use assistive software used by disabled web users (e.g. a screen reader) to effectively carry out the audit, along with the Web Accessibility Toolbar.
You could hire an external accessibility consultancy to do this, but it's also possible to conduct the audit in-house. By reading through the W3C accessibility guidelines and attending a web accessibility training course (the latter to learn 'real-world' accessibility) you can gain a base level of accessibility knowledge. You should then be able to get your website up to a reasonable level of accessibility.
The main benefit of using an accessibility audit to evaluate your website is that accessibility audits are significantly cheaper and quicker than accessibility testing. Accessibility audits are often more comprehensive than accessibility testing in their depth and breadth of recommendations.
The main disadvantage of accessibility audits is that they're not designed for knowledge transfer. As such, your web team won't gain a great understanding of web accessibility nor are they likely to get much extra buy-in into accessibility. Both of these can be remedied through effective accessibility training.
There are a number of organisations in the accessibility world that swear by accessibility testing. The Disability Rights Commission, for example, have consistently said that testing a website with real disabled users is the only way to ensure it offers optimum accessibility. Their formal investigation into the accessibility of 1000 websites and the PAS 78 document they spearheaded both very strongly state that accessibility testing is the way to uncover all accessibility issues.
The main benefits of conducting accessibility testing include:
There are also some disadvantages of using accessibility testing as a way of evaluating your website's accessibility:
Generally speaking, it's not necessary to conduct accessibility testing most of the time. To conduct accessibility testing properly is extremely expensive and simply not worth the return on investment for most organisations. It's unlikely to highlight any major accessibility issues that don't come out of the accessibility audit (provided that the audit is carried out by an accessibility expert).
Are we really recommending to not do accessibility testing then? Well mostly, yes. Accessibility is far more guideline-driven than usability so accessibility solutions can easily be transferred from one website to the next. Provided that some organisations continue to carry out accessibility testing, you can 'piggy-back' off the knowledge that they've gained. By employing an accessibility expert to carry out an accessibility audit you'll quite quickly have this 'real-world' accessibility knowledge applied to your website.
If you do wish to carry out accessibility testing then it should only take place after the findings from an accessibility audit have been implemented. There's no point in conducting the accessibility tests if users fall down straightaway on basic accessibility issues that would have been highlighted in the audit.
Generally speaking, automated accessibility testing tools are very limited in being able to uncover real problems. They can't adequately check for the majority of accessibility guidelines and can sometimes report problems that aren't actually problems. For more on the disadvantages of automated testing tools, read our article, The problem with automated accessibility testing tools.
The only real benefit of using an automated testing tool is to get a top level feel of how accessible (or inaccessible) your website is. In terms of getting real recommendations for fixing real problems then an accessibility audit is vastly superior.