The main problem with user evaluation in designing interactive systems is that information about what, in usability terms, is wrong with a system comes very late in the design process. Recall that we argued that rectifying mistakes becomes more and more expensive the later in the design process they are identified. This means that usability issues identified by evaluation can be very expensive to remedy. Indeed in a lot of cases, too expensive to remedy and the product will get shipped full of usability bugs.

There is plenty of evidence of developers identifying problems by user testing, realising that it is now too expensive to fix them, papering over the cracks and shipping the product. A very popular way of papering over cracks is by creatively writing the user manual for the product. If a usability problem is identified then the developers can usually think of a way of getting around the problem, though this get around is usually complicated and difficult. The problem and the get around can be described in the manual, and to a rather trivial extent, this ‘solves’ the problem. Usability expert Harold Thimbleby claims that the usability of a product is inversely proportional to the size of its user manual; the bigger the manual, the less usable the product.

Many of the criticisms that can be aimed at rapid prototyping can also be aimed at evaluation techniques. In particular evaluation techniques identify where a system is unusable, but not why.