Concurrent protocols involve the subjects of the observation verbalising what they are doing and why whilst they perform the task that is the subject of the observation. Retrospective protocol is used after the observation – subjects recount what they did and why.

Concurrent protocols may distract the subject whilst they are attempting to perform their task. On the other hand, retrospective protocols rely on the subject remembering what they did and why, and involve the subject in extra time as part of the evaluation.

There is only one independent variable which is the type of user interfaces. It has three levels: 1) command-line, 2) desktop metaphor, and 3) virtual environment.

There is one dependent variable – the time it takes to complete the take using a given interface.

As there is only one independent variable and it has three levels there are three conditions – the three levels of the independent variable.

Confounding variables might include: subjects' experience of any of the interfaces, and typing experience (for command-line interface).

Heuristic 1.1: The interaction is concise, but not too clear about how to proceed from step to step

Heuristic 1.2: The language is not natural – users have to press PGRM to start the video programming process, also, OK does not do what users might expect (accept a value).

Heuristic 1.3: Users must remember both what PGRM does, and that the first set of numbers presented are not the channel numbers.

Heuristic 1.4: Consistency seems good – the user always presses Þ to move to the next entry.

Heuristic 1.5: Feedback is good – something always flashes when there is something to be done.

Heuristic 1.6: There are no clearly marked exits. In fact the user must press PGRM again to exit.

Heuristic 1.7: Shortcuts are not supported – its not clear how relevant they would be to a remote control anyway.

Heuristic 1.8: There are no error messages.

Heuristic 1.9: It is not possible to enter erroneous information (this is not apparent from the example).

Heuristic 1.10: There is no help or documentation provided on the remote control.

This shows that the heating panel is better for checking all the rooms than the lighting panel as the ‘on' buttons are all in the same group.

Model based approaches use simple models of human cognitive processes to predict how users would react to systems. As such they do not require users, but are often complex to use. Other approaches rely on observing, or questioning users about their use of a system. This means that users have to be found who are willing to try to use a system. Moreover, their actions need to be interpreted by analysts, and sufficient users need to be found to give generalisable results. Finally, some approaches do not rely on users, but use trained analysts. These benefit from not requiring users, but results may be skewed by analysts prior understandings of what the evaluation is about – what sort of problems they expect to find.

Approaches which involve models are appropriate where we want to make quite specific predictions about user behaviour, possibly before any systems have been built. These should help us to decide whether certain interfaces would be more appropriate than others for given tasks.

Four advantages of the waterfall model:

Fourth generation tools allow systems to be specified at a high level of abstraction and then convert this specification directly into runnable code. It is similar in structure to the iterative prototyping discussed previously. However, instead of prototyping, complete systems are generated from the requirements. These can then be evaluated to determine whether they are suitable. If not, the specification (derived from consultation with clients) will need to be changed, or the style of system generation altered i.e. changing the kinds of interfaces the code generator tool produces.

Out experiment would have to take into account the kind of user, the working environment, and the kinds of tasks and interaction users have with computers. For example, a user who relies heavily on email correspondence, works in their own office, and does not perform intensive interaction with their computer may prefer a dialogue box with the sound of a trumpet. In contrast, someone who does not rely on email and moreover, performs intensive interaction with their computer e.g. an air traffic controller, may want no visual notification, and only slight audio indication such as a coughing noise. Therefore, our hypothesis might be that there is a correlation between the users' tasks, email reliance, and notification style.

Independent variables:

Dependent variables:

A large problem will be finding enough subjects to complete the experiment – there are four independent variables, each with several levels. Another problem would be determining how intensive a task is in order to assign the subjects appropriately – maybe this could be determined by some pre-experiment tests. Finally, there is the problem of measuring user preference (the dependent variable) – this could be ascertained from post-experiment interviews or questionnaires.

Using a questionnaire we could ask people what kinds of noises get their attention, what noises they would consider suitable for a working, home, or recreational environment. Maybe this could be structured using multi-choice answers, or possibly open questions in order to find a greater range of sounds. A harder aspect of the questionnaire would be how to find out which visual indicators people might prefer. In order to cater for non-computer users we would have to think of some metaphors for email arrival. For instance, likening it to post arriving every five minutes – would they was some one coming up to them and announcing it (like a dialogue box), or would a simple indicator such as a lamp lighting be better (like a flashing icon).

Understanding non-computer users' preferences can help in the design of new systems – such understandings can help us to make new systems more friendly to potential new users.

Experimentation typically requires a large number of subjects who are given quite restricted tasks to perform. User observation, on the other hand, requires a smaller number of subjects who may carry out larger and more realistic tasks. Moreover, whereas experimentation typically requires quite specific physical surroundings such as a usability lab, user observation is more flexible and can even be employed in the workplace. Results from the two approaches also differ - experiments provide quantitative data, whereas user studies typically give qualitative data.

The table below details possible problems with the remote control in terms of the first ten heuristics

The main interface improvements would be some clearly marked exits, less jargon, and the possibility of flexible data entry. It would also be beneficial to have on-line help which may be audio in the case of such a small device.

The third and fourth steps are much more coherent than the first two. For a start, the number blinks to indicate that it can be changed (question 2, 3). Also, the user can enter the number directly on the keypad, e.g. 22, or can use the arrows to increase or decrease it. The numbers themselves immediately update, so meeting question 4. As with the previous steps, it is a bit odd that the user has to press Þ instead of OK, but by this point the user may have become accustomed to this interaction.

In terms of redesign, the main changes would be to allow OK to be used to confirm entries. Also, it would be clearer if the video timers were referred to by letters rather than number which may be confused with channel numbers – maybe these could even be entered later in the task so that the desired channel is set first.

In setting the video two approaches can be taken in entering numeric values such as time and date – using the numeric keys, or the navigation keys. Both of these approaches indicate possible problems due to the structure of the interface. Using the navigation keys, the OK and Cancel buttons are in the same group, so implying that they might be used to confirm entry of values – this is not the case. Conversely, entering numeric values using the numeric keys means that moving to a different entry point (e.g. from start time to end time) requires shifting focus to the navigation group to press the Þ key which requires effort on the part of the user.

Even though we assume expert users when evaluating systems using GOMS there are still plenty of tasks for which a mouse is more efficient than a keyboard.

For example, supposing a user wants to delete the files bib_info.jpg, cs_search.jpg, and greenstone_digital_lib.jpg from the file directory illustrated below. One way of doing this using a mouse driven interface would be to select bib_info.jpg, then drag it to the trash, then select cs_search.jpg, and drag it to the trash, and finally select greenstone_digital_lib.jpg and drag it to the trash. This is 3(Tp + Tb÷2 + Tp + Tb÷2) = 7.2s. To perform the same task using a command-line interface such as DOS would require the user issuing a series of commands such as ERASE bib_info.jpg¿, ERASE cs_search.jpg¿, ERASE greenstone_digital_lib.jpg¿. This would take 19Tk + 20Tk + 34Tk = 8.76s. Therefore, in this case, it would be quicker for an expert to use a mouse to delete these files.

Clearly an expert DOS user could use wildcards to shorten the filename and so reduce the number of keys pressed e.g. ERASE g*¿ would erase greenstone_digital_lib.jpg. However, we could argue that this would involve a mental operator (Tm). Moreover, the mouse based approach could be made more efficient by multiply selecting the files and then dragging them all at once to the trash so saving click and drag operators. The key to this example is that mouse based interfaces are better for multiple file operations where wild-cards would be difficult to use.