Print versionDetailed testing method
During the user based one-on-one evaluation, participants were asked to complete a number of tasks (see 5.2.3 Task Summary).
A total of six participants (see Table 5.0) were recruited to take part in the one-on-one evaluation sessions. During these sessions participants sat with an RMIT consultant who acted as a host. The host guided the evaluation process and encouraged the participant to ‘think aloud’ as they went through the evaluation session. The host also provided limited prompting if the participant was unable to understand or progress any further. A second consultant acted as an observer and operated the filtered environments.
The participants were asked to undertake a number of typical tasks without knowing whether the Content Filter was being used or not. As the testing progressed the users were questioned about their subjective view of the performance of the Internet performance for each task.
The test was a simple ‘blind’ test with the users unaware of what was on test. The lab polled six users and ran through a series of scripted tests, that the users followed, the user’s reactions were then recorded for each filter with specific emphasis placed on noting whether any of them mentioned a performance issue.
The test was a simple ‘blind’ test with the users unaware of what was on test. The lab polled six users and ran through a series of scripted tests, that the users followed, the user’s reactions were then recorded for each filter with specific emphasis placed on noting whether any of them mentioned a performance issue.
Seven different environments were set up and the users were exposed to all filters. Six of the environments involved using a different filter and the seventh was a non-filtered environment.
Note that this test was run in a LAN environment at the TestLab, a further formal user experience test, if required, should attempt to run across a simulated ISP network with a larger section of end-user participants to ensure that any performance issues are noted and statistics can then be drawn.
This test was not designed to be a comprehensive user experience test, merely to gauge whether or not the majority of the six participants noticed some/any performance difference. The results of this test show that one out of six detected a performance reduction. In the real world on an ISP network, running at speeds of 256kbps or 512kbps the question must be asked how many individual users would detect a difference in speed, unless the upstream aggregate pipe was congested.
Participants were asked to complete a questionnaire providing additional demographic information (see Appendix 7.2). Results of this questionnaire are provided in Appendix 7.3.
5.2.1. Test configuration
Figure 5.0 - User experience test system
Please view the larger figure (New window)
5.2.2. Test tools
WAPT
WAPT 3.0 (www.loadtestingtool.com/features.html) is a load and stress testing tool that provides a consistent way of testing web sites and intranet applications that have a web interface. WAPT can be used to test and analyse the performance characteristics and bottlenecks of a web site under various load conditions.The tool simulates the activity of human users with virtual users. A test run can contain tens to hundreds of virtual users to simulate the load experienced by a web site when the corresponding numbers of real users browse it. The behaviour of each virtual user, the number of them and other test parameters are specified in a test scenario.
The tool includes the following components.
- Test recorder,
- Scenario editor,
- Test runner,
- Results viewer.
The Test recorder is used to record or capture browser activity when stepping through the web site to be tested in a Microsoft Internet Explorer window. When this process is finished the captured test sequence can be edited to adjust the virtual user’s behaviour and add dynamically changing data such as names or passwords, by using the Scenario editor. The Test runner is used to specify additional parameters for the test and runs the test while the Results viewer displays the results of the test.
Multiple iterations of test sequence and multiple test runs can be set up and run. The tool also allows the specification of the duration of test or the number of hits to perform.
5.2.3. Task summary
Tasks 1-7
For each of the 7 tasks, the users were asked to find a specified photograph on the web site http://www.photodisc.com and download it. Each of the 7 tasks was conducted in a different environment with one being non-filtered, and each of the remaining 6 tasks being run through a different filter. Participants were asked to comment on any noticeable performance differences of the web site’s behaviour between the individual environments.
Tasks 8-10
Tasks 8-10
For each task the users were asked to book a specified flight on the web site http://www.virginblue.com.au . Each of the 3 tasks was conducted in a different filter environment which were:
- Not filtered,
- With the fastest filter identified in performance testing,
- With the second slowest filter identified in performance testing.
For a detailed overview of tasks and the order in which filters were tested with the individual participants see Appendix 7.2.
* The decision was made to test the second slowest filter, as the slowest filter proved unreliable and continually crashed during previous Lab tests.
5.2.4. Participants
Number of participants
Six participants were selected for the one-on-one usability sessions.
This was done because, as a rule, a small sample that clearly reflects the demographic characteristics of interest is sufficient to observe and make judgments on typical user behaviours.
For web site usability sessions, up to six participants are selected to observe general trends in user behaviour. More users should be selected if a web site has several highly distinct user groups that may view the web site in different ways. For example, a University web site would need to consider users from groups such as existing students, staff, alumni, new students, overseas students etc.
Any more than twelve users for one-on-one sessions display diminished returns; as user differences plateau at approximately twelve users and 80% of user-related issues are already detected by this stage (see Nielson15, Jakob 2000; Rubin, Jeffrey ‘Handbook of Usability Testing’, John Wiley & Sons ISBN 0471594032; Kuniavsky, Mike16, 8 April 1998).
Hence a total of six participants were used with the following demographics, as being typical users of Internet services.
| Participant | Name | Occupation | Internet at Work | Internet atHome |
|---|---|---|---|---|
| 1 | Andrew | Recent University Graduate | Other | Dial-Up |
| 2 | Troy | Federal Police Officer | ADSL | ADSL |
| 3 | Athy | 3rd year University Student | ADSL | Cable |
| 4 | Nadine | 3rd year University Student | None | Cable |
| 5 | Michael | Writer | Dial-Up | Dial-Up |
| 6 | Steve | Secondary School Teacher | ADSL | Cable |
5.2.5. User comments and observations
Tables 5.1 through to 5.6 contain comments made by the individual user during the user based evaluation session. These comments were captured through the video taping of the sessions.
Table 5.1 - User 1 comments
| Task | Filter | User Comments | Observations |
|---|---|---|---|
| 1 | None | ‘It’s fast. My home connection is slower. ‘ | Positive reaction from participant. |
| 2 | 1 | ‘Just as fast. No difference.’ | Positive reaction from participant. |
| 3 | 2 | ‘Not very good. Blanked out all the images.’ ‘[If this happened at home] I would probably swear a huge amount.’ ‘Slower loading, big pause between loading the header and loading images.’ | User gets frustrated. All images have been greyed outby filter, even during search for generic terms such as ‘bus’ or ‘cars’. Filter behaves slower than others. |
| 4 | 3 | ‘Again broken images’. ‘[If this happened at home] I would assume something went wrong. Assume my connection played up.’ | No clear feedback provided by filter as to why images appear broken. User assumes problem lies with ISP. |
| 5 | 4 | ‘Bit slower than first time [non-filtered].’ ‘Getting to the page is slower.’ ‘The change of speed is bearable – nothing ridiculous.’ | Filter appears slower, but acceptable. |
| 6 | 5 | ‘Assume I am behind a firewall.’ ‘Worried why this page is being considered [forbidden].’ | BLOCKED: Filter does not allow access to ‘Art/Culture/Heritage’. |
| 7 | 6 | ‘Very slow.’ ‘Incredibly slow.’ ‘This was the slowest.’ | User notices slow down of connection. ERROR: Filter crashes when user tries to access search. |
| Task | Filter | User Comments | Observations |
|---|---|---|---|
| 8 | 5 | ‘Very quick, processed everything, no lost images.’ | Positive reaction from participant. |
| 9 | 3 | ‘Was very quick.’ ‘Slightly faster loading images.’ | ERROR: Filter crashes when user tries to access SSL page (credit card information) |
| 10 | None | ‘Very quick.’ | Positive reaction from participant. |
Note: at the conclusion of the User 1 session the following changes were made to the setup of filters:
- Filter 05 was changed to allow 'Art/Culture/Heritage'
- 'Not Work Related' Content was allowed on Filter 03
- 'Intimate Apparel/Swimsuit' and 'Nudity' were allowed on Filter 03 to show Photodisc images.
These changes were made to overcome blocking by the respective filters.
Table 5.2 - User 2 comments
| Task | Filter | User Comments | Observations |
|---|---|---|---|
| 1 | None | No comments by user in regards to speed of site’s performance. | |
| 2 | 2 | ‘Seems to have stalled.’ ‘Not sure if it is still downloading.’ ‘No difference [in speed].’ | Photos have been greyed outby filter without clear feedback. User doubts himself, starts task over several times. |
| 3 | 3 | ‘First one was similar to this one. Seems to be just as fast.’ | Positive reaction from participant. |
| 4 | 4 | ‘Not much difference between speeds.’ | Positive reaction from participant. |
| 5 | 5 | ‘No difference.’ ‘Speed and quality much the same.’ | Positive reaction from participant. |
| 6 | 6 | ‘Looks like it crashed.’ ‘[If this happened at home] I would retry address or ring up Bigpond to find out if lines are down.’ | ERROR: Filter crashes when user tries to access search. User assumes problem lies with ISP. |
| 7 | 1 | ‘No difference.’ ‘No need to worry about speed.’ | Positive reaction from participant. |
| Task | Filter | User Comments | Observations |
|---|---|---|---|
| 8 | 3 | ‘[If this happened at home] I would retry from the beginning. Type in all the details again.’ ‘Call up Virgin, even if it costs 10 AU$ extra for the flight.’ | ERROR: Filter crashes when user tries to access SSL page (credit card information) |
| 9 | None | ‘Good, straight forward.’ ‘Speed was really good.’ | Positive reaction from participant. |
| 10 | 5 | ‘No difference in speed.’ | Positive reaction from participant. |
Note: at the conclusion of the User 2 session the following change was made to the setup of filters:
- Filter 02 was modified to allow images on www.photodisc.com
Table 5.3 - User 3 comments
| Task | Filter | User Comments | Observations |
|---|---|---|---|
| 1 | None | No comments by user in regards to speed of site’s performance. | |
| 2 | 3 | ‘Some delay between pages.’ | Filter appears slower. |
| 3 | 4 | ‘Longer delay between pages.’ ‘Not really annoying – understandable.’ | Filter appears slower, but acceptable. |
| 4 | 5 | ‘Nothing different.’ ‘Less of a time delay between this and previous [filters].’ | Positive reaction from participant. |
| 5 | 6 | ‘Did I break it?’ ‘[If this happened at home] I would go back to the front page and do a different search.’ | ERROR: Filter crashes when user tries to access search. User initially blames herself for the problem. |
| 6 | 1 | ‘Image delay.’ ‘Loading images was slowest on this machine, but loading pages was straight away’. ‘Not annoying.’ | Filter appears slower, but acceptable. |
| 7 | 2 | ‘No difference.’ | Positive reaction from participant. |
| Task | Filter | User Comments | Observations |
|---|---|---|---|
| 8 | None | ‘Pretty quick in the search.’ | Positive reaction from participant. |
| 9 | 5 | ‘Much the same.’ | Positive reaction from participant. |
| 10 | 3 | ‘Pretty much the same.’ | ERROR: Filter crashes when user tries to access SSL page (credit card information)Positive reaction from participant in regards to speed. |
Table 5.4 - User 4 comments
| Task | Filter | User Comments | Observations |
|---|---|---|---|
| 1 | None | ‘Loaded nice and fast.’ | Positive reaction from participant. |
| 2 | 4 | ‘Longer to load.’ | Filter appears slower, but acceptable. |
| 3 | 5 | ‘All the images load together.’ ‘Faster than the previous one.’ | Positive reaction from participant. |
| 4 | 6 | ‘Slower than the rest of them.’ | Filter appears slower. ERROR: Filter crashes when user tries to access search. |
| 5 | 1 | ‘Faster again.’ | Positive reaction from participant. |
| 6 | 2 | ‘Speed is good.’ | Positive reaction from participant. |
| 7 | 3 | ‘Not as fast.’ | Filter appears slower, but acceptable. |
| Task | Filter | User Comments | Observations |
|---|---|---|---|
| 8 | 5 | ‘Nothing unusual.’ ‘Comfortable fast.’ | Positive reaction from participant. |
| 9 | None | ‘Faster than the previous one.’ | Positive reaction from participant. |
| 10 | 3 | ‘Faster than the previous one [Filter 05]’ | ERROR: Filter crashes when user tries to access SSL page (credit card information)Positive reaction from participant in regards to speed. |
Table 5.5 - User 5 comments
| Task | Filter | User Comments | Observations |
|---|---|---|---|
| 1 | None | No comments by user in regards to speed of site’s performance. | |
| 2 | 5 | ‘No difference.’ | Positive reaction from participant. |
| 3 | 6 | ‘[Except for the error] I didn’t notice anything.’ ‘[If this error happened at home] I would try again. | ERROR: Filter crashes when user tries to access search. |
| 4 | 1 | ‘No difference in loading.’ | Positive reaction from participant. |
| 5 | 2 | ‘I am on Dial-up connection [at home]. It’s all fast to me.’ | Positive reaction from participant. |
| 6 | 3 | ‘Not really different.’ | Positive reaction from participant. |
| 7 | 4 | ‘No difference.’ | Positive reaction from participant. |
| Task | Filter | User Comments | Observations |
|---|---|---|---|
| 8 | 3 | ‘No difference.’ | ERROR: Filter crashes when user tries to access SSL page (credit card information)Positive reaction from participant in regards to speed. |
| 9 | 5 | ‘No difference in speed.’ ‘I don’t pay much attention [to speed]. It only bothers me if it is ridiculously slow.’ | Positive reaction from participant. |
| 10 | None | ‘Scrolling [of the page] seemed slower last time.’ ‘A little bit faster than the other one [Filter 05]’. | Positive reaction from participant. |
Table 5.6 - User 6 comments
| Task | Filter | User Comments | Observations |
|---|---|---|---|
| 1 | None | No comments by user in regards to speed of site’s performance. | |
| 2 | 6 | ‘[If this error happened at home] I would restart the browser.’ | ERROR: Filter crashes when user tries to access search. |
| 3 | 1 | ‘Not sure if this one was quicker.’ | Positive reaction from participant. |
| 4 | 2 | ‘Seems a little bit slower, but doesn’t affect me.’ | Filter appears slower, but acceptable. |
| 5 | 3 | ‘No obvious differences.’ | Positive reaction from participant. |
| 6 | 4 | ‘Probably slowest out of the lot.’ ‘Takes a while.’ ‘Longer, but not annoying.’ | Filter appears slower, but acceptable. |
| 7 | 5 | ‘Transition of pages [appears] more blocky than smooth.’ | Filter appears slower, but acceptable. |
| Task | Filter | User Comments | Observations |
|---|---|---|---|
| 8 | None | ‘Probably quicker than before.’ ‘Was really quick.’ | Positive reaction from participant. |
| 9 | 3 | ‘Except for error, quick, fine.’ | ERROR: Filter crashes when user tries to access SSL page (credit card information)Positive reaction from participant in regards to speed. |
| 10 | 5 | ‘No difference.’ | Positive reaction from participant. |
15 ‘Why you only need to test with 5 users’ (http://www.useit.com/alertbox/20000319.html)
16 'Why User Testing Is Good', HotWired. (http://webmonkey.wired.com/webmonkey/98/14/index3a.html)
Back to Introduction to user experience testing | Table of contents | Forward to Summary of findings