Methodology

2.3.1. Overview

A NetAlert commissioned scoping study, mentioned in Section1.1, was undertaken to review the viability, processes and procedures associated with a potential broadband filtering trial that may be undertaken in conjunction with the Telstra Broadband eLab in Launceston, Tasmania.

The outcomes from the scoping study were required to provide NetAlert with sufficient information to determine the importance (or otherwise) of conducting a broadband filter trial to determine the performance degradation experience by broadband users when using server based Internet content filters.

The original proposal from RMIT TestLab suggested that the outcomes of the study would be as below.

‘Conduct research into any existing trials of Internet content filter performance. Extend existing knowledge and expertise in Internet content filter and broadband performance testing to ensure that all aspects of a potential trial are addressed. The research would enable RMIT TestLab to define:

• Clear objects for a trial,
• Technical test parameters for a potential trial including the criteria used to best evaluate the impact of server level filters on user experience in taking up broadband services,
• Identify hardware test resources and software tools required to conduct a trial,
• Skill sets and role of participants in the trial,
• Identify the types of filters to be assessed in the trial,
• Define the actual test procedures that are required to produce the specified results.

This study also recommended that an Internet content filter performance trial be conducted to provide quantitative information about any broadband performance degradation that may occur from the use of filters. In addition, the report recommended that accuracy testing of the filters should be conducted at the same time as the performance testing. This testing should provide details on how well restricted sites were blocked. Finally, the report recommended that the trial be extended from just conducting laboratory based testing (which will provide the actual level of any degradation) to the use of typical users in a phase of the testing so that feedback about the user experience is obtained.

2.3.2. Stage One: Selection of filters

Internet content filters, six filters were recommended to be selected for the Testing Phases. In order to achieve test coverage across the different product types, filters were selected in the following categories:

• Windows server based,
• Linux server based,
• Appliance.

In order to provide an equal opportunity for filter vendors to take part in the testing, an advertisement requesting Expressions of Interest was placed in The Australian newspaper, on 19 April 2005, inviting participation. However, only one vendor responded to this advertisement, hence other vendors were contacted directly and invited to participate.

No filter vendor was able to provide a product that was specifically designed for ISP usage. At least one vendor had an ISP based Internet content filter listed in its catalogue but the product was not locally available within the project timescale.

The products supplied by vendors and used in the trial were designed for use at an enterprise level. It was decided that the project should continue using these products as they would provide an indication of performance and accuracy in an environment that would not be very different to that of a smaller ISP. One product was still being developed, but it was decided to use it for the testing as it provided an example of an open source product.

2.3.3. Stage Two: Test planning

A comprehensive Test Plan was developed to provide an overall structure to the testing processes and to provide an indication of the higher-level strategic test issues such as scope, approach, risks and contingencies and test conditions. The Test Plan typically:

• Defined the purpose, scope, roles and responsibilities and test processes,
• Developed the final evaluation methodology, determined potential methods, identified choice and provide justification,
• Determined suitable test equipment and software tools to provide adequate virtual user⁷ resources that acted as a load for the performance testing and user experience phases of the project,
• Identified potential participants for user experience testing and defined test demographic range,
• Finalised the range of information collected and how this would be done (e.g. one on one user sessions, face to face user focus groups or online user focus groups etc),
• Provided details of the types of tests that will be conducted on identified test items,
• Identified test documentation to be produced,
• Identified the reporting process and any classification criteria,
• Developed any Test Scenarios and Tasks,
• Identified any metrics that to be used,
• The test environments to be used,
• Decide on which automated test tools to be used,
• Define any other test tools, if any, required.

2.3.4. Stage Three: Expert review

As part of the preparation for the following stages of the project methodology, RMIT TestLab staff conducted an expert review on each of the Internet content filters selected for the testing. In the expert review the tester undertook a ‘walk through’ of the product features and made a comparative assessment between the filters as well as assessing individual features based on the tester’s expertise and experience.

2.3.5. Stage Four: Performance testing

This stage of the methodology was designed to provide an indication of the reduction in typical network performance or throughput when using server based content filters. This performance reduction occurs as the filter has to make an assessment of the Internet site requested, usually by comparison with a list of URLs of inappropriate sites and then any URL requests matching the list are blocked, this process takes time and hence slows the process of requesting sites.

Performance testing was conducted at the Telstra Broadband eLab in Launceston. This stage involved the build of test resources and programming of test scripts to be run on automatic test tools in order to provide a significant load of virtual users for running performance tests.

A test facility or rig was assembled and a test was run with just the test rig itself and the results averaged to ensure that the performance results were consistent. This provided a baseline of performance for the test rig. Once the baseline was recorded, the filtering application/appliance was connected in-line with the server to filter all traffic transparently. The test routine was then run again to show the filtered results. Once the results were recorded and checked for consistency the filter was removed from the network and the server re-tested to ensure that the baseline test results were consistent. This test was repeated with each filter selected for the test.

As recommended in the test plan, engineers for each of the content filter vendors were involved in the initial setup and configuration to ensure their filter was tested in its optimal configuration and that any performance issues were solved via configuration changes with minimum time loss.

2.3.6. Stage Five: Accuracy testing

Whilst the overall aim of the project was to make an assessment of performance degradation caused by the use of server based content filter and the effect that this degradation has on the user experience, it was recognised that basic accuracy testing could be conducted at the same time.

The methodology for testing server based content filter accuracy testing was conducted in two phases which included testing against:

• A restricted content list, and
• A category list.

The restricted content list consisted of 96 URLs that had been rated by ACMA as content that had been found to be prohibited content (or potential prohibited content), following a complaint made to ACMA under Schedule 5 of the Broadcasting ServicesAct 1992 (BSA) and hence provided a representative sample of the content to be blocked by the content filtering engines.

The category list included 461 sites that were spread across 27 categories. The selection of URLs and categories in this second list were created by NetAlert and the (then) ABA for previous filter research³ and deemed to be relevant for this research.

Prior to conducting the testing, vendors of the application or appliance filters were given the opportunity to ‘tune’ their product in order to achieve optimal results. They were provided with a representative selection of 41 sites selected from the above category list and nine sites that would be expected to be passed by the filters (e.g. Commonwealth Bank, News Limited and the Australian Broadcasting Commission) to use for the tuning process and were told to block the 50 sites contained in the list. Following a first round of tuning the vendors noticed the sites that should be passed and adjusted their filter settings accordingly to allow the nine sites. This prevented the vendors from setting their filters to simply block everything.

This created a scenario of a business/ enterprise looking to block content according to their employee acceptable usage policies.

The raw accuracy results were tabled which showed simply which sites were blocked.

This test required vendors to be present to assist the RMIT TestLab staff in properly configuring their filters for maximum accuracy prior to the testing taking place. This was done to provide an equal opportunity for each vendor to adequately set up their product as the filters need to be tuned or tuned to match the user’s requirements.

This accuracy testing was bracketed by the performance testing as it provided an indication to whether or not the tuning created any additional overhead on the performance results.

2.3.7. Stage Six: User experience testing

The NetAlert commissioned Scoping Study (Section 1.1) was unable to identify any previous testing conducted to determine whether the level of degradation in broadband performance, caused by the use of server based Internet content filters, would adversely affect the user experience of broadband. The study recommended that a user based testing method be used to practically test the user experience with and without content filters enabled. A number of different user experience test methodologies were discussed in the study. It was not technically possible to take advantage of the large sample broadband audience available through the Telstra Broadband eLab in Launceston and hence a smaller scale test was undertaken and run at the TestLab facility at RMIT. This testing used a well proven usability user based testing technique which provides rich feedback regarding a user’s experience when using a software application or web site. In this process the participant was required to attend the RMIT TestLab where they undertook a number of test tasks (with and without the Internet content filtering turned on) with a usability specialist (or Host) sitting with them. The participant was requested to ‘think aloud’ so that the Host could determine the user’s thought processes and the logic in navigating the sites. At key times the Host would ask questions related to the performance of thenetwork to determine the user reaction to any degraded performance. This process was video taped so that a permanent record of the session was obtained.

The user based methodology has the advantage that a considerable amount of information can be achieved using on a small number of participants from the main user demographics for the product under test.

³ ‘Effectiveness of Internet Filtering Software Products’, http://www.netalert.net.au/00379-CSIRO-Filter-Report.pdf

⁷ Virtual User: A virtual user is a way of simulating real users carrying out real activity.

Back to Objectives | Table of contents