Detailed testing method

Accuracy testing was conducted by selecting a number of sites, some of which were either restricted or otherwise inappropriate and should be blocked and others that were ‘legitimate’ and shouldn’t be blocked. These sites were accessed sequentially using a test system that automatically stepped through the list of test sites and recorded whether they were blocked or not.

The two lists of URLs used for the accuracy testing were:

• Restricted content list consisting of 96 URLs sourced from ACMA as content that had been found to be prohibited content (or potential prohibited content), following a complaint made to ACMA under Schedule 5 of the Broadcasting Services Act 1992 (BSA)
• Category list consisting of 461 URLs sourced from NetAlert. This list was a subset of a larger list of active URLs originally created by NetAlert and the (then) Australian Broadcasting Authority (ABA) used for previous filter research conducted by the CSIRO¹³

It should be noted that these lists consisted of active URLs when the testing commenced however, the lists were further reduced during the testing as some of the URLs became unavailable during the test period.

4.2.1. Restricted content list

The first list contained 96 URLs and was a subset of a larger list of URLs supplied by ACMA. The URLs were divided into 4 categories.

The URLs used here were also typical of the type of site names that are sent out to filter vendors by ACMA advising them that they should be blocked.

4.2.2. Category list

The second list of URLs was provided by NetAlert as containing representative categories for sites in which some sites would be expected to be blocked and others that should be passed through. While some of the material contained in the list would not be prohibited under the Online Content Scheme, it may be considered to be “inappropriate”. Other categories and sites however, may well be considered “appropriate” depending on the requirements of an individual business or organisation.

Sites were therefore allowed and blocked by filters, as a result of the unique policies set for the filter by the filter vendor in the tuning process mentioned in 4.1.1.

Of the categories listed below, a significant number of URLs were not available and out of the 900 URLs that were supplied in the original list only 461 URLs were currently active and so were used to form the category list. This list was divided up into 27 unique categories containing a mixture of potential “appropriate” and “inappropriate” categories and URLs.

The URLs used to form this list were not part of the list of sites ACMA sends out to filter vendors.

* Redirectors¹⁴ were included as part of the list to test whether the filters could easily be bypassed

4.2.3. Test configuration

Figure 4.0 Accuracy filter test configuration

View the larger figure (New window)

Using the representative selection of sites as detailed in section 4.1.1, the application or appliance vendors were able to tune their filters, if necessary, before conducting the accuracy testing.

The testing process required vendors to be present to assist the RMIT TestLab staff in properly configuring their filters prior to the testing taking place in order to maximise accuracy. This was done due to the nature of filters and their ability to be tuned to match the user’s requirements.

A single computer (Intel P4 2.8Ghz with 1GB RAM) running on a 1Gbps switched network was used to visit a total of 557 URLs consisting of the 96 sites from the restricted list and 461 from the category list. Internet Explorer 6 was used as the browser to fetch all URLs.

An automatic test tool (Mercury Quick Test) was used to automate the accuracy testing process. This tool reduced the amount of time it would have taken for a tester to manually visit every URL. A test script was prepared that emulated a user and Internet browser requesting all 557 web sites with a 10 second delay between each web site. The tool also recorded the content that was retrieved from the web server. The raw accuracy scores were then tabled to show which sites passed and those that failed.

All the URLs chosen were available at the time the lists were constructed. There were times, however, where some of the URLs were unavailable during the testing. This problem was further compounded by the time it took to run through all the URLs on all the filters. The list of URLs that was initially constructed comprised of 603 URLs and from the time the first filter was tested to the 6th and last filter; 46 URLs were unavailable. Because of this the 46 URLs were deleted from the initial list so that they didn’t skew the results in any way and the filters were then scored on the 557 URLs (both lists included) that remained.

¹³ ‘Effectiveness of Internet Filtering Software Products’, http://www.netalert.net.au/00379-CSIRO-Filter-Report.pdf

¹⁴ Redirectors: Proxy servers set up as a publicly available service accessible from anywhere on the Internet. Web users can send their browsing requests to these remote proxies, and can possibly bypass any filtering imposed by their local ISP.

Back to Introduction to accuracy testing | Table of contents | Forward to In depth results and analysis