Print versionIntroduction to accuracy testing
This project was primarily concerned with the performance impact and issues associated with server based content filtering systems. However, testing was also undertaken to determine the relative accuracy of the filters.
It was not considered useful to undertake the performance test and discover that vendor A’s product performed twice as fast as vendor B’s without knowing that both could actually deliver a level of accuracy in filtering content. Tuning was required for all applications in this test, in most cases taking some time before the filters could be considered acceptable for the intended operating environment.
4.1.1. Issues with accuracy testing - Business Environment
The main issue raised during this testing process was that of judgment, particularly in relation to the subject of restricted content (sites that would be prohibited under the Online Content Scheme). The TestLab, for its part as an independent body, could not judge whether the content used in the accuracy testing should be blocked or not, therefore a business use scenario was created. Hence the results obtained were a measure of what sites were blocked and the results do not address false positives as this can only be determined by the administrators who set the policies.
It was assumed that a list of sites provided by ACMA, some 96 live unique sites, (the ‘restricted content list’) should all be blocked as content that has been found to be prohibited content (or potential prohibited content), following a complaint made to ACMA under Schedule 5 of the Broadcasting Services Act 1992. This provided the most accurate level of content blocking of illegal material, say in an ISP environment.
Additionally the TestLab selected 461 sites and added those to a ‘category list’ which the vendors attempted to tune their products against (to block), using a selection of 41 sites from the category list and 9 sites, such as RMIT University, Commonwealth Bank and News Limited as sites that should be passed . This category list was a subset of a larger list of active URLs originally created by NetAlert and the (then) Australian Broadcasting Authority (ABA) used for previous filter research conducted by the CSIRO3.
This smaller list was selected to cover many categories allowing more granular results to be produced indicating where each filter was most/least accurate at that point in time. The vendors were told to block the sites contained in the list. The vendors ran the list of 50 sites with the default settings on the filter and were able to see what URLs were being filtered and those that weren’t. The vendors then configured their filter to block the inappropriate sites. Following a first round of tuning the vendors noticed the sites that should be passed and adjusted their filter settings accordingly to allow the nine sites. This process was repeated once more and then the full list of sites was run.
This test simulated a scenario of a business/enterprise blocking employee access to content according to an acceptable usage policy. There were twenty seven categories in this test. This provided the most accurate indication of control that could be exercised over the applications in a business/enterprise specific policy enforcement environment.
Two other factors primarily affected the ability to exercise control. As content filters employ a variety of technologies to filter content, they may not necessarily be able to block all levels of content. Secondly, there is also the risk of ‘false positives’ occurring where content that should be allowed through the system is blocked. In this scenario the end user does not have the option of tuning the filtering product to suit their needs and instead must accept the upstream policies which are controlled by the ISP or policy administrator in a business environment.
3 ‘Effectiveness of Internet Filtering Software Products’, http://www.netalert.net.au/00379-CSIRO-Filter-Report.pdf
Table of contents | Forward to Detailed testing method