Combining Information from Multiple Internet Sources

4. Tests of the three approaches
This chapter presents the tests of the three approaches: Game theory, Auction and
Consensus. There were three queries issued for the testing purposes: consensus decision
making, consensus decision making for conflict solving and is
consensus decision making for conflict solving good enough or
maybe Game theory or auction is better. The idea was to take three queries which
relate to the same topic; however first was to be simple, second more complex and third was to be
very complex, while retaining coherence.
There were 5 search engines queried. Four of them were English-language-based: Google,
Ask.com, Live, Yahoo! and one of Polish origin – Interia, which in fact is a Google based engine;
however very often it produces results which differ from its parent engine. Search engines were set
up to return 20 results for each query. This means that as input to tested algorithms there were 5
result sets provided; each comprising of 20 URLs. This allowed for fast algorithm processing.
The first phase of result evaluation was to compare the result sets of each of the three tested
approaches against result sets produced by each search engine individually. There are two measures
of comparison: Set Coverage and URL to URL coverage. Set Coverage measures how many URLs
from the result of the algorithm is contained in the result set returned by the search engine
regardless of the position of the URL. URL to URL measures how many URLs were at the same
position in both results – of the algorithm and that of the search engine. Those measures however;
were taken only for the 10 top results returned by each search engine. This means that in the
algorithms result sets there may be answers which are not shown in the result set of any search
engine. Afterwards, algorithms, for each query, were compared with each other and then with
MySpiders system [11].
42