DEliverable 2.3 - the School of Engineering and Design - Brunel ...

ICT Project 3D VIVANT– Deliverable 2.3
Contract no.:
248420
User Acceptance Validation Plan
2.3 QUALITY ASSESSEMNT ON 3D AUDIO
For studying the perceptive quality of audio of content reproducing, processing algorithms and
content generation, listening tests are the current state-of-the-art.
For conducting listening tests, there are several standardised and established methods. Most of them
are used for validating the quality level of coding systems. There are no up-to-date standardised
methods for 3D audio assessment, but the known and established principles can easily be transferred
for assessing certain attributes of 3D audio.
All methods offer the test subjects one or more test-signals, so called stimuli, which they have to
value on the basis of predefined attributes. Multiple stimuli, which have to be valued simultaneously,
form a so-called trial. The different trials of a listening test should be validated back-to-back and the
order of the trials has to be accidental and unknown to the subjects (ITU-R BS.1284-1).
Participating test persons are grouped in expert listeners and normal listeners. The assigned category
depends on the experience of the subject with listening tests and used methods. If the differences
between the stimuli under test are small and difficult to detect, the listening test should be conducted
with expert listeners. Test with normal listeners may be consulted for the evaluation of a new audio
reproducing system’s overall perceptive quality. For the evaluation it is necessary to separate the
valuations of expert and normal listeners to draw conclusions from the results to the listening
experience of the test persons. In principle the number of test persons should be as big as possible, but
in practice a number of 12 to 30 test subjects is considered to be sufficient.
Listening tests can be conducted in three different environments:
• Anechoic chamber: Advantageous for tests without any influence of the room.
• Headphone listening: Avoids most room influences and provides the detection of subtle
distinction.
• Room with predefined conditions referred to ITU-R BS.1116-1: Volume, reverberation
time, etc. are defined to prevent possible unwanted room influences. These parameters also
guarantee the reproducibility of the listening tests.
In the preparation of the listening test it is necessary to define the attributes, which are to be rated.
These can be for example the sound colour or the spatial impression. A list of possible attributes can
be found at ITU-R BS.1284-1, but using own attributes is also allowed as long as they are explicit and
clear defined.
2.3.1 Methods for the Assessment of Sound Colour
Every method for proceeding listening tests has advantages and disadvantages and has to be selected
on the basis of the criteria to be evaluated, the number of stimuli and the expected perceptibility.
Typical methods are:
• “ABX”: Two known stimuli “A” and “B” and one further “X”, which is either identically
with “A” or “B”, are offered to the test person. The test subject has to allocate “X” to the
corresponding stimulus. The different stimuli can be listened to as often as wanted. This
method is qualified for subtle distinction. The number of correct and wrong allocations is a
measure of the perceptibility of the differences, but doesn’t provide any quantitative
information.
• “Double-blind triple-stimulus with hidden reference” referred to ITU-R BS.1116-1:
Three stimuli “A”, “B” and “C” are presented to the test subject. “A” is always the known
01.09.11 12