DEliverable 2.3 - the School of Engineering and Design - Brunel ...

More documents

Recommendations

Info

ICT Project 3D VIVANT– Deliverable 2.3 Contract no.: 248420 User Acceptance Validation Plan 2 METHODOLOGY There are major differences between testing functions and features involving interactivity and the assessment of acceptance of perception quality (using vs. viewing). While, for instance, usability testing requires tests on whether users understand what they are expected to do and how easy it is for them to find out and actually do, validating acceptance of a novel perception of 3D video and audio will require very different research questions. These differences will influence the choice of methods to a large extent. The following section will describe the objectives and the adopted approach for conducting user validation tests for the development in 3D VIVANT. 2.1 QUALITATIVE VS. QUANTITATIVE ASSESSMENT METHODS As the main objective of the User Validation Activities in 3D VIVANT is to investigate whether the results of 3D VIVANT, from holoscopic 3D to novel opportunities to view and interact with video content, will be accepted by potential users as an added value to their previous experience, many subjective aspects will have to be taken into account. In order to explore the limitations of 3D VIVANT’s features and functions, the chosen evaluation methods will have to allow maximum openness and flexibility. For both interactivity as well as viewing characteristics there is broad agreement that they could not be sufficiently validated with objective measures alone. While performance, colour display and various other aspects may be measured according to objective criteria, subjective terms like acceptance are best evaluated with qualitative methods. All functions and features to be validated by project externals will be pilot tested by the partners involved in the development of these features before they will be subject to tests involving external users. These internal tests will include several objective tests, which will be dictated by the specifications determined in the respective work packages. User acceptance, on the other hand, will be validated primarily through qualitative methods. While quantitative evaluation methods require huge samples of testers in order to create statistics, deduct objectivity from subjective statements, and confirm or falsify the scientist’s expectations, qualitative methods are a source for recommendations and inspirations. Focus group discussions, open interviews and the thinking aloud method, for instance, may produce answers that were not planned, let alone prepared by the scientists leading the evaluation. This means, at the same time, that test results are less predictable as testers can come up with answers to questions that were not asked or not even thought of. This relative openness, however, does not contradict the will and possibility to achieve generalisation (Seale 1999) and transferability (Lincoln and Guba 1985) of the test results. Approved quality criteria ensure that both, the choice of the instrument and the evaluation of the test results will lead to an understanding of positive and negative potentials and give valuable input as to how to avoid negative and achieve positive results (De Abreu et al 2006). In terms of the data to be obtained from users, these typically fall under two categories: • Process data – Observations about what the users were doing and thinking as they proceeded through the tests. • Summary data – ‘Bottom-line’ information like the time taken to finish a task, error rate, etc. Relevant data collection methods are • Questionnaires – Comprising of both closed and open ended questions. Questionnaires in this 01.09.11 8
ICT Project 3D VIVANT– Deliverable 2.3 Contract no.: 248420 User Acceptance Validation Plan context are useful for collecting summary data, such as user data (e.g., age, background), user satisfaction, ease of use of specific tasks, performance of the system, etc. This can be collected in a fairly quick manner and processed and analysed at a later stage. Questionnaires will be given to the users to complete during tests or after they have used the tools in question. • Interviews – Semi-structured (prepared and spontaneous questions) interviews comprising of a selected user sample and one/two interviewees. The main aim here is to gain a deeper insight into the user perception of the system. Interviews are very useful in this context, as they facilitate confidentiality and allow users to express their views on the system. This is very useful from the interviewer’s perspective too, as they can further probe topics of interest with the user and ascertain valuable data that is very difficult to gain otherwise (e.g., via questionnaires). These are also useful in getting so-called user stories of their experiences. Interviews provide testers with process data. • Focus groups – Composed of a small number of users and a moderator. These are particularly useful in the process of user perception and form one of the main ways of getting feedback from professional users. They help to establish a feeling of group membership and provide a unique opportunity for users to express their views in an informal but informative manner. It is often the case in user groups that one response from a user triggers another question or response from other users or the moderator creating an iterative type of group interview/discussion. Focus groups provide evaluators with information on what users thought during tests and where it specifically did not match their expectations (process data). Inspiring and constructive remarks, especially from professional users, on how to improve the situation under assessment can be another very important outcome of a focus group discussion. The following sections will explain in details which methods will be used for which tests and why. 2.2 QUALITY ASSESSMENT ON 3D VIDEO While there are standardised objective methods for quality assessment of 2D video coding, display, etc., such as Perceptual Evaluation of Video Quality (PEVQ) and “Peak-Signal-to-Noise-Ratio” (PSNR), metrics for validating 2D video, these cannot be easily transferred to 3D (Veit 2011). Although ITU-R has initiated discussions on ‘Digital three-dimensional (3D) TV broadcasting’ (ITU- R Question 128/6 (2008)), there are no agreed standards for testing the perceived quality of 3D video, yet. Therefore, 3D VIVANT’s user validation tests will be inspired by standards on traditional validation scenarios on Quality of Experience of 2D video and/or TV as well as by tests performed in the course of recent research activities and currently proposed standards. ITU-R Recommendation BT.500, a standard which describes various methods of showing test footage and recording testers’ opinions will be a good starting point. The follow-up ITU-R BT.1438 did a very first step for standardizing subjective tests of 3D video but leaves many questions unanswered. The viewing conditions during the envisaged tests will be largely based on these recommendations and on Chen’s catalogue of subjective video quality assessment methodologies for 3DTV (Chen 2010), but may have to be adapted to 3DVIVANT-specific aspects in some details. In addition, methods of quality assessment of 3D displays, 3D content, and 3D devices based on human factors, are being discussed in IEEE P3333, the draft Standard for Test Procedures for Electric Energy Storage Equipment and Systems for Electric Power Systems Applications 1 , and may be considered in order to optimise the validation settings, if available at the time. Validating user acceptance will also have to involve comparison tests with other available technologies. 1 For more details check http://standards.ieee.org/develop/project/3333.html 01.09.11 9
Page 1 and 2: ICT Project 3D VIVANT- Deliverable
Page 7: ICT Project 3D VIVANT- Deliverable
Page 33: ICT Project 3D VIVANT- Deliverable

DEliverable 2.3 - the School of Engineering and Design - Brunel ...

Create successful ePaper yourself

Delete template?

Save as template?