03.12.2012 Views

Proposal for the tutorial at ISMAR 2012 Adaptive Augmented Reality ...

Proposal for the tutorial at ISMAR 2012 Adaptive Augmented Reality ...

Proposal for the tutorial at ISMAR 2012 Adaptive Augmented Reality ...

SHOW MORE
SHOW LESS

Create successful ePaper yourself

Turn your PDF publications into a flip-book with our unique Google optimized e-Paper software.

<strong>Proposal</strong> <strong>for</strong> <strong>the</strong> <strong>tutorial</strong> <strong>at</strong> <strong>ISMAR</strong> <strong>2012</strong><br />

<strong>Adaptive</strong> <strong>Augmented</strong> <strong>Reality</strong> (A 2 R): where AR meets user’s interest<br />

Nenad Stojanovic, FZI Karslruhe, Germany<br />

Areti DAMALA, Centre d’Etude et de Recherche en In<strong>for</strong>m<strong>at</strong>ique et Communic<strong>at</strong>ions, Conserv<strong>at</strong>oire<br />

N<strong>at</strong>ional des Arts et Métiers, Paris, France<br />

Tobias SCHUCHERT, Fraunhofer-Institute of Optronics, System Technologies and Image<br />

Exploit<strong>at</strong>ion, Karlsruhe, Germany<br />

Ljiljana Stojanovic, FZI Karslruhe, Germany<br />

Stephen FAIRCLOUGH, School of N<strong>at</strong>ural Sciences and Psychology, Liverpool John Moores<br />

University, UK<br />

Abstract<br />

<strong>Adaptive</strong> behavior is one of <strong>the</strong> main challenges in building computerized systems, especially<br />

in <strong>the</strong> case of systems which are delivering in<strong>for</strong>m<strong>at</strong>ion to <strong>the</strong> end users. Indeed, since <strong>the</strong><br />

in<strong>for</strong>m<strong>at</strong>ion overload has become <strong>the</strong> main drawback <strong>for</strong> <strong>the</strong> future development of such<br />

systems (cf. Big D<strong>at</strong>a challenge), <strong>the</strong>re is a huge movement in <strong>the</strong> research community to<br />

develop concepts <strong>for</strong> better adapt<strong>at</strong>ion of <strong>the</strong> <strong>for</strong>m and size of in<strong>for</strong>m<strong>at</strong>ion th<strong>at</strong> will be<br />

delivered to a user (usually taking different <strong>for</strong>ms of <strong>the</strong> personaliz<strong>at</strong>ion). However, <strong>the</strong> main<br />

ef<strong>for</strong>t has been dedic<strong>at</strong>ed to <strong>the</strong> contextualiz<strong>at</strong>ion of <strong>the</strong> user’s task in order to determine wh<strong>at</strong><br />

is <strong>the</strong> best way to tailor/adapt <strong>the</strong> present<strong>at</strong>ion of in<strong>for</strong>m<strong>at</strong>ion to <strong>the</strong> user, neglecting <strong>the</strong> role<br />

of <strong>the</strong> user’s internal context, expressed as <strong>the</strong> user’s (short-term) interest. The same is valid<br />

<strong>for</strong> <strong>the</strong> AR systems.<br />

In this <strong>tutorial</strong> we present novel results in modeling users’ interest in <strong>the</strong> context of AR<br />

systems and demonstr<strong>at</strong>e some practical results in realizing such an approach in a multisensor<br />

AR system based on <strong>the</strong> usage of <strong>the</strong> see-through AR glasses. Due to <strong>the</strong> need <strong>for</strong><br />

continuously adapt <strong>the</strong> AR content to <strong>the</strong> user’s interest, such models are facing many<br />

challenges in sensing <strong>the</strong> user’s behavior (using acoustic-, video-, gesture- and bio-sensors),<br />

interpreting it as an interest and deciding in real-time wh<strong>at</strong> kind of <strong>the</strong> adapt<strong>at</strong>ion to per<strong>for</strong>m.<br />

We argue th<strong>at</strong> this lead to a new class of AR system th<strong>at</strong> we coined as adaptive AR (A 2 R)<br />

systems.<br />

This work has been partially realized within <strong>the</strong> scope of <strong>the</strong> FP7 ICT research project<br />

ARtSENSE (www.artsense.eu), th<strong>at</strong> is developing new AR concepts <strong>for</strong> improving<br />

personalized museum’s experience. The <strong>tutorial</strong> will present practical results from applying<br />

<strong>the</strong> approach in three cultural heritage institutions in Europe (Paris, Madrid and Liverpool)<br />

Motiv<strong>at</strong>ion<br />

There are two main novelties to be presented in this <strong>tutorial</strong>:<br />

- It will introduce new concepts rel<strong>at</strong>ed to <strong>the</strong> real-time adapt<strong>at</strong>ion of <strong>the</strong> AR systems<br />

th<strong>at</strong> is becoming ever more important <strong>for</strong> <strong>the</strong> realiz<strong>at</strong>ion of <strong>the</strong> AR systems in different<br />

scenarios, especially where <strong>the</strong> content to be presented is huge and changing in realtime<br />

(so called streaming d<strong>at</strong>a). We will present a multisensing environment using<br />

acoustic-, video-, gesture- and bio-sensors which are combined in a complex way in<br />

real-time using methods from <strong>the</strong> complex event processing in order to detect changes<br />

in <strong>the</strong> user’s interest on time and calcul<strong>at</strong>e <strong>the</strong> best possible adapt<strong>at</strong>ion. We argue th<strong>at</strong><br />

such a combin<strong>at</strong>ion of sensors <strong>for</strong> calcul<strong>at</strong>ing changes in <strong>the</strong> user’s st<strong>at</strong>e is beyond<br />

existing AR systems and our results and experiences can be quite interesting <strong>for</strong>


<strong>ISMAR</strong> community. We find <strong>the</strong> role of biosening in detecting interest in <strong>the</strong> AR<br />

context as very interesting/novel <strong>for</strong> <strong>at</strong>tendees.<br />

- The presented AR system is based on <strong>the</strong> new gener<strong>at</strong>ion of AR see-through glasses,<br />

th<strong>at</strong> has been developed by Fraunhofer, initially in <strong>the</strong> project iSTAR and continued<br />

within ARtSENSE. The experience with glasses would be quite interesting <strong>for</strong> <strong>the</strong><br />

<strong>ISMAR</strong> community, especially because of <strong>the</strong> recent huge hype around <strong>the</strong> Google<br />

Glas project (https://plus.google.com/111626127367496192147/posts). We hope to be<br />

able to present a comprehensive analysis about wh<strong>at</strong> can (and cannot) be realized in<br />

<strong>the</strong> future AR glasses (e.g. based on advertised in<strong>for</strong>m<strong>at</strong>ion, Google AR glasses will<br />

not support eye-tracking, like in our case).<br />

Target Audience<br />

The content of <strong>the</strong> <strong>tutorial</strong> should be interesting to everyone, emphasizing <strong>the</strong> <strong>at</strong>tention of <strong>the</strong><br />

research community <strong>for</strong> <strong>the</strong> topic 1 from <strong>the</strong> Motiv<strong>at</strong>ion section and <strong>the</strong> industry/applic<strong>at</strong>ion<br />

community <strong>for</strong> <strong>the</strong> topic 2. Since <strong>the</strong> examples th<strong>at</strong> will be presented in <strong>the</strong> <strong>tutorial</strong> will be<br />

taken from <strong>the</strong> Cultural Heritage and Art domain, <strong>the</strong>se communities will be benefiting much<br />

from our experience in applying cutting edge technologies in <strong>the</strong>se domains<br />

Outline of <strong>the</strong> Tutorial:<br />

1. <strong>Adaptive</strong> <strong>Augmented</strong> <strong>Reality</strong> (A 2 R): 1.5 h<br />

1.1. Motiv<strong>at</strong>ion, St<strong>at</strong>e of <strong>the</strong> Art and Use cases overview<br />

1.2. Requirements from use cases<br />

1.3. A 2 R: Found<strong>at</strong>ion<br />

1.3.1. Sensing<br />

1.3.1.1. Acoustic sensing<br />

1.3.1.2. Visual sensing<br />

1.3.1.3. Gesture recognition<br />

1.3.1.4. Biosensng<br />

1.3.2. Interpret<strong>at</strong>ion<br />

1.3.2.1. Signal Fusion<br />

1.3.2.2. Complex Event Processing<br />

1.3.2.3. User’s Attention and Interest<br />

1.3.3. Action<br />

1.3.3.1. Recommend<strong>at</strong>ion<br />

1.3.3.2. Execution<br />

1.3.3.3. Adapt<strong>at</strong>ion Loop<br />

1.4. A 2 R: Challenges<br />

1.4.1. Met<strong>at</strong>ad<strong>at</strong>a provisioning<br />

1.4.2. Per<strong>for</strong>mances<br />

1.4.3. Precision of models<br />

2. Demonstr<strong>at</strong>ion of <strong>the</strong> technology: 30 min<br />

Selected scenarios (to be detailed l<strong>at</strong>er on)<br />

3. Potential of and Applic<strong>at</strong>ions <strong>for</strong> A 2 R: 45 min<br />

3.1. Applic<strong>at</strong>ion potential: overview


3.2. Cultural heritage: Better In<strong>for</strong>m<strong>at</strong>ion provisioning<br />

3.3. Art: Fostering Cre<strong>at</strong>ivity<br />

3.4. Google Glas project: go’s and no go’s<br />

4. Concluding remarks: 15 min<br />

Learning Objectives<br />

1. To learn wh<strong>at</strong> is challenging <strong>for</strong> <strong>the</strong> AR community when going into Big D<strong>at</strong>a scenarios<br />

2. To understand <strong>the</strong> role of multisensing and real-time adaption in such scenarios<br />

3. To understand <strong>the</strong> limit<strong>at</strong>ion/potential of <strong>the</strong> biosensing technologies <strong>for</strong> engaging and nonintrusive<br />

AR systems<br />

4. To learn particip<strong>at</strong>ory design methods and tools requested <strong>for</strong> developing A 2 R systems<br />

It will be an interactive <strong>tutorial</strong>. Experienced presenters will be provoking questions from <strong>the</strong><br />

audience in order to monitor learning success. M<strong>at</strong>erial will be provided well in advance<br />

Form of Present<strong>at</strong>ion<br />

The <strong>tutorial</strong> will be provided in <strong>the</strong> <strong>for</strong>m of slides and demonstr<strong>at</strong>ion activities<br />

Expected # of <strong>at</strong>tendees<br />

Around thirty <strong>at</strong>tendees can be accommod<strong>at</strong>ed<br />

Planned demo has been presented several times on different occasions (most recently, ESWC<br />

conference, May <strong>2012</strong>, next planned DEBS conference, July <strong>2012</strong>)<br />

An early version of this <strong>tutorial</strong> was presented in SIMESITEM <strong>2012</strong> in Louvre, Paris, January<br />

<strong>2012</strong>. (Video: http://www.youtube.com/w<strong>at</strong>ch?v=BnbGllVQMYQ).<br />

Publicizing Tutorials<br />

There will be <strong>the</strong> web page dedic<strong>at</strong>ed to <strong>the</strong> <strong>tutorial</strong>: <strong>ISMAR</strong><strong>2012</strong>-<strong>tutorial</strong>.iCEP.fzi.de.<br />

Different email lists will be used <strong>for</strong> announcing <strong>the</strong> <strong>tutorial</strong>. Twitter will be used <strong>for</strong><br />

spreading <strong>the</strong> word.<br />

Presenter Bios and Contact In<strong>for</strong>m<strong>at</strong>ion <strong>for</strong> Presenters<br />

Nenad Stojanovic, FZI Karslruhe, Germany, Nenad.Stojanovic@fzi.de<br />

Dr. Nenad Stojanovic is <strong>the</strong> project leader in IPE. He received <strong>the</strong> MSc in computer science<br />

from <strong>the</strong> University of Nis / Serbia and <strong>the</strong> PhD degree from <strong>the</strong> University of Karlsruhe<br />

(<strong>the</strong>sis on Ontology-Based In<strong>for</strong>m<strong>at</strong>ion Retrieval). He has worked in and has had <strong>the</strong> technical<br />

management of several EU as well as bmbf propjects (n<strong>at</strong>ional German funding) in <strong>the</strong> area of<br />

applying semantic technologies <strong>for</strong> knowledge based systems. In this area, he also particip<strong>at</strong>es<br />

in industry consulting projects. He published more than 70 technical papers in intern<strong>at</strong>ional<br />

journals, conferences, and workshops in <strong>the</strong> areas of applic<strong>at</strong>ions of ontologies and machine<br />

learning. He is initi<strong>at</strong>or and co-chair of several workshop serials in <strong>the</strong> area of Semantic Web<br />

and Complex Event processing on prominent research conferences. He was co-organizer of<br />

<strong>the</strong> AAAI 2009 Spring Symposia on Intelligent Event Processing<br />

Teaching experience: 1996-2000, University of Nis, Serbia, 2001-2006 University of<br />

Karlsruhe, Germany<br />

Recent conference <strong>tutorial</strong>s:<br />

ESWC 2011, Semantic technologies <strong>for</strong> real-time d<strong>at</strong>a processing: st<strong>at</strong>e of <strong>the</strong> art, applic<strong>at</strong>ion<br />

opportunities and challenges<br />

DEBS 2011, Grand Challenge: The Global Event Processing Fabric and its Applic<strong>at</strong>ions


Web of Events: Challenges and Opportunities <strong>for</strong> <strong>the</strong> event-driven world, Networking Session<br />

<strong>at</strong> ICT Event 2010<br />

Public<strong>at</strong>ions:<br />

Nenad Stojanovic, Ljiljana Stojanovic, Darko Anicic, Jun Ma, Sinan Sen, Roland Stühmer<br />

Semantic Complex Event Reasoning -- Beyond Complex Event Processing<br />

book, Springer Berlin Heidelberg, 2011, 253-279<br />

Yongchun Xu, Nenad Stojanovic, Ljiljana Stojanovic, Ana Cabrera, Tobias Schuchert<br />

An Approach <strong>for</strong> using Complex Event Processing <strong>for</strong> <strong>Adaptive</strong> <strong>Augmented</strong> <strong>Reality</strong> in<br />

Cultural Heritage Domain, Proceedings of <strong>the</strong> 6th ACM Intern<strong>at</strong>ional Conference on<br />

Distributed Event-Based Systems (DEBS <strong>2012</strong>) July <strong>2012</strong> Berlin<br />

Darko Anicic, Paul Fodor, Sebastian Rudolph, Nenad Stojanovic, EP-SPARQL: A Unified<br />

Language <strong>for</strong> Event Processing and Stream Reasoning proceeding, WWW 2011: Proceedings<br />

of <strong>the</strong> Twentieth Intern<strong>at</strong>ional World Wide Web Conference<br />

Areti Damala, Centre d’Etude et de Recherche en In<strong>for</strong>m<strong>at</strong>ique et Communic<strong>at</strong>ions,<br />

Conserv<strong>at</strong>oire N<strong>at</strong>ional des Arts et Métiers, Paris, France<br />

Dr. Damala works as a Post-Doctoral Associ<strong>at</strong>e Researcher in Cultural Engineering, Museum<br />

Studies and Interaction Design, in <strong>the</strong> CEDRIC labor<strong>at</strong>ory of <strong>the</strong> Conserv<strong>at</strong>oire N<strong>at</strong>ional des<br />

Arts et Métiers, in Paris, France. Her research interests are focused on <strong>the</strong> interpret<strong>at</strong>ive uses<br />

of New Technologies in Museums and Cultural Heritage Institutions with a particular focus<br />

on Mixed and <strong>Augmented</strong> <strong>Reality</strong> applic<strong>at</strong>ions and mobile learning in in<strong>for</strong>mal and <strong>for</strong>mal<br />

learning environments. Dr. Damala has a dual background in Museum and Visitor Studies<br />

and In<strong>for</strong>m<strong>at</strong>ion and Communic<strong>at</strong>ion Technologies. She has particip<strong>at</strong>ed in several n<strong>at</strong>ional<br />

and European research projects and Networks of Excellence -particularly rel<strong>at</strong>ed with <strong>the</strong> use<br />

of <strong>Augmented</strong> <strong>Reality</strong>, and mobile and context aware interpret<strong>at</strong>ive applic<strong>at</strong>ions <strong>for</strong> <strong>the</strong><br />

museum visit- and has collabor<strong>at</strong>ed with many European Museums, CH institutions and<br />

research academic and industrial labor<strong>at</strong>ories. During <strong>the</strong> winter semester 2011-<strong>2012</strong> Dr.<br />

Damala taught <strong>the</strong> post-gradu<strong>at</strong>e course "Museums and Educ<strong>at</strong>ion: Digital Media in Museum<br />

Pedagogics" <strong>at</strong> <strong>the</strong> Department of Cultural Technology and Communic<strong>at</strong>ion of <strong>the</strong> University<br />

of <strong>the</strong> Aegean, in Mytilene, Greece (http://www.ct.aegean.gr/en) while in <strong>the</strong> spring semester<br />

she taught as an adjunct lecturer <strong>at</strong> <strong>the</strong> Université Paris 1 (Panthéon, Sorbonne) <strong>the</strong> use of ICT<br />

and document<strong>at</strong>ion technologies to undergradu<strong>at</strong>e History of Art and Archeology students<br />

(http://www.univ-paris1.fr/en/ufr/ufr03/).<br />

Public<strong>at</strong>ions:<br />

I. Astic, C. Aunis, A. Damala, E. Gressier-Soudan, A ubiquitous mobile edutainment<br />

applic<strong>at</strong>ion <strong>for</strong> learning science through play. In: Museums and <strong>the</strong> Web 2011, Archives and<br />

Museum In<strong>for</strong>m<strong>at</strong>ics, April 2011, Philadelphia, USA. Available <strong>at</strong>:<br />

http://conference.archimuse.com/mw2011/papers/a_ubiquitous_mobile_edutainment_applic<strong>at</strong><br />

ion_fo<br />

O. Haberman , A. Damala, R. Pellerin , U. Haberman , E. Gressier-Soudan. Exploring<br />

Contemporary Painting through Sp<strong>at</strong>ial Annot<strong>at</strong>ions Using RFID Tags. In: VAST'10, Virtual<br />

<strong>Reality</strong>, Archaeology and Cultural Heritage, The 8th EUROGRAPHICS Workshop on<br />

Graphics and Cultural Heritage, Short and Project Papers, Paris, France, September 21-24,<br />

2010., January 2010, pp.49-52.<br />

Ljiljana Stojanovic, FZI Karslruhe, Germany, Ljiljana.Stojanovic@fzi.de<br />

Dr. Ljiljana Stojanovic is a senior researcher and project manager <strong>at</strong> <strong>the</strong> FZI - Research<br />

Center <strong>for</strong> In<strong>for</strong>m<strong>at</strong>ion Technology <strong>at</strong> <strong>the</strong> University of Karlsruhe. She received her B.Sc. in<br />

electrical engineering and M.Sc. degrees in computer science from <strong>the</strong> Faculty of Electronic<br />

Engineering, University of Nis, Serbia, in 1994 and 1998, respectively. In 2004 she received<br />

her PhD degree from <strong>the</strong> University of Karlsruhe, Germany <strong>for</strong> <strong>the</strong> <strong>the</strong>sis on "Methods and


Tools <strong>for</strong> Ontology Evolution". Since 2001, she works <strong>for</strong> FZI and has carried out her<br />

research activities in many projects funded by <strong>the</strong> European Commission in <strong>the</strong> FP6/FP7 IST<br />

Programme. Currently she is <strong>the</strong> coordin<strong>at</strong>or of <strong>the</strong> FP7 CP-STREP ALERT project and <strong>the</strong><br />

technical and scientific coordin<strong>at</strong>or of <strong>the</strong> FP7 CP-STREP ARtSENSE project. She has<br />

published more than 50 papers in intern<strong>at</strong>ional journals and conferences in <strong>the</strong> areas of<br />

semantic technologies, knowledge management, e-government and event processing. She has<br />

organized many workshops in <strong>the</strong> area of Semantic Software Engineering and Semantic<br />

EGovernment.<br />

Recent conference <strong>tutorial</strong>s:<br />

ESWC 2011, Semantic technologies <strong>for</strong> real-time d<strong>at</strong>a processing: st<strong>at</strong>e of <strong>the</strong> art, applic<strong>at</strong>ion<br />

opportunities and challenges<br />

Public<strong>at</strong>ions:<br />

Nenad Stojanovic, Ljiljana Stojanovic, Darko Anicic, Jun Ma, Sinan Sen, Roland Stühmer<br />

Semantic Complex Event Reasoning -- Beyond Complex Event Processing<br />

book, Springer Berlin Heidelberg, 2011, 253-279<br />

Tobias Schuchert, Fraunhofer-Institute of Optronics, System Technologies and Image<br />

Exploit<strong>at</strong>ion, Karlsruhe, Germany<br />

Dr. Tobias Schuchert received his diploma degree in Electrical Engineering from <strong>the</strong><br />

Technical University Dortmund, Germany in 2005 and <strong>the</strong> PhD in Electrical Engineering<br />

from <strong>the</strong> RWTH Aachen University in 2010. He joined <strong>the</strong> Fraunhofer IOSB in 2009 and<br />

works as research employee in <strong>the</strong> research group “Image-based Real-time Systems”.<br />

Public<strong>at</strong>ions:<br />

Tobias Schuchert, Fabian Oser: Optical Flow Estim<strong>at</strong>ion with Confidence Measures <strong>for</strong><br />

Super-resolution based on Recursive Robust Total Least Squares. ICPRAM (2) <strong>2012</strong>: 463-<br />

469<br />

Tobias Schuchert, Hanno Scharr: Estim<strong>at</strong>ion of 3D Object Structure, Motion and Rot<strong>at</strong>ion<br />

Based on 4D Affine Optical Flow Using a Multi-camera Array. ECCV (4) 2010: 596-609<br />

Tobias Schuchert, Til Aach, Hanno Scharr: Range Flow in Varying Illumin<strong>at</strong>ion: Algorithms<br />

and Comparisons. IEEE Trans. P<strong>at</strong>tern Anal. Mach. Intell. 32(9): 1646-1658 (2010)<br />

Stephen Fairclough, School of N<strong>at</strong>ural Sciences and Psychology, Liverpool John Moores<br />

University, UK<br />

Dr. Stephen Fairclough is <strong>the</strong> Subject Leader <strong>for</strong> Neuroscience and Health Psychology in <strong>the</strong><br />

School of N<strong>at</strong>ural Sciences and Psychology <strong>at</strong> Liverpool John Moores University (LJMU). He<br />

has been involved in applied psychophysiological research almost twenty years and has over<br />

fifty research public<strong>at</strong>ions in psychophysiology and HCI.<br />

Public<strong>at</strong>ions:<br />

Nikola B. Serbedzija, Stephen H. Fairclough: Reflective pervasive systems. TAAS 7(1): 12<br />

(<strong>2012</strong>)<br />

Stephen H. Fairclough: Fundamentals of physiological computing. Interacting with<br />

Computers 21(1-2): 133-145 (2009)

Hooray! Your file is uploaded and ready to be published.

Saved successfully!

Ooh no, something went wrong!