WebCenter User Experience and Interaction From iPads to Xbox

WebCenter User Experience and Interaction
From iPads to Xbox
John Sim, Fishbowl Solutions
@JRSim_UIX
Introduction
In recent years User Experience (UX) has become increasingly important. Organizations can no longer get away
with ignoring the ways users interact with applications both inside and outside the office. By investing time in
prototyping and usability testing, organizations can help their users be as productive as possible. As we look to the
future the traditional mouse and keyboard interaction model is simply not enough. Market innovators are
developing new methods to enhance applications with technology like Voice, Touch and Motion Gestures that enable
the user to interact with and locate the information they need quickly and easily in new and interesting ways.
There has already been an explosion of new requirements and support for the latest touch devices; we see evidence
of this even in Oracle WebCenter PS5. Recognized standards are coming into play more and more often as users
become increasingly familiar with touch devices and technologies like the Xbox Kinect. These technologies allow
users to interact with innovative UIs like the Windows Metro interface used within the Windows 8 operating system,
the Xbox gaming platform and Windows Mobile.
The Metro UI
This whitepaper will showcase how we can now support multiple devices and use new input methods to interact and
enhance WebCenter’s capabilities. By using custom implemented features to provide Touch, Motion, and voice
interactions within a web browser, users can escape from the keyboard while exploring new approaches to
interacting with information.
Cross-Device Interface Support: Desktop to Tablet to Mobile
Whether you are designing for a desktop display, or a Blackberry, or the latest iPad3, supporting multiple devices
within one interface can be a challenge. This type of interface design is now widely referred to as responsive or
adaptive design—an interface design where the user’s environment, screen resolution, platform and orientation are
all supported by one template. Clients have been asking for this support for years and, although it’s not easy, it has
been made easier with CSS3.
© 2012. Fishbowl Solutions, Inc.

Fluid grids (liquid layout), flexible images, breakpoints,
media queries and a touch of JavaScript are now the key
elements to getting started with responsive design.
Fluid Grids / Liquid Layouts
Fluid Grids provide the ability for a web site or
application to scale its containing region elements based
on the width of the browser viewport. This allows for
elements to be flexible and reposition on the site to
support multiple resolutions ranging from desktops to
mobile devices.
The new Smashing Magazine Website is a perfect
example of a responsive site – on the right you can see
an example of the site and it’s breakpoints as the
browser is scaled down from 1900px wide to 614px.
Initially all the navigation is vertical, positioned left of
the content region. This provides a nice flow from site
sections to section navigation elements. As we reduce
the size of the site you will notice that the navigation
model transforms to position these items in the header
above the content; providing a constant fixed
proportional width for the content information area of
the site.
As we scale further down you will notice the right hand
side of the site which previously contained
SmashingMagazine.com Working Example Responsive Site.
© 2012. Fishbowl Solutions, Inc.

Advertisements, Tags, and highlights is removed, and
the header now contains a large search field allowing
users to filter and find required content easier whilst still
having access and providing an equal amount of real-
estate for the main content portion of the site which
users would find more relevant.
Flexible Images
Generally felixble image use involves setting the image width to 100% and allowing the containing
DOM element to manage the size of the image as it scales down or up. Another option is to use
JavaScript to define which image to download based off the screen resolution. This is especially
useful when implementing support for mobile users in areas where mobile internet net speed is limited.
This option means that users are not forced to download large bandwidth-heavy heavy images.
There are
browsers like Amazon Silk and Opera Mini that will proxy and optimize the content delivered to the
device. Some mobile providers even do this
through their network,
but do you want to risk having users who do not
use those browsers or providers and thereby risk the responsiveness of your site appearing
slow? WebCenter
Content can provide this optimization functionality out of the box and can create multiple files optimized from a
source image and at different resolutions
for you to pull content into your website.
A good example and source for Responsive-Images - https://github.com/filamentgroup/Responsive-Images
For any img elements that offer a larger desktop-friendly size, reference the larger image's source via a ?full=
query string on the image url. Note that the path after ?full= should be written so that it works directly as the src
attribute as well (in other words, include the same path info you need for the small version).
Breakpoints
As you work with your design in a fluid grid, you will find that your site just doesn’t look right in certain resolutions.
A breakpoint is where you define how the template should be altered based on the resolution and orientation. This is
generally done with CSS Media Queries although more complex designs will use JavaScript to handle this.
Example Breakpoints
© 2012. Fishbowl Solutions, Inc.

These breakpoints will usually reflect the resolutions you would want to support – 240px, 320px, 480px, 640px,
800px, 960px, 1280px+. This is not definitive, but servers as a useful guide. As devices change, graphics cards
improve, and manufactures compete to have the highest resolution displays, web developers must continue to
evaluate the resolutions supported by their sites.
If you’re not using the web developer toolbar plugin within Firefox then you can use http://responsivepx.com/ that
allows you to define the dimensions and see how your site works within your predefined breakpoints.
CSS3 Media Queries
CSS media queries have been around for some time now. CSS2.1 allows developers to define stylesheets to be used
for printing a page, WebTV, projector display and daily device use, but now with CSS3 we can define the styles that
are used based on the following additional queries: max-width, min-width, device-width, orientation and color.
In the past web designers have either had a single global stylesheet or have broken them into multiple styles
(layout, colours, fonts) to be reused and allow easy corporate brand management across their WebSite, Intranet,
Extranet and Portal; but they have never really been widely used to support more complex multi-resolution capable
devices.
With CSS3 we can do this much more easily and can apply required transformations for breakpoints on the fly. For
example, if I rotate the screen I can load in a style to reposition crucial site elements, and if I resize the screen I can
load in another style to make the site adaptive.
2 ways of using CSS Media queries:
1. Inline within the style sheet you can encapsulate the required css within the query –
@media only screen and (max-device-width: 480px) {
body {
}
}
background:red;
2. Within the Media attribute of the html tag
For browsers that do not support the CSS3 Media queries a javascript patch is available from here
http://code.google.com/p/css3-mediaqueries-js/or https://github.com/scottjehl/Respond
Voice Enabled Integration (Microphone)
Speech input is one of the latest innovative browser technologies to appear.
It’s easy to implement and there are several obvious uses:
o
o
o
o
Assistive dictation for those with impaired mobility
An alternative input option for mobile phones and tablets
Provides support for an environment where a keyboard or mouse is impractical
Enhance web site features like Search and Navigation.
© 2012. Fishbowl Solutions, Inc.

Example Xbox Metro UI with Voice Enable Kinect Integration
There are number of methods to enable browser voice integration.
- here are some I have used:
1. Google Chrome Native Support
March of last year Google released the HTML speech input API within their Chrome browser giving developers the
ability to transcribe voice to text from a webpage—a prototype for the HTML Speech Incubator Group.
Architecture Diagram
© 2012. Fishbowl Solutions, Inc.

How does this work - Overview?
HTML Source
Speech test:
Example of Enabled Speech Integration
With the following attribute x-webkit-speech a microphone is displayed within the input field.
1. The user selects the microphone and the “Speak now” model window is displayed; as you speak you will see
the sound range highlight.
2. After you stop speaking the sound wave is passed to Google’s servers where speech recognition software is
used to analyze your voice.
3. An XML file is then passed back to the browser with the transcribed text and a few additional parameters
such as how close the recognition match was.
4. This text is then inserted into the selected text field.
The problem with this approach is that is currently only supported by the latest webkit browsers.
1. Dragon Speak, Kinect Voice SDK
If you need cross-browser device support for voice integration, you can now develop your own custom solution with
a few limitations.
Both Nuance and Microsoft can supply a voice recognition engine (Dragon Speak or Kinect Voice) that you can
setup on your webserver. The challenge lies in getting the browser to capture the voice input. Here are a few
options here for talking this:
1.1 Flash
Currently the only easily supported method is to use a Flash plugin as this plugin has access to the audio and video
input device – an open source framework is available from http://speechapi.com. It communicates with its own voice
engine, although you can configure it to point to your servers if desired.
function onLoaded() {
speechapi.setupRecognition(
"SIMPLE",
document.getElementById('words').value,
false,
false
);
}
© 2012. Fishbowl Solutions, Inc.

var flashvars = {speechServer : "http://www.speechapi.com:8000/speechcloud"},
params = {allowscriptaccess : "always"},
attributes = {};
attributes.id = "flashContent";
swfobject.embedSWF(
"http://www.speechapi.com/static/lib/speechapi-1.6.swf",
"myAlternativeContent",
"215", "138", "9.0.28",
false,
flashvars,
params,
attributes
);
speechapi.setup(
"eli",
"password",
onResult,
onFinishTTS,
onLoaded,
"flashContent"
);
function onResult(result) {
document.getElementById('answer').innerHTML = result.text;
speechapi.speak(result.text,"male");
}
function onFinishTTS() {
//alert("finishTTS");
}
function ResetGrammar() {
speechapi.setupRecognition(
"SIMPLE",
document.getElementById('words').value,
false);
}

};
audioStream = this.data;
recordCtlBut.disabled = false;
// in window.onload
recordCtlBut = document.getElementById("record_ctl_but");
recordCtlBut.onclick = function () {
if (!recorder) {
// start recording
recordCtlBut.value = "Stop";
recorder = audioStream.record();
// set the maximum audio clip length to 10 seconds
recordTimer = setTimeout(stopRecording, 10000);
} else
stopRecording();
};
function stopRecording() {
clearTimeout(recordTimer);
var audioFile = recorder.stop();
useAudioFile(audioFile);
// reset to allow new recording session
recorder = null;
recordCtlBut.value = "Record";
}
You can implement the above feature using the Web Real-Time Communication API that gives you early access to
Experimental Browser Features.
https://labs.ericsson.com/developer-community/blog/beyond-html5-audio-capture-web-browsers
1.1 Mobile Web App Frameworks
Finally, if you are creating a mobile web application with a framework like PhoneGap-Callback you have access to
the capture method that will enable you to record and transmit audio or video without the requirement of Flash.
The following is an example using phonegap 1.4.1 – The methods have changed since 1.0 which ADF Mobile currently
resides on.
http://docs.phonegap.com/en/1.4.1/phonegap_media_capture_capture.md.html#Capture
// Called when capture operation is finished
//
function captureSuccess(mediaFiles) {
var i, len;
for (i = 0, len = mediaFiles.length; i < len; i += 1) {
uploadFile(mediaFiles[i]);
}
© 2012. Fishbowl Solutions, Inc.

}
// Called if something bad happens.
//
function captureError(error) {
var msg = 'An error occurred during capture: ' + error.code;
navigator.notification.alert(msg, null, 'Uh oh!');
}
// A button will call this function
//
function captureAudio() {
// Launch device audio recording application,
// allowing user to capture up to 2 audio clips
navigator.device.capture.captureAudio(captureSuccess, captureError, {limit: 2});
}
// Upload files to server
function uploadFile(mediaFile) {
var ft = new FileTransfer(),
path = mediaFile.fullPath,
name = mediaFile.name;
}
//AJAX Upload to be replaced with Socket implimentation
ft.upload(path,
"URLToVoiceServer",
function(result) {
console.log('Upload success: ' + result.responseCode);
console.log(result.bytesSent + ' bytes sent');
},
function(error) {
console.log('Error uploading file ' + path + ': ' + error.code);
},
{ fileName: name });
Capture Audio
© 2012. Fishbowl Solutions, Inc.

Touch Events & Gestures: Integrating with Touch Screens
Touch integrations with both websites and applications are now playing a major role in both the mobile and tablet
world with touch screen monitors and laptops just around the corner. We can already see that Microsoft has put in a
great amount of effort with Windows 8 allowing its next generation OS to be fully touch interactive and
incorporating Rich HTML5-driven applications. More importantly Oracle also is also recognizing this with its latest
WebCenter PatchSet5 released in February.
ADF DVT Touch Graph Solution
Current Oracle PS5 ADF Touch Enhancements
o
Data Visualization (DVT)
Graph and Gauge now support HTML5 output format, supporting touch gestures for all the major
interactivity features, such as selection, zoom and scroll, legend scrolling, time selector, data cursor and
magnify lens.
A new web.xml context parameter oracle.adf.view.rich.dvt.DEFAULT_IMAGE_FORMAT was introduced to
change the default output format to HTML5. In addition, a new value for the imageFormat attribute
imageFormat="HTML5" is now supported to allow for explicit usage.
o
Redistributing Touch Events for Tablets
For tablet devices that support touch screens and do not have a mouse, the browser simulates some mouse
events, but not all. In order to achieve functional equivalency for these platforms, client components need to
be broadcast touch events. A new touch event object AdfComponentTouchEvent has been made available to
components when agents support single or multiple "touchScreen" capabilities.
Component peers can conditionally register for touch event handling using the same mechanism.
© 2012. Fishbowl Solutions, Inc.

ADF Example:
AdfDhtmlPanelSplitterPeer.InitSubclass = function(){
// Register event handlers specific to touch devices
AdfRichUIPeer.addComponentEventHandlers(this,
AdfComponentTouchEvent.TOUCH_START_TYPE,
AdfComponentTouchEvent.TOUCH_END_TYPE,
AdfComponentTouchEvent.TOUCH_MOVE_TYPE);
};
AdfDhtmlPanelSplitterPeer.prototype.HandleComponentTouchMove
= function(componentEvent) {...}
AdfDhtmlPanelSplitterPeer.prototype.HandleComponentTouchStart
= function(componentEvent) {...}
AdfDhtmlPanelSplitterPeer.prototype.HandleComponentTouchEnd
= function(componentEvent) {...}
o
o
Simulating Context Menu and Tooltip Activation from Touch Gestures for Tablets
Webkit on iOS and Android platforms does not raise contextMenu events. Under the desktop platform the
contextMenu event is derived from the right mouse click. The tooltip is also not supported by these
platforms. For desktop browsers the tooltip is shown on mouse over of elements that have a title attribute.
Since these tablet devices do not fire contextMenu events or show tooltips, Oracle has added an
enhancement to simulate this event from touch gestures. The default gesture is tap+hold (500ms).
However, this gesture is also used to active component drag-and-drop. To resolve this conflict in the cases
where drag-and-drop behaviors exist for a component, the context menu and tooltip will be activated on
tap+hold+finger-up. Only single finger gestures can be used to active context menus and tooltips.
Drag and Drop on Touch Devices
On touch devices like tablets, the component drag-and-drop has a different gesture than with the mouse. An
item that can be dragged must be activated by a tap-and-hold gesture. The item will change its appearance
to indicate that it can be dragged once held long enough. The same gesture applies to reordering table
columns. Tap-and-hold the column header, then drag it to reorder the columns.
Developing With Other Frameworks
When developing with WebCenter Content, Sites or Portal you can incorporate your own touch events if ADF Mobile
does not provide the functionality out of the box, and you are looking for a lightweight solution to be supported
across devices.
o
jQuery & ZeptoJS
jQuery is a fast and concise JavaScript Library that simplifies HTML document traversing, event handling,
animating, and Ajax interactions for rapid web development and is one of the most widely used Javascript
Frameworks. Currently it does not have support for touch guestures (although you can write your own and
there are a wide variety of touch plugins available).
If you are planning to support mobile devices, I would recommend taking a look at ZeptoJS; it is a minimalist
JavaScript framework for modern web browsers, with a jQuery-compatible syntax aimed to support mobile
devices and also incorporates a number of Touch Guestures – [tap, doubleTap, swipe, swipeLeft, swipeRight,
swipeUp, swipeDown, pinch, PinchIn, PinchOut].
© 2012. Fishbowl Solutions, Inc.

o
Other Libraries
There are other libraries available like SenchaTouch, jQueryMobile, jQTouch, etc., however these provide a
UI with the library for mobile devices and are not recommended for cross platform interfaces such as
desktop to Mobile.
Example of some of the standardized touch gestures
o
Touch points Device Support t and Considerations
In iOS you can capture 11 points of simultaneous contact with the device (The eleventh is a mystery to
everyone…)
Other operating systems capture a lot less, although this is improving. Currently when designing for touch
interfaces I do not apply support for more than 2 simultaneous touch interaction due to support across
devices and the requirements. I have never come across a requirement for more than this, and limiting to 2
touches also allows me to also reuse my methods for Motion Events as I standardize using only 2 hands to
interact with a screen.
The main JavaScript Touch Events are:
• Touchstart – fires once
• Touchmove – fires continuously
• Touchend – fires once
© 2012. Fishbowl Solutions, Inc.

• Example:
element.addEventListener("touchstart", myTouchFunction, false);
Do not use iOS gesture events unless you are only developing for iOS. IMPORTANT gesture events are not
supported on any other OS device. [gesturestart, gesturechange, gestureend].
For information on available touch event use the Apple site is a great resource, just keep in mind it’s
designed for iOS and may not work in Android or other platforms.
https://developer.apple.com/library/IOs/#documentation/AppleApplications/Reference/SafariWebContent/Han
dlingEvents/HandlingEvents.html
Experimental Motion activated Gestures (Kinect)
Spacial Operating Environment
This is more for the innovators and hackers out there.
Browsers are now escaping the standard Mouse and keyboard conventions as we can see with touch interactions
and voice integrations. There are also new API specifications and implementations using the latest custom browser
builds (Chrome and Firefox) such as the Gamepad API, Mouse Lock API, and Full Screen API - you can review the
draft gamepad API here –
http://dvcs.w3.org/hg/webevents/raw-file/default/gamepad.html.
A fun preview of this can be seen here using a wireless Xbox controller to interact with a browser -
http://vimeo.com/31906995
Minority Report 2002 Conceptual Touch Screen Innovation
For those of you that have seen Minority Report, Tom Cruise navigates through a set of enormous screens of the
future by gesturing his hands through the air.
The Kinect sensor within the latest Microsoft Xbox console works in a similar way.
A camera sensor plugs into the console or PC via a USB input enabling users to interact with an interface via a
motion gesture.
© 2012. Fishbowl Solutions, Inc.

How does this Work
There are currently no browser APIs available for the Kinect; however, there are two ways I have managed to
integrate the Kinect with the browser.
1. WebSocket Push
1. Plug the Kinect into your PC or Mac.
2. Install the Kinect drivers OpenNI and NITE
3. From device manager you will see you Kinect is now recognized.
4. Setup a WebSocket server (I use nodeJS)
(If you prefer, you can setup nodeJS to run directly on the clients machine within a FireFox XUL Extension -
a tutorial can be found here so an external server is not required.)
http://rawkes.com/blog/2011/12/05/running-node.js-from-within-a-firefox-xul-extension
a.) WebSocket Server plugin for WebLogic is soon to be released.
5. I wrote a small C++ app that passes the information from the Kinect to the socket server.
6. On the webpage I use the socket.io.client JS Lib that opens a connection to the WebSocket server.
7. I pass an array X,Y,Z co-ordinates from my C++ app connected to the Kinect through the WebSocket
gateway that transfers the information to the browser in realtime allowing me to create my own gestures
and integrations with HTML5 elements like canvas.
2. DepthJS
DepthJS in action interacting with The New York Times Website
Students at the Massachusetts Institute of Technology have gone further, inventing DepthJS, a browser extension
(currently Chrome & Safari) that allows the Microsoft Kinect to talk to any web page. It provides the low-level raw
access to the Kinect as well as high-level hand gesture events to simplify development. These allow the Kinect to
recognise hand and finger motions, allowing users to surf the internet and ''handle'' computer files. And, unlike Tom
Cruise's character in Minority Report, no gloves are required.
© 2012. Fishbowl Solutions, Inc.

1. Plug the Kinect into your PC or Mac.
2. Install the Kinect drivers OpenNI and NITE
3. Setup Browser
a.) Chrome, Firefox install the FireBreath plugin + DepthJS
b.) Safari install the DepthJS extension
c.) No support currently for IE
4. Include the depthjs lib on your page.
5. A demo can be seen here with a few sample events –
https://github.com/doug/depthjs/blob/master/developer-api/BasicDemo.html
Kinect for Windows
If you know C++, Windows provides a great resource for their open SDK allowing you to integrate with the Kinect
Voice Recognition engine and more – You can find information on this here –
http://www.microsoft.com/en-us/kinectforwindows/develop/
Conclusion
For the last 30 years the mouse and keyboard have been the main input devices for interacting with desktop
interfaces. While other technology such as graphics cards, processors, and network infrastructures have
significantly evolved during this period, it is only within the last couple of years that we have stepped out of the
confinement of mouse-and-keyboard interactions and have begun to look at how other input devices can help
improve our interaction with data. Touch integrations and spatially aware devices that were not even conceivable a
decade ago will now let us push user experience to the next level.
© 2012. Fishbowl Solutions, Inc.