A multi-finger haptic interface for visually impaired people - Robot ...

Proceedings of the 2003 IEEE lntematlonal Workshop on
Robot and human lnteracliv~ Communcation
Millbrae. Califomla. USA. Om 31 - hov 2.2003
A Multi-Finger Haptic Interface for Visually Impaired People
C.A. Avizzano, S. Marcheschi, M. Angerilli, M. Fontana, M. Bergamasco*
T. Gutierrez**, M. Mannegeis***
* PERCRO, Scuola Superiore S. Anna
Piazza Martin d e b Liberta, 33. Pisa, Italy, mailto:carlo@sssup.it
** Labein, Bilbao, Spain, *** National Council of Blind of Ireland
Abstract
The present paper deals with a novel type of dual point
haptic system. The system exploits a novel kind of
kinematics to achieve a high isotropy which improves the
force rendering. Other features of the system are its high
stiffness, high peak forces together with a zero backlash
cable based transmission. The device can be used by
meam of a set of sizeable thimbles by any pairs of user’s
fingertips. Such a device has been developed within the
EU GRAB project, for testing a navel sei of application
with blind users. Such a project investigates to which
extent a purely haptic environment can be employedfrom
users wirh visual impainnenrs in ierms OJ? the effective
capabilities of the device to exchange high level
information with the users: the real degree of inieractioii
which can be achieved in this ope of system; the effective
opponuniry for the user to profit from the system using
just this type of interncrion.The basic system concepts, the
applicntion environmenr and the system perfonwnces are
given in the following.
1.Introduction
Haptic interfaces have proven to be extremely useful
during the interaction of users within Virtual Environment
(VE). Witbin these environment the use of the haptic sense
is employed for enhancing the sense of presence by
reproducing the contact forces and other specific forces on
the user band (Emery 161). The use of such systems
improves the interaction with the VE and increases the
sense of presence and the dexterity of the users with the
virtual objects.
Within almost all the existing VE, the haptic
interfaces work in cooperation with a visual feedback
which integrates the contact information with
complementary visual information. It is commonly
accepted the position that the visual interaction is
dominant in these types of interaction as well as it is
known that haptic feedback can be used to improve the
interaction of people with visual impairments. In (41
OModhrain transcribed the visual information of the
screen and made it available to the haptic senses for blind
person, by using a prototype two-axis HI, the Moose. In
151 Melchiorri proposed a haptic tactile display to help
people to navigated environments by means of an
0-7803-81 36-W03/$17.00@2003 IEEE 165
automated conversion of camera images to haptic “bas-
relief”. In I71 Tzovaras experimented some of the effects
of haptic interfaced when Visual Impaired (VI) users
interacts with them in VE.
Figure 1. Typical system layout
The forerunner research on use of haptic devices as
force display tools when targeted to VI people
demonstrated that such a feedback can be conceived as
being independent from visual cues. While in the case of
sighted people such an approach is not very useful, it is
specifically true for the case of VI people. In the specific
case of VI, it can be conceived to associate audio feedback
to haptic one in order to have a more immersive
interaction. Such an idea is not completely new, In 191
Ramstein presents a system in witch force feedback is
introduced by enhancing the mouse through the addition
of a Pantograph HI and “earcons”, along with voice
synthesis, are used to identify objects and events. In 1101
Pai experimented an interface having audio haptic
capabilities during the contact interaction. The integration
of haptic feedback together with the audio feedback can
provide an environment powerful enough to allow a
comfortable interaction to the computer interfaces for
blind people.
Since last five year, a lot of research activities have
been carried out for building applications and systems well

'.
:
suited for being used from VI people. For example in [SI
Kopacek set up a system for teaching to blind users.
Notwithstanding the research efforts in this field and
the encouraging results provided, it was commonly
acknowledged by the authors' that the present haptic
systems were not sophisticated enough to allow blind
people to effectively enabling VI people to interact with
computer application. On the basis of these considerations,
the GRAB consortium has been motivated to create a new
type of system which is purposely developed for ihese
users and that tries to overcome the technological limits in
its most significant aspects.
The main aim of GRAB is to allow blind people to
access to the 3D graphic computer world through the sense
of touch and with audio help. by means of a Haptic and
Audio Virtual Environment (HAVE). The proposed
HAVE would allow the user to move his fingers over the
virtual objects, to interact with them, to recognize specific
.features (comers, edges,..), as if he was manipulating a
real mock-up of the object. Further features specifically
thought for VI people enhance the level of interaction by
means of a smart fusion of the audio-haptic feedback with
the speech recognition capabilities included in the system.
The GRAB system has been produced in a limited
number of copies and it is currently subject to several
validation tests in four users group laboratories all over
Europe. The research discussed in this paper has been
supported within the framework of the GRAB project The
GRAB research has been canied on with the economic
support of the European Union. In the present paper the
basic structure of the GRAB system will be described, the
design of the new haptic device discussed and the overall
performances of the system reported. The functional test
and the user validation of the device together with its
application have been theme of another international paper
r1.1.
. .
. . 2.Motivation
Before starting the research activities, previous
consortium experiments 121 had shown the difficulties in
operating practical application with almost all
commercially available HI?. The need for a large
workspace, high manipulating forces, high isotropy, access
to multiple fingers and a good force resolution were not
addressed by any of the existing commercial devices on
the market (PHANTOM, CyberGrasp, PCS Haptic Master,
Delta, Virtuose [IZ]). A deep analysis of the features of
these devices when applied to a VI population has been
conducted in [3]. In particular, this preliminary user
analysis, based on the mock-up evaluation and on the user
survey, indicates that the possibility of separating the
haptic information among hands would allow the user to
make a separate use for fingers. For example, one finger
can be dedicated to pan the environment while the other
166
can be used, for exploring the objects. On the basis of these
considerations the Consortium decided to proceed in the
development of a novel haptic system which is based on a
new concept haptic device and a novel interacting
environment. The figure 1 represents the components of
the system assembled in a typical system layout.
. .
1 HAPTIC- AUDIO VIRTUAL ENVIRONMENT 1
Figure 2iBasic Operation of the HAVE.
The;+VE is based on the integration of:
a new force feedback haptic interface capable of
replicating independent force vectors on two fingertips,
wi,th 'a workspace covering a large part of the user
desktop and with a high force feedback fidelity to
perceive tiny geometrical details of the objects:
a haptic geomevic modeller (HGM) to provide haptic
stimuli to be rendered by the HI and sound aids to
improve the interaction among the user, the new HI
d d any 3D virtual object.
The figure 2 shows the basic operation of the GRAB
HAVE and the interaction between the user and the two
main components of the system (haptic interface and
haptic geometric modeller) during the interaction tasks.
Using.the sensor readings of the haptic interface, the
system can compute the spatial position of the user fingers
and compare them with the description of a virtual space.
3.Description of the system
The main hardware components of the system consist
in the following elements (Figure 3): a twc-finger haptic
interface, a driving unit and a control unit.
The haptic interface is composed by two identical
robotic arms. Each ann has 6 Degree of Freedoms
(DOF's), of which 3 are required to track the position of
the fingertip in 3D space and the remaining 3 are required
to track its orientation. The arms will interact with the
user's fingers by means of a couple of thimble the user can
suit.

1 1
Figure 3. Hardware component of the system
Due to the large workspace requirements, a serial
kinematics has been selected the first 3 DOF's are used to
track the position while the last 3 DOF's to track the
orientation. In this case, only the first 3 DOF's have to he
actuated in order to exert forces of arbitrary orientation at
the fingertip, while the last DOF's can he passive because
it is not required to exert moments. The following figure
shows briefly the kinematic scheme adopted for the
solution:
Figure 4. Kinematics scheme of the MACRO
This kinematic solution allows a very high degree of
isotropy of each arm w.r.1. other kinematics solutions, like
that used for the PHANTOM. A high degree of isotropy is
important in order to achieve a uniform use of the
actuators in the workspace of the device and to have a
uniform reflected inertia.
In order to improve the transparency of use of the
device, the following design guidelines have been adopted,
for the design of the anns:
Remote localization of the motors
Selection of motors with high torque to mass ratio and
high torque to rotor inertia ratio
Use of tendon transmissions
Use of low reduction ratio geared reducer
Use of light materials for the construction of the
moving links
Low or zero backlash implementation of the joints.
167
These guidelines have been implemented as follow:
The first 2 actuators have been mounted integral with
the fixed link (base), while the third actuator has been
mounted integral with the second moving link (link2)
having its center of gravity very close to the
intersection of the yaw and pitch axes of the
mechanism that is fxed w.r.t. the ground;
As actuators, brushed DC setvomotors have been
selected, with an iron-less construction of the rotor;
Metallic in tension tendons routed on idle pulleys have
been used as means of transmission of forces form the
actuators to the joints;
No geared reducer have been used
All the structural parts have been realized in
aluminum.
In fig.5 an assonomeuic view of the final device is
reported.
Figure 5. Assonometric view of the haptic interface.
The control unit provides the required real time
support for running a Real time operating system. A
850MHz PENTIUM m provides the required computing
power for a 2.5Khz control frequency. The unit is capable
of managing the following U 0 ECP Xbit parallel U 0 for
communicating to a remote PC, 24 bit digital IN digital
OUT for device control, 6 Digital Analog Converters for
drivers and motor control, 6 24-bit encoder inputs for
encoders monitoring.
The control system kernel will be based on Linux and
a real-time micro-kernel which ensures high performances
capabilities (up to 5 0 m z typical switching time with few
microsecond latency [141). All the data VO will be
provided with real-time device drivers.
Connection with the HGM (Haptic Geometric
Modeller) that runs on a remote PC computer is achieved
by means of an ECP parallel port interface which allow
high data throughput (>=500 Kbytekec), bi-directional
communication, low CPU resource request (automatic
FIFO and handshake) and no latencies. The ECP standard
(defined by the IEEE 1284 specs) has been selected as the
target communication hardware for GRAB due to its low
costs and the availability to common personal computers.

Law-level control software has been implemented in
order to provide system the following functionalities:
manage the ECP communication
verify, change and store the received value for system
tunable parameters
provide sound feedback
generate the correct HI control motors signal for
moving two arms and for activating the force feedback
functionality
. Furthermore the control system identifies, describes
and compensates a wide set of non linear features such as
i the arm kinematics and dynamics, the unbalanced weight
.. effects, the transmission model of the elastic cable, the
distributed friction effects, the electrical features of the
power supply, the errors due to swctural deflection under
: load conditions.
...
All the features available have been embedded into
the control system and masked with respect to the external
usage so that they can be transparently used by a remote
.'application (the HGM).
Figure 6. Schematic of the control architecture.
a.Control architecture
For executing all tasks and procedures which involve
HI actuation, that is to generate the correct HI motors
control signals starting from the received HGM command,
HI controller architecture is implemented on 3 different
level.
Four control modes have been implemented
position control, used when is necessary to move HI
along a predefined trajectory and/or with a predefined
speed
force control, used 'when the system should be
actuated in order to apply the desired forces on finger
compensation control, this allows to compensate
different system non linearity
calibration control, this features enhance the accuracy
of the force feedback by improving the. spatial
resolution of the devices and synchronizing the arm
thimbles position measurements. In a visionless VE
this instruments improves coherence of devices and the
168
quality of the interaction.' Being based on a novel
synchronization method il' will be discussed in a
forthcoming paper.
'Ibe control software is arranged as a hybrid hard real-
time hierarchically organized control. The highest-level
control scheme is arranged io a few number of subsystems
containing specific software which implements each of
functionalities below.
Figure I. HI Controller functionalities.
As shown in figure 6, the heart of the GRAB control
system model is represented by the Master Main
Machine which provide to manage and coordinate whole
system functionalities according to command and data
received from HGM computer and to monitor the system
status.
HI controUer, is the main part of control software
which provides to controls the HI in order to execute
specific task established by Master State Machine; it
includes positiodforce control and compensation of HI,
models of different system components, U 0 board drivers
which interfaces the control software with hardware
signals (encoder and signals to motor power supply) ...
Sound Driver, a set of simple midi songs are
implemented in order to indicate the current status of
control system (movement, ready, run ...).
ECP communication, contains a fast real time driver
and a local state machine which provides to manage and to
monitor ECP data transmission: thus, it allows the Master
State Machine to receivekend commandheply and data
buffer f r d t o HGM computer according to a simple
internal protocol; this module is most important during
force feedback phase because it should provide to receive
from the HGM the desired forces for the arms and then to
send to the HGM the position of each arm.
Tunable Parameters, by predefined sets of
command, HGM computer can changes the current values

for some control parameters, i.e. position and orientation
of reference system respect to which are defined positions
and forces of thimbles.
Main State Machine also provides to monitor the
system status in order to stop and recover system in safe
way if some malfunctions occur. In particular, the
following error conditions are kept into consideration:
moton are not correctly actuated or sensorized, (i.e.
some cables are not connected): by sending a
calibrated low torque on each motor and by reading the
correspondent rotor position, it is possible to detect any
error condition; this feature test is executed before
activating any HI position or force control;
. during all control phases in which two arms are moved
along predefined path, collisions whit obstacle (i.e.
user of some object placed on the desktop) can occur:
low saturation levels are applied on motor currents in
order to avoid any possible damages and the movement
is stopped if the time required for reaching a single
step along the path is greater than predefined one;
when the force feedback is activated, force data
transmission is constantly monitored in order to detect
any data error and too long transmission timeout.
4.System at work
When integrated with the application software
running on the remote computer the system can work at a
control frequency up to 2KHz (currently IKHz) and
according to the software a wide number of functionalities
can be perceived. The running software is the HGM it is a
VE editor which allow user to design and haptically test
the workspaces. In order to be coherent with the user’s
requirements and the interface features, the current version
of the HGM supports multiple contact points. As show in
figure 8, the HGM aggregates these haptic features to a
coordinate audio recognition and synthesis engine. The
modeller is completely modular and supports a scalable
number of objects and features. The modeller bas been
developed so that a large number of task could performed
without requiring any extemal coding, among them:
Find simple objects and identify the shape;
Explore different features: weight, haptic textures and
buttons, stiffness, stickiness (guided adherence to
surfaces);
Constraining movement to: follow trajectories, tie
movement to the boundary of an object and explore
surface’s attraction and repulsion forces;
Unexplored objects: audio and force feedback;
Grasp and move objects detecting collisions;
Zoom in and zoom out the environment and panning
the virtual workspace.
At present, on the top of the HGM software three
different types of computer interaction may be run:
169
Exploration of chart data (with automatic import
utilities from common data sources);
City Map explorer, that enables user to haptically
explore the city maps from a GIS representation;
A simple adventure game, used for testing and
assessing the to which extent VI people can haptically
interact with computer application.
Figure 8. HAVE dum@ 3 valid3iion test.
5.Performances of the system
Before testing the system with the HGM environment
and with the users’ group, at the end of the assembling
phase a wide set of test has been performed. Here they are
some of the achieved results:
Workspace: the shared workspace of the ann twins is a
box 400 mm depth, 400 mm high, 600 mm wide. Such
a box is achieved when the arms are placed as shown
in figure 1. By using the special calibration tool offered
in the controller the arms can be placed everywhere on
the users desktop with a resulting change in the shared
workspace:
Continuous force: the motorisation group make use of
PM-DC motors that provide a maximum continuous
torque that gives a force at the end-effector of 4 N in
the worst condition (tyyical is 6-7 N);
Peak forces: for short time periods peak forces as
higher as 20 N (in the worst workspace .~ position) can
&-exerted;
Stiffness: as for the force, worst stiffness is 2 Nlmm,
but typical ranges from 6-8 Nlmm;
e Friction: the mechanical solutions adopted for
motorisation and transmission systems allow to obtain
perceived friction less than 200 mN without
compensation and less than 20 mN with compensation:
Position resolution: worst condition offers a position
resolution of 0.1 mm;
Joint accuracy and coherence: once adequately
calibrated, inter thimble position errors are below 0.5
“/IO0 mm (less that 1 %);

Force resolution: dynamic compensation of
gravitational effect, static and viscous friction and of
the elastic transmission model allow to achieve
residual error which are about 2-4 % of the exerted
force;
Mechanical Inertia at the fingertips: the inertial
properties at the fingertips are highly improved with
the isotropy of the mechanical design. Design values
for this performance are less than 0.4 Kg along the
directions orthogonal to the third DOF in the worst
case (typical values are about 0.2 Kg), and less than
0.5 Kg constant along the direction of the thud DOF.
financial assistance of the EU which co-funded the project
within the IST-2000-26151 GRAB (www.grab-eu.com),
GRaphical Access for Blind people, project. The authors
are gateful to the EU for the support given in carrying on
these activities.
References
[ I ] Man&, M. el Al. CRAB - A new haptic and audio virrual
6.Conclusion
The evaluation of the first prototype of system
undertaken by end users has shown preliminary
information on the interface and on the degree of
interaction the user can have with such a system.
The interface has some key features and
functionalities appear being unique or superior w.r.t
current commercial haptic systems. The main issues that
users identified as being the most valuable are: the
simultaneous use of two fingers in a common workspace,
the larger shared workspace, the smoothly refined
movement, the robustness of the device, the position
accuracy, the high peak forces, the fidelity in terms of
judging sizes and comparative distances, audio input and
output, interaction with a changing environment (buttons,
moving objects, organising objects, detecting
collisions...), the interaction and exploration of objects
including features like: weight, stiffness, stickiness,
curvatures, attraction forces, following a trajectory,
utilities to find, recognise and explore small objects.
[Z]
171
enviromenr cmbling vision impaired people ro access rhe three
dinknsionol compurer graphic world, WP) Advancement of
Asistive Technology in Europe ( M A T E ) 2003, D u b h , lreland
Gutiwrez T.. Carill0 A., Femandaz J.L. Munoz J.A., Compurer
GraDhicr ADD/icafionr for rhe Blind Peode, . DO. .. 1-5, SenrAble
c0rporation;im -
GRAB Proctiml Fensibiliry Study. Rojm intemal repon. GRAB
EU IST-ZwO-26tSI
OModhraio M. S., Gillespie B.. n e Mwre: A Hopric User
Interface for Blind Person$. CCRMA. Depmmmt of Music
Smford Universily. Stanford. CA, USA, 1997.
di-Srefano L.. Melchioni C., Vassura G.. Haptic inrerfmes os aids
for visually impnired persou, 81h-IEEE-Inlemational-Workshopon-Robol-and-Human-Inferaction.-RO-MAN-P9-C~t.-
Na99TH8483. 1999 2914. IEEE, Piscamway. NJ, USA
Virense H.S.. Jack0 J.A., Emery V.K., Foundorion for impoved
inrerncrion by individuals with visual impairmenrs rhrough
mulrimodol feedback. Univenal-Access-in-the-lnfomrauon-
Sociey Nov. 200% XI): 76-87, Springer-Verlag
Tzovaras D.. Niliolakir G., Fergadir 0.. Malasiotis S., Slaw&%
M., Design and implsmenrarion of virruol eiwimnmenr~ for
rmining of rhc visually impaired, lacko,-J.-A. (ED). ASSFTS-
2 0 0 2 . - P m e e e ~ g ~ g s - o f - ~ ~ - ~ ~ h - l n t e m a l i o n a
Conferrnce-on-Assistive-Teehnalogies. 2W2 41-8. ACM, New
York NY, USA
E D Kopaeek P., Red-rime L i n u conrrol of0 hopric inrerfo~efor
visuolly impaired persons, Robot-Con~ol-ZWO-SYR0C0~.-
~ccdings-uolume-fmm-the-61h-IFAC-Symp~i"~. 2001: 669-
The technologies developed for the GRAB system
have demonstrated to be robust and safe. The authors
considered of taking them as a reference p i n t for future
systems developments. At present PERCRO is cooperating
with Haptica Ltd in order to offer the benefit and the
advantages of this device to the market.
74 "01.2
Ramstein C.. M d a l 0.. Dufceesne A., Cariyan M., Chasrd P..
Mabilleau P., Touching & hewing. GUIs: Design issues for rhe
PC-Access rprcm ASSEI' 96, ACWSIGCAPH. L 2nd Annual
ACM Conference an Assistive Teehnolo@es (pp. 2-9). Vancouver.
BC, Canada
DiFilippo D., Pai D.K.. me MI: on audio and haptic inrrrfocc for
c o t " inrerortions, ULFT.-Proceedingsgs-of-1h~-13~-~"al-
7.Acknowledgments
.'
The activities described in this paper have been
ckried out in collaboration and with the help of the GRAB
consortium. The GRAB consortium is composed of three
European user group: the National Council of Blind of
Ireland (NCBD, The Royal National Institute for Blind
A C M - S y m p o r i u m - o o - U s ~ - ~ r - I n f e r f a c e - S o f t w a r e ~ ~ ,
2000: 149-58, ACM. New York, NY. USA
Jansson G., Billbergs K.. et Al.. Haptic Virrul Ofviro-nrsfor
Blind People: ~xplororory Erperimnrr wirh Two Devices,
lntematioaal Joumal of Vinual Reality, 1999 4.10-20
Gosselin E, Riwan A., Design of Virruose 3D; a new hoptic
interface for nleoparorion and v i n u l rea1i.y. Bejczy,-A.-K.;
Korlowski-K.; Rudas.-L-l., 10th-International-Conferenceim-
Advanced-Robotics.-ICAR-2001, 2001: pp205-12, Budapest
(RNIB) and ONCE (Organization National de Ciecos de
Espana), the Spanish organization of Blind people: two
technical developers: PERCRO and the LABEIN
foundation of Spain; and one commercial partner: Haptica
LtD (Dublin).
The authors are grateful the whole consortium for
their contribution in the development of the activities. The
activities of this project have been canied on with the
[I31
I141
Polytechnic. Budapest, Hungary
Robem J.C.. Franhlio K.M., Cullinane I., Virtual haptic
exploraron visuolimion of line graphs ond chnns, Roceedingso
f - t h e - S P ~ ~ ~ - I n t u n a t i o n a l - S o c i e t y f o r - ~ ~ .
2002:4660:401-10
Bianchi E., Dozio L. Ghuinghelli G.L. Mantegazza P., Complex
Conrrol Sysrem, Applications of DIAPM-RT4J or DIAPM,
Realtime Linux Workhop Vienna 1999
170