Gesture Recognition - Pritpal Sarlech

Introduction

Gesture Recognition interprets human gestures by using mathematical algorithms. The face or hand is the most common bodily motions used in Gesture Recognition. One of the main focuses nowadays is emotion recognition by recognising the different gestures from the hand and face. There have also been attempts in understanding sign language by the use of cameras and computer vision.

Using Gesture Recognition computers are able to understand the human body language, therefore making the relationship computers and humans already have stronger. Gesture Recognition allows humans to interface with machines (HMI) and interact without any mechanical devices. Gesture Recognition allows the user to point their finger at a monitor on a PC, and by doing this the mouse pointer will move to where it’s supposed to go, because of this it could make PC input devices no longer required.

Uses of Gesture Recognition

Gesture Recognition is good for processing data from humans which is not input as speech or on a keyboard. Computers can identify many different types of gestures such as:

- Sign language recognition – Just as speech recognition can convert speech to text, different types of Gesture Recognition software can convert the symbols represented through sign language into text.

- For Socially Assistive Robotics – By using sensors such as Accelerometers and Gyros worn on the user and then reading the results given by the sensors, the robots are then able to assist in patient treatment, for example stroke rehabilitation.

- Directional indication through pointing – Using Gesture Recognition to show where a person is pointing is useful for identifying the instructions given.

- Control through facial gestures – Controlling a computer by using facial gestures is very useful for people who may not be able to physically be able to move a keyboard or mouse, for example Eye tracking is an excellent example of how the user would be able to control the cursor by focusing on different parts of the monitor.

- Alternative computer interfaces – Prior to the keyboard and mouse setup to interact with the computer, strong Gesture Recognition would allow the user to complete tasks by using hand or face gestures to a camera.

- Immersive game technology – The Nintendo Wii is a prime example of immersive game technology, the remote makes the game player’s experience more interactive.

- Virtual controllers – For systems where finding a physical controller could take up too much time, gestures can be used as another way to control, for example, a devices in a car or a television.

- Remote control – Thanks to Gesture Recognition, controlling a remote control with the wave of a hand is now possible. The signal must give the desired response. Playstation’s sixaxis controllers and the Nintendo Wii’s Remote are a good example of this.

Gesture Recognition Challenges

Gesture Recognition has many challenges associated with the accuracy and usefulness of Gesture Recognition software. With image-based Gesture Recognition there are limits as to the equipment used and image noise. The lighting used in images or video may not be very consistent, or in the same location. Objects in the background, or anything that stands out may make it more difficult for recognition.

In order to capture human gestures by the use of visual sensors, computer vision methods are also required, such as with hand tracking and hand posture recognition and also for capturing movements of the head and facial expressions.

Below are a couple of examples of gesture recognition:

http://www.youtube.com/watch?v=F8GVeV0dYLM

This video shows that Gesture Recognition is used to sense the users hand and detect how many fingers are being displayed, when the sensors pick up the users hand it then shows how many it thinks are being shown, and as you can see it is very accurate apart from when the user makes sudden movements or changes to how many fingers are being displayed.

http://www.youtube.com/watch?v=D2BcRblGVVM

This video shows Gesture Recognition software that speeds up/slows down/changes direction depending on which way the user is pointing their hand.

Below is an image of a Nintendo Wii Remote:

http://www.progex.at/progex/index.php/studies/4-flying-jj/33-the-wii-controller

Below is an image of a sixaxis Playstation 3 controller:

http://news.cnet.com/8301-17938_105-9781566-1.html

References

YouTube (2008) Simple Hand Gesture Recognition [online] available from [5^th May 2009]

YouTube (2007) Gesture recognition [online] available from [6^th May 2009]

::work in...projex:: (no date) the wii controller [online] available from [5^th May 2009]

cnet news (2007) PS3 controller is ready to rumble [online] available from [5^th May 2009]

Speech Recognition - By Balroop Bhogal

What it is

Speech recognition (also known as automatic speech recognition or computer speech recognition) converts spoken words to be machine readable. Speech recognition applications include voice dialling e.g. “Call home”, call routing e.g. "I would like to make a collect call", domestic appliance control and content-based spoken audio search, simple data entry e.g., entering a credit card number, preparation of structured documents speech-to-text processing e.g. word processors or emails, and in aircraft cockpits. The performance of speech recognition systems is usually specified in terms of accuracy and speed. Dictation machines can be used to achieve very high performance in controlled conditions. Commercially available speaker-dependent dictation systems usually require only a short period of training and may successfully capture continuous speech with a large vocabulary at normal pace with a very high accuracy. Most commercial companies claim that recognition software can achieve between 98% to 99% accuracy if operated under optimal conditions. Optimal conditions usually assume that users:

• have speech characteristics which match the training data,



• can achieve proper speaker adaptation, and



• Work in a clean noise environment (e.g. quiet office or laboratory space).

This explains why users with accents might have lower recognition rates. Speech recognition in video has become a popular search technology used by several video search companies. Limited vocabulary systems, requiring no training, can recognize a small number of words as spoken by most speakers. Such systems are popular for routing incoming phone calls to their destinations in large organisations. Both acoustic modelling and language modelling are important parts of modern statistically based speech recognition algorithms. Hidden Markov models (HMMs) are widely used in many systems. Language modeling has many other applications such as smart keyboard and document classification

How it works

To understand how speech recognition works it is desirable to have knowledge of speech and what features of it is used in the recognition process. In a human brain, thoughts are constructed into sentences and the nerves control the shape of the vocal tract which includes the jaws, tongue, mouth, vocal cords etc, to produce the desired sound. The sound comes out in phonemes which are the building blocks of speech. Each phoneme resonates at a fundamental frequency and harmonics of it and thus have high energy at those frequencies. The first three harmonics have significantly high energy levels and are known as formant frequencies. Each phoneme have a unique fundamental frequency and hence unique formant frequencies and it is this feature that enables the identification of each phoneme at the recognition stage. In general , speech recognition systems have stored reference templates of phonemes or words with which input speech is compared and the closest word or phoneme is given out. Since it is the frequencies that are to be compared , the spectra of the input and reference template are compared rather than the actual waveform.

Speech is preferred as an input because it does not require training and it is much faster than any other input. Also information can be input while the person in engaged in other activities and information can be fed via telephone or microphone which are relatively cheaper compared to current input systems. But there are several disadvantages in the recognition process. The same difficulty occurs when words are stored in reference template in continuous speech recognition. As already mentioned, in speaker-independent systems only isolated word recognition is commercially available. Most buyers would like the system to be speaker independent, and uttering words in isolation can be quiet irritating especially when the input is bulk and the processing speed may not be very fast. Even in speaker-dependent connected word recognition system (limited vocabulary) the speed of input is only up to 50 words per minute which is not very fast.

Other Disadvantages include;

• TIME; Typing is much faster than voice recognition.

• MONEY; In addition to the cost of the software and the microphone, there has been very little success using voice recognition on a machine with less than 512 MB of RAM.

• ACCURACY. This is related to the time issue--part of what makes voice recognition slower than typing is the need to correct misrecognition errors. In addition, any errors that are not caught by the author will not be caught by a spell checker since they will consist of the wrong word, spelled correctly
  
  
  
  

Design Issues

Abstracted view of reality - separate processing for speech - we hear what we expect to hear background noise, directional or broadcast modality
Human speech recognition tolerates mispronunciations, non-grammatical sentences, dialects
Sound/speech discrimination varies with age, depends on frequency (pitch), amplitude (loudness) and contrast (foreground/background - dB ratio) The future As with any automation systems, Automatic Speech Recognition (ASR) systems will be employed when their speed and efficiency is higher than the current input method so that savings can be made. But as mentioned above ASR systems have not quite reached that competitive position. On the other hand ASR systems are now more affordable than ever before. And when speaker-independent continuous speech recognition systems are developed speech recognition will be one of the popular methods of data input and will lead to the development of vocally interactive computers.

Conclusions

Initially looking at Speech recognition, it seemed simple and straight forward. It has become apparent that it would be a very difficult task to accomplish, and would require much more time, effort, and background on the subject than first thought.

http://video.google.co.uk/videoplay?docid=8823126335312817761&ei=By8ESq_qFcvT-QbIvdT6AQ&q=speech+recognition&hl=en

Being able to determine what is spoken or who a speaker is with near perfect accuracy is an extremely difficult task. Preventing another individual from breaking into the system can be just as difficult, as it requires a system dependent on text and a system that will not accept anything other than what it specifies. The initial idea of being able to determine what word was being spoken is, at best, naïve, and at worst not at all feasible.

Links
http://www.youtube.com/watch?v=kX8oYoYy2Gc&feature=related

http://www3.edc.org/spk2wrt/hypermail/5371.html

http://en.wikipedia.org/wiki/Speech_recognition

3D Interaction (Virtual Reality / Augmented reality) posted by Saagar Parmar

A clear and concise description and explanation of your chosen technologyThere are many different aspects of Virtual Reality, such as ‘Immersive VR’, ‘Desktop VR’, ‘Command and control’ and ‘Augmented Reality’. I will briefly explain the differences between the various aspects of Virtual Reality below. (Dix, Finlay, Abowd and Beale 2004)Immerse VRImmerse VR allows the user to be fully “immersed” into the virtual world. This could mean they are using such equipment like ‘VR Goggles’, ‘VR Helmet’, ‘VR full body kit’ and a ‘VR Dataglove’. (Dix, Finlay, Abowd and Beale 2004) Being fully immersed into the virtual world allows the user to be completely inside this world whereby they can interact with the objects around them.Desktop VR
Desktop VR allows the user to interact with 3D objects using the mouse and the keyboard. Examples of where this has been used is in ‘football’ games and other games like “Flight Simulator”, “DOOM” and “Quake 3” too, however with Quake 3 the default maps can be transformed using ‘VRML’ which stands for Virtual Reality Markup Language. Figure 1 illustrates the transformed Quake 3 map in VRML and Figures 2 and 3 illustrate the use of VRML in a football game. VRML allows virtual worlds to be spread across the Internet which can be integrated with other virtual worlds. The user can have the option of navigating through these worlds and interacting with the objects in front of them using both the keyboard and the mouse. Furthermore the use of these interactions can also take the user from one virtual world to another. (Dix, Finlay, Abowd and Beale 2004)

Figure 1 - Quake 3 VRML (Grahn)

Figure 2 - Football Game VRML (Virtual Reality Laboratory 2004)

Figure 3 - Football Game VRML (Virtual Reality Laboratory 2004)

Command and Control VR
Command and Control VR allows the user to be put in a virtual world but be surrounded by real physical surroundings. For e.g. the use of flight simulators. The user is in a pretend cockpit where the windows are replaced with large screens that have the terrain projected to them in which case the cockpit moves around to simulate being in a real flight simulation. (Dix, Finlay, Abowd and Beale 2004)
Augmented RealityAugmented Reality is where both VR and the real world meet. Virtual images are projected over the user as an overlay whereby the user can interact with the objects in front of them. (Dix, Finlay, Abowd and Beale 2004) The use of similar technology has been used in ‘X-Men: The Last Stand’ in the war simulation at the beginning of the film.The disadvantage of Augmented Reality is that both the overlay of the virtual world and the physical objects must be exactly aligned otherwise problems could occur whereby the interaction of objects could be miscalculated and would most definite confuse the user but also could be fatal depending on the interaction carried out. The advantage of such technology is that with the use of the user’s gaze and position it is detected by the virtual world in which case the environment is safe. (Dix, Finlay, Abowd and Beale 2004)
An insight into how your chosen technology ‘breaks’ the paradigm of desktop computing
In relation to the other aspects of VR I have decided to choose the use of Desktop VR in particular the use of VRML. One example I have found which is supported with evidence is the use of surgery. Operations can be carried out by surgeons in a virtual world. The use of carrying out surgery in such a way is to perfect the technique in carrying out a certain procedure. The patient’s body is scanned and the data is passed and transformed into the virtual world. What’s more is that the use of haptic feedback is incorporated in this simulation whereby the surgeon can feel the texture and the resistance whilst the incision is being made in the “virtual body”. See Figures 4 – 6 for examples of where this has been used. (Dix, Finlay, Abowd and Beale 2004)

Figure 4 - Surgery VRML (State and Ilie 2004)

Figure 5 - Surgery VRML (State and Ilie 2004)

Figure 6 - Surgery VRML (State and Ilie 2004)

An analysis of the usability and HCI problems still to be overcome before your chosen technology becomes widely adopted in the marketImmerse VR can be costly as it requires a lot of processing power and thus it is still not ready for mass market. (Dix, Finlay, Abowd and Beale 2004) Furthermore it could be uncomfortable to wear the gear that comes with immersive VR. (Prashanth)Furthermore, the user in the virtual world could also suffer from ‘motion sickness’ if there is latency in the system relaying the images to the user, whereby the user will become disorientated from the dizziness. (Dix, Finlay, Abowd and Beale 2004)With augmented VR the registration of the overlay and the physical objects need to be exact as disussed above as it could be disastrous if these images are not correctly aligned. (Dix, Finlay, Abowd and Beale 2004)

References

Websites

Grahn, H. N/A [online] available from <http://home.snafu.de/hg/vrml/q3bsp/q3mpteam3_shot.jpg> [25 April 2008]– Uses VRML

Prashanth, B.R. AN INTRODUCTION TO VIRTUAL REALITY IN SURGERY [online] available from<http://www.edu.rcsed.ac.uk/lectures/Lt12.htm#Applications> [25 April 2008]State, A. and Ilie, A. (2004)

3D+Time Reconstructions [online] available from <http://www.cs.unc.edu/Research/stc/Projects/ebooks/reconstructions/indext.html> [25 April 2008]– Uses VRML

Virtual Reality Laboratory (2004) The Virtual Football Trainer [online] available from<http://www-vrl.umich.edu/project/football/> [25 April 2008]– Uses VRML

Books

Dix, A. , Finlay, J. , Abowd, G. and Beale, R. (2004) HUMAN-COMPUTER INTERACTION. 3rd ed. Essex:Pearson Education Limited

Online Papers For VR

Brewster, S. and Pengelly, H. Visual Impairment, Virtual Reality and Visualisation [online] available from <http://www.dcs.gla.ac.uk/~stephen/visualisation/>[25 April 2008]– VR for blind people

Villanueva, R., Moore, A. and Wong, W. (2004) Usability evaluation of non-immersive, desktop, photo-realistic virtual environments [online] available from <http://eprints.otago.ac.nz/152/01/28_Villanueva.pdf> [25 April 2008]

Weaver, A., Kizakevich, P., Stoy, W., Magee, H., Ott, W. and Wilson, K. Usability Analysis of VR Simulation Software [online] available from <http://www.rti.org/pubs/Usability.PDF> [25 April 2008]

Eye Tracking (5) - By Kalpesh Chavda

References


EYE TRACKING

http://thethinkingmother.blogspot.com/2008/09/eye-tracking-problem-links.html

http://en.wikipedia.org/wiki/Eye_tracking

http://psychology.exeter.ac.uk/images/eyetracker.jpg

http://thinkeyetracking.com/eyetracking-20/

http://www.a-s-l.com/Site/

http://www.a-s-l.com/site/Products/EYETRAC6Series/DesktopRemote/D6RemoteTracking/tabid/66/Default.aspx

Eye Tracking (4) - By Kalpesh Chavda

Leading developers in Eye Tracking Technology

ASL has been a pioneer in the examination of human eye movement and pupil dynamics for over 30 years. Founded by M.I.T. scientists in 1962, ASL developed the first video based eye tracker in 1974.

ASL is the first company to develop head mounted optics, eye/head integration, parallax free optics, magnetic head tracking, assisted remote optics and many features that are now industry standard. Our innovative spirit continues to flourish. ASL offers the broadest and most comprehensive line of video based eye trackers.

ASL is the leader in the design, development, manufacturing and distribution of eye tracking equipment worldwide by providing specialty design and development services for the research and consumer marketplace.
The product range of ASL is continually being developed to incorporate new technology with the noteworthy goal of advancing the understanding of eye movement and dynamics. ASL has designed smaller, less expensive and more flexible devices to assist researchers operating under stringent budget guidelines.

The ASL range of systems represents the most complete line of eye measurement and recording available today. ASL eye tracking applications are far reaching and include:

• Usability

• WEB Design

• In-Vehicle Research

• Human Factors

• Sports Training

• Medical Research

http://www.a-s-l.com/site/Company/Overview/tabid/115/Default.aspx




http://www.youtube.com/watch?v=tFdOSODLrDs




http://www.youtube.com/watch?v=fFbT7bAVI8w

Eye Tracking (3) - By Kalpesh Chavda

The problems Eye Tracking is Suffering

Firstly the eye tracking equipment and software is very expensive, some of the most complex and advanced products can easily go into four figures which the average person may find too expensive.

Not everyone can work with eye tracking software, every person is different and no type of eye is the exact same, issues in the past have been different shape eyes, contact lenses, and even to certain extent eye lashes have been seen as a problem. People with poor vision and elderly people have shown very bad results when it comes to using the software.

The whole process of setting up the eye tracker on a person and the calibrating it can be a very lengthy process, people can become disinterested, and when you compare this to the set up of a mouse it seems like a lifetime.

Eye movements as accurate as they may seem not always produce the results we wish to hope for. Random eye movements may occur resulting in unexpected results. Eye movement are natural to us (subconscious) and compared to a mouse it can be very difficult to precisely gaze the eye on a point.

If these issues can be ironed out then it could widely be adopted in the market.

Eye Tracking (2) - By Kalpesh Chavda

How does Eye Tracking Break the Paradigm of Desktop Computing?

Leading developers in the world associated with Eye Tracking technology have been working hard to try and make eye tracking a reliable and competitive technology along with other technologies such as motion and touch. ASL (Applied Science Laboratories) have been working hard at applying this technology to standard everyday desktop use.

They have developed a device called the D6 Remote Tracking. The device is based on a humans gaze and works with people of all ages. The ‘gaze’ is picked up on a computer screen and can show the movement of a persons eye based on the stimulus of the eye.



http://www.youtube.com/watch?v=ticWZ0ad8sc&feature=PlayList&p=B5546130824D9F07&index=35

ASL have envisaged an idea that eventually the eye tracking software can be a competitive option to normal computing technologies such as the mouse. Their idea is to basically substitute the mouse for the eye tracking system. This would help with disabilities and could help the ease of computing.

Current Technologies within Eye Tracking

Eye tracking is currently being used in a wide variety of fields such as

• Cognitive Studies
• Medical Research
• Human Factors
• Computer Usability
• Translation Process Research
• Vehicle Simulators
• In-vehicle Research
• Training Simulators
• Virtual Reality
• Adult Research
• Infant Research
• Adolescent Research
• Geriatric Research
• Primate Research
• Sports Training
• fMRI / MEG / EEG
• Commercial eye tracking (web usability, advertising, marketing, automotive, etc)
• Finding good clues
• Communication systems for disabled
• Improved image and video communications
  
  
  
  
  
  
  
  
  
  
  
  

Eye Tracking Vs Eye Gaze

Eye trackers necessarily measure the rotation of the eye with respect to the measuring system. If the measuring system is head mounted, as with EOG, then eye-in-head angles are measured. If the measuring system is table mounted, as with scleral search coils or table mounted camera (“remote”) systems, then gaze angles are measured.

In many applications, the head position is fixed using a bite bar, a forehead support or something similar, so that eye position and gaze are the same. In other cases, the head is free to move, and head movements are measured with systems such as magnetic or video based head trackers.

For head-mounted trackers, head position and direction are added to eye-in-head direction to determine gaze direction. For table-mounted systems, such as search coils, head direction is subtracted from gaze direction to determine eye-in-head position.

Eye Tracking - By Kalpesh Chavda

http://psychology.exeter.ac.uk/images/eyetracker.jpg

What is Eye Tracking?

Eye tracking is the process of measuring either the point of gaze ("where we are looking") or the motion of an eye relative to the head. An eye tracker is a device for measuring eye positions and eye movements. Eye trackers are used in research on the visual system, in psychology, in cognitive linguistics and in product design.

Why did I choose Eye Tracking?

Eye tracking claims to have been around for over a 100 years but its improvement has significantly been noticed over the last 5 years. Early versions were built in the form of a contact lens which contained a hole for the pupil, the lens was connected to an aluminium point which tracked the eyes movements. Studies found that eye movements consisted of rapid side to side movements rather than smooth movements as previously thought.

Due to its complex and fascinating nature it led me to choose this topic within the pervasive field. In my next blog I will be discussing the different types of trackers, how and were it is used in practise as well as how it breaks the paradigm of desktop computing.

Multi-Sensory Feedback (7) - By Yasmin Aftab

Ernst, Banksô and BÏlthoff in their publication: “Haptic feedback affects visual perception of surfaces.” (http://www.kyb.tuebingen.mpg.de/publication.html?publ=228, present effects of the introducing, Haptic feedback and it is influence on visual perception of surfaces. Their work shows that all designers which try to merge more than one feedback in one solution should remember about the influence of every feedback on each other. Introducing new feedback changes the weights of the rest of elements in the final output presented to the user. In the research mentioned above, this situation occurred during implementing Haptic feedback to existing visual output.

We conclude that giving Haptic feedback consistent with one cue causes its weight to increase in a purely visual task”.

Therefore, new feedbacks affect the perception of the whole product what should be taken into account during the design process. There are also other problems which designers meet during creating multi feedback interfaces. Those additional feedbacks affect the whole product, can in certain situations make it more difficult to use and discourage users from new technologies when, in fact, they should always improve products’ usability. Some difficulties come from the fact that humans perceive things differently while receiving different types of signals or multiple feedbacks instead of a single one, by introducing additional feedback we can loose some information being passed to the user.

It can happen because not every feedback transmits the information with the same accuracy and when several feedbacks are implemented, the user concentrates on ‘the easier to perceive’ feedback – not necessarily the most accurate one. As an example we can state here two feedbacks: visual and auditory. The user generally will not concentrate on the content of the screen as it is easier for him to hear the results instead of to analyse the screen and additionally he does not have to stay in front of the computer all the time.

The imperfection of digitally created speech as a feedback has been observed by Nicole Yankelovich, Gina-Anne Levow and Matt Marx during the preparation of an experimental conversational speech system and described in the document: “Designing SpeechActs Issues in Speech User Interfaces

http://people.cs.uchicago.edu/~levow/papers/CHI_95/chi95.pdf). In the summary of the document they state that finally we have strong evidence to suggest that translating a graphical interface into speech is not likely to produce an effective interface. This conclusion was made by analysing the accuracy of this feedback and dealing with problems like, the difficulty users have in interpreting silence, lack of “immediate and informative feedback” and much more.

Therefore, implementing new feedback does not necessarily improve product’s usability as this ‘new way’ of communication can be less accurate and more error prone. It is a serious problem especially, when the accuracy and correctness of the data obtained from the product have a critical value for the success of the whole process.

Multi-Sensory Feedback (6) - By Yasmin Aftab

The multi-sensory feedback is also used in computer games. Since the beginning computer games used two ways of providing a user with feedback – sound and video. In 1985 a first attempt to add touch sensors (haptics) was made by SEGA. It was integrated with their game called Hang-On and it was a motorbike that you had to sit on and lean right/left in order to turn. It also had complete handlebars with throttle and break lever.

Figure 1: Hang-On screenshotSource:

http://www.ilovethe80s.com/hangon.jpg

http://www.brentradio.com/images/Other/Top50Cabinets/hang-on.jpg

It used visual, sound and haptic feedback in order to provide a user with better experience while playing the game. One of the reviews by Brett Alan Weiss states that “Although available in a standard upright cabinet, Hang-On must be played using the deluxe sit-down model to be fully enjoyed. This device despite being very old had great usability and was highly regarded by players. In present times there are much more multi-sensory interfaces implemented in games. They are using various pads (e.g. Playstation 2 with force feedback), Microsoft Sidewinder Force Feedback joystick or Logitech Racing Force Feedback Wheel and Pedals set.

Figure 2:MicrosoftSidewinder Force Feedback joystickSource: http://www.cs.unc.edu/~stewartm/images/joystick.jpg

Figure 3: Logitech Racing Force Feedback Wheel and Pedals

http://www.horace.org/blog/wp-content/uploads/2007/09/momo.jpg

Together with visual and sound effects they are used to create a more realistic game experience. In racing games the wheel reacts differently for each surface and maneuver that user performs. It is the same with joystick and flight simulators – it can simulate bad weather conditions and turbulences. Such devices generally get great usability ratings. Shane McGlaun from about.com when reviewing one of the force feedback wheels writes.

Playing games that support force feedback with the Logitech MOMO Racing Steering Wheel is cool in the extreme. The addition of force feedback makes the games seem so much more realistic and makes driving games much more fun. The force feedback action helps you to feel what your vehicle is doing on the track, very much like you would feel in a real car if you were driving.

It is exactly the same with joysticks used for flight simulators. One of the product reviews from amazon.com states:“I am a pilot, and this is about as real as you can get without a yoke and pedals. You can even feel the cracks in the pavement as you zoom down the runway. The throttle is a bit small, but other than that, I love it!”Those examples show how far multi-sensory feedbacks (video + sound + haptics) can improve the usability of a game. Most of the users like it.Unfortunately there are also some problems with it. There are few constraints that need to be taken into consideration. Those constraints are cost, time and portability. Giving a user more output, from the virtual environment, can be beneficial but it can also annoy him. Imagine playing a car game with a wheel that has force feedback. In the beginning it seems to be fun and pleasant, but after spending more hours, the vibrations are becoming inconvenient.As Yasuhisa Kato from University of Stanford writes.

The joystick simulated the vibration of the engine. As the speed increases, the vibration was getting bigger. It was very annoying during the concentration on the driving.

This is a point which needs to be considered by designers of such devices. A lot of tests need to be performed in order to classify which feedback is beneficial for the user and which annoys him. That’s why most devices are now configurable.The cost is also important constraint. As an example we can look at the Sony’s replicas of Mitsubishi Lancer to give a user an experience of driving a real car.

Figure 4: Mitsubishi expositionSource:
http://www.gizmag.com/picture/hero/3833_01.jpg

Development of it cost AUD$250,000. It lets users play a playstation game – Grand Turismo 3. The experiences are extremely pleasant but the cost involved was very high. Developing such technologies absorb a lot of time.Multi-sensory feedback is also used to train pilots. Virtual cockpits are prepared for them in order to improve their skills before entering a real machine. Multi sensory feedback is extremely useful in this situation as it provides better feel of realism of the test machine. Its usability is also very high, as it reflects the exact plane controls and presents high quality textures.

Figure 5: A training cockpitSource:
http://www.confederationc.on.ca/ace/images/Frasca-B58-Beech-Baron-cockpit.jpg

Visual, sound and haptics interfaces additionally provide a vital tool for surgeons. In future such technology could allow performing surgeries on great distances. At the moment it can be used to simulate an operation and for training purposes. As we can read on the pages of Ecole Polytechnique Federale de Lausanne “Virtual-reality based surgery simulation opens up new fields in medical training and education.

Figure 6: Presentation of multi-sensory interface simulating an operationSource:

http://www.cmis.csiro.au/mediapix/hapticdemo/hapticstomachdemo.jpg

Multi sensory feedback can be also used to stimulate the “out-of-body” feeling. Michael Persinger from Laurentian University in Canada did an experiment so as to simulate a feeling of virtual reality. He used a 3D camera with goggles, sound and haptics in order to achieve the desired effect. ‘"Wow!" giggled one subject. "I felt as though I was outside my body and looking at myself from the back!"’. It was evaluated by the test participants as very interesting experience.

Figure 7: Michael Persinger during the experiment.
http://www.youtube.com/watch?v=MPG-LYoW27E&feature=player_embeddedSource:
http://img.slate.com/media/17/070823_Sci_ExperimentEX.jpg

Near Coventry (in Staffordshire) in the Shire Hall Gallery a multi-sensory room was opened. It consists of musical hand wall, interactive bubble tube, pinspot projector, mirror ball, infinity tunnel and sensory drawers. It can be used for therapy, relaxation or to stimulate learning and development. A survey was made among the people that visited this room.

The results showed that:

98% of users said that they enjoyed their visit

100% said that they would come again

42% of users rated it as ‘excellent’

47% as ‘great fun’

11% as ‘good’

Some of the user comments:

The Multi-Sensory Room is a fantastic idea and much appreciated. I'm sure it will be beneficial to lots of people and I hope it proves very successful. The kids really enjoyed themselves exploring and I found it relaxing!As the survey shows the usability of this room was highly regarded by the participants.

Figure 8: Music creation hand wallSource: http://www.staffordshire.gov.uk/NR/rdonlyres/7919BC43-2691-4AA2-9ADD-2C9C93C16C2D/45345/Musiccreationhandwall.jpg

Figure 9: Interactive infinity tunnelSource:

http://www.staffordshire.gov.uk/NR/rdonlyres/7919BC43-2691-4AA2-9ADD-2C9C93C16C2D/64631/MSERoom91.jpg

Generally the research into multi-sensory feedback can bring us not only the benefits of making some experiences (e.g. video games) more pleasant and realistic. Can you imagine machines (e.g. aircraft fighters) being controlled by human remotely, without risking his life. A surgeon performing operation on a distance of many miles going into virtual reality to relax, better understand a given technology/process or fight with your fears? All of this is already possible or a very near future, and multi sensor feedback is extremely vital in all of these situations. We are looking forward to its development.
References:

Engber, D. (2007) The Out-of-Body Electric [online] available from http://www.slate.com/id/2172694/pagenum/all/ [23 August 2007]
Kato, Y. Force Feedback devices [online] available from http://ldt.stanford.edu/~yasukato/portfolio/class/cs147/as8/
Gizmag (2005) The World’s most advanced (and expensive) Sony Playstation peripherals [online] available from http://www.gizmag.com/go/3833/ [11 March 2005]
Amazon.com, Inc. (2002) Customer Reviews Microsoft Sidewinder Force Feedback 2 Joystick [online] available from http://www.amazon.com/review/product/B00005NIMB/ref=cm_cr_pr_hist_5?%5Fencoding=UTF8&filterBy=addFiveStar [7 January 2002]
McGlaun, S. (2008) Logitech MOMO Racing Steering Wheel [online] available from http://peripherals.about.com/od/gamingperipherals/fr/momoracingwheel.htm [2008]
Weiss, B. (2008) Hang On: Synopsis [online] available from http://www.allgame.com/cg/agg.dll?p=agg&sql=1:10234 [2008]
Aviation Centre of Excellence Flight Simulator Training [online] available from http://www.confederationc.on.ca/ace/simulator.asp
EPFL (2008) A Flight Simulator for Surgeons [online] available from http://lsro.epfl.ch/page65765-en.html [2008]
Staffordshire County Council (2007) Multi-sensory room [online] available from http://www.staffordshire.gov.uk/leisure/museumandgalleries/shirehallgalleries/mseroom.htm [26 November 2007]
Staffordshire County Council (2006) Feedback on the Multi Sensory Room at the Shire Hall Gallery [online] available from http://www.staffordshire.gov.uk/NR/rdonlyres/7919BC43-2691-4AA2-9ADD-2C9C93C16C2D/45321/FeedbackontheMultiSensoryRoomnovtodec06.pdf [December 2006]