Image/Video Processing Technologies

Image Processing Technologies
for the 21 st Century
Lawrence Rabiner
Professor, Rutgers University and the
University of California at Santa Barbara

Image/Video Processing Technologies
• FAX (Bilevel)/Image Compression Standards—JBIG,
JBIG2, JPEG, JPEG2000
• Document Compression—DjVu, pdf
• Image Understanding—image libraries, image retrieval,
image search
• Image (character) Recognition—OCR, handwriting
• Video Compression Standards—MPEG1, MPEG2,
MPEG4, MPEG7
• Video Segmentation and Classification—finding faces,
shot/scene/… based on content
• Video Conferencing—between groups of individuals, and
between individuals and groups
• Video Telephony—narrowband and broadband
• Visual TTS—avatars and sampled faces
• Image-based Info Services—networking of multimedia
information—searching and browsing

AT&T Image, Video and Multimedia
Technology Services/Products
Prototype
Trial
Developed
Deployed
New in 1999
WorldWorx
Vistium (NCR)
H.261/AVC
Chipset
POTS Video
Phone
Kansas City
Bill Reader
“FIRST” Fax
Form Reader
H.263/MPEG-1
Chipset
Digital TV
(MPEG -2)
NCR Check
Reader/Verifier
Pictorial
Transcripts
FusionNet
NCR Check
Reader-Sorter
Digital Video
Libraries
VideoTalks
Language ID
for FAX
Catalog.com
DjVu Clients
DjVu Web Site
SALSA
DjVU SDK
SBTV
1992 1993 1994 1995 1996 1997 1998 1999+
Year

Image Processing Systems
• DjVu: high quality document compression for web display of
document images
• Shoebox: image libraries with searchable access using audio and
textual content description
• DVL: digital video libraries for searchable, browsable video
content
• Video Talks: high quality, real-time, audio-visual presentations to
remote locations and the desktop
• SALSA: utilization of cable upstream capability for TV-based
video telephony over 384 kb/s channels
• Visual TTS: video agent-based technology for user interactions on
the desktop and for portable devices
• netFridge: home information center using cyber-fridge as central
access and control point for information delivery and access
• Broadband Phone: IP-telephone with thin client access to
network-based services including telephony, image transport and
display, games, video, handwriting, messaging, fax

DjVu-- DjVu--Document
Document Compression

What is DjVu?
• A New Method of Image Compression from AT&T Labs
– separates the high-contrast detail from the general
graphics
– compresses each with separate algorithms
– decompresses and reassembles the layers on-the-fly
with a browser plug-in
– progressive rendering
– multi-page documents
– hyperlinks

Size Comparisons
TIFF/BMP 24MB
GIF 4.8MB
JPEG 1.7MB
* Size Based on 300DPI scanned DjVu Brochure page
PDF
17MB
DjVu
55KB

DjVu Segmentation and Coding

Background
graphics uses
wavelet
compression to
compress at 100
DPI using IW44.
How it Works
Mask created from high
contrast portions of image.
Compressed at 300 DPI
using JB2.
Foreground
colors
compressed at
25DPI using
IW44.
Image recombined - now
greatly reduced up to 5 to
10 times smaller than
JPEG and decompressed
on-the-fly in browser.

JPEG
DJVU
(Same file sizes)

300 dpi
DJVU Document Image Sizes

Declaration of
Independence.djvu
yellowstone.djvu
> DEMO <
technology
article.djvu
French Comic
Strip.djvu
stanley Catalog.djvu

• Compressors
DjVu Software
– Online Compression Server
– Linux, Solaris, IRIX: “djvuencode”
– Windows (third-party)
– Windows/UNIX/Mac SDK
• Plug-in Viewers for Netscape, IE, or Opera
– Linux, Solaris, IRIX, OSF, FreeBSD
– Windows 95/98/NT
– MacOS8
• Editor
– Linux, Solaris, IRIX, HP-UX, (Windows soon)
• Reference Library: Source Code Release

DjVu in Action
http://www.att.net/
http://www.feith.com/
BOOTH #1604
http://www.monarchis.com/

DjVu in Action (cont.)
http://www.roottech.com/
http://www.hp.com/
http://www.cgk.de/
BOOTH #3438

DjVu in Action (cont.)
AI http://www.teletrade.com/
http://www.umi.com/
http://www.art.com/

Digital Photo
Management Technology

What is Shoebox? Shoe ox?
• Digital photo management system
• Allows convenient and intuitive organization,
annotation, and navigation of digital photos
– Features:
• Browse by thumbnail
• Spoken or text annotation of photos
• Text query on all annotations
• Timeline index
• Concept index
• Visual similarity and content search
• Import/Export via XML, publish via HTML

Underlying Technology
• ODMG-compliant OODB
– Text indexing
– B-trees, M-trees
– Extensible and scalable
• Windows front end
• Speech recognition
• Image processing

Why here?
Wireless cameras are coming: one click anywhere and it’s safe in your Shoebox
Web
GSM
UMTS
PCS
GPRS
...
Central
Office
Shoebox
Home Shoebox
Email
Broadband Service Opportunities

Image Processing
• Designed to give the computer access
to the ‘content’ of an image
– Segmentation into regions
• Region description
• Whole image similarity metrics
• Composed queries
– Learning: Neural network based semantic
classification of region descriptors

Indexing paradigms
• Generate one “word” per region
– Use text retrieval models (B-trees)
– “smudged queries” to address boundary problems
• Store region properties in a metric space
– M-trees
• Compact summary of an entire image
– Intended to support nearest neighbour browsing

Visualisation of
inter-image inter image
distance:
• Summary vector
representing:
– Dominant region
in each 9th of
the image
– Colour and
texture summary
• MDS algorithm
used to order
images

DVL-Digital DVL Digital Video Libraries

Digital Video Library (DVL)
• DVL is a digital video management system that
enables the content-based retrieval and intelligent
browsing of video information
– applies text, image, video, audio, and speech
processing techniques to organize, index, and
condense video information
– generates multiple representations of the video
contents to enable the delivery of information over
a range of information appliances and a wide
range of available bandwidth.

Applications of DVL
• Digital media asset management
– effective and efficient utilization of large archives
of video material (e.g., TV content)
• Content “repurposing”
– automatic conversion of TV content to Web
content
• On-demand and content-based retrieval of video
– information
– entertainment
– educational material

Capabilities of DVL
• Image and video processing
– Content-based sampling of video, image
matching, image spotting, people detection
• Text processing
– Case restoration (capitalization), automatic
hyperlink generation, parallel text alignment
• Speech and audio processing
– audio classification, speech recognition, speaker
identification, text-speech alignment
• Multimodal processing
– story segmentation (using speech, text, and video)

Current Activities and Future Directions
• Extraction of high-level semantic information to
describe the visual content of video
• Video summarization
• Multimodal story segmentation and categorization
• Audio-based searching and browsing
• Speaker-based audio segmentation
• Natural language based interfaces
• Speech-based interfaces

Automated Extraction of the Semantic
Hierarchy of News
Linear
Retrieval
Audio
Table of Contents
Broadcast News Categories
News Summary
Story 1 Story 2
Anchor Detailed News Reporting
News Commercials
Content-based Searching
and Browsing
Multi-Modal
Categorization Segmentationusing
audio and video
and text
Video
Text This is the broadcast content transcribed by human. It is used to illustrate the construction of semantics using automated techniques based on multimedia
Broadcast news programs: across multiple media; linear in time; flat structure.
...
Topic detection
and categorization
Story extraction
by text processing

WorldNet DVL Home Video Trial
DVL Demo

Video Talks

A Comprehensive Multimedia Conferencing System
A VideoTalks Conference Room
☯ VT Interactive - high-quality
multimedia interaction
☯ VT Desktop - multimedia
presentations on desktop
☯ VTJukebox - multimedia
presentations recorder/player
• VTonDemand - VoD with
indexing
• VT Lite - low-bandwidth
(modem) access to VideoTalks
☯ completed • prototype

Current Networking
Desktop
Desktop
multicast
virtual channel
unicast
Unicast to Conference Rooms:
--presenter and audience audio/video
Multicast to Offices:
--mixed audio/video from presenter/audience
--vugraphs and pointer
Florham Park LAN
LAN
Mcast Tunnel
ATM
SW
Mcast Tunnel
Conference Room
IP /ATM
Guaranteed QoS connection
ATM
SW
Desktop
IP /ATM
LAN
Newman
Springs LAN
Audience: M-JPEG,
VBR, 500-1000 Kb/s
Presenter: M-JPEG,
VBR, 500-1000 Kb/s
NetVG: M-JPEG
(640 x 480 pixels)
with 4 second refresh
at 100-200 Kb/s
Conference Room
Desktop

Video Codecs
• Developed in AT&T Labs - Research
• Low delay (

NetVG *
World’s only high resolution viewgraph presentation system featuring a live pointer
Document Camera
Ethernet
Network Interface
Video Capture
VGA Interface
Mouse Interface
PC
Mouse (used as a pointer)
Projector
• High resolution, color viewgraphs
• Full-motion pointer synchronized with A/V
• Paper, transparency, or computer presentations
• Real-time annotation
• No security risk
* Patent pending
Display
Pointer

VT Desktop - Desktop Viewing
• Software only (a soundboard is assumed)
• IP Multicast based
• 320x240, 10 -15 fps H.320 video with mu-law PCM audio using:
– MBONE tools for audio/video, or
– IP/TV viewer from Cisco
• NetVG viewer
• High packet loss tolerance
• Not interactive but, text based or telephone based questions are
possible
• Total bandwidth < 400 Kbps
VTJukebox
• On-demand re-multicasting of recorded multimedia presentations
• Can be received using the same software as VT Desktop
• The playback is available to everyone on the intranet
• Requests are played immediately if a channel is available, otherwise queued
• Number of channels depend on available bandwidth
• Improved performance through intelligent recorder/player deployment

VTonDemand - Indexed presentations on the desk top
• Viewgraphs based indices for on demand playback of a presentation
– derived automatically
– displayed on a web browser
– presentation starts from the selected viewgraph

SALSA-- SALSA--Video
Video Telephony over
Cable Systems

SALSA* - A Prototype to Enable Upstream Video over Cable TV
• Targets enabling video services using the enhanced upstream
bandwidth to be provided by the multiplexed fiber passive coax
network (LightWire) technology and guaranteed QoS over IP.
• Potential service examples include:
• Videophone with family and friends
• Video dating
• Personal TV studio at home
• Virtual gatherings, video-telecommuting, virtual town hall
• Video messaging
• Interactive shopping: personal tailoring/fashion, customized home shopping assistant,
home design/interior decorator
• Monitoring/surveillance: babysitter camera, elderly citizens’ watch, children at parks and
swimming pools, remote medical monitoring and diagnostics
• Auctions/garage sales
• Distance learning, remote tutoring
• Interactive customer service
• Remote interviewing
• Small business monitoring
• Remote video archiving, video clip-art library
• Play along game shows
• ...
* System for Audio/Visual Live Services & Applications
Video Telephony;
Distance Learning;
Home Monitoring

SALSA
• Architectural Elements
– Based on a reduced, off-the shelf PC platform as an add-on
component for a set-top box (STB)
– Software-only codecs
• packet loss resilient coding (H.263+/MPEG-4 and RTP over UDP/IP)
• layered coding for economical use of QoS (optimal split between
guaranteed QoS and best effort bandwidth)
– UI is through the STB
– All signaling and communications is IP based
– Multipoint ready
• Advantages
– Low cost
– Stable
– Easy to use
– Best possible quality

SALSA Video Codec & Logical
Transport
H.263+ / MPEG4
SW Only
Layered Codec
High Priority Bitstream,
Guaranteed QoS Delivery
< 100 Kb/s, fixed rate
IP Network
with
Guaranteed QoS
Low Priority Bitstream,
Best Effort Delivery
~ 300 Kb/s, variable rate
H.263+ / MPEG4
SW Only
Layered Codec

First working prototype:

Microphone
Camera
DOCSIS Modem
SALSA Block Diagram
Analog
USB
10baseT
audio,
video,
data
Echo Canceller
plug-in card
VGA/NTSC
500 MHz
Pentium
Motherboard
(compress/
decompress)
SALSA Box
Audio
Baseband
Video
RS232
Set Top Box
(CISCO
DCT 5000)
TV
NTSC
Salsa Video
comparison of
salsa and polycom

STB Main Screen Featuring VideoPhone

Simple VideoPhone UI on the STB

Auditory and Visual TTS

TTS Demos
U.S. English Female:
U.S. English Male:
Spanish Female:
TTS: AT&T Lucent Acuvoice

Visual Text-to Text to-Speech Speech Synthesis
• Personalized friendly agents provide an entertaining and
effective user experience.
• Subjective Tests confirm:
– Agents are preferred over text and audio interfaces
– Agents are more trusted
than text and audio interfaces
• Applications:
– Virtual Customer Service
– Virtual Newscaster
– E-commerce
– Games

Non-AT&T Non AT&T Players
• Computer/Video Game Industry
– Motion capture; emphasis on fast rendering
– Results are still far from looking natural
• Animation Movie Studios
– Synthetic worlds
– Photo realistic manual animations
• Several Talking Head Companies
– Synthetic models
– Marginal lip synchronization

AT&T Synthetic and Sample-Based
Sample Based
• Synthetic models
Models
• Sample-based models

Two Rendering Techniques
Synthetic
models
parametrized
shapes
Sample-based
parametrized
textures
+ -
Keeps correct
appearance under full
range of views
Reproduces photo
realistic appearances;
fast
hard to reproduce
minute skin details
like wrinkles, that
look absolutely natural
Range of views
limited by planar
approximation of parts

AT&T VTTS
• Synthetic (3D) face animation part of MPEG-4 and H-anim
standards
• Most advanced data-driven talking heads
– Sample-based coarticulation and head movements.
– Face recognition without sensors.
• Only sample-based models look natural
– Synthetic models improve fast in quality, but are still orders of
magnitude away from looking natural;
• Option to combine sample-based heads with synthetic
environments to get best of both worlds
• Strong AT&T patent portfolio

AT&T VTTS Challenges
• Business Challenge:
– Need to establish business needs: where and under what scenarios
is VTTS likely to have an impact?
• Technical Challenges:
– Sample-based VTTS needs to be speeded up by at least 10 times
– Not clear how to combine 3D (e.g., full-body animation) and
sample-based (e.g., talking head) technologies
– “Easy” plug-in as a component of MMUI output
– Scripting too low level: need to develop a high-level scripting/text
tagging language that allows anticipatory planning of gestures
(“reach full pointing amplitude at this syllable in the input text;
‘normal’ speed, but don’t care when you have to start”; “reach
maximum gaze at object 13 on page at this syllable in the input
text; start 1 second earlier”)

Text
Text to
Speech
Synthesizer
Conversation
module
phonemes
stress
VTTS Process
emotions
Coarticulation
Model/Library
Movements
Emotions
Model/Library
Lip shapes
Emotions
Movements
( = FAPs)
Rendering
3D model
Samplebased
model

Model-Independent Model Independent Animation in
• Facial Animation Parameter Units (FAPU)
– Low-level Faps (3-68)
– High-level Faps
• Visemes
• Expressions
MPEG-4
MPEG
ES0
MW0
ENS0
MNS0
IRISD0

Model-Independent Model Independent Animation
• Facial expressions
Anger
Joy
Disgust
Sadness
Fear
Surprise

Sample-Based Sample Based Model
• Concatenate snippets of video to
synthesize talking heads
• Reduce the number of samples to
store by decomposing recorded
head into sub-parts.
• Use a background image of the
whole head onto which parts are
warped.
• Feathering (transparency gradient
at border) helps smooth blending
• Smooth transitions of each object
(e.g., mouth shape) across unit
boundaries by using advanced
morphing techniques

Parameterizing Samples
• Record a “unit selection”
VIDEO database
• Label database with a set of
parameters:
– phonetic information (Audio TTS)
– measured features
• Define a low dimensional space
that is quantized, creating a
“codebook”.
• Populate each “bin” with many
samples.
• Use unit selection approach from
Audio TTS to select appropriate
units on the fly

Head Movements
• Background sequence (4)
– typical short movements
(nods, …)
• Match mouth sequence to
the background sequence
– Extract mouth sequence
(1) -> (2)
– compute pose on background
sequence (4)
– warp sample to match pose (3)
– “Feather” (3) + (4) => (5)

Giving Machines High Quality Voices and Faces
‘Natural Speech’

VTTS Perspective
• In the future…
– Multimedia Customer Care

Recent VTTS
Au Clair de la Lune Virtual Secretary

Communicator

netFridge-- netFridge--a
a New
Communications Paradigm?

Why a netFridge?
• The kitchen is the focus of the home
• The refrigerator is the focus of the
kitchen
• Lots of the family’s stuff get posted
on the fridge door:
– notes, artwork, photos, postcards,
coupons
• Why not have one in cyberspace?

Kitchen2005
• Always-on high-speed internet
connection to every room of the house
• Large flat-panel touch screen with pen,
camera and microphone mounted on
the fridge door
• But people aren’t necessarily much
more technologically savvy

Benefits to Consumer
• “easy to use” security/access control
• simple post-it like messaging in
visual media (like an offline chat
room)
• sharable across households
• easy access via telephone
• download to PDA
• simple interface hides complex
technology

Smith Family
Page
Schedule Family Events
Smith Family
Page
Check the
family calendar
netFridge
( o
Check the
grocery list
Web TV
Smith Family
Page
Keep up with
extended family

• Notes
What’s in the Fridge?
– Typed
– Handwritten
• Pictures
– Doodles
–Photos
–Faxes
• Sounds
• Calendar

WorldNet Beta Site trial of netFridge sm

Browser
access
IE 4, Netscape 4
(JavaScript, Java
required)
Need special camera
control app (in VB)
to use PC camera.
POTS
access
PhoneWeb
Server
(PML)
Internet
Array Systems (PC) box
with Dialogics board(s)
running SCO Unix
netFridge Architecture Overview
netFridge Server
Web Server
fridge.cgi
donote.cgi
calendar.cgi
family.cgi
personal.cgi
admin.cgi
Apache 1.3; should easily port
Static Site Content
HTML pages
Images (gifs, jpegs)
Java Applets
fridgelib.pl
Fridge Databases
Jones Family fridge
Smith Family fridge
Armstrong Family fridge
Unix or Win32 platform; tested on Linux, IRIX, Solaris, Windows 95/98
fridgedb.pl
Fridge Metadatabase
Global map of fridge databases;
Text box
fridge names ⇔ telephone exts.
data (a single binary file)
•All note/calendar event
text and parameters
•All global fridge data
and parameters
•All per-user data
Uploaded objects in separate
files (images and sound)
Each family fridge database is in its own directory

How does this benefit AT&T?
• We surely won’t be in the business of
supplying the hardware
• We may not even be in the business of
supplying the software
•But we ought to be in the business of
supplying the service
– keep it multi-modal, network-centric

Fridge Door
Add a new note
View deleted notes
Turn brief mode on
Grocery List
Xmas Wish List
Needed for tomorrow
Kim needs notebook paper
Window on the World
Stock Exchange
T45¼ IBM 120¼ GRC 37¼
News 3 Articles
Sports
Traffic
Calendar Family Personal Options Leave Fridge
The Brown netFridge
Calendar Events for Tuesday, April 27, 1999
PTA Meeting 6pm at Kim’s School
Go to calendar
Kid Cam
Family
Photo
Album
File Drawer
We Won
Dentist appointment
Pick up Jimmy
Urgent fax
Notes
A fax came in for
you it looked important
from Kesler and Kesler
it is on your desk in the
rec-room
Family Chat
Brown’s Family Room
Grandmas’ recipe for pie
Alert Monitor
Page
Home Alarm
Family Connect
Joe stop by for cards tonight - Dave
I saw Sarah yesterday give me a
call - Barbara

The Broadband Phone

Philosophy
It’s a phone not
a computer
Architecture
100% network centric
Cute feature
Both parties can see
the same thing
Sound bite
Simple phone
Smart network
Broadband phone
Enhancing everyday communications
No operating system
No web browser
No downloadable code
Nothing to go wrong
Broadband
IP Network
IP telephony
The screen comes from the
network
The services are on the screen
The network is the phone
Remote graphics protocol
Enhanced applications
Home shopping
Photo Albums
Fax & Mail
Web browsing
Reservations &
Information
Chat ‘n’ draw
Live video

Summary
• Image and Video Processing technologies are
beginning to mature and reach the point where
they can be utilized in mass market, high
volume services
• Image and Video Processing technologies will
drive broadband technologies into the home,
the office, and for mobility applications
• Real-time Image and Video Signal Processing
will enhance the rate at which improvements
are made to the technology, at the same time
lowering the cost of service and increasing the
penetration of these services