Sumit Basu - MIT Media Laboratory

Sumit Basu
Ph.D. Candidate, MIT Dept. of Electrical Engineering and Computer Science, MIT Media Laboratory
Personal Data
Address: E15-383, 20 Ames Street
Cambridge, MA 02139 USA
Office: (617) 253-0370
Home: (617) 784-4594
Email: sbasu@media.mit.edu
WWW: http://www.media.mit.edu/~sbasu
Date of Birth: February 12, 1974
Citizenship: USA
Educational Background
Massachusetts Institute of Technology
Candidate, Ph.D. in Electrical Engineering and Computer Science, minor in
Brain and Cognitive Sciences
Advisor: Professor Alex (Sandy) Pentland, MIT Media Laboratory
Thesis: Conversational Scene Analysis
GPA: 5.0/5.0
Massachusetts Institute of Technology
Master of Engineering in Electrical Engineering and Computer Science
Advisor: Professor Alex (Sandy) Pentland, MIT Media Laboratory
Thesis: A Three-Dimensional Model of Human Lip Motions
GPA: 5.0/5.0
Massachusetts Institute of Technology
Bachelor of Science in Electrical Science and Engineering, Phi Beta Kappa
Thesis: Hyperacuity Sensing for Image Processing
GPA: 5.0/5.0
Research/Industrial Experience
MIT Media Laboratory, Vision and Modeling Group
Research Assistant
Advisor: Professor Alex (Sandy) Pentland
Topics: modeling conversational interactions from low-level features, prosodic
feature estimation, active interfaces, bayesian networks (exact and approximate
inference), speech detection, wearable phased arrays, source localization,
maximum likelihood tracking for deformable meshes (applied to lip tracking),
finite element priors for 3D meshes, optical-flow regularization with 3D models
(applied to head tracking), vision-steered beamforming.
Perceptive Network Technologies
Perceptual Engineer
Supervisor: Dr. Julian Center, Jr.
Topic: developed a real-time speech detection module.
1997 -
Summer, 2002
(expected)
1995 – 1997
1991-1995
1995 – present
August 2000

Microsoft Research, Vision Group
Research Consultant
Advisor: Kentaro Toyama
Topic: developed a real-time version of my 3D lip tracking work, began the
development of a mesh-based smoothing algorithm.
MIT Media Laboratory, Speech Interfaces Group
Undergraduate Research Assistant
Advisor: Chris Schmandt
Topics: audio interfaces, speaker identification and segmentation.
Xerox PARC, Electronic Materials Laboratory (EML)
Research Intern
Advisors: Warren Jackson, David Biegelsen, David Jared
Topic: image enhancement using position-sensitive detectors (PSD) and their
mechanisms as applied to standard scanning elements.
Xerox PARC, Computer Science Laboratory (CSL)
Research Intern
Advisor: David Goldberg
Topics: handwritten character recognition for a palmtop device, pen-based
interfaces
Iowa Department of Transportation, Information Systems
Developer
Supervisors: Ron Laird, Robert Klopping
Topics: developed a C++ application, BIAS (Bid Item Automation System), for
entering/editing contract items, to be used by all contractors working for the
state of Iowa.
Research Interests
Please see http://www.media.mit.edu/~sbasu for a full statement of my research interests
• Machine Perception (Vision and Audition)
• Speech Processing
• Machine Learning
• Computer-Human Interfaces
Teaching Experience
MIT Deparment of Electrical Engineering and Computer Science
Graduate Teaching Assistant
Class: Probabilistic Systems Analysis (6.041/6.431)
Supervisor: Professor Dimitri Bertsekas
Description: Undergraduate/Graduate introduction to probability, going from
basic axioms through central limit theorems, markov chains, basic stochastic
processes.
Duties: Each week, taught one 40-student recitiation (undergraduates), eight 5student
tutorials (interactive problem solving), held two-four office hours
(undergraduates/graduates), and attended a two hour staff meeting with other
TAs. Additionally graded papers, led several quiz reviews for the entire class
(300+ students) and developed handouts for the students. Received an excellent
evaluation in the students’ “Underground Guide:” “Recitation instructor S.
Basu was extremely well-liked by his students, receiving comments such as
August 1998
1993-1995
June, 1994-
August, 1994
June, 1993-
August, 1993
May, 1992-
August, 1992
Fall, 1998

"Sumit rocks the house!" from his students. He was very available, very friendly,
and very helpful, taking the time to help his students whenever it was needed.
He was considered an excellent teacher, who was always prepared and had
good examples to clarify his points. His tutorials were "great fun" and very
helpful for learning. His explanations were clear, and very helpful.”
MIT Media Laboratory
UROP (Undergraduate Research Opportunities Program) Supervisor
Supervisor: Alex (Sandy) Pentland
Description: Held regular meetings with undergraduate researchers under my
supervision, taught relevant theory, gave guidance in choosing research goals
and coursework/career paths, engaged in many one-on-one help sessions,
evaluated their progress.
MIT F/ASIP (Freshmen/Alumni Summer Internship Program)
Counselor
Supervisor/Organizer: Professor Arthur Steinberg
Duties: Worked with groups of freshmen in the F/ASIP program. Led resume
workshops, simulated work scenarios, helped students develop simulated
consulting reports. Also advised students in course selection and career
strategies.
MIT ISP (Integrated Studies Program)
Guest Lecturer
Supervisor: Professor Arthur Steinberg
Gave a guest lecture on “Artificial Intelligence and the Future of Computing” to
a group of 40 freshmen.
MIT ESP (Educational Studies Program)
Teacher for Songwriting Workshop
Supervisor: MIT ESP
Co-taught (with Regalp Sen) a two-hour workshop on songwriting skills for 25
high school students. Led warm-up exercises, discussed elements of rhyme and
meter, helped students combine words and music in group activities.
Teaching Interests
This is a subset of the possible courses I would be interested in teaching:
• Introduction to Probability and Statistics (Undergraduate/Graduate)
• Introduction to Pattern Recognition/Machine Learning (Undergraduate/Graduate)
• Computational Perception – Vision and Audition (Undergraduate/Graduate)
• Signal Processing for Voice and Music Applications (Undergraduate/Graduate)
• Speech Processing/Speech Recognition (Graduate)
• Machine Perception Seminar (Graduate)
• Adaptive Interfaces (Graduate)
• Bayesian Networks (Graduate)
• Machine Learning Seminar (Graduate)
Awards, Honor Societies, and Fellowships
• NSF Graduate Research Fellowship – Sept.1995-Aug., 1998
• Member, Phi Beta Kappa (Undergraduate Honor Society) – May, 1995-present
• Member, Sigma Xi (Scientific Research Society) – May, 1995-present
Fall 1995 –
present
Spring, 1998;
Spring, 1999
Spring, 2001
Fall, 2001

Patents
• Member, Tau Beta Pi (Engineering Honor Society) – March, 1994 - present
• Member, Eta Kappa Nu (EECS Honor Society) – March, 1994 - present
• Winner, MIT 6.270 Robotics Competition (team: Loren Shih, myself) – January, 1993
• United States Presidential Scholar - 1991
Warren B. Jackson, David A. Jared, Sumit Basu, and David K. Biegelsen. "Macrodetector-Based
Image Conversion System." US Patent No. 5,790,699. Granted August 4, 1998. Available at
http://www.uspto.gov/patft/index.html.
Warren B. Jackson, David A. Jared, Sumit Basu, and David K. Biegelsen. "Position Sensitive
Detector Based Image Conversion System Capable of Preserving Subpixel Information." US
Patent No. 5,754,690. Granted May 19, 1998. Available at http://www.uspto.gov/patft/index.html.
Julian L. Center, Jr., Christopher R. Wren, Alex Pentland, Trevor Darrell, Sumit Basu, and
Evgeniy Gusvatin. "Method of Establishing a Communications Link Using Perceptual Sensing of
a User's Presence." Filed November 10, 2000 (Pending).
Julian L. Center, Jr., Christopher R. Wren, Alex Pentland, Trevor Darrell, and Sumit Basu.
"Method of Extending Image-Band Face Recognition Systems to Utilize Multi-View Image
Sequences and Audio Information." Filed November 10, 2000 (Pending).
Refereed Journal Publications
Sumit Basu, Nuria Oliver, and Alex Pentland. "3D Lip Shapes from Video: A Combined Physical-
Statistical Model." Speech Communication 26, 1998. pp. 131-148.
Refereed Conference/Workshop Publications
Sumit Basu*, Tanzeem Choudhury*, Brian Clarkson*, and Alex Pentland. "Towards Measuring
Human Interactions in Conversational Settings." In Proceedings of the IEEE Int’l Workshop on
Cues in Communication (CUES 2001) at CVPR 2001. Kauai, Hawaii. *The first three authors
contributed equally to this work and are listed alphabetically.
Sumit Basu, Brian Clarkson, and Alex Pentland. "Smart Headphones: Enhancing Auditory
Awareness through Robust Speech Detection and Source Localization." In Proceedings of the
IEEE Conf. on Acoustics, Speech, and Signal Processing (ICASSP ’01). Salt Lake City, Utah.
2001.
Sumit Basu, Brian Clarkson, and Alex Pentland. "Smart Headphones." In Proceedings of the
Conference on Human Factors in Computing Systems (CHI ’01). (Short Paper). Seattle,
Washington. April, 2001.
Sumit Basu, Steve Schwartz, and Alex Pentland. "Wearable Phased Arrays for Sound Localization
and Enhancement." In Proceedings of the IEEE Int’l Symposium on Wearable Computing (ISWC
’00). Atlanta, Georgia. October, 2000. pp. 103-110.
Jacob Strom, Tony Jebara, Sumit Basu, and Alex Pentland. "Real Time Tracking and Modeling of
Faces: An EKF-based Analysis by Synthesis Approach." In Proceedings of the IEEE Modeling
People Workshop at the IEEE Int’l Conf. on Computer Vision 1999 (ICCV '99). Kerkyra, Greece.
September, 1999.

Christopher R. Wren, Sumit Basu, Flavia Sparacino, and Alex Pentland. "Combining Audio and
Video in Perceptive Spaces." In Proceedings of the First Int’l Workshop on Managing Interactions
in Smart Environments. Dublin, Ireland. 1999.
Sumit Basu, Nuria Oliver, and Alex Pentland. "Coding Human Lip Motions with a Learned 3D
Model." In Proceedings of the Int’l Workshop on Very Low Bitrate Video Coding (VLBV ’98).
Urbana, Illinois. October, 1998.
Sumit Basu, Nuria Oliver, and Alex Pentland. "3D Modeling and Tracking of Human Lips." In
Proceedings of the IEEE Int'l Conf. on Computer Vision (ICCV ’98). Mumbai, India. January,
1998. pp. 337-343.
Christopher R. Wren, Sumit Basu, and Alex Pentland."Perceptive Spaces: Learning Dynamic
Models of Human Behavior." In Proceedings of the Workshop on Perceptual User Interfaces (PUI
'97). Banff, Canada. 1997.
Sumit Basu and Alex Pentland. "Recovering 3D Lip Structure from 2D Observations Using a
Model Trained from Video." In Proceedings of the ESCA Workshop on Audio-Visual Speech
Processing (AVSP'97). Rodos, Greece. 1997.
Sumit Basu and Alex Pentland. "A Three-Dimensional Model of Human Lip Motion." In
Proceedings of the IEEE Non-Rigid and Articulated Motion Workshop at the IEEE Conference on
Computer Vision and Pattern Recognition (CVPR '97). San Juan, Puerto Rico. June, 1997.
Irfan Essa, Sumit Basu, Trevor Darrell, and Alex Pentland. "Modeling, Tracking, and Interactive
Animation of Faces and Heads Using Input from Video." In Proceedings of Computer Animation
’96. Geneva, Switzerland. 1996.
Sumit Basu, Irfan Essa, and Alex Pentland. "Motion Regularization for Model-Based Head
Tracking." In Proceedings of the 13 th IEEE Int'l Conf. on Pattern Recognition (ICPR '96). Vienna,
Austria. September, 1996. pp. 611-616.
Sumit Basu, Michael Casey, Bill Gardner, Ali Azarbayejani, and Alex Pentland. "Vision-Steered
Audio for Interactive Environments." In Proceedings of the 1996 Image Communications
Conference (IMAGE'COM '96). Bordeaux, France. 1996.
Michael Casey, William G. Gardner, and Sumit Basu."Vision Steered Beamforming and
Transaural Rendering for the Artificial Life Interactive Video Environment (ALIVE)." In
Proceedings of the 99th Convention of the Audio Engineering Society. 1995.
Selected Technical Reports
Sumit Basu*, Tanzeem Choudhury*, Brian Clarkson*, and Alex Pentland. "Learning Human
Interactions with the Influence Model." Vismod Technical Report #539. June, 2001. *The first
three authors contributed equally to this work and are listed alphabetically.
Sumit Basu. “ICA: A Critical Review of Three Prominent Approaches.” Technical Report. April,
2000.
Sumit Basu. "Empirical Results on the Generalization Capabilities and Convergence Properties of
the Bayes Point Machine" December, 1999.
Sumit Basu, Kentaro Toyama, and Alex Pentland. "A Consistent Method for Function
Approximation in Mesh-based Applications." Vismod Technical Report #486. 1999.

Sumit Basu. "Efficient Multiscale Template Matching with Orthogonal Wavelet Decompositions."
May, 1997.
Trevor Darrell, Sumit Basu, Christopher Wren, and Alex Pentland. "Perceptually-Driven Avatars
and Interfaces: Active Methods for Direct Control." Vismod Technical Report #416. 1997.
Invited Talks
Sumit Basu and Alex Pentland, "Concept Formation in Multi-Modal Learning." In Alex Pentland,
Tony Jebara, Brian Clarkson, and Sumit Basu, Learning Techniques in Audio-Visual Information
Processing, a tutorial at the Int’l Conf. on Pattern Recognition (ICPR ’00) Barcelona, Spain.
September 3, 2000.
Sumit Basu. "Empirical Results on the Generalization Capabilities and Convergence Properties of
the Bayes Point Machine." Invited talk at Tomaso Poggio’s group meeting, MIT AI Lab/CBCL.
May 12, 2000.
Sumit Basu, Deb Roy, Brian Clarkson, and Alex Pentland. "Learning the Structure of Human
Behavior from Sensory Inputs: Language, Daily Patterns, and Conversations." At Grounded
Intersensory Language Learning in Sign and Speech (GILLS ’00). Grenoble, France. March 24,
2000.
Musical Projects
I am an avid songwriter/singer/keyboardist/guitarist, and am involved in a number of musical projects:
References
08:29:06, a solo album I released in June 2000 under the name deepoceanblue, available at
http://www.mp3.com/deepoceanblue
Sonovar/Bodybeat, an electronic music project experimenting with convolution and natural sounds
as musical building blocks (joint work with Brian Clarkson). We received an MIT Council for the
Arts Grant for this project in February 2000. Samples are available on my website.
Additional projects are listed at http://www.media.mit.edu/~sbasu/music.html
Listed in a separate document.