C0S 597d - Advanced Topics in CS Sensing for Graphics

Fall, 2001

Summary

The course will be organized around four major data types that may be measured from the real world:

• The plenoptic function, images, light
• Reflection and scattering
• Shape
• Motion and deformation

For each, we will discuss:

• Data types and models
• Sensors, issues in acquisition, postprocessing
• Algorithms and applications in graphics

The following detailed schedule should be regarded as preliminary and likely to change.

Course Outline

Monday, September 17	Introduction: real-world data in computer graphics Plenoptic function: definition, taxonomy Sources of error in acquired images
Wednesday, September 19	Image formation, lenses, focus, geometric distortion Image sensors (CCD, CMOS), dynamic range Color and chromatic aberration Slides available as PPT or as images.
Monday, September 24	High-dynamic-range photography and applications in graphics Student presentations of papers: Debevec, P. and Malik, J. "Recovering High Dynamic Range Radiance Maps from Photographs," Proc. SIGGRAPH, 1997. Debevec, P. "Rendering Synthetic Objects into Real Scenes: Bridging Traditional and Image-based Graphics with Global Illumination and High Dynamic Range Photography," Proc. SIGGRAPH, 1998.
Wednesday, September 26	Panoramic mosaics and environment mapping Student presentations of papers: Szeliski, R. and Shum, H. "Creating Full View Panoramic Image Mosaics and Environment Maps," Proc. SIGGRAPH, 1997. Schechner, Y. and Nayar, S. "Generalized Mosaicing," Proc. ICCV, 2001.
Monday, October 1	Intro to image-based modeling and rendering Lightfields and lumigraphs Background reading: Levoy, M. and Hanrahan, P. "Light Field Rendering," Proc. SIGGRAPH, 1996. Gortler, S., Grzeszczuk, R., Szeliski, R., and Cohen, M. "The Lumigraph," Proc. SIGGRAPH, 1996. Slides available as PPT or as images.
Wednesday, October 3	Other image-based representations Student presentations of papers: Wood, D., Azuma, D., Aldinger, K., Curless, B., Duchamp, T., Salesin, D., and Stuetzle, W. "Surface Light Fields for 3D Photography," Proc. SIGGRAPH, 2000. Shum, H. and He, L. "Rendering with Concentric Mosaics," Proc. SIGGRAPH, 1999.
Monday, October 8	Taxonomy of reflectance and scattering measurements The BRDF: definition, models, representation Other data types: BTF, BSSRDF
Wednesday, October 10	Guest lecture by Daniel Aliaga: "Reconstructing 3D Interactive Walkthroughs from Omnidirectional Video"
Monday, October 15	Devices and methods for measuring general BRDFs, parameters in specific BRDF models, "gloss", etc. Background reading: Ward, G. "Measuring and Modeling Anisotropic Reflection," Proc. SIGGRAPH, 1992. Slides available as PPT or as images.
Wednesday, October 17	BRDF of skin and faces Student presentations of papers: Marschner, S., Westin, S., Lafortune, E., Torrance, K. and Greenberg, D. "Image-Based BRDF Measurement Including Human Skin," Proc. EGRW, 1999. Debevec, P., Hawkins, T., Tchou, C., Duiker, H., Sarokin, W., and Sagar, M. "Acquiring the Reflectance Field of a Human Face," Proc. SIGGRAPH, 2000.
Monday, October 22	BRDF, lighting, and BTF measurement Student presentations of papers: Ramamoorthi, R. and Hanrahan. P. "A Signal-Processing Framework for Inverse Rendering," Proc. SIGGRAPH, 2001. Dana, K., van Ginneken, B., Nayar, S., and Koenderink, J. "Reflectance and Texture of Real-World Surfaces," ACM TOG, vol. 18, no. 1, 1999. Additional reading: Yu, Y. and Malik, J. "Recovering Photometric Properties of Architectural Scenes from Photographs," Proc. SIGGRAPH, 1998.
Wednesday, October 24	Recovering BRDF and texture on arbitrary geometry Student presentations of papers: Sato, Y., Wheeler, M., and Ikeuchi, K. "Object Shape and Reflectance Modeling from Observation," Proc. SIGGRAPH, 1997. Yu, Y., Debevec, P., Malik, J., and Hawkins, T. "Inverse Global Illumination: Recovering Reflectance Models of Real Scenes from Photographs," Proc. SIGGRAPH, 1999.
Monday, October 29	Fall break - no class
Wednesday, October 31	Fall break - no class
Monday, November 5	Measurement of light transmission and scattering Student presentations of papers: Zongker, D., Werner, D., Curless, B., and Salesin, D. "Environment Matting and Compositing," Proc. SIGGRAPH, 1999. Chuang, Y., Zongker, D., Hindorff, J., Curless, B., Salesin, D., and Szeliski, R. "Environment Matting Extensions: Towards Higher Accuracy and Real-Time Capture," Proc. SIGGRAPH, 2000. Wann Jensen, H., Marschner, S., Levoy, M., and Hanrahan, P. "A Practical Model for Subsurface Light Transport," Proc. SIGGRAPH, 2001.
Wednesday, November 7	Taxonomy of methods for acquiring 3D shape Brief overview of passive methods Active range scanning: touch probes, defocus, time-of-flight, photometric stereo, structured-light triangulation Slides available here.
Monday, November 12	Guest lecture by Claudio Silva on surface reconstruction
Wednesday, November 14	Practical issues in the design of 3D laser scanners Scan data postprocessing pipeline: scan alignment, global registration, merging, surface reconstruction, holefilling, etc.
Monday, November 19	Recent applications of range scanning: scanning sculptures Background reading: Rushmeier, H., Bernardini, F., Mittleman, J., and Taubin, G. "Acquiring Input for Rendering at Appropriate Levels of Detail: Digitizing a Pietà," Proc. EGRW, 1998. Levoy, M. et al. "The Digital Michelangelo Project: 3D Scanning of Large Statues," Proc. SIGGRAPH, 2000. Miyazaki, D. et al. "The Great Buddha Project: Modelling Cultural Heritage through Observation," Proc. VSMM, 2000.
Wednesday, November 21	Thanksgiving - no class
Monday, November 26	Range scanning technologies: stereo with manual correspondence and shape from silhouette Student presentations of papers: Debevec, P., Taylor, C., and Malik, J. "Modeling and Rendering Architecture from Photographs," Proc. SIGGRAPH, 1996. Matusik, W., Buehler, C., Raskar, R., Gortler, S., and McMillan, L. "Image-Based Visual Hulls," Proc. SIGGRAPH, 2000.
Wednesday, November 28	Range scanning technologies: shape from defocus and photometric stereo Student presentations of papers: Nayar, S., Watanabe, M., and Noguchi, M. "Real-Time Focus Range Sensor," Trans. PAMI, vol. 18, no. 12, 1996. Rushmeier, H., Taubin, G., and Guéziec, A. "Applying Shape from Lighting Variation to Bump Map Capture," Proc. EGRW, 1997.
Monday, December 3	Motion capture technologies: mechanical, inertial, ultrasonic, magnetic, optical (marker-based and markerless)
Wednesday, December 5	Measurement of position and motion for augmented reality Student presentation of papers: State, A., Hirota, G., Chen, D., Garrett, W., and Livingston, M. "Superior Augmented Reality Registration by Integrating Landmark Tracking and Magnetic Tracking," Proc. SIGGRAPH, 1996. Foxlin, E., Harrington, M., and Pfeifer, G. "Constellation: A Wide-Range Wireless Motion-Tracking System for Augmented Reality and Virtual Set Applications," Proc. SIGGRAPH, 1998.
Monday, December 10	Processing motion capture data and measuring deformation Student presentation of paper: Gleicher, M. "Retargetting Motion to New Characters," Proc. SIGGRAPH, 1998. Pai, D., van den Doel, K., James, D., Lang, J., Lloyd, J., Richmond, J., and Yau, S. "Scanning Physical Interaction Behavior of 3D Objects," Proc. SIGGRAPH, 2001.
Wednesday, December 12	Presentations of student projects and course wrapup