COS 429 - Computer Vision

Fall 2005

Assignment 3

Note that Thursday Nov. 24 and Friday Nov. 25 will not be counted as late days for this assignment.
Submitting on Saturday will count as two days late.

1. Colorspaces (30%)

(Note: the previous part (a) was a bad question - it has been removed from the assignment.

For this question, you'll evaluate the use of RGB vs. HSV for skin color detection. To do this, start by looking back at your database of faces from Assignment 2. For each image, crop it tightly around the face and convert it to HSV using Matlab's rgb2hsv function (assume that the image gamma is 2.2, so you need to convert RGB to the 0..1 range, then raise the values to this power). Assemble all the pixels from all the images into 6 vectors, one for each of R, G, B, H, S, and V (hint: reshape). Plot the histogram for each of the 6 color channels (hint: use the hist command, but you probably want more bins than the default of 10). Comment on what you see, and the implications for skin color detection and tracking. Please include the histogram plots in your writeup.

2. Color histograms for tracking (20%)

The remainder of the assignment deals with implementing a blob tracker for objects in video sequences. The first cue you will investigate is color histograms. Implement the following:

1. Read in the first image of the sequence (see datasets below), and allow the user to select the object(s) to be tracked (using the getrect function).
2. Compile a histogram of whichever of R, G, B, H, S, and V you found to be most effective in section 1 (hint: if the answer wasn't H, go back and try again :-)
3. Read in each subsequent frame from the sequence in turn. You can read frame number i using a call such as

   imread(sprintf('img%02d.jpg', i));

4. For each pixel, look up into the target histogram to determine how well that pixel matches the foreground object's colors.

   (Note: simply using the value of the histogram looked up for each pixel's hue is good enough - you don't need to use the full Bayes's rule to evaluate a probability...)

3. Background subtraction (10%)

The next thing to implement is background subtraction: for each frame, find the absolute color difference between the frame and the background. For some of the datasets the background frame will be given, while for others you should determine the background from the video frames themselves by taking the median of all the frames.

4. Basic EM Tracking (30%)

Allow the user to specify any number of objects to be tracked in the first frame, and, as shown in class, propagate their locations forward through the video using expectation maximization and an anisotropic Gaussian mixture model (details here, here, here, here, and here). You should be able to use the results of steps 2 or 3 above, or use a combination of both cues by multiplying the outputs together.

5. Dynamic model (10%)

Implement simple prediction by keeping track of the velocity of each blob on each frame, and updating the velocity using a rolling average. Show that this allows you to keep track of objects more robustly than not using prediction (if necessary, skip frames in the datasets, or construct synthetic datasets).

Writeup

Show sample outputs of each stage of the tracking, as well as at least one complete tracked sequence (where you have marked each object at each frame by overlaying a box on the video frame - make sure to use a different color for each object being tracked). If you want, you can draw nice ellipses using this .m file.

Data

Here are a few datasets. Stay tuned for more...

• book.zip, 50 frames, static camera, one moving object - try both background subtraction (background image provided) and color histograms
• playdough.zip, 41 frames, static camera, one moving object - try background subtraction (use median image as background)
• hands.zip, 100 frames, static camera, two moving objects - try color histograms + dynamic model
• lawn.zip, 42 frames and 62 frames (two sequences), people walking around - try color histograms + dynamic model
• lawn3.zip, 145 frames, people walking around - try color histograms + dynamic model; try and track Benedict's head including his shirt