In this paper an optimal framework for summarization of stereoscopic video sequences is presented, which extracts a meaningful set of video frames. Each sequence is first partitioned into shots, the disparity field, occluded areas and depth map are estimated and then a hierarchical color and depth segmentation scheme is applied to each shot, based on a multiresolution implementation of the RSST algorithm. Color and depth segment fusion is employed for achieving high-quality semantic segmentation, and feature vectors are constructed using a fuzzy classification formulation. For a given shot, key frames are extracted using an optimization method, namely, a genetic algorithm, for locating frames of minimally correlated feature vectors. Experimental results indicate the reliable performance of the proposed scheme.