In this paper, an interactive framework for efficient browsing and transmission of video sequences is presented, based on an optimal content-based video decomposition scheme. In particular each video file is analyzed to provide a multiscale structure of different" content resolution levels". This structure can be seen as a tree structure, each level of which corresponds to a particular content resolution, while the tree-nodes contain viewing elements, representing the visual content of a segment of the sequence. The multiscale optimal video organization is performed by minimizing a cross correlation criterion so that the most representative shots (key-shots) from a video sequence or frames (key-frames) from a video shot are extracted. Experimental results on real-life video sequences show that the proposed multiscale video organization technique enables users to detect content of interest much faster, compared to the conventional sequential video scanning method, and thus it leads to significant reduction of the viewed/transmitted information.