In this paper, an optimal and efficient algorithm for video summarization is proposed by exploiting temporal variations of video visual content. In particular, the most characteristic frames/shots (key-frames/shots) are extracted by estimating appropriate points on the feature vector curve, which represent in an optimal way the corresponding trajectory. This is performed by minimizing the approximation error of the feature vector curve and the respective curve formed by the estimated points using an interpolation scheme. A genetic algorithm is used for the minimization task, since the complexity of an exhaustive search is too large to be implemented. Furthermore, a fast technique for increasing the number of extracted key-frames/shots is presented.