In this paper, an adaptive neural network architecture is proposed for efficient video object segmentation and tracking of stereoscopic video sequences. The scheme includes (a) a retraining algorithm for adapting network weights to current conditions, (b) a semantically meaningful object extraction module for creating a retraining set and (c) a decision mechanism, which detects the time instances of a new network retraining. The retraining algorithm optimally adapts network weights by exploiting information of the current conditions and simultaneously minimally degrading the obtained network knowledge. The algorithm results in the minimization of a convex function subject to linear constraints, thus, one minimum exists. Furthermore, a decision mechanism is included to detect the time instances that a new network retraining is required. Description of the current conditions is provided by a segmentation fusion algorithm, which appropriately combines color and depth information.