In this paper, an adaptive neural network architecture is proposed for efficient video object segmentation and tracking of stereoscopic video sequences. Object extraction is a very important issue, addressed by the emerging multimedia applications, since it provides a meaningful description of the visual content. The scheme includes:(A) A retraining algorithm that optimally adapts the network weights to the current conditions and simultaneously minimally degrades the previous knowledge.(B) A semantically meaningful object extraction module for constructing the retraining set of the current conditions and (C) a decision mechanism, which detects the time instances when network retraining is required. The algorithm results in the minimization of a convex function subject to linear constraints. Furthermore description of the current conditions is achieved by appropriate combination of color and depth information. Experimental results on real life video sequences indicate the promising performance of the proposed adaptive neural network-based scheme.