A novel human video object watermarking scheme is proposed in this paper, providing copyright protection of semantic content. The proposed method focuses on the existence of face and body regions within an initial image, however other cases can also be addressed. Initially detection of human video objects is achieved using two adaptive 2-D Gaussian models, one for skin color distribution modelling and the other for body localization modelling. A watermark is then designed using invariant Hu moments of each human video object, and the watermark insertion procedure is driven by an iterative encryption module based on chaotic functions. Performance of the proposed object based secure watermarking system is tested under various signal distortions and known cryptanalytic attacks.