In this paper we address some of the issues commonly encountered in automatic image annotation systems such as simultaneous labeling with keywords corresponding to both abstract terms and object classes, multiple keyword assignment, and low accuracy of labeling due to concurrent categorization to multiple classes. We propose a hierarchical classification scheme which is based on predefined XML-dictionaries of tree form. Every node of such a tree defines a particular classification task while the childs of the node correspond to classification categories. The winning class (subnode) defines the subsequent classification task and the process continues until the leafs of the tree are reached. The final classification task is performed at image segment level; that is every image segment is assigned a particular keyword corresponding to a tree leaf. The path followed from the root of the XML tree to the leafs along with the union of labels assigned to the image segments compose the list of annotation