Web-based image search engines and CBIR techniques are blind to the actual content. As a result querying for a specific object is often cluttered with irrelevant data, leading to low precision. Furthermore recall rates are also very low since retrieval procedures are usually based either on context (surrounding text) and file captions or on low-level visual features. In this paper an automatic multi-agent image retrieval system is proposed. Our novel system exploits the format of multimedia sharing web sites to discover the underlying structure in order to finally infer and extract multimedia files and corresponding associated keywords from the web pages. The system first identifies the section of the web page that contains the multimedia file to be extracted and then extracts it by using clustering techniques and other tools of statistical origin. Experimental results on real-world image sharing web sites are presented and discussed in this paper, indicating the promising performance of the proposed system.