This work presents a novel approach to chest x-ray characterization. It is based on the generation of a visual words dictionary to represent x-ray images, and similarity-based categorization with a kernel based SVM classifier. Two main tasks are addressed: First, the extraction of chest images from a large radiograph archive, i.e. an organ identification task; Second, the detection and identification of chest pathologies, i.e. shifting from the organ level to a pathology level analysis. We used a large generic archive of 12,000 radiographs (IRMA) to tune the system parameters. We demonstrate automated organ detection on the IRMA collection as well as the generalization to a new data collection. The application is shown to discriminate between healthy and pathology cases, as well as identify specific pathologies on a set of 223 chest radiographs taken from a routine hospital examination. Results indicate detection of pathology at a sensitivity of 88.4% and a specificity of 81%. This is a first step towards similarity-based categorization that has a major clinical importance in computer-assisted diagnostics.