Individuals with Autism Spectrum Disorders (ASD) show deficiencies in prosodic abilities, both pragmatic and affective. Their deficiencies in affective prosody have been mostly related to cognitive difficulties in emotion recognition. The current study tested an alternative hypothesis, linking vocal emotion recognition difficulties in ASD to lower level auditory perceptual deficiencies. Twenty one high functioning male adults with ASD and 32 male adults from the general population, matched on age and verbal abilities, and screened for normal hearing thresholds, undertook a battery of auditory tasks. Results demonstrated that individuals with ASD scored significantly lower than controls on vocal emotion recognition. Psychoacoustic abilities were strong predictors of vocal emotion recognition in both the ASD and control groups. Psychoacoustic abilities explained 48.1% of the variance of vocal emotion recognition scores in the ASD group, and 28.0% of the variance in the general population group. These results highlight the importance of lower level psychoacoustic factors in the perception of prosody in autism.