An application note (PDF!) from TI on optimizing the filter design to minimize the losses:
Filtering plays an essential part in nearly all communications systems. Removing unwanted signals is part of system design to eliminate unwanted noise or distortion. The signal presented to the analog to digital converter should have no spurs or distortion products, especially outside the desired Nyquist band. Noise and distortion spurs that may be safely outside the band of interest may not remain outside of that band after the ADC sampling process folds them around the sampling frequency.
The primary targets for the final analog filter are broadband noise and harmonic distortion spurs. Broadband noise, in particular, can be problematic since the bandwidth of many ADCs is typically several times the bandwidth of the signals to be processed. Likewise, harmonic spurs from the upper end of the band of interest are typically still within the input bandwidth of the ADC, even though they are far removed from the signal band. If not filtered out these spurs and noise can fold back onto weaker desired signals
and mask them.
