Yes, there is. On your PANELBY statement, specify these two options:
panelby flag sex / uniscale=row columns=4;
The UNISCALE option make only the ROW axes uniform, allowing the column axes to vary independently. By setting COLUMNS=4, ech column axis is only getting on histogram. that way, the "outlier" histogram does not have any affect on the other histograms.