What is the issue you're seeing that leads you to wonder about the limit for the BY variable? Are you doing one-to-one merging, one-to-many merging or many-to-many merging in the "real" data. With just 3 test observations in each data set, it is hard to understand your issue or what leads you to ask about BY variables.
If you are seeing this message in the log with your "REAL" data,
NOTE: MERGE statement has more than one data set with repeats of BY values.