You can't list multiple matrices on the RETURN statement (that's MATLAB syntax) but you can return as many values as you want by writing a subroutine module instead of a function module.
For an example, see the bottom of p. 2 of http://support.sas.com/resources/papers/proceedings10/329-2010.pdf
Incidentally, you can do this because SAS/IML passes arguments by reference, not by value.
The SAS/IML convention is that output arguments are listed first, so your module might look like
proc iml;
start ols (beta, residual, y,x);
beta=inv(x`*x)*(x`*y);
residual=y-x*beta;
finish ols;
/* define data */
x= { 1 1 1, 1 2 4, 1 3 9, 1 4 16, 1 5 25, 1 6 36, 1 7 49, 1 8 64 };
y= {3.929,5.308,7.239,9.638,12.866,17.069,23.191,31.443};
/* call module */
run ols(b, r, y, x);
print b, r;