You can't list multiple matrices on the RETURN statement (that's MATLAB syntax) but you can return as many values as you want by writing a subroutine module instead of a function module.
For an example, see the bottom of p. 2 of http://support.sas.com/resources/papers/proceedings10/329-2010.pdf
Incidentally, you can do this because SAS/IML passes arguments by reference, not by value.
The SAS/IML convention is that output arguments are listed first, so your module might look like
proc iml;
start ols (beta, residual, y,x); 
  beta=inv(x`*x)*(x`*y);
  residual=y-x*beta;
finish ols;
/* define data */
x= { 1 1 1,     1 2 4,     1 3 9,     1 4 16,     1 5 25,     1 6 36,     1 7 49,     1 8 64 }; 
y= {3.929,5.308,7.239,9.638,12.866,17.069,23.191,31.443};
/* call module */
run ols(b, r, y, x);
print b, r;