Efficient way of naming and designating a series of matrices with a common prefix

[Season]

I am using SAS/IML to calculate the covariance matrix of data by hand. A problem I encountered was on how to name a bunch of matrices with a common prefix in IML. I summarized my concerns as the two following questions:

(1) Is the horizontal concatenation operator (i.e., "||"), which is valid in the DATA step and the operation of matrix elements of blocks, valid in the naming process of matrices in IML? I was using a macro and was hoping to give my matrices names like cov1, cov2, etc. I tried "cov||1" but the IML reported an error in the log.

(2) How to designate the bunch of matrices with a common prefix in the loading and summation steps? I was using the LOAD statement in IML and attempted to designate the matirces I wished to load with the argument "cov1-cov35". This works in the DATA step, but unfortunately it also did not work in the LOAD statement in IML. So I used the 

load _all_;

statement to get round typing the names of every matrix after the "load" clause. But I do not think it a good way if the storage designated is stockpiled with more matrices.

Similarly, I also got into trouble as I was trying to sum all the matrices with the same prefix. This time, there did not seem to be a way to get-around the monotonous typing of matrix names, so I had to type them one by one and join them with the plus sign. But is there a more efficient way?

Thanks!

[quickbluefish · Accepted solution]

I don't know if there is an IML-specific solution (though might help to see more of your code), but you could probably do this sort of thing with either macro statements or with a separate macro, e.g.:

%macro load_matrices(prefix=, start=, end=);

    %do mi=&start %to &end;
        &prefix&mi
    %end;

%mend; *load_matrices();

* sample call ;

proc iml;
load %load_matrices(prefix=cov, start=1, end=35) ;
...;
quit;

[quickbluefish]

* if you only need particular matrices rather than the whole range ;
%macro load_matrices(prefix=, mlist=);

    %let mlist=%cmpres(&mlist);
    %let nm=%sysfunc(countW(&mlist,' ');

    %do mi=1 %to &nm;
        &prefix%scan(&mlist,&mi,' ')
    %end;

%mend; *load_matrices();

* same idea for adding, etc ;
%macro add_matrices(prefix=, mlist=);

    %let mlist=%cmpres(&mlist);
    %let nm=%sysfunc(countW(&mlist,' ');

    %do mi=1 %to &nm;
       %if &mi>1 %then +;
        &prefix%scan(&mlist,&mi,' ')
    %end;
%mend; *add_matrices();

proc iml;
load %load_matrices(prefix=cov, mlist=5 6 8 11 15 16 22) ;
*... ;
sumM = %add_matrices(prefix=cov, mlist=8 11 15 16) ;
quit;

[Season]

Thank you for your further explanation! I really appreciate your kind help.

[Season]

Thank you for your rapid reply! Embedding a macro in the IML codes is a good choice. It solves my problem.

[ardehg]

You can use this simple method to create names with a prefix in SAS/IML.

proc iml;
 names="cov1":"cov10";
 print names;

[Season]

Thank you for your reply! I skimmed Rick's blog and found it was about naming the columns of a matrix instead of the matrix itself. Still, it is of use in a more delicate manipulation of matrices in other settings, but it seems that this does not solve my problem.

[Rick_SAS]

Someone else has already discussed how to store/load matrices with a common prefix, but I think a more important issue is WHY you would want to do such a thing? It makes it difficult to write code to analyze the various covariance matrices. 

Base SAS programmers experience this same difficulty if they try to write datasets that are named DS1-DSn. The better paradigm is to have one dataset that contains an indicator variable and use BY-group processing to analyze the groups in the data.

For example, if you have matrices cov1-cov5, and you want to analyze them (maybe convert them to correlation matrices or compute eigenvectors), it will be difficult to write code that analyzes cov1-cov5, but it is easier to write code that extracts the various covariance matrices and analyze them in a loop.

Please let us know about your application so we can suggest a better way to structure your analysis.

[Season]

Thank you for your reply! I do not think that the reason why I am conducting such matrix operations is a matter worth disclosing in detail. I think statistical softwares are designed to facilitate easy and rapid calculations to the most probable extent. Moreover, I think assigning matrices with a common name is a very common practice, just like what we do with datasets. Till now, I have not received an IML-specific solution. Admittedly, I am disappointed by the finding that such a mature statistical package as SAS/IML does not seem to have a quick and easy solution in IML as they do in the DATA step.

Anyway, since you asked the reason why I ran into this problem, I hereby say it. I am dealing with complex survey data and am calculating covariance matrices by the jackknife method. As you can see in the documentation in the SURVEYLOGISTIC procedure, the covariance matrix thus generated is a sum of covariance matrices calculated on each replicate. In my code, I manually calculate the covariance matrices on each replicate, multiply each of them by scalar factors and then sum them to get the estimated covariance matrix, which is what I want. By the way, I am calculating the covariance matrix of the difference between observed cumulative percentages and the CDF of a specified distribution, whose covariance matrix is not able to be calculated by built-in SAS SURVEY proceudres.

[Rick_SAS]

> I do not think that the reason why I am conducting such matrix operations is a matter worth disclosing in detail.

OK, that's fine. I was just trying to be helpful. Good luck on your project. 

[Season]

Never mind, I am a bit disappointed and slightly shocked by my finding. As I was compiling my second set of IML program (i.e., I hardly knew anything about IML before), I pre-emptively assume that it is because of my ignorance rather than the deficiency of the software that caused the problem. But till now, the latter seems to be the case. That's why I am disappointed and a bit shocked. Still, I will retract these remarks whenever I receive an IML-specific solution.

Anyway, I am grateful for your helping hand offered to not only this question but also the questions I previosuly raised. I wish you a happy and pleasant day!

[Tom]

Are you asking how to generate code?  Can you be more specific about what code you want to generate? In general the SAS macro processor is the tool to use for generating SAS code.

It sounds a little like you want to generate either a statement to load one matrix from storage.

load cov1;

Or perhaps a single statement to load multiple matrixes?

load cov1 cov2;

In this case it looks like something similar to this blog post https://blogs.sas.com/content/iml/2015/07/27/vector-to-string.html

is what you want.

If you define that MakeString function (or something similar) then you could just do:

 call symputx('names',MakeString("cov1":"cov20"));
 load &names;

[Season]

I think I have made myself clear in the thread. But I skimmed Rick's blog you cited and found this blog did not seem to fit my question as well. So I will explain my question again. There are two questions: (1) is the horizontal contactenation operator valid for naming matrices in a loop in the same way it is valid in the CALL EXECUTE routine in the DATA step? (2) how to efficiently load and sum a batch of matrices with a common prefix? In the DATA step, we can use arguements like 

proc delete data=cov1-cov35;
run;

to designate and delete a batch of matices with a common prefix. But arguments like

load cov1-cov35;

does not seem to work. In addition, if I want to calculate the sum of the 35 matrices, I have to name them one by one like "cov=cov1+cov2+cov3+...+cov35" or use the macro by @quickbluefish. Are there any IML-specific solutions to handle this summation problem more efficicently?

[Tom]

CALL EXECUTE() is another way to do code generation.  And it works in PROC IML.  So why not just use it?

Try this example program:

* Store a matrix ;
proc iml;
cov1=4.5;
store cov1;
quit;

* Use CALL EXECUTE() to generate LOAD statement ;
proc iml;
call execute('load cov'+'1'+';');
print cov1;
quit;

As to your other issue I am not sure why when they added the ability to use name lists to reference datasets they did not also add a similar feature to the IML commands that reference matrixes.  You could raise it as a enhancement request.

[Season]

Let us put my second question aside for the time being and focus only on the first one. First of all, I want to thank you for pointing out the availability of CALL EXECUTE in SAS/IML. But it seems not as powerful as it is in the DATA step in the sense that in the latter situation, CALL EXECUTE can loop and name matices with a common prefix.

For instance, the following code generates a series of datasets named a1, a2, ..., a35 that all have a single observation of two:

data a;
do x=1 to 35;
output;
end;
run;
data _null_;
set a;
call execute ('data a'||x||'=2;run;');
run;

But the following code does not make sense in SAS/IML:

data a;
do x=1 to 35;
output;
end;
run;
proc iml;
use a;
read all var {x} into a;
close a;
print a;
do i=1 to 35;
call execute ('a'||a[i]||'=2;');
call execute ('print a'||a[i]||';');
end;
quit;

The log reads:

ERROR: All specified variables must be the same type.

Of course, you can argue that CALL EXECUTE is not necessary here, as a macro can also generate a series of matrices named a1, a2, ..., a35. In fact, that is the way I adopted for my project.

But the problem lingers in my attempt of using the CALL EXECUTE routine to iteratively load matrices:

proc iml;
libname u "C:\";
reset storage='u';
do i=1 to 35;
call execute ('load a'||i||';');/*suppose that a1 through a35 has been generated previously and has been stored in the library named u*/
call execute ('print a'||i||';');
end;
quit;

The log also reads:

ERROR: All specified variables must be the same type.

So it seems that CALL EXECUTE cannot help us efficiently (iteratively or simultaneously) load a1 to a35 via the LOAD statement. With CALL EXECUTE, you still have to type the numbers following the letter "a" one by one.