topic Re: Efficient way of naming and designating a series of matrices with a common prefix in SAS/IML Software and Matrix Computations

Efficient way of naming and designating a series of matrices with a common prefix

Season — Wed, 13 Nov 2024 14:17:21 GMT

I am using SAS/IML to calculate the covariance matrix of data by hand. A problem I encountered was on how to name a bunch of matrices with a common prefix in IML. I summarized my concerns as the two following questions:

(1) Is the horizontal concatenation operator (i.e., "||"), which is valid in the DATA step and the operation of matrix elements of blocks, valid in the naming process of matrices in IML? I was using a macro and was hoping to give my matrices names like cov1, cov2, etc. I tried "cov||1" but the IML reported an error in the log.

(2) How to designate the bunch of matrices with a common prefix in the loading and summation steps? I was using the LOAD statement in IML and attempted to designate the matirces I wished to load with the argument "cov1-cov35". This works in the DATA step, but unfortunately it also did not work in the LOAD statement in IML. So I used the

load _all_;

statement to get round typing the names of every matrix after the "load" clause. But I do not think it a good way if the storage designated is stockpiled with more matrices.

Similarly, I also got into trouble as I was trying to sum all the matrices with the same prefix. This time, there did not seem to be a way to get-around the monotonous typing of matrix names, so I had to type them one by one and join them with the plus sign. But is there a more efficient way?

Thanks!

Re: Efficient way of naming and designating a series of matrices with a common prefix

quickbluefish — Wed, 13 Nov 2024 14:49:51 GMT

I don't know if there is an IML-specific solution (though might help to see more of your code), but you could probably do this sort of thing with either macro statements or with a separate macro, e.g.:

%macro load_matrices(prefix=, start=, end=);

    %do mi=&start %to &end;
        &prefix&mi
    %end;

%mend; *load_matrices();

* sample call ;

proc iml;
load %load_matrices(prefix=cov, start=1, end=35) ;
...;
quit;

Re: Efficient way of naming and designating a series of matrices with a common prefix

quickbluefish — Wed, 13 Nov 2024 15:05:38 GMT

* if you only need particular matrices rather than the whole range ;
%macro load_matrices(prefix=, mlist=);

    %let mlist=%cmpres(&mlist);
    %let nm=%sysfunc(countW(&mlist,' ');

    %do mi=1 %to &nm;
        &prefix%scan(&mlist,&mi,' ')
    %end;

%mend; *load_matrices();

* same idea for adding, etc ;
%macro add_matrices(prefix=, mlist=);

    %let mlist=%cmpres(&mlist);
    %let nm=%sysfunc(countW(&mlist,' ');

    %do mi=1 %to &nm;
       %if &mi>1 %then +;
        &prefix%scan(&mlist,&mi,' ')
    %end;
%mend; *add_matrices();

proc iml;
load %load_matrices(prefix=cov, mlist=5 6 8 11 15 16 22) ;
*... ;
sumM = %add_matrices(prefix=cov, mlist=8 11 15 16) ;
quit;

Re: Efficient way of naming and designating a series of matrices with a common prefix

Season — Wed, 13 Nov 2024 15:09:53 GMT

Thank you for your rapid reply! Embedding a macro in the IML codes is a good choice. It solves my problem.

Re: Efficient way of naming and designating a series of matrices with a common prefix

Season — Wed, 13 Nov 2024 15:11:29 GMT

Thank you for your further explanation! I really appreciate your kind help.

Re: Efficient way of naming and designating a series of matrices with a common prefix

ardehg — Wed, 13 Nov 2024 15:12:50 GMT

You can use this simple method to create names with a prefix in SAS/IML.

proc iml;
 names="cov1":"cov10";
 print names;

Re: Efficient way of naming and designating a series of matrices with a common prefix

Season — Wed, 13 Nov 2024 15:16:44 GMT

Thank you for your reply! I skimmed Rick's blog and found it was about naming the columns of a matrix instead of the matrix itself. Still, it is of use in a more delicate manipulation of matrices in other settings, but it seems that this does not solve my problem.

Re: Efficient way of naming and designating a series of matrices with a common prefix

Rick_SAS — Wed, 13 Nov 2024 15:43:11 GMT

Someone else has already discussed how to store/load matrices with a common prefix, but I think a more important issue is WHY you would want to do such a thing? It makes it difficult to write code to analyze the various covariance matrices.

Base SAS programmers experience this same difficulty if they try to write datasets that are named DS1-DSn. The better paradigm is to have one dataset that contains an indicator variable and use BY-group processing to analyze the groups in the data.

For example, if you have matrices cov1-cov5, and you want to analyze them (maybe convert them to correlation matrices or compute eigenvectors), it will be difficult to write code that analyzes cov1-cov5, but it is easier to write code that extracts the various covariance matrices and analyze them in a loop.

Please let us know about your application so we can suggest a better way to structure your analysis.

Re: Efficient way of naming and designating a series of matrices with a common prefix

Season — Wed, 13 Nov 2024 16:09:57 GMT

Thank you for your reply! I do not think that the reason why I am conducting such matrix operations is a matter worth disclosing in detail. I think statistical softwares are designed to facilitate easy and rapid calculations to the most probable extent. Moreover, I think assigning matrices with a common name is a very common practice, just like what we do with datasets. Till now, I have not received an IML-specific solution. Admittedly, I am disappointed by the finding that such a mature statistical package as SAS/IML does not seem to have a quick and easy solution in IML as they do in the DATA step.

Anyway, since you asked the reason why I ran into this problem, I hereby say it. I am dealing with complex survey data and am calculating covariance matrices by the jackknife method. As you can see in the documentation in the SURVEYLOGISTIC procedure, the covariance matrix thus generated is a sum of covariance matrices calculated on each replicate. In my code, I manually calculate the covariance matrices on each replicate, multiply each of them by scalar factors and then sum them to get the estimated covariance matrix, which is what I want. By the way, I am calculating the covariance matrix of the difference between observed cumulative percentages and the CDF of a specified distribution, whose covariance matrix is not able to be calculated by built-in SAS SURVEY proceudres.

Re: Efficient way of naming and designating a series of matrices with a common prefix

Rick_SAS — Wed, 13 Nov 2024 16:11:47 GMT

> I do not think that the reason why I am conducting such matrix operations is a matter worth disclosing in detail.

OK, that's fine. I was just trying to be helpful. Good luck on your project.

Re: Efficient way of naming and designating a series of matrices with a common prefix

Season — Wed, 13 Nov 2024 16:21:34 GMT

Never mind, I am a bit disappointed and slightly shocked by my finding. As I was compiling my second set of IML program (i.e., I hardly knew anything about IML before), I pre-emptively assume that it is because of my ignorance rather than the deficiency of the software that caused the problem. But till now, the latter seems to be the case. That's why I am disappointed and a bit shocked. Still, I will retract these remarks whenever I receive an IML-specific solution.

Anyway, I am grateful for your helping hand offered to not only this question but also the questions I previosuly raised. I wish you a happy and pleasant day!

Re: Efficient way of naming and designating a series of matrices with a common prefix

Tom — Wed, 13 Nov 2024 19:01:54 GMT

Are you asking how to generate code? Can you be more specific about what code you want to generate? In general the SAS macro processor is the tool to use for generating SAS code.

It sounds a little like you want to generate either a statement to load one matrix from storage.

load cov1;

Or perhaps a single statement to load multiple matrixes?

load cov1 cov2;

In this case it looks like something similar to this blog post https://blogs.sas.com/content/iml/2015/07/27/vector-to-string.html

is what you want.

If you define that MakeString function (or something similar) then you could just do:

 call symputx('names',MakeString("cov1":"cov20"));
 load &names;

Re: Efficient way of naming and designating a series of matrices with a common prefix

Season — Thu, 14 Nov 2024 01:02:44 GMT

I think I have made myself clear in the thread. But I skimmed Rick's blog you cited and found this blog did not seem to fit my question as well. So I will explain my question again. There are two questions: (1) is the horizontal contactenation operator valid for naming matrices in a loop in the same way it is valid in the CALL EXECUTE routine in the DATA step? (2) how to efficiently load and sum a batch of matrices with a common prefix? In the DATA step, we can use arguements like

proc delete data=cov1-cov35;
run;

to designate and delete a batch of matices with a common prefix. But arguments like

load cov1-cov35;

does not seem to work. In addition, if I want to calculate the sum of the 35 matrices, I have to name them one by one like "cov=cov1+cov2+cov3+...+cov35" or use the macro by @quickbluefish. Are there any IML-specific solutions to handle this summation problem more efficicently?

Re: Efficient way of naming and designating a series of matrices with a common prefix

Tom — Thu, 14 Nov 2024 03:21:13 GMT

CALL EXECUTE() is another way to do code generation. And it works in PROC IML. So why not just use it?

Try this example program:

* Store a matrix ;
proc iml;
cov1=4.5;
store cov1;
quit;

* Use CALL EXECUTE() to generate LOAD statement ;
proc iml;
call execute('load cov'+'1'+';');
print cov1;
quit;

As to your other issue I am not sure why when they added the ability to use name lists to reference datasets they did not also add a similar feature to the IML commands that reference matrixes. You could raise it as a enhancement request.

Re: Efficient way of naming and designating a series of matrices with a common prefix

Season — Thu, 14 Nov 2024 09:35:57 GMT

Let us put my second question aside for the time being and focus only on the first one. First of all, I want to thank you for pointing out the availability of CALL EXECUTE in SAS/IML. But it seems not as powerful as it is in the DATA step in the sense that in the latter situation, CALL EXECUTE can loop and name matices with a common prefix.

For instance, the following code generates a series of datasets named a1, a2, ..., a35 that all have a single observation of two:

data a;
do x=1 to 35;
output;
end;
run;
data _null_;
set a;
call execute ('data a'||x||'=2;run;');
run;

But the following code does not make sense in SAS/IML:

data a;
do x=1 to 35;
output;
end;
run;
proc iml;
use a;
read all var {x} into a;
close a;
print a;
do i=1 to 35;
call execute ('a'||a[i]||'=2;');
call execute ('print a'||a[i]||';');
end;
quit;

The log reads:

ERROR: All specified variables must be the same type.

Of course, you can argue that CALL EXECUTE is not necessary here, as a macro can also generate a series of matrices named a1, a2, ..., a35. In fact, that is the way I adopted for my project.

But the problem lingers in my attempt of using the CALL EXECUTE routine to iteratively load matrices:

proc iml;
libname u "C:\";
reset storage='u';
do i=1 to 35;
call execute ('load a'||i||';');/*suppose that a1 through a35 has been generated previously and has been stored in the library named u*/
call execute ('print a'||i||';');
end;
quit;

The log also reads:

ERROR: All specified variables must be the same type.

So it seems that CALL EXECUTE cannot help us efficiently (iteratively or simultaneously) load a1 to a35 via the LOAD statement. With CALL EXECUTE, you still have to type the numbers following the letter "a" one by one.

Re: Efficient way of naming and designating a series of matrices with a common prefix

WeiChen — Thu, 14 Nov 2024 10:51:24 GMT

When I analyze lots of coviarance matrixes I use method in Rick's blog

https://blogs.sas.com/content/iml/2015/02/09/array-of-matrices.html

Reason Rick says don't need many matrixes with different names is because you cna write loop and reuse same matrix as shown at end of blog. Just store each result as flat row in result matrix.

I also like example of simulation (like jackknife) in this blog https://blogs.sas.com/content/iml/2012/05/16/the-curious-case-of-random-eigenvalues.html Rick makes matrix in loop and stores it (or related result like eigenvalsi) flat in row. That is a good way. Then easy no trouble to store and load results. The ith row holds ith matrix.

ALso, I like to store results matrix as sas datset instead of use STORE command. Datset more flexible and cna be grphed and used in other procudures. If use STORE command, only is good inside IML.

Re: Efficient way of naming and designating a series of matrices with a common prefix

Season — Thu, 14 Nov 2024 12:38:19 GMT

Thank you for taking time to read and learn about my problems and sharing your experience. I generated matrices (in fact, vectors in my case) with different names because they actually came from independent loops enclosed in a macro written by me. Assigning them independently with different names and a common prefix makes me easier to find mistakes if there is any. I have the habit of randomly choosing entries of a matrix and match them to previous SAS procedure outputs to see if there is any programming error. As the matrix becomes larger, it is difficult for you to find entries in the middle of the matrix and you may only focus on the very outer entries of the big matrix.

As for the debate of dataset vs. matrices, I chose matrices because the task I was about to perform right after I compute the covariance matrix is to calculate its eigenvalues. I agree that if the task varies, storing matrices as datasets might be a good alternative.

Re: Efficient way of naming and designating a series of matrices with a common prefix

Tom — Thu, 14 Nov 2024 13:04:25 GMT

The || operator is not what you want in the IML code (also not what you want in your data step example either but let's ignore that). The + operator is what is used to append strings. This was discussed in the blog post from Rick about building a string with names.

proc iml;
use a;
read all var {x} into a;
close a;
print a;
do i=1 to 3;
  name='a'+strip(putn(a[i],'5.'));
  call execute(name+'=2;');
  call execute('print '+name+';');
end;
quit;

Re: Efficient way of naming and designating a series of matrices with a common prefix

Season — Thu, 14 Nov 2024 13:18:44 GMT

@Tom wrote:

The || operator is not what you want in the IML code (also not what you want in your data step example either but let's ignore that).

No, that's exactly what I want. The "||" operator works perfectly in CALL EXECUTE routine of the DATA step. Try run the codes I provided a few hours ago and verify that this does generate datasets named a1, a2, a3, ..., a35. It is both relatively simple and efficient. So I wish to go on using it in IML. That is why I originally mentioned it in my post and repetitively referred to it in my various replies.

As for your codes provided... Yes, it does solve my problem with the help of string functions. I really appreciate your help. But I still grumble about the lack of simpler codes like the CALL EXECUTE code I composed above.

Re: Efficient way of naming and designating a series of matrices with a common prefix

Season — Thu, 14 Nov 2024 13:21:35 GMT

Now that my first question is resolved, I turn to the second. I am not sure of what your sentence of "I am not sure why when they added the ability to use name lists to reference datasets they did not also add a similar feature to the IML commands that reference matrixes." means. I am not a native English speaker. Please forgive me for my lack of fluency in English to understand this sentence.