topic Re: use Array in a proc fcmp function in SAS Programming

use Array in a proc fcmp function

rhapsody — Wed, 06 Feb 2019 12:38:43 GMT

I have the table Bhat which has the variable theta according to

theta1 theta2 theta3

0.2 0.5 0

0.3 0.6 0.023

0.5 0.1 0.05

0.7 0.2 0.01

I want to use a function which knows it's an Array and takes the dimensions of the Array automatically into account. Below is what I've done so far but it doesn't work. The function I've done is called "add". Any ideas?

proc fcmp outlib=work.myfuncs.dates;

function add(theta);

trans1 = theta1;

do i=1 to dim(theta)-1;

trans(i+1) = theta(i+1) - theta(i);

end;

return(trans);

endsub;

run; quit;

/* set search path for custom functions */

options cmplib=work.myfuncs;

/* test with data set of values */

data test;

set Bhat;

/*array theta(*) theta1-theta2;*/

result=add(theta);

run;

Re: use Array in a proc fcmp function

RW9 — Wed, 06 Feb 2019 13:04:58 GMT

Why would you want to use fcmp for that? To be honest, I have never seen a single scenario where fcmp is useful, it only ever adds obfuscation both to the code, and to any code you create (macro can at least be highlighted as macro for instance). I would highly advise to do this in a datastep and avoid catalogs/compiled obfuscated and hidden code completely.

Re: use Array in a proc fcmp function

rhapsody — Wed, 06 Feb 2019 13:24:48 GMT

I'm using SAS base 9.2

Re: use Array in a proc fcmp function

RW9 — Wed, 06 Feb 2019 13:37:52 GMT

Not sure how that is relevant?

Re: use Array in a proc fcmp function

KachiM — Wed, 06 Feb 2019 14:05:39 GMT

Show your output Data Set as a result of your calculation as I am unable to visualize it.

regards

DATAsp

Re: use Array in a proc fcmp function

rhapsody — Wed, 06 Feb 2019 14:11:02 GMT

trans1 trans2 trans3

0.2 0.7 0.5

0.3 0.9 0.623

0.5 0.6 0.15

0.7 0.9 0.21

Re: use Array in a proc fcmp function

KachiM — Thu, 07 Feb 2019 00:21:22 GMT

rhapysody

Most simple solution is to use a Data Step to get what you want.

You seem to be interested to understand FCMP functionality. So,
I pass my little knowledge on FCMP to you! A FCMP function or
Subroutine consists of Data Step statements. Few matrix functions
exclusively belong to FCMP Environment. One big advantage is that we can hold
a Data Set as a two-dimensional Array (Matrix) in memory. Data
staying in memory reduces I/O cost. I am giving a function
later that reads a Data Set, holds in memory, manipulates cells.
Then I write out the resulting Array as a Data Set.

For SAS programmers: As long as you keep the source code of the function in safe place,
you need not worry about how it is saved as compiled code. There
is no secrecy or threat. It requires user-familiarity.

Your OUTPUT shows adding of columns of theta.

The simplest way is to pass a _temporary_ array from the Data Step
with values of theta and an another empty array to hold to-be-summed
values in FCMP. I filled the theta: values to t[]. I created empty trans[]
to pass to a Subroutine.

First the Subroutine ADD is made to receive two arrays t[] and
tr[]. Addition is made using t[] and the sum is assigned to tr[].
The use of OUTARGS shares the arrays both in FCMP Environment and
Data Step.

Before calling the Subroutine in the Data Step you tell the
location where the compiled one is saved by:

options cmplib = work.func;

In the Data Step, read theta: and save to the _temporary_ array
t[]. Create trans[] as a _temporary_ array.

You call the Subroutine Add, passing the two arrays.

Then copy the values from trans[] to tran[] array which writes out to WANT.

data have;
input theta1-theta3;
datalines;
0.2  0.5  0
0.3  0.6  0.023
0.5  0.1  0.05
0.7  0.2  0.01 
;
run;

proc fcmp outlib = work.func.arr;
   subroutine add(t[*], tr[*]);
      outargs t, tr;
      tr[1] = t[1];
      do i = 2 to dim(t);
         tr[i] = t[i-1] + t[i];
      end;
   endsub;
quit;

options cmplib = work.func;

data want;
   array t[3] _temporary_;
   array trans[3] _temporary_;
   
   set have;
   array th theta1-theta3;
   do i = 1 to dim(t);
      t[i] = th[i];
   end;
   call add(t,trans);
   array tran trans1-trans3;
   do i = 1 to dim(t);
      tran[i] = trans[i];
   end;
   drop i th:;
run;

The Output:

Obs 	trans1 	trans2 	trans3
1 	0.2 	0.7 	0.500
2 	0.3 	0.9 	0.623
3 	0.5 	0.6 	0.150
4 	0.7 	0.9 	0.210

A better way is to just pass the Data Set (HAVE) to a FCMP
function which reads the Data Set to a Dynamic Array and
after additions are done in all rows, write out the Dynamic
Array as a Data Set. Though your problem does not need this approach
but you can use this example to try bigger problems later on.

I have given explanations for most of the statements as COMMENT
in the following FCMP Function, DYNAMIC_ADD(). You are welcome
to ask any doubts in the code.

proc fcmp outlib = work.myfunc.lib;
   function dynamic_add(ds $);
   file log;
   array theta[1] / nosymbols; *Array head for THETA;
   array trans[1] / nosymbols; *array head for TRANS;

   rc = read_array('have', theta); *Dynamically sizes THETA and stores HAVE in a Matrix;
   put theta =; * See the row-col values of HAVE;
   rows = dim1(theta); cols = dim2(theta); * gets rows and cols of Matrix;
   call dynamic_array(trans, rows, cols);  * Create a Dynamic Array(TRANS) just like THETA;  
   do i = 1 to rows;
      do j = 1 to cols - 1;
         trans[i,1] = theta[i,1]; * First cell is copied;
         trans[i,j+1] = theta[i, j+1] + theta[i, j]; * Here comes your calculations;
      end;
   end;
   rc = write_array('WANTDYN', trans); * Write Out TRANS as a Data Set, WANTDYN;
   
   return(1); * A Function needs a return statement and a dummy value is returned;/*EDITED*/
   endsub;
quit;

options cmplib = work.myfunc; /*EDITED*/

data _null_;
   rc = dynamic_add('have');
run;

proc print data = wantdyn;
run;

You simply need one statement in the Data Step. The FCMP Function writes out the Data Set as WANTDYN which gives the same OUTPUT as shown earlier.

Best regards,

DATAsp

Re: use Array in a proc fcmp function

ErikLund_Jensen — Wed, 06 Feb 2019 19:22:36 GMT

Hi @RW9

I think you are a little unjust to fcmp, It is very useful in a SAS data warehouse, where all development is done in DI Studio.

Best practice in DI Studio is to use standard transformations and avoid userwritten code as much as possible, which makes data steps the last resort. As a result, a typical DI Studio job contains many proc sql extracts and joins, where recoding is limited to sas functions and case constructs, stored deep inside the metadata server. But as a fcmp function works as built-in functions, they can also be used in proc sql, and do things that would be impossible otherwise, like transforming UTM coordinates to longitude/latitude.

And with 20 developers working in different departments and about 10000 extracts and joins in the daily batch, a centrally maintained and documented fcmp catalog gives much less obfuscation than all developers trying to solve problems with overly complicated case constructs and scores of unnecessary steps. So in this scenario the use of a few fcmp functions is a big help.

Re: use Array in a proc fcmp function

RW9 — Thu, 07 Feb 2019 09:34:17 GMT

Never used DIS, so can't comment on that. What I can say is two things:

1) "a centrally maintained and documented" - whilst I have often seen central macro libraries and I agree that is definately a plus point, documentation is nothing more than a ghost of whisper from my experience. Even the macro and variable naming is non-informative. At one of the big conferences we should ask for a show of hands who has documentation, coding standards, common object interface agreements in place...

2) I would have thought the recent migration from 32-64bit would have shown the real dangers surrounding proprietary file formats. To my knowledge there is still no upgrade path for a catalog (formats/macros, compiled functions etc.) from 32bit to 64bit - and as outsourced work is considered third party owned, we only get catalogs. Hence anything before the switch is now unusable. Hence I can never recommend catalogs for any purpose, only ever use plain open text which is openable in pretty much anything and cross portable. Sure if you have the source code you can go back and recompile, if you have the source code...

The reason I dislike fcmp more than compiled macros is that they don't even jump out as being non-standard SAS for example:

data want;
   set have;
   a=catx("x",d,e);
   b=cats("x",d,e);
   c=catr("x",d,e);
   d=catq("x",d,e);
run;

One of the those is a user fcmp, how do you easily pick it out - and note this is without the standard all in upper case, all on the same line with no indentations and surrounded by masses of macro code?

</rant>

Re: use Array in a proc fcmp function

ballardw — Thu, 07 Feb 2019 21:44:33 GMT

@rhapsody wrote:

I have the table Bhat which has the variable theta according to

theta1       theta2          theta3

    0.2           0.5               0

    0.3           0.6               0.023

    0.5           0.1               0.05

    0.7           0.2               0.01

I want to use a function which knows it's an Array and takes the dimensions of the Array automatically into account. Below is what I've done so far but it doesn't work. The function I've done is called "add". Any ideas?

proc fcmp outlib=work.myfuncs.dates;

function add(theta);

trans1 = theta1;

do i=1 to dim(theta)-1;

trans(i+1) = theta(i+1) - theta(i);

end;

return(trans);

endsub;

run; quit;

/* set search path for custom functions */

options cmplib=work.myfuncs;

/* test with data set of values */

data test;

set Bhat;

/*array theta(*) theta1-theta2;*/

result=add(theta);

run;

First thing is it doesn't work because you have errors:

When I run your fcmp code:

1393  proc fcmp outlib=work.myfuncs.dates;
1394
1395  function add(theta);
1396  trans1 = theta1;
1397  do i=1 to dim(theta)-1;
ERROR: Argument number 1 to the DIM function must be an array.
1398  trans(i+1) = theta(i+1) - theta(i);
1399  end;
1400
1401  return(trans);
1402  endsub;
1403  run;

So you have declared your variable THETA incorrectly. I believer you are mixing array addressing trans(I+1) with variable trans1 and theta(I) and theta1. Since you did not pass a parameter theta1 it is not defined. To use the first value of a passed array you would use theta[1] or theta(1).

Trans has not be declared as an array so you can't actually use the trans(I+1) either.

Are you attempting to calculate a single value value or to pass an array back?

Your example call in the data step

result=add(theta);

would not work as if theta, at that point represents an array you cannot assign an array of values to a single result variable.

It may help to provide a data set with variable names that do not match the variables in your function so you can trace out referencing the data set variable for value and what you want to accomplish.

If you want to pass a modified array back I believe you would be in the realm of a SUBROUTINE, not function (generic definition of function often involves a result of a single variable). Example sum(a,b,c) returns a single value in a data step. Subroutine will let you specify OUTARGS. From the documentation:

OUTARGS

specifies arguments from the argument list that the subroutine should update.

Re: use Array in a proc fcmp function

KachiM — Fri, 08 Feb 2019 03:18:18 GMT

Hello ballardw

Your answer is excellent to fill the gap I left in my answer ! Thanks to you.

Kind regards,

DATAsp

Re: use Array in a proc fcmp function

ErikLund_Jensen — Fri, 08 Feb 2019 10:25:02 GMT

Hi @RW9

Thanks for your comments.

It never occured to me that fcmp functions could be difficult to pick out, because i know them by heart. We have named our fcmp's after the type of conversion they do, such as utm2lat or ssn2bd, and the digit in the middel of the name makes them recognizable. But that is sheer luck and works only if you know the rules, so you have certainly got a point there.

As to the catalog problem, we don't use catalogs for anything else than fcmp and formats in our production environment, and catalogs are read-only to users. The only way to get a catalog entry (format or fcmp) promoted to production is in the form of a program, that creates the entry, and we have jobs to run the programs and recreate the full content if necessary.

Re: use Array in a proc fcmp function

rhapsody — Wed, 13 Feb 2019 07:15:15 GMT

Could you please provide a full working example to your answer?

Re: use Array in a proc fcmp function

KachiM — Wed, 13 Feb 2019 09:37:30 GMT

Hi Rhapsody,

You are returning to ask for a working example from Ballardw.

I have been thinking that my solution is not useful to solve your problem. Will not be good to share your thoughts? Do come and say your issues.

Regards,

DATAsp

Re: use Array in a proc fcmp function

rhapsody — Wed, 13 Feb 2019 09:59:39 GMT

To make it simple. What I want is a function which can take an Array as an input and and return an Array as output. The example doesn't really matter.

Y=something(X)

where "X" is an input Array and "Y" is an output Array. Something is the function.

Re: use Array in a proc fcmp function

KachiM — Wed, 13 Feb 2019 10:51:28 GMT

Hi Rhapsody,

Your desire:

Y=something(X)

can be achieved as a Function (something), only if Y can take a SINGLE value where X can be an array. But you desire an array to be returned and so it can be achieved by a Subroutine only.

See the first part of my earlier solution. I reproduce that part:

data have;
input theta1-theta3;
datalines;
0.2  0.5  0
0.3  0.6  0.023
0.5  0.1  0.05
0.7  0.2  0.01 
;
run;

proc fcmp outlib = work.func.arr;
   subroutine add(t[*], tr[*]);
      outargs t, tr;
      tr[1] = t[1];
      do i = 2 to dim(t);
         tr[i] = t[i-1] + t[i];
      end;
   endsub;
quit;

options cmplib = work.func;

data want;
   array t[3] _temporary_;
   array trans[3] _temporary_;
   
   set have;
   array th theta1-theta3;
   do i = 1 to dim(t);
      t[i] = th[i];
   end;
   call add(t,trans);
   array tran trans1-trans3;
   do i = 1 to dim(t);
      tran[i] = trans[i];
   end;
   drop i th:;
run;

The Output:

Obs 	trans1 	trans2 	trans3
1 	0.2 	0.7 	0.500
2 	0.3 	0.9 	0.623
3 	0.5 	0.6 	0.150
4 	0.7 	0.9 	0.210

Since we want the Array, TRANS[ ], to be filled by FCMP program from the Array, T[ ], it is possible by a SUBROUTINE only. Hence, in Data Step we pass both the arrays by the subroutine (Subroutine ADD). This is done by passing one observation with theta1 to theta3 at a time where Array T[ ] holds them. Array TRANS[ ], while holding missing values on input, gets filled by the FCMP call routine:

call add(t,trans);

Note in FCMP parentheses ( ), are reserved for function arguments. Square brackets ([ ]) is a natural choice for Arrays.

Regards,

DATAsp

Re: use Array in a proc fcmp function

rhapsody — Wed, 13 Feb 2019 11:36:28 GMT

Oki.

The reason why I was looking into this option in the first place is to simplify the code. Instead of writing 30 lines of code you write just one function and everytime you wanna use that function you just write something like

data;

set input;

Y=something(X);

run;

And that gives you the output Array Y.

But since your solution implies about 30 lines of code I suppose that this kind of code simplification just isn't available in SAS?

Re: use Array in a proc fcmp function

RW9 — Wed, 13 Feb 2019 12:06:18 GMT

Just re-assess your data modelling. Normalised (i.e. data going down the page) is far simpler to work with than transposed (across, which is what you have). It would save you all the hassel of array processing and lots of coding. Alternatively if you have a lot of matrix data, then IML should be something you look at, that is built for matrix processing.

Simply put, if you find something is difficult, or requires lots of coding, then you likely have something else wrong which is causing you work later on, be that data modelling or choice of processing.

Re: use Array in a proc fcmp function

KachiM — Wed, 13 Feb 2019 12:12:06 GMT

Hello,

FCMP functions and call routines are written once, compiled and saved in a Library with special Dataset Name. You don't write the function every time to run it. Then, give the LibraryName.Dataset name as option and use the function.

The second part of the solution, I gave previously, is one way to achieve your need.

I am not aware of any other software doing it in one or two statements. I will be thankful to you if you can describe such a software.

Regards,

DATAsp

Re: use Array in a proc fcmp function

rhapsody — Wed, 13 Feb 2019 12:59:35 GMT

In your example you have to write quite a lot of lines ta call the function again which makes it pointless to work with functions in the first place. I'll give you an example in Matlab where you can write a function and then call it using only one row of code.

I have the input Array X

0,67	393,96	833,78	48,81	3,61	0,85	0,00	1,47
22,04	167,80	2189,82	8,60	1,97	2,44	0,00	0,04
8,83	39,58	1563,34	42,64	46,33	1,76	0,84	0,00
6,00	81,53	1039,95	12,26	25,77	0,76	0,94	0,00
1,88	87,12	386,61	11,52	19,83	2,85	0,03	0,00
6,12	61,22	195,15	47,00	0,14	0,92	0,00	0,00
0,38	109,71	649,68	93,17	13,05	0,00	0,70	0,00
2,31	17,99	1416,38	92,77	5,44	0,98	0,58	0,00
3,29	55,23	1542,51	35,90	19,83	1,21	0,00	0,88
4,04	130,80	1417,13	63,79	16,54	0,98	0,00	0,00
13,17	5,18	963,50	69,60	34,62	1,31	0,98	0,00
3,34	160,36	1341,45	135,73	31,93	0,00	0,04	0,00

And then I want a function which normalizes every row such that it sums to one. The function looks like

function Y = num2porcent(X)

[row col]=size(y);

sumY=sum(y,2);

Y = X./(sumY*ones(1,col));

end

Every time I wanna call it I just wirite

Y = num2porcent(X)

No more lines required.

The output is

0,001	0,307	0,650	0,038	0,003	0,001	0,000	0,001
0,009	0,070	0,915	0,004	0,001	0,001	0,000	0,000
0,005	0,023	0,918	0,025	0,027	0,001	0,000	0,000
0,005	0,070	0,891	0,011	0,022	0,001	0,001	0,000
0,004	0,171	0,758	0,023	0,039	0,006	0,000	0,000
0,020	0,197	0,628	0,151	0,000	0,003	0,000	0,000
0,000	0,127	0,750	0,107	0,015	0,000	0,001	0,000
0,002	0,012	0,922	0,060	0,004	0,001	0,000	0,000
0,002	0,033	0,930	0,022	0,012	0,001	0,000	0,001
0,002	0,080	0,868	0,039	0,010	0,001	0,000	0,000
0,012	0,005	0,885	0,064	0,032	0,001	0,001	0,000
0,002	0,096	0,802	0,081	0,019	0,000	0,000	0,000

So this is a good example where I use an Array as input and get an Array as output and everytime I call it the only thing needed is one row of code.