BookmarkSubscribeRSS Feed
🔒 This topic is solved and locked. Need further help from the community? Please sign in and ask a new question.
PerNielsen
Calcite | Level 5

Hi

I have a data set that I have been told I need to perform a concordance correlation coefficient test on. It is a correlation between two measures in a time serie. I have tried to find a totutial here on how to perform this test in SAS but have not been able to find one. I do therefore hope that there is someone on this cummuniti that can help me with this problem.

I have performed a test here https://services.niwa.co.nz/services/statistical/concordance but since this is a web based test where I have no idea of what is going on behind the screen, I would prefere to perform a proper test in SAS.

Kind regards,

Per Nielsen

1 ACCEPTED SOLUTION

Accepted Solutions
21 REPLIES 21
art297
Opal | Level 21

You can find an example at: https://onlinecourses.science.psu.edu/stat509/node/161

 

Art, CEO, AnalystFinder.com

 

Reeza
Super User

I'm not sure Cohens Kappa is the correct test for a time series, my stats is getting rusty. 

First determine the appropriate tests that need to occur and then figure out how to perform the test in SAS. 

PerNielsen
Calcite | Level 5

Worked perfectly, got the same results as from the home page, just feels a lot better to have a SAS program doing the analysis. Thanks for this suggestion!

PerNielsen
Calcite | Level 5

Hi Art,

 

I hope it is okay that I email you with an additional question to this one.

I have been asked to include repeated measures in my analysis, but have been discussing this with a statistician who on the other hand suggested that I did a CCC analysis for each animal (I have 20) and then made an additional coloum to the table with results for min-max for these 20 CCCs'.

Normally when you run a Proc statement in SAS you can include a BY statement, do you know if this is possible in a PROC IMLas this one:

 

proc iml;
use data.ice_cs;
read all var {iceMinlying} into var1;
read all var {csMinlying} into var2;
start concorr;
nonmiss=loc(var1#var2^=.);
var1=var1[nonmiss];
var2=var2[nonmiss];
free nonmiss;
n=nrow(var1);
mu1=sum(var1)/n;
mu1=round(mu1,0.0001);
mu2=sum(var2)/n;
mu2=round(mu2,0.0001);
sigma11=ssq(var1-mu1)/(n-1);
sigma11=round(sigma11,0.0001);
sigma22=ssq(var2-mu2)/(n-1);
sigma22=round(sigma22,0.0001);
sigma12=sum((var1-mu1)#(var2-mu2))/(n-1);
sigma12=round(sigma12,0.0001);
lshift=(mu1-mu2)/((sigma11#sigma22)##0.25);
rho=sigma12/sqrt(sigma11#sigma22);
rho=round(rho,0.0001);
z=log((1+rho)/(1-rho))/2;
se_z=1/sqrt(n-3);
t=tinv(0.975,n-3);
z_low=z-(se_z#t);
z_upp=z+(se_z#t);
rho_low=(exp(2#z_low)-1)/(exp(2#z_low)+1);
rho_low=round(rho_low,0.0001);
rho_upp=(exp(2#z_upp)-1)/(exp(2#z_upp)+1);
rho_upp=round(rho_upp,0.0001);
crho=(2#sigma12)/((sigma11+sigma22)+((mu1-mu2)##2));
crho=round(crho,0.0001);
z=log((1+crho)/(1-crho))/2;
if sigma12^=0 then do;
   t1=((1-(rho##2))#(crho##2))/((1-(crho##2))#(rho##2));
   t2=(2#(crho##3)#(1-crho)#(lshift##2))/(rho#((1-(crho##2))##2));
   t3=((crho##4)#(lshift##4))/(2#(rho##2)#((1-(crho##2))##2));
   se_z=sqrt((t1+t2-t3)/(n-2));
end;
else se_z=sqrt(2#sigma11#sigma22)/((sigma11+sigma22+((mu1-mu2)##2))#(n-2));
t=tinv(0.975,n-2);
z_low=z-(se_z#t);
z_upp=z+(se_z#t);
crho_low=(exp(2#z_low)-1)/(exp(2#z_low)+1);
crho_low=round(crho_low,0.0001);
crho_upp=(exp(2#z_upp)-1)/(exp(2#z_upp)+1);
crho_upp=round(crho_upp,0.0001);
Results=n//mu1//mu2//sigma11//sigma22//sigma12//rho_low//rho//rho_upp//
        crho_low//crho//crho_upp;
r_name={'SampleSize' 'Mean_1' 'Mean_2' 'Variance_1' 'Variance_2' 'Covariance' 'Corr LowerCL'
       'Corr' 'Corr UpperCL' 'ConcCorr LowerCL' 'ConcCorr' 'ConcCorr UpperCL'};
print 'The Estimated Correlation and Concordance Correlation (and 95% Confidence Limits) IceTag vs CowScout lying';
print Results [rowname=r_name];
finish concorr;
run concorr;

 

Kind regards,

Per

art297
Opal | Level 21

Per: Unfortunately, I've never learned IML. However, I'm sure that @Rick_SAS@Ksharp or others (who are) could easily answer your question.

 

That said, worse case, since the IML use statement accepts dataset options, you could always wrap the code in a macro that uses a %do loop, and supply the by condition in a where=() dataset option.

 

Art, CEO, AnalystFinder.com

 

Rick_SAS
SAS Super FREQ

@art297 wrote:

That said, worse case, since the IML use statement accepts dataset options, you could always wrap the code in a macro that uses a %do loop, and supply the by condition in a where=() dataset option.

No. You almost never use a %DO loop in IML programs. SAS/IML contains DO loops, IF-THEN statements, etc, as part of the language, so you can perform loops and conditional logic within the language.

 

Although SAS/IML does not support a BY statement, there are many ways to perform BY-group processing:

PerNielsen
Calcite | Level 5

Thanks for your swift responses.

I assume the UNIQUE-LOC is the procedure that might work on my data and I will test it with my limited knowledge of programming out of my comfort zone in SAS.

Otherwise I just have to perform 300 runs of the procedure 🙂 I will specify the animal number, out of 30, that I have in my dataset and then run the analysis. Would however have been nice to have a procedure in sas that does this for me in the proc iml, but I understand that it is not that simple 🙂

 

Kind regards,

Per

art297
Opal | Level 21

@Rick_SAS: Thanks for providing the info. However, I do have to ask, 'Why are macros almost never used' in IML?

 

Since, like @PerNielsen, I don't know anything about IML, if I was asked to take the IML code you provided in one of your loop blogs and run the code separately for each gender, I'd do something like the macro wrapper shown below. I created a numeric field called gender simply because @PerNielsen mentioned that he had to repeat the code for animals 1 thru 30.

 

Is there a problem doing it this way, or is it just preferred to take advantage of the IML functionality?

 

Art, CEO, AnalystFinder.com

data testdata;
  set sashelp.class;
  if sex eq 'M' then gender=1;
  else gender=2;
run;

%macro loopiml;
%do i=1 %to 2;
title "gender eq &i";
proc iml;
use testdata (where=(gender eq &i.)); 
   read all var {age height} into m;
close;
call sort(m, 1);         /* 1. Sort data by first column */
C = m[,1];               /* finished with m: extract sorted categories */
x = m[,2];               /* and sorted data */

/* 2. Obtain row numbers for the first observation in each level. */
b = uniqueby(C, 1);     /* b[i] = beginning of i_th category */
u = C[b];               /* get unique values (if needed) */
print b u;

s = j(nrow(b),1);         /* 3. Allocate vector to hold results */
b = b // (nrow(C)+1);     /* trick: append (n+1) to end of b */
do i = 1 to nrow(b)-1;    /* 4. For each level... */
   idx = b[i]:(b[i+1]-1); /* 5. Find observations in level */
   s[i] = mean(x[idx]);   /* 6. Compute statistic on those values */
end;
lbl = putn(u, "Best4.");  /* convert numeric values to character */
print s[rowname=lbl];
quit;
  %end;
%mend loopiml;
%loopiml;
Rick_SAS
SAS Super FREQ

I do have to ask, 'Why are macros almost never used' in IML?

 

Because macro replication is unnecessary and is inefficient compared to native IML processing. In a macro, you start/stop the procedure &N times, open/close/read the data set &N times, and allocate/free all the matrices &N times. Using native IML code you start/stop the procedure once, open/close/read the data once, and can make more efficient use of memory. A lot of these reasons are the same reasons that BY-processing is preferred over macro loops when performing computationally intensive tasks such as simulation and bootstrapping.

Ksharp
Super User

Arthur.T ,

In IML , you can define a function which can generate the code dynamically just as MACRO did .

Rick_SAS
SAS Super FREQ

I think @Ksharp is making a good point about reusability. A macro loop generates multiple copies of code. Each copy must be parsed. In contrast, if you define an IML function (or block of code) and call it in a loop, the code is parsed once but executed multiple times with different arguments. 

ChristosK
Quartz | Level 8

Rick:

 

I have been using your macro to generate concordance coefficients but in one example the lower confidence interval is negative.

Is there a problem somewhere here? 

Whilst I understand that the direction of effect of any correlation coefficient just describes the direction of the relationship, I am puzzled in this example as this is the only model where  I have found a negative value .

 

Christos 

 

title "Concordance Correlation Coefficient";
proc sgplot data=Schumann.NumBP noautolegend;
scatter y=i3min_LL_DIA   x= i3min_A_DIA;
lineparm x=0 y=0 slope=1 / clip;
yaxis label='i3min_LL_DIA' min=28 max=108;
xaxis label='i3min_A_DIA' min=38 max=111;
run;

proc iml;
*******************************************************************************
*  Enter the appropriate SAS data set name in the use statement and enter the *
*  appropriate variable names in the read statements.                         *
*******************************************************************************;
use Schumann.NumBP;
read all var {i3min_LL_DIA} into var1;
read all var {i3min_A_DIA} into var2;

start concor(x1, x2);nonmiss=loc(x1#x2^=.);var1=x1[nonmiss];var2=x2[nonmiss];free nonmiss;n=nrow(var1);
mu1=mean(var1);mu2=mean(var2);Sigma = cov( var1||var2 );sigma11=Sigma[1,1];sigma22=Sigma[2,2];sigma12=Sigma[1,2];
lshift=(mu1-mu2)/((sigma11#sigma22)##0.25);rho=sigma12/sqrt(sigma11#sigma22);z=log((1+rho)/(1-rho))/2;se_z=1/sqrt(n-3);
t=tinv(0.975,n-3);z_low=z-(se_z#t);z_upp=z+(se_z#t);rho_low=(exp(2#z_low)-1)/(exp(2#z_low)+1);rho_upp=(exp(2#z_upp)-1)/(exp(2#z_upp)+1);crho=(2#sigma12)/((sigma11+sigma22)+((mu1-mu2)##2));
z=log((1+crho)/(1-crho))/2;if sigma12^=0 then do;t1=((1-(rho##2))#(crho##2))/((1-(crho##2))#(rho##2));t2=(2#(crho##3)#(1-crho)#(lshift##2))/(rho#((1-(crho##2))##2));t3=((crho##4)#(lshift##4))/(2#(rho##2)#((1-(crho##2))##2));se_z=sqrt((t1+t2-t3)/(n-2));end;
else se_z=sqrt(2#sigma11#sigma22)/((sigma11+sigma22+((mu1-mu2)##2))#(n-2));t=tinv(0.975,n-2);
z_low=z-(se_z#t);z_upp=z+(se_z#t);crho_low=(exp(2#z_low)-1)/(exp(2#z_low)+1);crho_upp=(exp(2#z_upp)-1)/(exp(2#z_upp)+1);
Results=n//mu1//mu2//sigma11//sigma22//sigma12//rho_low//rho//rho_upp//crho_low//crho//crho_upp;r_name={'SampleSize' 'Mean_1' 'Mean_2' 'Variance_1' 'Variance_2' 'Covariance' 'Corr LowerCL'
'Corr' 'Corr UpperCL' 'ConcCorr LowerCL' 'ConcCorr' 'ConcCorr UpperCL'};print Results [rowname=r_name format=best6. L='Estimated Correlation and Concordance Correlation'];finish concor;

run concor(var1, var2);
Rick_SAS
SAS Super FREQ

@ChristosK Sorry, but this program was not written by me. It was posted by @PerNielsen. I do not know if he wrote it himself or downloaded it from somewhere else.  My only contribution was converting the PROC GPLOT code to PROC SGPLOT so that the graph could be generated in SAS University Edition.

HongyanLiu
Calcite | Level 5

Hi,

The link https://onlinecourses.science.psu.edu/stat509/node/161 cannot open anymore except with PSU ID.

Could anyone share  the codes?

Thank you!

 

Hongyan Liu

sas-innovate-2024.png

Join us for SAS Innovate April 16-19 at the Aria in Las Vegas. Bring the team and save big with our group pricing for a limited time only.

Pre-conference courses and tutorials are filling up fast and are always a sellout. Register today to reserve your seat.

 

Register now!

What is ANOVA?

ANOVA, or Analysis Of Variance, is used to compare the averages or means of two or more populations to better understand how they differ. Watch this tutorial for more.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 21 replies
  • 6876 views
  • 1 like
  • 7 in conversation