Clinical analysis: PROC PSMATCH (SAS/STAT or SAS/OR) results differ between SAS 9.4 and Viya

[Stephanvd]

We are migrating an existing analysis from SAS 9.4 to SAS Viya. The user requirement is that results are reproduced 1-to-1 (clinical analysis).

The PS matching procedure does not reproduce exactly:

• The estimated propensity scores differ slightly (small numerical differences)
• In several cases the assigned Match ID differs

 I have attached a small test program. It is not the original code, but it does reproduce the differences.

My questions
1. Are others seeing comparable differences in the output of SAS/STAT or SAS/OR procedures when migrating to Viya?
2. How would you deal with such differences in a migration of clinical analyses to Viya, what do you consider acceptable, and how do you document or justify it?

Any experience or guidance is very welcome.

Appendix 1: psmatch_test_anon.sas
Appendix 2: compare between output test code, SAS PC (9.04.01M8, WX64) vs SAS Viya (V.04.00M0, LIN X64) 

[sbxkoenk]

So you notice very small discrepancies between PSMATCH statistical results in SAS 9.4 on PC (Microsoft) and SAS Viya 4 on a Unix/Linux server.

These discrepancies - if any - usually stem from hardware-level floating-point arithmetic differences. Or do you have any indication that there’s more to it than that?

See these links:

• Numeric Precision : SAS Help Center
• Numerical Accuracy in SAS Software : SAS Help Center
• 275-2011: Why .1 + .1 Might Not Equal .2 and Other Pitfalls of Floating-Point Arithmetic

When you compare output datasets with PROC COMPARE, 

• try METHOD=relative Criterion=0.000000001; for example
• or look at the fuzz option

BR,

Koen

[JuanS_OCS]

Hi @sbxkoenk, Koen, indeed.

How do you manage what is on an acceptable level within a clinical trial analysis? Do you have any experience around it?

Thanks in advance!
Best regards, Juan

[sbxkoenk]

Hello Juan,

I was seeking to find out whether there are any FDA and/or EMA regulatory guidelines around this topic, but I couldn't find any (maybe my investigation was too superficial !).

I also checked the CAMIS project (https://psiaims.github.io/CAMIS/) because it deals with discrepancies and conflicting results in (pharma) statistical analyses between different statistical software packages. I was hoping to find some FDA/EMA statements there on how to handle tiny differences, but that's not the case.

BR,
Koen