Significant Figure

[CCW]

We can use PROC FCMP to make user defined function to generate the significant figure result for a number in Character type. It's like a format through a function.

There are 3 functions. We use the first and second function in the third one. The last data step is for unit test.

******************************************************************************
  Significant Figure(digit) 
******************************************************************************;
* show a number in its significant figure form which could be widely used for 
  scientific number presentation ;
  
libname my "YOUR PHYSICAL LOCATION";

proc fcmp outlib=my.func.format;
  function rndgt(x, sf);
  * Round to the specified digit according to significant figure ;
    I=if x < 1 then 0 else 1;
    rd=int(log10(x)) - sf + I;
    return(round(x, 10**rd));
  endsub;
  
  function extsf(x) $30;
  * Extract significant digits from a number ;  
    attrib sfc length=$30;
    xc=compress(put(x, BEST30.)); /* Char. of x */
   * Eliminate the decimal point and leading 0's;
    sfc=prxchange("s/\.//", -1, xc); /* Eliminate decimal point */
    sfc=prxchange("s/0*([1-9][0-9]*)/$1/", -1, sfc); /* Eliminate leading 0's */
    return(sfc);
  endsub;
    
  function sigFig(x, sf) $30;
  * Show significant figure result as a Char. ;   
    attrib sfc length=$30;  
    * 1. Round number ;
    xr=rndgt(x, sf); /* Rounding result */
    xrc=compress(put(xr, BEST30.)); /* Char. of xr */
    * 2. Adding trailing 0's ;
    n0=sf - length(extsf(xr)); /* Number of adding 0's */
    if n0 > 0 then do; /* The case we need to add 0's */
      zeros=repeat("0", n0 - 1); /* zeros we add*/
      out=if index(xrc, ".") then cats(xrc, zeros)
        else cats(xrc, ".", zeros); /* For integer, adding decimal point */
    end;
    else out=xrc;
    return(out);
  endsub;
  
options cmplib=my.func;

* TEST ;
data one;
  length result $30.;
  input x;
  result=sigFig(x, 3);
  l=length(result);
  datalines;
0.012
0.01235
0.01234
0.0001
1.1
1.125
100.1
;

[RW9]

And the question is?

Now I personally strongly disagree with the use of fcmp.  There is an industry wide disease called code hiding, which causes unimaginable suffering to programmers.   It has long been that code would either not be passed back to the owning company, or that some code would be "hidden" in compiled macro libraries.  This is bad in several ways - first that code can neither be seen nor edited by the end user, hence can never be considered in terms of proper oversight.  Also, future versions of the software have made these libraries no longer function at all - 32bit versus 64bit.  This is a big problem where companies are responsible for the product long after studies have finished.  Finally the concept of hiding code, and thankfully this is starting to be addressed, is in itself very bad.  With the need to supply code to the FDA and such groups this will hopefully eradicate this practice and bring about a harmonious open source plain text code files world.

Now why do I mention this?  Well, macros can be quite easily picked up in code, by use of special characters, so reading through the code you can pick out parts which are hidden in these compiled libraries.  Fcmp however doesn't have that limitation, and that makes it far worse from a programmer point of view.  Take the code here, if 5-10 years down the line, some other programmer gets this code alone:

data one;
  length result $30.;
  input x;
  result=sigFig(x, 3);
  l=length(result);
  datalines;
0.012
0.01235
0.01234
0.0001
1.1
1.125
100.1
;
run;

Where will the user find the SAS provided documentation for the sigFig() function?  There is nothing to indicate its not a SAS provided function, nor (assuming compiled somewhere, not provided etc.) any way of understanding what that is doing.  Sure your name is pretty good in that it confers some information, but it could just as easily randomise the number and print a tweety bird to the log.

/rant, but you can see why compiled libraries and moreso proc fcmp are really bad for the open source, open practice world we should be moving towards.

[ecofred]

Dear RW9,

I stumble on your answer while looking if some restriction exists for the use of Proc FCMP in FDA submission (none so far).

And the 'strongly disagree with the use of fcmp' push the urge to post a reply.

But first, yes dumping compiled library as stand alone (may this be proc fcmp or macros) violates FDA requirements and open codes principle. no question here.

However I don't see the point against proc fcmp (only against compiled library of any kind) if the .sas files to compile the functions are provided in the same ways as the macro files.

About the 'easy macro pick up in code': with Proc Fcmp you just check for 'word(.*)', of course you get also all the SAS standard function, but the same applies for & or % when checking for macros.

And the documentation problem applies to any kind of poorly documented user abstraction (not relevant only for fcmp)

In your example, I changed 'sigFig' for '%sigFig' and could turn the conclusion to make it as relevant for macros

Where will the user find the SAS provided documentation for the %sigFig() macro? There is nothing to indicate its not a SAS provided macro.

So my question is: should your first sentence be read as:

... I personally strongly disagree with the use of fcmp compiled library ...

?

or is there any critics only relevant to proc FCMP and not to macros?

Some context maybe: most of the resistance I encountered against proc fcmp only goes in the direction 'It is not the SAS I am used to, so I won't use it'. I find it sad as a programmer, but also for the projects given what marvel proc fcmp can do (... even more when mixed with macros...).

Kind Regards,

frederic

[RW9]

First off, please refer to the date of posts when re-opening old posts.  I rarely visit the site currently and don't need emails about old topics.

In terms of your question, hopefully you never have to deal with this, however the recent switch from 32 to 64 bit systems highlighted how dangerous compiled catalogs are.  If you have items compiled into catalogs - in my case it was third party - and then had to switch to a new system, those catalogs no longer work.  As they were third party there was no source, and no documentation other than calls.  This rendered all of it totally unusable on the new system. 

I would also say the opinion is also in alignment with the open context that is becoming prevalent.  Ask yourself is there any reason to not use plain text code files?  Benefit/loss evaluation may work that you still want to go with compiled libraries.  Plain text, open, clean files are easier in many ways, in fact other than possibly speed.

This is not a "its new and I don't like it", in fact quite the opposite, compiled binary file formats are prevalent in the past, a quick example would be the move from proprietary XLS files to OpenOffice XML XLSX files, i.e. the new shiny way of doing things is plain open file formats.

Finally, macro is generally pretty easier to pull out from code, even more so if you have color coding editors as they should be higglighted different colors and if you keep macros in plain text files also then they are pretty easy to work with.  Compiling them and hiding the code however also falls under the same thinking as for fcmp.  Code hiding/obfuscation is bad.

And yes, this is my opionion, based on various roles where compiled catalogs/binary-proprietary files have been big issues, thus I avoid them and recommend to avoid them, although as with anything its always best to use the right tool for the task.  If speed is of the utmost as the expense of open source, then sure compiled would be the answer (perhaps big data for instance), but not for general use anymore than writing code directly in assembler.  Code should be open clean and easy to read for all programmers down the line.

[ecofred]

I considered that my post was clearly orienting the reply in the FCMP/macro topic leaving the 'compiled library behind', my bad?

I did check the post date, I also went to check your profile for more recent question/answer on your use of proc fcmp. I couldn't find any so yes despite the relative age I decided to post here. Fact is this forum is timeless and my July 2019 search pointed to your specific post. Now the choice to follow-up on a post, or not, is up to the author and if email to old topics annoys you, please un-tick the 'Email me when someone replies' options.

For these reason I'm just thankful that you answered a follow up on an old post as fast as you did.

I went through a compile library issue similar to the one you mentioned, and I know how it hurts. As mentioned in the 3rd sentence of my post, i agree with you on the drawbacks of compiled SAS library, without questions!

The 'it's new I don't like it' refers to critics I get when pointing to the use of proc fcmp (unrelated to 'compile', just plain proc fcmp as was the topic of the thread).

I was honestly curious and interested to hear from someone with the badge 'Grand Advisor' (check mine to see the gap in our experience) why he strongly disagree with the use of proc FCMP as you started your first post on the topic with:

Now I personally strongly disagree with the use of fcmp

Now, back on track: (yes opinion coming): My point is:

Functions will initially be defined in plain text, they can be delivered as plain text, and can be made transparent as plain text in the log.

So I found the good point you attributes to Macros should also be relevant to proc FCMP and wondered if there was more than that to disagree with the use of fcmp when they do the job.

Please fill free not to answer, but if you do, i have a request, let's leave the compiled topic aside (again: I just agree with you on this point), to focus on your opinion on why Macros present an advantage when FCMP can do the job. (I'm fine with opinions but they might be challenged).

Kind regard,

Frederic

[RW9]

FCMP: Function Compiler - this is used to extend the SAS language by providing compiled functions to further code.  Its only plus is that it can be better speed wise in processing data, so useful perhaps in big data where speed is essential.

Macro: Inbuilt text/find/replace system - this is used to create code which can be expanded by the pre-processor into full Base SAS code by use of special terminologies.

As we can see the two systems have similarities, but are quite different.

In terms of why I do not recommend fcmp:
1) It is a compiled to catalog language, that is the one main point of it and where the one key benefit of it is.  Ignoring that is not optional.

2) Macro code, whilst it can be compiled to a catalog (bad, code hiding really is bad), does not have to be, it is processed as part of the SAS subsystem called the pre-compiler, and macro does not get "compiled" it is simply a find/replace system whereby special character sets are picked up in the text and replaced or expanded per macro rules.

3) Users coming into SAS learn Base SAS and macro SAS as a minimum.  Therefore being able to pick out in code where & and % are and knowing that is macro is fundamental, anything else is assumed to be part of Base SAS.  Therefore creating your own functions - unless they are named specially - will simply confuse users, why can I not find documentation on the function printl on SAS's website?

4) As yet I have not seen any "special" benefits of using compiled functions other than on special niche circumstances.  Therefore why use it?  Base SAS can do almost all tasks (in general faster and more efficiently than user code), and where that is lacking then simple macro can be used.  Where I see macro being used a lot is to cover up badly modelled data or a lack of knowledge of Base SAS (excluding SAS atttempts to macrotize everything in their systems as well now, but that is another topic).  This does not need to be compounded by further levels of badly written, unsupported, hidden code.

So as previously posted, I have no real benefits of using fcmp (except in niche scenarios) that Base or Base + Macro do not cover, and plenty of downfalls, hence the conclusion for me is to not recommend it.

[ecofred]

FCMP: Function Compiler - this is used to extend the SAS language by providing compiled functions to further code. Its only plus is that it can be better speed wise in processing data, so useful perhaps in big data where speed is essential.

Yes 'Compiler' it is. But FCMP pushes the function to a data-set with plain text (outlib=) and not the 'obfuscate my code to make it unreadable' kind. I can't check the %macro() as easily with a so simple proc print on 'outlib' data-set. The FCMP Documentation stresses more than once that function => data in SAS data set..

We can share the source file creating the FCMP functions (as we do for macros), Therefore we have cross-platform guarantee if we follow the rules.

Never tried to dump/load the data-set on an other platform. The SAS documentation goes full length on the 'save' part, but not much on the 'load/share' part. How would you just dump a %macro() without the source file?

If under proc FCMP, you call platform specific compiled libraries (like C) now yes this is the 'bad bloat' trap, not because of FCMP but  the choices made/imposed on the users.
Side note: I wouldn't be surprised that standard function (e.g. length ) call platform specific binaries 🙂

Macro: Inbuilt text/find/replace system - this is used to create code which can be expanded by the pre-processor into full Base SAS code by use of special terminologies.

Here, the 'expand' phases are indeed different (is the word 'expand' even relevant for FCMP functions?). I need to first let the data speaks (or load them in PDV if you prefer) and then choose a dynamic path based on the data (more post-processor way). The use of %macro() pre-processor is just to often getting in my way by freezing the decision path to early.
For almost all the PROC options, functions are not an options and one must rely on macros… or wait… I can't resist:

PROC FCMP outlib=work.fun.fun;
    Function printds5(x $) $;
        Length sascode $ 200;
        sascode = catb('proc print data=', x, '(obs=5); run;');
       Return (sascode);
    endsub;
run;
option append=(cmplib=work.fun);

data _null_;
    set sashelp.vcolumn(obs=10 where=(libname='SASHELP'));
    libds = catx('.', libname,memname);
    rc = dosubl(printds5(libds));
run;

proc print data=work.fun;
run;

FCMP as an alternative non macro SAS code pre-processor !

(dummy example to make a point)

1) It is a compiled to catalog language, that is the one main point of it and where the one key benefit of it is. Ignoring that is not optional.

Again, from the documentation it is saved as a standard SAS data-set, the 'language catalog' is SAS. That is quite powerful: you can query your function data-set with PROCs ( meta funky ).

2) Macro code, whilst it can be compiled to a catalog (bad, code hiding really is bad), does not have to be, it is processed as part of the SAS subsystem called the pre-compiler, and macro does not get "compiled" it is simply a find/replace system whereby special character sets are picked up in the text and replaced or expanded per macro rules.

That is one of my issues with macros. One level & or % is almost always fine, but passed this point you have to resort on context dependent use of &&&myvar. or %str, %nstr. You can trust the user to go past your defenses, add a new pre-process cycle and find a blind spot. That is to me obfuscation and maintenance bargain. But sometimes, in SAS, there's no way around of course.

3) Users coming into SAS learn Base SAS and macro SAS as a minimum. Therefore being able to pick out in code where & and % are and knowing that is macro is fundamental, anything else is assumed to be part of Base SAS. Therefore creating your own functions - unless they are named specially - will simply confuse users, why can I not find documentation on the function printl on SAS's website?

Well well. user defined macros surely don't have any SAS documentation neither. Or am I missing something?
I agree that 'write your own macro' comes before the 'write your own function'. That is a fact.

However the application of function (length, trim, max, date…) comes before in beginner SAS curriculum: let spot the '()' in DATA STEPs.

Out of the 'Little SAS book':

• Write
  • %macro(): p209
  • function: ???? (nooooothing)
•  Application:
  • function: p70 (first!!!)
  • %macro(): p209 (waaaaaay after)

I tend to think that function application is more intuitive and don't need much attention. With %macros() you have to be careful and take the time to learn the many pitfalls.

4) As yet I have not seen any "special" benefits of using compiled functions other than on special niche circumstances. Therefore why use it? Base SAS can do almost all tasks (in general faster and more efficiently than user code), and where that is lacking then simple macro can be used. Where I see macro being used a lot is to cover up badly modelled data or a lack of knowledge of Base SAS (excluding SAS atttempts to macrotize everything in their systems as well now, but that is another topic). This does not need to be compounded by further levels of badly written, unsupported, hidden code.

My special benefit for FCMP function is: run 'proc' and pass value at Data Step time: the power of late binding in the almighty DATA STEP, that is where my data are. I find it super powerful mega plus. Or recursion. Or use of hash-table or... (no need to quote the FCMP SAS Documentation).

Badly written code/decision is on the side of the user, not FCMP. You surely have seen badly written %macro().

So as previously posted, I have no real benefits of using fcmp (except in niche scenarios) that Base or Base + Macro do not cover, and plenty of downfalls, hence the conclusion for me is to not recommend it.

And so my conclusion diverge from yours, and they can coexists.

I would recommend it more often than not, PROC FCMP is a BASE SAS procedure after all.

[PaigeMiller]

Code to compute significant digits needs to be hidden? I could understand somehwat if you had some proprietary algorithm that you want to hide ... but significant digits??

Anyway, there are much simpler (in fact, 1 line of code) ways to compute significant digits. For example, message 8 here: https://communities.sas.com/t5/Base-SAS-Programming/counting-significant-digits/td-p/90609

[Reeza]

@CCW if you're trying to share resources, you can post your functions in the Library, rather than as a forum. The forum is primarily for questions. 

[CCW]

Thank you! I've contacted the Admin to let me share materials in Library.