topic Re: Blank injection in a string in SAS Programming

Blank injection in a string

deleted_user — Fri, 30 Oct 2009 16:11:23 GMT

I have a weird thing happening, and am hoping another SAS expert, with more knowledge and experience than myself, can shed some light on the subject.

I have a data step that uses a call execute statement to "call" a macro.
The controlling dataset has hundreds of observations in it.
The last parameter in the macro call is a path to a directory, like \\localhost\folder1\dev\content\unix_servers\daily\loc-servername\interfacename.

A sample of the "call execute" is

call execute('%superq(make_chart)(combined,' ||"'"|| s_where ||"'"|| ', inpkts, in_packets, ' || s_title || ' , ' || base_path || ')' );

it is the "base_path" that is being corrupted.

I have 4 of these data steps in the program.
I get three errors, which are caused by a blank suddenly appearing in the path.
1) \\localhost\folder1\dev\content\ unix_servers\daily\loc-servername\interfacename
2) \\localhost\folder1\dev\content\unix_servers\daily\ loc-servername\interfacename
3) \\localhost\folder1\dev\content\unix_servers\daily\loc- servername\intervacename
Each occurs only once in one of the 4 data steps.
the observation immediately before the error and the observation immediately following the error do not contain the injected blank.
The blank is inserted, it is not an overwrite.

Any clues? Message was edited by: Chuck

Re: Blank injection in a string

deleted_user — Fri, 30 Oct 2009 18:43:51 GMT

Is there a limit to the number stuff added after a data step by a call execute?

Re: Blank injection in a string

Patrick — Sat, 31 Oct 2009 06:53:14 GMT

Are you sure this blank is not already stored this way in "base_path " before you pass it as a macro parameter?

No clue why this might happen - but as UNIX paths never contain blanks a compress(base_path) might help. May be you also might want to use some of the additional "options" and remove all whitespace characters.

HTH
Patrick

Re: Blank injection in a string

sbb — Sat, 31 Oct 2009 08:18:13 GMT

We have seen no code related to your program - no explanation about how variable base_path is constructed. Are you using FILENAME PIPE engine to get your information or is it stored in a SAS dataset to start? So, here's some code to add somewhere in your program to diagnose the imbedded blank condition:

IF INDEXC(TRIM(BASE_PATH),' ') THEN PUTLOG '>DIAG-GOT_BLANK>' / _ALL_;

Scott Barry
SBBWorks, Inc.

Re: Blank injection in a string

deleted_user — Sat, 31 Oct 2009 21:00:01 GMT

I am absolutely certain my code is not injecting the blank.

Re: Blank injection in a string

sbb — Sat, 31 Oct 2009 21:06:12 GMT

This is not voo-doo. So, did you run the recommended diagnostic that was suggested or not?

Scott Barry
SBBWorks, Inc.

Re: Blank injection in a string

deleted_user — Sat, 31 Oct 2009 21:10:03 GMT

[pre]
%let make_chart = %qsubstr('%make_chart',2,11);

data _null_;
length s_where s_title $64 test_path base_path $128;
set things;

test_path = "&base_path\" || trim(left(node_name)) ;
if not fileexist(test_path) then base_path = dcreate(node_name, "&base_path");
else base_path = test_path;
if base_path > " " then do;
test_path = cats(base_path,'\',intrf_name);
if not fileexist(test_path) then base_path = dcreate(intrf_name, base_path);
else base_path = test_path;
if base_path > " " then do;
s_where = 'node_name = "' || trim(left(node_name)) || '" and intrf_name = "' || trim(left(intrf_name)) || '"';
s_title = 'Out Packets for '|| trim(left(intrf_name)) || ' on ' || trim(left(node_name));
put base_path=;
call execute('%superq(make_chart)(combined,' ||"'"|| trim(s_where) ||"'"|| ', outpkts, out_packets, ' || trim(s_title) || ' , ' || base_path || ')' );
end;
end;
run;
quit;
[/pre]

Because of security restrictions, since this deals with our servers, I cannot share any log messages as that would reveal internal server names.

I can absolutely assure you that my code does not insert a blank.
The problem has to be with SAS itself, and I am suspecting some sort of internal buffer thing.

%Make_Chart is a simple macro that runs a Proc GPLOT.
The reason I am not using a BY statement in the GPLOT is that for each call, the resulting chart is placed in a different directory, and an HTML drilldown file is also created. Not using a BY statement simplifies the process of putting things where I want/need with the names that I want/need.

Re: Blank injection in a string

sbb — Sun, 01 Nov 2009 15:16:10 GMT

I recommend to declare a temporary SAS CHARACTER variable in your DATA step, perform the necessary assignment statements to build your CALL EXECUTE argument, and instead of invoking CALL EXECUTE with a series of concatenations, pass a SAS CHARACTER variable as the function invocation argument.

If you cannot share the exact SAS code and log output being executed, you're left to debug the condition mostly on your own, which I'd submit is a good opportunity to get the most out of learning about the SAS language for current and future code support and maintenance operations.

Scott Barry
SBBWorks, Inc.

Re: Blank injection in a string

Peter_C — Sun, 01 Nov 2009 23:43:29 GMT

not superq, but nrstr, and repeat the %, like
call execute('%nrstr(%%make_chart)(combined

Re: Blank injection in a string

deleted_user — Mon, 02 Nov 2009 13:07:10 GMT

I have four of these data steps in the program.
I have already done that last week.

[pre]
data _null_;
length s_where s_title $64 test_path base_path $128 statement $250;
set things;

test_path = "&base_path\" || trim(left(node_name)) ;
if not fileexist(test_path) then base_path = dcreate(node_name, "&base_path");
else base_path = test_path;
if base_path > " " then do;
test_path = cats(base_path,'\',intrf_name);
if not fileexist(test_path) then base_path = dcreate(intrf_name, base_path);
else base_path = test_path;
if base_path > " " then do;
s_where = 'node_name = "' || trim(left(node_name)) || '" and intrf_name = "' || trim(left(intrf_name)) || '"';
s_title = 'In Packets for '||trim(left(intrf_name)) || ' on ' || trim(left(node_name));
statement = cats( '%superq(make_chart)(combined,' , "'" , s_where , "'" , ',inpkts,in_packets,' , s_title , ',' , base_path , ')' );
call execute(statement);
end;
end;
run;
quit;
[/pre]

Apparently you don't believe me that the my code is not the problem.
I began programming in 1977.
I began SAS programming in 1996.
While it is true that I am always learning, this is not the first time that I have found problems in a vendor's product -- bugs in Novell, C compilers, OS's (Solaris) and within the last 12 months, some bugs in SAS itself (Proc Robustreg).

So my question is if my fellow expert SAS programmers have run into limitations involving call execute before, what were they, and how did they get around them.
I am not asking how to "debug" my program, because that is not the problem. I know how to debug programs. I am generally also debugging other people's programs in addition to my own -- I know I'm not perfect, never have been, never will be.

Re: Blank injection in a string

deleted_user — Mon, 02 Nov 2009 13:10:27 GMT

Why not superq?
It works.

I need to insure that all of the called macro runs after the data step, not within it.

I'll have to try your suggestion, though.
What would make that a better way?

Re: Blank injection in a string

sbb — Mon, 02 Nov 2009 13:53:05 GMT

Getting interesting - more code revealed this time. And, also, nothing personal, strictly feedback on technique with the SAS language in particular - you should be able to code the PUTLOG as demonstrated in this thread, place it just before the CALL EXECUTE and you should have what you need to answer the problem/question. For this thread, I'm stepping away from the mouse cursor - good luck.

Scott Barry
SBBWorks, Inc.

Re: Blank injection in a string

deleted_user — Mon, 02 Nov 2009 15:22:22 GMT

1) I don't understand your comment about "stepping away from the mouse cursor".

2) Your "PUTLOG" suggestion is worthless to me for this instance (although, I am putting it into my bag of tricks for the future). It simply detects if a blank appears in the base_path variable before the call execute, which I had already determined wasn't happening by simply placing "put baselog=;" immediately prior to the call execute. As I said, I know how to troubleshoot my own code, and that I was absolutely certain that my code was not causing the problem. I know for absolute certain that it has something to do with "call execute" itself, and probably the number of items it is injecting after the "run;" of the current data step.

3) I got an email response this morning, finally, from SAS support about issues with call execute.

Re: Blank injection in a string

sbb — Mon, 02 Nov 2009 16:00:59 GMT

Sounds like you have SAS support's input, finally. My other point is you have sufficient feedback, at least from my perspective, so I'm exitting the discussion.

Scott

Re: Blank injection in a string

deleted_user — Mon, 02 Nov 2009 16:46:20 GMT

Ok.
Thanks for trying.

So, you have not had any issues with call execute in your past experience?

Re: Blank injection in a string

sbb — Mon, 02 Nov 2009 17:30:24 GMT

Definitely no problem with constructing either a macro, PROC, or DATA step invocation data-string, and not with a SAS variable passed as an argument. With the most recent code you contributed, it appears you are using a mix of SAS data variables and macro variables, sometimes with functions to construct your literal argument being passed to the CALL EXECUTE. Another opportunity would be to output your code to a temp-allocated sequential file, and then %INCLUDE the allocated to invoke your code -- remember to end your main DATA step with a RUN; statement, then your %INCLUDE statement.

Scott Barry
SBBWorks, Inc.

Re: Blank injection in a string

deleted_user — Mon, 02 Nov 2009 19:11:42 GMT

I'm familiar with the %include technique.
We use that in some other programs.
I also use a form of threading in another one where the call execute calls a macro that issues a signon and rsubmit to run a set of processes on a remote server.

I am not having problems with building the string.
The string is correct in the data step, as evidenced by using a put statement in front of the call execute.
But, then, in the log, what SAS records as the generated statement, the blank insertion appears, and only once in > 600 observations. One time, the insertion was in a part of the string that had been hard coded between single quotes.

It's a real mystery.
SAS tech support is still not understanding/acceptng what I'm seeing.
Their first response was for a proc sql execute statement. eesh.
It would be nice if people would actually pay attention.

Re: Blank injection in a string

Peter_C — Mon, 02 Nov 2009 23:57:35 GMT

Chuck
I think it is a defect.
Do the generated code lines in the log, exceed 255 characters?
Where the MFILE option generates code wider than 255, a problem has occured in the past with null characters being generated at the "final" end of the "wrapped" line.
There is a minor overlap between MFILE and call execute code generation..
Of course, this is just guesswork

hope customer support comes up with the solution...

good luck
PeterC

Re: Blank injection in a string

deleted_user — Tue, 03 Nov 2009 13:31:08 GMT

The response back from customer support is that there is a hotfix/patch for a totally different problem that seems to also clear up some "call execute" issues as a side benefit.

I have put in a request with our people to obtain and apply the patch, but I'll have to way a number of days for the change management process to do its thing.

As a work around, per support's suggestion, I am going to us a temp file and %include to mimic the call execute affect. This is simple is all I have to do is

[pre]
* call execute(statement);
file tempfile;
put statement;
file log;
...
run;
quit;

%include tempfile;
[/pre]

For the four affected data steps.

Well, not quite that simple, I had to make some minor changes to the creation of statement as will, but nothing huge.

Thanks for the note Peter.C
The generated lline is < 255 characters, but, that doesn't mean that there isn't some sort of wrapping occurring.
The problem isn't showing up an every line, and some lines are longer than ones being corrupted.
I think it may have something to do with the total buffer space for something within SAS. My test run yesterday produced only one error, whereas last week, I was getting 3 of them. Message was edited by: Chuck

Re: Blank injection in a string

Flip — Wed, 04 Nov 2009 14:17:28 GMT

Call execute will do strange things as code reaches boundries in the editor. It is quite often better to construct the string ina temporary variable, then pass that variable to the Call Execute. It is also easier to diagnose since you can put the string to the log to insure it was constructed correctly.