topic Re: Output Directory in Batch Server properties in Administration and Deployment

Output Directory in Batch Server properties

jklaverstijn — Fri, 25 Mar 2016 10:19:44 GMT

Hi,

Since 9.4M3 (I think) there is a new entry field in the properties of the DATA Step Batch Server in tyhe Management Console. Below the Logs Directory we now have one for Output Directory.

I can imagine what this field can do (in fact have been hoping for this a long time) but I cannot see how it works out. The online docs and the in-app help do not mention the field. As far as I can tell it is not used when calling SAS. I am on RHEL7 Linux.

Many thanks in advance for you thoughts,

- Jan.

Re: Output Directory in Batch Server properties

JuanS_OCS — Fri, 25 Mar 2016 10:24:31 GMT

Hi @jklaverstijn,

this directory stands out for the default location of lst files generated by batch processes (proc print).

The .LST file is the output file for the program that has been executed. This file is created only if there is output for the program.

Before this, all the logs directory was filled with lst files, by default. Now it is possible to define a different location for those files, which can be useful as they can be used for controlling the batch processes and reporting.

I hope this can help a bit 🙂

Re: Output Directory in Batch Server properties

jklaverstijn — Fri, 25 Mar 2016 11:10:57 GMT

Hi @JuanS_OCS,

Thanks for chiming in. That does make sense and it is (almost) exactly what I was hoping for. However, the expected does not happen. At least not for me. I have set it up as in the image but a job containing a proc print fails with

ERROR: Insufficient authorization to access /srv/SASConfig/Lev1/SASApp/Options_Libnames.lst.

So SAS tries to stick the .lst file in the root of the batch server config, ignoring the setting in the metadata. Also the command given to LSF to be execute does not contain the metadata setting. I would have expected a -print parameter.

I have taken care of .lst file locations many times, mostly by editing the sasbatch.sh script, but was hoping for things to become more automatic in 9.4M3. So I must be doing something wrong or missing something here. But I don't see it.

- Jan.

Re: Output Directory in Batch Server properties

LinusH — Fri, 25 Mar 2016 21:01:19 GMT

Have you redeployed your jobs? Since the batch server doesn't call the meta data server during execution.

Re: Output Directory in Batch Server properties

jklaverstijn — Sat, 26 Mar 2016 12:50:38 GMT

Thanks @LinusH. Yes I have. Many times.

The batch server is in fact not even a real server. It is a fancy way of defining the command line that's sent to the OS, in my case via LSF. The command is determined it two stages: first from metadata when deploying (eg from DI Studio) and later by adding the job to a flow. I have noticed that the latter tweaks the log line by adding the names of flow user to the logfile name. I have not added logconfigloc so no log4sas.

In all this I see no sign of adding or handling procedure output via the -print parameter, as the new server metadata property would suggest.

I hope for someone from the forum to shed some light in this. If not I will stalk the developer(s) at SGF 😉

- Jan.

Re: Output Directory in Batch Server properties

jakarman — Mon, 28 Mar 2016 10:00:30 GMT

Jan. You are the question "what is an application". Please try to get one correct asap. It is very basic business process organisation.
I am having for you:

++ http://support.sas.com/publishing/pubcat/chaps/58172.pdf Tuning SAS Applications in the OS/390 and z/OS Environments. " What Is a SAS Application? The word application, as used in this book, is defined as one or more related SAS language programs that perform a specific function for an organization on a cyclic basis."
++ https://en.wikipedia.org/wiki/Application_software "An application program (app or application for short) is a computer program designed to perform a group of coordinated functions, tasks, or activities for the benefit of the user. "

Perhaps better to indicate it as https://en.wikipedia.org/wiki/Business_logic

- logic tier = code , the business sas programs (also map schedule informatie etc)

- data tier = data , the business data-sets (SAS-datasets Excel TD Oracle or whatever )

- presentation layer = results (lst or other like pdf/html/ods catalogs)

Very classic but not less true. It is te Foundation for ICT services when adding scheduling SLA's with technical performance.

After that you find the business goal is lost. All technical references are making the tools / utilities the leading subject (eg the SAS System) ignoring business requiements and business semantics. What are those business appplication in their world?
Examples of SAS applications can be: IH OB DM (just abbrevations for used business logic)

It is getting data in applying some business logic and getting results with logs in a scheduled way. (4 categories). Those are commonly build on the Develop/Test/Acceptance/Production stages (4 categories).

Now the question: "What is a SAS Appserver about.."

1/ Is it for the sake of an IT-department delivering the SAS-system as tool?

..... (won't proceed wiht this one)

2/ Is for fullfilling the business needs like OB IH DM?

..... this one makes the most sense for getting aligned with the business.
Yup some more work to do for IT departments. Can be automatized that is what it professionals can do.

Suppose you have those three businsess applications and some adhoc work how to organize storage locations?

You probably know this one: http://support.sas.com/kb/50/345.html The often mentioned bypass is hardcoding phyiscal names into business logic code. The result is frustrating the requirement of release management process.
You can solve it all by having four appservers (adhoc and three business logic) app-servers.

The destination of the logs/results/source repositories van be tailored as of each app-server. The batch server script can be aligned wiht that according to needed scheduling and security requiements.

Re: Output Directory in Batch Server properties

Resa — Tue, 29 Mar 2016 08:01:33 GMT

Hi ja karman,

Regardless of the fact whether what you state is relevant or not I fail to see the relation of your reply to the question of the OP.

IMHO the OP indicates that the new option that is currently available for the batch server, that is to redirect output (.lst files) to a location of your own choosing, does not seem to live up to expectations.

Basically the OP would like to know if he is overlooking something causing it not to work for him.

I have seen the same option and also for me it is not working yet.

So if anyone has find the solution I am also curious to hear it. 😉

Re: Output Directory in Batch Server properties

jakarman — Tue, 29 Mar 2016 11:24:55 GMT

Resa, there is some more background known than you can see following this thread question.
Those business applications OB DM IH are real ones aside of that adhoc to come a live in the infrastructure Jan is helping to deliver.
That one of the current directory ws is a real experienced one in that environment (temp files). Adding an option in the batch server for lst redirection is a technical one a previous described. I am answering the functional requirements. There is difference between technical working and functionality within business.
As weird it is. Do not use spaces in namings. Unix and Windows are allowing that, but the scripting by SAS fails.

Re: Output Directory in Batch Server properties

Resa — Tue, 29 Mar 2016 11:43:09 GMT

Hi ja karman,

The fact that there may be more background within a project you are working on may very well be the case.

However your information is only taking this thread off-topic. Your technical and/or functional requirements may be different to what the feature provides but that is a topic for discussion inside your project. This thread is about a feature currently available.

IMHO the question as stated by the OP is simple:

There is seemingly a new feature in the SMC for the batch server and it is not working for him.

All the OP is interested in (and so am I) is how to get this feature working or, if misinterpreted, what is the actual meaning of the new feature.

Re: Output Directory in Batch Server properties

jakarman — Tue, 29 Mar 2016 12:13:35 GMT

Resa, did you verify not having spaces in the namings somewhere? This is a problem with the batch server script still existing. It is caused by the logging settings.
The lst file also did cause issues as you must have full write access to the current directory.
Both issues are solved in our project with Jan. The lst is a nasty one. You have to have write access to the current map even if you do not create one. They are hardly used arround here and when it are functional logging.

I think when you take care setting the current dir as stated by that other note your batch script will work.

Re: Output Directory in Batch Server properties

jklaverstijn — Wed, 30 Mar 2016 09:12:53 GMT

Sorry @jakarman but your assumptious about what is being asked here are completey wrong. We are trying to get an answer to one single question: what is the usage of the Output Directory property of the Batch Server. Several things go wrong in your line of answering that result in total derailment of the thread that make me afraid I will not get an answer at all:

1) You assume a different question should be asked and answer the question you think should be asked. Your assumption is wrong and thus your answer.
2) Your answer is lengthy and its language confusing and inconcistent. You lost me at the second sentence.
3) You assume local knowledge from the workfloor ("OB DM IH ...") that we share to be the basis of my question. This assumption is also wrong. Otherwise approach me on that workfloor and not on the forum.
4) You refer to local knowledge that no one will be able to comprehend. Wrong again.
5) You target @Resa based on an incorrect assumption of what I am asking. You also incorrectly assume Resa has some batch thing that is not working. He doesn't, nor have I.

How can answering a simple question go wrong in so many ways?

Another simple question to you: will you please never answer any of my questions?

Regards, Jan.

Re: Output Directory in Batch Server properties

PaulHomes — Wed, 30 Mar 2016 11:13:17 GMT

Hi Jan,

I also think it's being ignored. I tried it out with a SAS 9.4 M3 install on Linux deploying and scheduling a trivial single job flow. Looking at the command line within SAS Management Console, whilst the -log parameter is specified, there was no -print parameter for controlling the output file location. This was confirmed as absent by looking in the LSF logs too. This particular job then failed because it was trying to write a .lst file into the main SASApp dir where the user had no permissions (instead of the specified SASApp/BatchServer/Output dir where they would have).

I'm wondering if this is a) a defect or b) an option that is only used by some products/solutions (or planned for future products)?. Perhaps SAS Tech Support might be able to advise?

Cheers

Paul

Re: Output Directory in Batch Server properties

jklaverstijn — Wed, 30 Mar 2016 11:28:12 GMT

Thanks @PaulHomes for confirming my observations. And for putting the thread on track again ;-).

I am still pondering if I will report this to TS as it is not a clear bug. Also there is no immediate impact and we already have a long list of tracks. I am now consider addressing the issue at SGF in the Quad. And in that case we'll have plenty of opportunity to dicsuss the outcome.

Either way I will keep you posted.

Cheers Jan

Re: Output Directory in Batch Server properties

Resa — Tue, 10 May 2016 12:46:50 GMT

Hi Jan,

In your last post you indicated:

jklaverstijn wrote:

I am now consider addressing the issue at SGF in the Quad. And in that case we'll have plenty of opportunity to dicsuss the outcome.

Out of curiosity (although I know what this did to the cat) ....

Did you find time to address this at SGF?

And if so, any news on this item?

Re: Output Directory in Batch Server properties

jklaverstijn — Mon, 16 May 2016 13:45:51 GMT

Hi Resa,

Just got back from SGF and a subsequent vacation in SW USA. Still buzzing from the jetlag and all the impressions.

I talked to several guys in the data management booth and they were all stumped. General advise was to raise the issue with tech support, which I will do asap.

Cheers Jan.

Re: Output Directory in Batch Server properties

jklaverstijn — Fri, 27 May 2016 10:08:01 GMT

So I talked about this at the Data Management booth in the Quad and a few eyebrows were raised but noone had the answer. So they suggested to open a track with tech Support which I did. And lo and behold the answer came swift and caused a facepalm moment for me.

The Output Directory property of the Data Step Batch server works in concert with the "Send the output to a file" option of the Schedule Manager in SMC. If you right-click on the Schedule Manager plugin you get access to that option. So if that option is checked and you have a location specified for the batch server than the -print option is added to the SAS command that is composed when deploying the job. Easy as that. I did not find any reference to this option in the online or in-app help. When naming the .lst file the "Prefix the SAS log file with the flow name" option is also honoured. Just no timestamps or anything so output files will be overwritten.

So finally the riddle is solved. Thanks for all chiming in and I hope we all learned something.

Regards,

- Jan.

Re: Output Directory in Batch Server properties

JuanS_OCS — Fri, 27 May 2016 19:59:01 GMT

Hey Jan,

This question it turned out to be a much more difficult question to answer, than what you might expect.

Thanks a lot for the feedback, I just learned something good.

Best regards,

Juan

Re: Output Directory in Batch Server properties

PaulHomes — Thu, 24 Nov 2016 00:52:35 GMT

Jan, thanks for posting your solution. I encountered this problem today (no permission to write .lst file to SASApp dir) and after spending some time tracking down the issue to a missing -print option on the SAS command line, I remembered this thread, used the solution you posted and it fixed it. SAS Communities rocks! 🙂

Makes me wonder if that SAS Management Console Schedule Manager option should be turned on by default.

BTW I found that I had to remove and re-add the job to the flow to get the -print option added. Did you have to do this too?

Re: Output Directory in Batch Server properties

jklaverstijn — Thu, 24 Nov 2016 12:44:32 GMT

Hi Paul,

Great that you found this helpful. I did have a few small issues after finding out the solution. What you refer to, the command line of the deployed job not being refreshed, was one of them.

It is not needed to remove and re-add a job (with the consequence of loosing all dependencies and triggers as well) or even redeploy. In SMC, open the flow and then open the Properties of the job you're looking for. In the Scheduling Details pane there is a Refresh button underneath the command line field. That button is supposedly available if clicking it would make sense (!). And if it is and you click on it the command line will be updated to include the -print option.

Another issue we found is the extension of the print filename. By default this is the trusted .lst. But if you visit that setting in SMC (it's in the batch server properties) and leave it unchanged or reset it to .lst the result is that the extension is entirely removed from the filename. Only setting this to something non-standard like .list will make it stick. Very nasty little bug. In the end we resorted to some metadata trickery in a datastep to force the situation so it gives us .lst again.

I hope this will help you, Paul. i consider this a great opportunity to give something back where I have received so much from you.

Kind regards,

Jan.

Re: Output Directory in Batch Server properties

PaulHomes — Fri, 25 Nov 2016 02:11:05 GMT

Thanks for those extra tips Jan - much appreciated!

Regarding the Reset button in the Scheduling Details tab, unfortunately it seems to be permanently greyed out for me. I was hoping that the change in Schedule Manager options would have been dynamically reflected in the command line, or that I could press that Reset button to trigger the update. Did you remember if you did anything special to enable it? I found the button wasn't enabled even after I toggled the Schedule Manager "Send the output to file" option. Not knowing what I had to do to get it enabled, I went with the remove and re-add the job technique. I was lucky in that the flow was very simple and so it was easy to do, but I definitely see what you mean about that approach being undesirable for more complex flows 🙂

Cheers

Paul