Output Directory in Batch Server properties

[jklaverstijn]

Hi,

Since 9.4M3 (I think) there is a new entry field in the properties of the DATA Step Batch Server in tyhe Management Console. Below the Logs Directory we now have one for Output Directory.

I can imagine what this field can do (in fact have been hoping for this a long time) but I cannot see how it works out. The online docs and the in-app help do not mention the field. As far as I can tell it is not used when calling SAS. I am on RHEL7 Linux.

Many thanks in advance for you thoughts,

- Jan.

[JuanS_OCS]

Hi @jklaverstijn,

this directory stands out for the default location of lst files generated by batch processes (proc print).

The .LST file is the output file for the program that has been executed. This file is created only if there is output for the program.

Before this, all the logs directory was filled with lst files, by default. Now it is possible to define a different location for those files, which can be useful as they can be used for controlling the batch processes and reporting.

I hope this can help a bit 🙂

[jklaverstijn]

Hi @JuanS_OCS,

Thanks for chiming in. That does make sense and it is (almost) exactly what I was hoping for. However, the expected does not happen. At least not for me. I have set it up as in the image but a job containing a proc print fails with

ERROR: Insufficient authorization to access /srv/SASConfig/Lev1/SASApp/Options_Libnames.lst.

So SAS tries to stick the .lst file in the root of the batch server config, ignoring the setting in the metadata. Also the command given to LSF to be execute does not contain the metadata setting. I would have expected a -print parameter.

I have taken care of .lst file locations many times, mostly by editing the sasbatch.sh script, but was hoping for things to become more automatic in 9.4M3. So I must be doing something wrong or missing something here. But I don't see it. 

- Jan.

[kmcnulty]

Could you be more specific about what you change in the file to get it to work?

[LinusH]

Have you redeployed your jobs? Since the batch server doesn't call the meta data server during execution.

[jklaverstijn]

Thanks @LinusH. Yes I have. Many times.

The batch server is in fact not even a real server. It is a fancy way of defining the command line that's sent to the OS, in my case via LSF. The command is determined it two stages: first from metadata when deploying (eg from DI Studio) and later by adding the job to a flow. I have noticed that the latter tweaks the log line by adding the names of flow user to the logfile name. I have not added logconfigloc so no log4sas.

In all this I see no sign of adding or handling procedure output via the -print parameter, as the new server metadata property would suggest.

I hope for someone from the forum to shed some light in this. If not I will stalk the developer(s) at SGF 😉

- Jan.

[jakarman]

Jan. You are the question "what is an application".  Please try to get one correct asap. It is very basic business process organisation.  
I am having for you:

++ http://support.sas.com/publishing/pubcat/chaps/58172.pdf   Tuning SAS Applications in the OS/390 and z/OS Environments.   "  What Is a SAS Application?   The word application, as used in this book, is defined as one or more related SAS language programs that perform a specific function for an organization on a cyclic basis."
++ https://en.wikipedia.org/wiki/Application_software "An application program (app or application for short) is a computer program designed to perform a group of coordinated functions, tasks, or activities for the benefit of the user. "

Perhaps better to indicate it as https://en.wikipedia.org/wiki/Business_logic 

- logic tier  = code , the business sas programs (also map schedule informatie etc)

- data tier  = data  , the business data-sets (SAS-datasets Excel TD Oracle or whatever ) 

- presentation layer = results (lst or other like pdf/html/ods catalogs)

Very classic but not less true. It is te Foundation for ICT services when adding scheduling SLA's with technical performance.

After that you find the business goal is lost. All technical references are making the tools / utilities the leading subject (eg the SAS System) ignoring business requiements and business semantics. What are those business appplication in their world?
Examples of SAS applications can be: IH OB DM     (just abbrevations for used business logic)  

It is getting data in applying some business logic and getting results with logs in a scheduled way. (4 categories). Those are commonly build on the Develop/Test/Acceptance/Production stages (4 categories). 

Now the question: "What is a SAS Appserver about.."

 1/ Is it for the sake of an IT-department delivering the SAS-system as tool? 

     .....  (won't proceed wiht this one) 

 2/ Is for fullfilling the business needs like OB IH DM? 

     .....  this one makes the most sense for getting aligned with the business.
   Yup some more work to do for IT departments. Can be automatized that is what it professionals can do. 

Suppose you have those three businsess applications and some adhoc work how to organize storage locations?

You probably know this one: http://support.sas.com/kb/50/345.html The often mentioned bypass is hardcoding phyiscal names into business logic code. The result is frustrating the requirement of release management process.
You can solve it all by having four appservers (adhoc and three business logic) app-servers.

The destination of the logs/results/source repositories van be tailored as of each app-server. The batch server script can be aligned wiht that according to needed scheduling and security requiements.

[Resa]

Hi ja karman,

Regardless of the fact whether what you state is relevant or not I fail to see the relation of your reply to the question of the OP.

IMHO the OP indicates that the new option that is currently available for the batch server, that is to redirect output (.lst files) to a location of your own choosing, does not seem to live up to expectations.

Basically the OP would like to know if he is overlooking something causing it not to work for him.

I have seen the same option and also for me it is not working yet.

So if anyone has find the solution I am also curious to hear it.   😉

[jakarman]

Resa, there is some more background known than you can see following this thread question.
Those business applications OB DM IH are real ones aside of that adhoc to come a live in the infrastructure Jan is helping to deliver.
That one of the current directory ws is a real experienced one in that environment (temp files). Adding an option in the batch server for lst redirection is a technical one a previous described. I am answering the functional requirements. There is difference between technical working and functionality within business.
As weird it is. Do not use spaces in namings. Unix and Windows are allowing that, but the scripting by SAS fails.

[Resa]

Hi ja karman,

The fact that there may be more background within a project you are working on may very well be the case.

However your information is only taking this thread off-topic. Your technical and/or functional requirements may be different to what the feature provides but that is a topic for discussion inside your project. This thread is about a feature currently available.

IMHO the question as stated by the OP is simple:

There is seemingly a new feature in the SMC for the batch server and it is not working for him.

All the OP is interested in (and so am I) is how to get this feature working or, if misinterpreted, what is the actual meaning of the new feature.

[jakarman]

Resa, did you verify not having spaces in the namings somewhere? This is a problem with the batch server script still existing. It is caused by the logging settings.
The lst file also did cause issues as you must have full write access to the current directory.
Both issues are solved in our project with Jan. The lst is a nasty one. You have to have write access to the current map even if you do not create one. They are hardly used arround here and when it are functional logging.

I think when you take care setting the current dir as stated by that other note your batch script will work.

[jklaverstijn]

Sorry @jakarman but your assumptious about what is being asked here are completey wrong. We are trying to get an answer to one single question: what is the usage of the Output Directory property of the Batch Server. Several things go wrong in your line of answering that result in total derailment of the thread that make me afraid I will not get an answer at all:

1) You assume a different question should be asked and answer the question you think should be asked. Your assumption is wrong and thus your answer.
2) Your answer is lengthy and its language confusing and inconcistent. You lost me at the second sentence.
3) You assume local knowledge from the workfloor ("OB DM IH ...") that we share to be the basis of my question. This assumption is also wrong. Otherwise approach me on that workfloor and not on the forum.
4) You refer to local knowledge that no one will be able to comprehend. Wrong again.
5) You target @Resa based on an incorrect assumption of what I am asking. You also incorrectly assume Resa has some batch thing that is not working. He doesn't, nor have I.

How can answering a simple question go wrong in so many ways?

Another simple question to you: will you please never answer any of my questions?

Regards, Jan.

[PaulHomes]

Hi Jan,

I also think it's being ignored. I tried it out with a SAS 9.4 M3 install on Linux deploying and scheduling a trivial single job flow. Looking at the command line within SAS Management Console, whilst the -log parameter is specified, there was no -print parameter for controlling the output file location. This was confirmed as absent by looking in the LSF logs too. This particular job then failed because it was trying to write a .lst file into the main SASApp dir where the user had no permissions (instead of the specified SASApp/BatchServer/Output dir where they would have).

I'm wondering if this is a) a defect or b) an option that is only used by some products/solutions (or planned for future products)?. Perhaps SAS Tech Support might be able to advise?

Cheers

Paul

[jklaverstijn]

Thanks @PaulHomes for confirming my observations. And for putting the thread on track again ;-).

I am still pondering if I will report this to TS as it is not a clear bug. Also there is no immediate impact and we already have a long list of tracks. I am now consider addressing the issue at SGF in the Quad. And in that case we'll have plenty of opportunity to dicsuss the outcome.

Either way I will keep you posted.

Cheers Jan

[Resa]

Hi Jan,

In your last post you indicated:

jklaverstijn wrote:

I am now consider addressing the issue at SGF in the Quad. And in that case we'll have plenty of opportunity to dicsuss the outcome.

Out of curiosity (although I know what this did to the cat) ....

Did you find time to address this at SGF?

And if so, any news on this item?