bsub comand, LSF issue

[SvetlanaP]

I have problem when I am executing the STP to run on LSF using the command “bsub”. It looks like some credential are changing for myuser. (and becomes storage issue apparently).

I execut bsub and the name of the store process and if the work tables in that storeed process are small I get log file and completet program. If the work tables are bigger, the program stops without writhing in the log file.

If I put some inputs that I have found on the IBM web site I get ERROR in my log file. here is what I write in the cod:

bsub -C 500MB -M 1G -S 1TB -F 1M -R "rusage[mem=512MB:swp=1GB:tmp=1TB]"

I have tested and played with the vaules here but I didn't suceed. 

I need help using this command "bsub", so it will not change the setings for my user on the server.

[Anand_V]

Hi @SvetlanaP 

Work is a SAS option and you should set it at SAS Configuration level. Using LSF properties to control resources can become fairly complex at times. You have mentioned that when the work tables are small, STP is working fine so it could be that you are running out of space on the assigned work library. You should raise this concern to your site administrator to help you allocate a different work location.

If you have access to login to the server you can set this library dynamically while invoking a SAS session from command line by providing the -work '<your-file-system-path>' option.

Ex: ..../SASFoundation/9.4/sas -sysin MySASCode.sas -work /path-to-my-filesystem

Hope this helps. Also, it would be great if you can post some errors you are facing to help us understand your issue better.

[SvetlanaP]

Hi,

This user is updating bilion rows evry night and it newer been problem with the work space. 

If I use only bsub , tho job just stops without writing in the log.

If I use bsub -C 500MB -M 1G -S 1TB -F 1M -R "rusage[mem=512MB:swp=1GB:tmp=1TB]" Than I get the ERROR in the log:

"ERROR: Insufficient space in file WORK.QUERY_FOR_.DATA"

I have plaeyed with all of this -C -W -M -S -F and If I set too big the job is pendinging and It newer starts.

The sheduled DI job calls a script like this:

'bsub "sh /opt/....../send_to_lsf.sh "SOME PARAMETARS WHERE THE STP's are" "';

and the script is doing following

/opt/.../batch.sh -log /opt/..../stp_#Y.#m.#d_#H.#M.#s.log -batch -noterminal -logparm "rollover=session" -sysin /opt/.../executestp.sas -sysparm "$1"

[Anand_V]

I really don't get it what are you trying to convey. The best I could understand is, when a user is submitting a job via normal SAS Session, it gets executed without any space issues.

However when you submit it via bsub, it just fails.
And when you submit it via bsub along with a bunch of parameters as you have shared, it fails with error related to insufficient space in work?

Can you share the exact command you are trying to execute using bsub and the parameters?

Also I can see that you busb command with parameters is exactly the same as example provided in the man page for bsub. It is also mentioned before this example that the conversion is not supported for tmp resource by the LSF_UNIT_FOR_LIMITS unless few changes are done. Have you made those changes?

[SvetlanaP]

yes is I run the same STP with the same user on the same mashin with proc STP comand works fine.

the idea using bsub is to Control the STPs That they don't run in the same time sing the Flow Manager have  queue system

If I use only -F for exaple and if the work table is too big, than it fails if I use -F 1000 (and no info in the log), If I user -F 8000 (I get ERROR in the log on the first wrok table in the cod), If I incerease this a lot than the job doesn't starts it is pending in the queue.

 bsub -F 1000 /opt/.../sbatch.sh -log /opt/..../STP.log  -batch -noterminal -logparm 'rollover=session' -sysin /opt/..../execute_user_stp_start.sas -sysparm '/Shared Data/......./NAME_OF_STP'

 bsub -F 8000 /opt/.../sbatch.sh -log /opt/..../STP.log  -batch -noterminal -logparm 'rollover=session' -sysin /opt/..../execute_user_stp_start.sas -sysparm '/Shared Data/......./NAME_OF_STP'

My problem is That I don't know how the credentiasl change with bsub.

to answer to your question regarding LSF_UNIT_FOR_LIMITS I havent done any Changes.

[Anand_V]

My suggestion is to use the SAS Grid Manager client utility to submit batch jobs to Grid. You can find more information about it here -> https://go.documentation.sas.com/?docsetId=gridref&docsetTarget=p06nu8c3qifl8vn15ya1ghu97lku.htm&doc...

I haven't tried invoking sasbatch.sh directly using bsub, however for anything other than sas programs bsub would fail. As the LSF license that comes with SAS is OEM and NOT full version. It allows you to submit .sas files only. 

There is a good answer by Patrick explaining this here -> https://communities.sas.com/t5/SAS-Data-Management/Scheduling-Unix-Script-in-LSF/td-p/103904

[SvetlanaP]

Thank you for your answer. I will lokk into SAS Grid Manager, It is something new for me.

Bsub works perfect with the STP That doesn't have big work tables.

this job is sheduled in LSF and it failes only when there is big work tables.