Slowness in DI jobs

[chandru1dev]

Hi

We have developed some simple DI jobs. When we developed the jobs were fine and performed well even till some repeated testings. But suddenly few of the DI jobs are taking much longer time to run and finally hangs or fails giving the reason - insufficient disk space.  We checked for any unkilled sas sessions, but still this happens.

Do you have any suggestions to resolve this?

Thanks

[SASKiwi]

The fact that some of your jobs are failing due to insufficent disk space is a very big clue as to why your DI jobs are slowing down or failing. Which SAS libraries get the insufficient space error? 

If it is your WORK library check how much disk space is available in the SAS WORK folder(s) - your SAS logs will tell you where these are located. It is quite possible that SAS is not clearing out your WORK folders when jobs hang so you may need to delete these manually.

Your SAS WORK folder needs to be adequately sized to cope with your current workload so it is also possible it is not big enough and needs to be increased in size.

[chandru1dev]

Thanks for the suggestion, I am deleting the intermediate datasets and even this occurs, so wanted to check whether it could be something else.

Will try to add on additional space to work and check.

[jakarman]

When you are on Unix (Linux) you have mountpoints with Inodes.
You can run out of space on mountpoints (logical volumes). When it gets filled above 80% the performance can decrease rapidly.

The Inodes is an other nasty thing, Above a number files you can run of this type of resource. What you will see is having enough space. trying to create a new file however create some strange error message that can get be interpreted as this Inode resource. There are more system error possible.

Every process is storing some information in the <home> directory. When this one is running out of space you can also get into performance troubles.

All DI proceses are run by some account.

Check your system integrity at the OS level for possible causes.     

[Kurt_Bremser]

What kind of operating system is your SAS server?

Where are your libraries located physically (local disks, SAN)?

Modern filesystems like JFS2 will work quite well until space is exhausted (JFS2 can't run out of inodes, BTW).

[jakarman]

Thanks Kurt for the JSF2 note. It was AIX and I could not figure out which limit to blame. As there were some 30.000 logfiles being created I noticed that weird behavior. Cleaning collecting logfiles solved the issue.  Could be a bug a btree limit or whatever, it could be also on the SAN as there was no clear message just no function on creating files.
I would have liked to show this impossible behavior to you (real life experience).  :smileyconfused:
I always believe someone telling me something impossible has happened until I have proven he is liar. 

An other time there was a volume dropped in the file systems, this was the answer I got when those storage guys "did something".         
This time the behavior was weird trying to copy one special file (they created) did cause a process hangup.

On a desktop I had a dying harddisk, it made some noise so we were convinced of a hardware failure.