Getting error "LOCK HELD BY PROCESS %D", but it's a new file, where is the lock coming from?

[Sven111]

I have a SAS job that runs every hour via cron to monitor various things.  The first major step that the SAS job does after declaring some LIBNAMES and setting up some macro vars is to create a new SAS dataset that imports the contents of a CSV file, the last thing the job does after various processing is delete this SAS dataset.  A few times a week however, the job fails while creating the dataset with the following:

ERROR: A LOCK IS NOT AVAILABLE FOR LIB_IN.SAMPLING.DATA.
ERROR: LOCK HELD BY PROCESS %D.
NOTE: THE SAS SYSTEM STOPPED PROCESSING THIS STEP BECAUSE OF ERRORS.
NOTE: SAS SET OPTION OBS=0 AND WILL CONTINUE TO CHECK STATEMENTS. THIS MIGHT CAUSE NOTE: NO OBSERVATIONS IN DATA SET.
WARNING: THE DATA SET LIB_IN.SAMPLING WAS ONLY PARTIALLY OPENED AND WILL NOT BE SAVED.

How can there be a lock on a file that is still in the process of being created?  Or is the lock really on the source CSV file despite the error text indicating otherwise?  What is "PROCESS %D"?  It looks like it's supposed to be doing some string interpolation there, but it's not working correctly.  I also checked the log file for the job that ran immediately prior to jobs that show this error and the log says the old version of the SAS dataset is being successfully deleted and there's nothing else unusual, so the lock doesn't appear to be being held from the previous iteration of the job.  There are no other jobs that would be accessing the CSV file or anything else in the directory specified in the LIBNAME statement.

Digging around the SAS help files I found the system or library option "FILELOCKWAIT", which defaults to zero.  Earlier today I added that option to the LIBREF statement with a wait time of 60, so I'll see if that makes a difference, but since the problem occurs randomly there's no real way to tell if it fixes it other than just waiting to see if the errors keep recurring.

Has anyone seen something like this before? Will the FILELOCKWAIT option help or is something else going on?

[snoopy369]

I think what you're probably running into is that the delete process occasionally doesn't fully delete. Something in the OS/filesystem/whatever is losing track of it, and so the lock persists.

One thing I'd recommend is if possible to use the work directory for something like this. If that's not possible (and I'd suggest it is _usually_ possible, even if you're sharing a dataset between processes), consider a dated dataset (where the name contains a date stamp). That way if something like this happens you don't have the risk of the process getting a lock.

[Sven111]

A dated filename (or even adding a random string to the dataset name) would definitely be feasible, although I'm still curious why the file wouldn't be fully deleted.  I could also move the file to the WORK directory, no other process is using the data I'm working with, but would that really resolve the problem?  If the dataset isn't being fully deleted why would not being fully deleted in the WORK directory make a difference?

In general I've always avoided using the WORK directory, I'm not sure why exactly, but I primarily use Libraries specific to the project I'm working on to keep all related files together and separate from other projects. Are there benefits to using the WORK directory over named libraries?

[snoopy369]

Moving it to work would resolve the problem assuming you have a new SAS session running each run(if this is running in batch, for example, as it sounds like it is). Each new SAS session has a different work directory, and each batch job starts a new SAS session.

The main benefit of work is just this: it is the place to put temporary files where you know they will be cleared for you, so you don't have to worry about it. You don't need to perform that delete, SAS will do so when your SAS session closes.  Work also may be in a location that's more appropriate to frequent read/writes, depending on how your administrator set up your filesystem and your SAS environment (i.e., it could be on faster storage that's not backed up).

As for why it's not deleting properly: especially with complicated or busy filesystems, sometimes things just get hung up. It doesn't happen to me often, but it's a more than once a month occurrence for sure.

[Kurt_Bremser]

Which platform are you working on, and what type of storage is the dataset in question on?

[Sven111]

It's on a Linux system, running SAS 9.4 M4.  The storage is over NFS to a spinning disk array for both the WORK and other libraries, although I believe the storage and SAS server are on the same rack with at least 2 10GB ethernet links between them.  There is a very limited amount of local disk storage using ext4.  We're in the process of upgrading to a new SAS server that'll have several TB of local, dedicated NVMe flash storage available though.

[Kurt_Bremser]

In that case, I'd remove the file with the OS. On UNIX systems, unlinking a file from a directory is possible even when the file is in use.

%macro remove_dataset(dsname);
data _null_;
path = pathname(scan("&dsname",1,'.'));
call symput('ds_filename',trim(path) !! '/' !! lowcase(scan("&dsname",2,'.')) !! '.sas7bdat');
run;

filename oscmd pipe "rm -f &ds_filename 2>&1";

data _null_;
infile oscmd;
input;
put _infile_;
run;

filename oscmd clear;

%mend;

or something like that.

[Kurt_Bremser]

BTW temporary datasets that don't need to exist after a job is run should always be created in work, and not on some remote storage. You are incurring the problems intrinsic to network shares, while you don't need their benefits (stability, RAID system, automated backups).

[Sven111]

I checked this and our WORK default location still points to a network share.  There's only like 35GB of local storage on the server, most of which either OS or home directories, so even if I overrode the default directory it'd likely do more harm than good since many of my datasets are larger than that.  Our new server will have several TB of local storage though.

[PopCorn14]

I know at one point we had encountered this issue when another process exceptionally was trying to access the same file that we were trying to copy.

We just wanted to copy to file anyways so we circumvent the problem this way:

lock &lib.&outTableName;

/* If a lock cannot be acquired then delete the locked file anyways to be able to copy the file. */

%if (&SYSLCKRC ne 0) %then %do;

   lock &lib.&outTableName clear;

   %let pathL = %sysfunc(pathname(&lib,l));

   %let file=&pathL/%lowcase(&outTableName).sas7bdat;

   /* File exists and can be deleted.*/

   data _null_;

      fname="tempfile";

      rc=filename(fname,"&file");

      if rc = 0 then

         rc2=fdelete(fname);

      rc=filename(fname);

   run;

%end; /* if dataset already locked. */

lock &lib.&outTableName clear;