Stacking data of 10 years

[Kurt_Bremser]

One of the "shortcomings" of proc append is that it fails when the base dataset is opened and can't be locked. Therefore I avoid append in batch jobs, instead combining the datasets in work with a simple data step, then using the OS to delete the base file physically (you can delete a file in UNIX when when opened), and then copy the temp file in work to the base file location.

[jakarman]

It looks you are having some IO performance problem in your system setup.

Proc append will read one dataset at the time and write one and there will be no PDV build. As it appends the data there is no rewriting or rereading of the areas that are finished.The result is a data set in de order data is specified.

The datastep will need to build a PDV (conversion step) first opening all datasets and then reading one by one while writing the destination.

If you are needing some analyses split by groups then grouping (ordering) at this stage of building the data is most effective.

Reordering a big dataset could be more effective by joining many smaller ordered ones in a match-merge process

[jakarman]

You did not mention your SAS version or OS type the SAs process is running on.

It could be the Eguide process having opened up the dataset and caused the locking.

Locking is something that can be handled by your OS. When all other possible cause are eliminated this is the nasty one.

By trying to optimize IO it can do the locks at a wrong moment and by that causing error message like your got.

This are very rare events, mostly you will find an other you having accessed that data.

[jakarman]

Would avoid relying on the often not activated locking process in Unix. In a real multi-user environment locking should be there.

When going for improvements in the OS you will find it is suddenly there.

SAS(R) 9.3 Companion for UNIX Environments   filelocks system option 9.3

This file locking is an internal SAS-one. The whole concept of locking is preventing accidents corruption of data.