SAS DI Import preventing user connections to metadata

[JoeLBG]

When importing a SAS package to SAS DI Studio, the Metadata become unrepsonsive. For example, it is not possible for users to connect to their Metadata profile in SMC or Enterprise Guide and existing sessions become unresponsive. As soon as the import is finished, it all comes back to life and works again. This happens when importing any SAS packages, even small ones with just a couple of jobs in.

This is only happening in the levs where we have grid launched sessions turned on for workspace sessions and not in those where we do not. This doesn't seem to make much sense to me though as this setting should only affect the connection to SASComp1, I wouldn't expect that this setting affects connecting to a metadata profile or importing a package, both of which I expect are only client - metadata connections.

From the look of the metadata logs the connection attempts are getting queued up while the import is happening and then being processed after the import is complete e.g. this 8 minute wait for the credential to be obtained for userid 8786746:

545903 2017-03-20T18:50:02,668 TRACE [00033530] 245:8786746 - IOM CALL {compRef:11122bc00}->OMI::GetMetadataObjects():

552416 2017-03-20T18:58:11,394 TRACE [00033530] 245:8786746 - Credential obtained for 8786746.

We also see from netstat that there are a lot more connections to the metadata server during the import process and some 'closed wait' connections:

normally:

$ netstat -an |grep 9561 | grep -i established | wc -l 
    1559
$ netstat -an |grep 9561 | grep -i close | wc -l       
       0

when importing:

$ netstat -an |grep 9561 | grep -i close | wc -l
     185

$ netstat -an |grep 9561 | grep -i established | wc -l
    3328

Server side is SAS 9.4 on Unix AIX 7.1 with LSF grid. Client side DI Studio 4.9 and Enterprise Guide 6.1 on Windows 7.

We have tried restarting everything of course and as suggested by SAS have ensured that various Unix limits such as nofiles and somaxconn are sufficiently high.

Any ideas why this is happening?

Thanks

[anja]

Hi,

did you check system resources at the time when the problem occurs, such as RAM? Disk Usage?

Does the Metadata Server run out of mem maybe?

Another thought - really just throwing out things here - is that for some reason, GRIDWORK fills up

in that environment when an export is done.

Thanks

Anja

[JoeLBG]

Yes, I have checked the system resources. It isn't running out of any of them.

I have actually recreated the issue in every Lev now so have rules out anything to do with the grid - when overwriting existing metadata in the import, new connections to metadata hang. Once the import moves past 'creating metadata' and into 'analysing metadata' everything comes back to life. It takes a lot longer to do the 'create metadata' part when overwriting metadata as opposed to creating new metadata, even for the exact same package. But even while the creating new metadata is happening it doesn't seem to hang, just when overwriting metadata.

I found the following 9.1.3 issue:

http://support.sas.com/kb/31/894.html

Obvioulsy we are on 9.4 so this 9.1.3 issue should be fixed but it still gives me some insight into how the file locks work for SAS on Unix - It seems like maybe whatever file locks are involved in an import that overwrites metadata, they prevent any other metadata requests from being processed until it is finished with those locks. I think our SAS track is going to Cary now so hoping they can shed some light on it....