Missing/Corrupted Cluster in SPDS ?

[jguzlecki]

We had a server hardware disk failure and now seem to have a corrupted Cluster in a SPDS library. I am unable to recover this cluster so I am reaching out to see if anyone has any thoughts on something I am missing.

The only successful thing I can do is a PROC CONTENTS on the library, which lists the Cluster DATA and INDEX as being members.

However, if I query the DICTIONARY.TABLES, the cluster is not shown as a member of that library, I also cannot manipulate the Cluster…

I’ve tried:

PROC CONTENTS of the cluster - returns: ERROR: One or more cluster member tables not found.

PROC SPDO; CLUSTER LIST - returns: ERROR: One or more cluster member tables not found.

PROC SPDO; CLUSTER UNDO – Seems to work, but there are no member tables listed in the library and the log indicates: NOTE: One or more cluster member tables not found.

PROC SPDO; CLUSTER DESTROY - returns: ERROR: One or more cluster member tables not found. / ERROR: The CLUSTER DESTROY CMRDETL is unsuccessful.

PROC SPDO; CLUSTER CREATE – returns: ERROR: Cluster Name already exists cannot overwrite.

There are other clusters in this library that are functioning just fine, are listed in the dictionary.table and can be manipulated with PROC SPDO commands.

Any ideas as to how I can attempt to fix this cluster, or is it a lost cause?

Thanks in advance for any insight.

[SASKiwi · Accepted solution]

Have you contacted Tech Support? I think they would be best qualified to deal with this type of problem.

Also have you done any restores from backups prior to the corruption?

[jguzlecki]

Thanks Kiwi,  we're pursuing a restore of the backup, if that fails and I am unsuccessful with recovering this, I will pursue Tech Support.

As I'm fairly new to SPDS, this was a great way to learn quite a bit about it (trial by fire), but if it turns out to be futile and I cannot repair, I'll have no other choice.

I wasn't sure if this is a somewhat common problem (doesn't seem to be) and I was missing a simple fix or something more sinister...

[Ksharp]

Can you repair it ?

proc datasets library=work;
repair have;
quit;

[jguzlecki]

Thanks ksharp, I was so focused on SPDS procedures, I hadn't thought of straight-up SAS solutions.

However it failed with the same type of issue...

proc datasets library=CICS;
repair CMRDETL;
quit;

returns:

NOTE: Repairing CICS.CMRDETL (memtype=DATA).
ERROR: One or more cluster member tables not found.
NOTE: The SAS System stopped processing this step because of errors.

I also tried with dldmgaction=repair, which seemed promising but SAS didn't seem to even try.

[LinusH]

If the data is corrupted due to a disk failure, no SAS in the world can fix that. Get functioning disks and perform a restore.
Also, how can you have a disk failure corrupting data in 2019? Are you using RAID0?

[jguzlecki]

Thanks Linus.  We've restored this data from the backup and written off the missing day of data.

Don't get me started on how a disk failure can occur...  Our Tech Ops guys identified two drives and a controller which failed.  The disk wasn't corrupted, it actually was able to rebuild once the bad drives were swapped.  

Somehow in the chaos, the cluster got corrupted and seemed to indicate that a member was part of the cluster, but couldn't find the member table.  Once that occurred, it became impossible to manage, undo, destroy, list or in any way manipulate the cluster...

Hopefully, by flagging this as answered, it'll go away and not waste anyone else's time.