File is not a SAS data set

[tmcrouse]

My computer crashed last week at work and the IT team took all my SAS tables that were on the crashed computer and saved them to a disk. I loaded them onto my machine and can open them up and see them as a SAS table. I have put them back in the library they were before. I am now running queries off those sets but get this error file is not a SAS data set. How do I repair this so I can use it?

[ballardw]

Posting code that generated the error and the error message might give you a better shot at an answer.

[tmcrouse]

Here is the code but the problem is that the computer crashed and what they took from my old and gave me on a disk has been placed on my new machine but for some reason it does not recogniz table final11 as a dataset. I can pull it up in SAS just fine and it looks like table. I cannot rerun the prior because this overall code takes 123GB and 5 days in all to run. thus the reason my computer crashed. Was just too much for it. Fan burned out and processor stopped. But because of the costs associated with running this overall code my manager prefers that I get what was on my C drive from old and go from there. I had all the final tables I needed with coding dropping the old due to size. Like I said the overall program is 123GB and my computer has only a 150GB hard drive. Cheap company that will not give programmers more space. Therefore our computers crash and we lose data. Anyway. Table final11 is fine and in there but it just does not understand that the table is a dataset. so I need to somehow make it understand that so I can run this code below.

proc sql ;

create table cardiac.final11b as

(select distinct a.*,

case when a.adj_hccc_cd ='07' or a.prov_mpin_pci=. then . else outpt_mpin_flg end as outpt_pci_mpin_flg,

case when a.adj_hccc_cd ='07' or a.prov_mpin_pci=. then . else outpt_mpin_prov end as outpt_pci_prov_mpin,

case when a.adj_hccc_cd ='07' or a.prov_mpin_pci=. then '  ' else outpt_tin_prov end as outpt_pci_prov_tin,

case when BUS_LINE_DESC='MEDICARE' and prov_tin_mbr=1 THEN 1 END AS MEDICARE_prov_tin,

CASE WHEN BUS_LINE_DESC='COMMERCIAL' and prov_tin_mbr=1 THEN 1 END AS COMMERCIAL_prov_tin,

case when BUS_LINE_DESC='MEDICARE' and member=1 THEN 1 END AS MEDICARE_nat,

CASE WHEN BUS_LINE_DESC='COMMERCIAL' and member=1 THEN 1 END AS COMMERCIAL_nat,

case when BUS_LINE_DESC='MEDICARE' and prov_mpin_mbr=1 THEN 1 END AS MEDICARE_prov_mpin,

CASE WHEN BUS_LINE_DESC='COMMERCIAL' and prov_mpin_mbr=1 THEN 1 END AS COMMERCIAL_prov_mpin,

CASE WHEN BUS_LINE_DESC='COMMERCIAL' and unique_fac_mpin=1 THEN 1 END AS COMMERCIAL_fac_mpin,

case when BUS_LINE_DESC='MEDICARE' and unique_fac_mpin=1 THEN 1 END AS MEDICARE_fac_mpin,

CASE WHEN BUS_LINE_DESC='COMMERCIAL' and unique_fac_mpin=1 THEN 1 END AS COMMERCIAL_fac_tin,

case when BUS_LINE_DESC='MEDICARE' and unique_fac_mpin=1 THEN 1 END AS MEDICARE_fac_tin

from cardiac.final11 a

left join

(select distinct

mbr_sys_id,

fst_srvc_dt,

prov_tin as outpt_tin_prov,

prov_mpin as outpt_mpin_prov,

case when prov_mpin>. then 1 end as outpt_mpin_flg

from cardiac.final11

where adj_hccc_cd='07' and prov_mpin_pci=1)b

on a.mbr_sys_id=b.mbr_sys_id

and

1. a.fst_srvc_dt=b.fst_srvc_dt);

quit;

[tmcrouse]

Oh and error message simply says:           Error: file cardiac.final11b.data is not a SAS data set.

[ballardw]

It sounds more like an OS problem. There are, as with many programs, issues about temporary file use and final. It looks like you may not have enough space for both in this instance. I would try runing the program and limiting the numbe observations used to something real small, say 10, to see if the logic is actually working.

If so, then I'd look at creating as much space as you disk as possible.

And I do understand the limitations. At one time I had a job where to run some models I actually had to uninstall the OS and strip the machine back to a DOS install to have enough space for the output files.

[tmcrouse]

Wow. Yeah IT will never do that. They just built this machine for me and shipping it out last week. I don't have admin rights either. I am just a grunt. So, I will try the 10 obs and see what happens. If that works I will have to do what I did last time with moving crap over to our share drive which is when my computer actually crashed. OMG will they freak if it crashes again. This is the downside of working in Corp America and they give you a 150GB hard drive to pull from a claim system where you insure 40 million people and you are pulling millions of Claims in a table and using SAS. They always try and get my team to use something else but all other software cannot handle the returns. We have tried it. They just need to bite the bullet and give me a machine with a terabyte. My home computer has a terbyte. I have even offered to buy my own computer to get the space and send it to them to format and they are like no. Too many healthcare violations doing that. So it is what it is. If they refuse to give me what I need to do my job appropriately then not sure what to say.

[tmcrouse]

Ok now this is odd. I moved all the SAS tables from the backup disks they provided and my final11B was on there which is derived from the above code. However when I tried to run the code below final11B which is summing stuff up it said that final11B is not a SAS data set. So this is why I went back to the code that created final11B from final11. I was reading the error thinking why on earth is it saying that final11B is not a data set now when I have not created it because I am creating it from final11. Well, final11B was moved over from the disk so I was like ok let's delete it and then I changed the name in my code to final11A and it is running. So that confuses me big time. I noticed that some of these SAS tables I can run code off of and do not get an error and others will say not a SAS data set. It is as if some of the tables are corrupt and some are not. So we shall see how this goes because another project I have that I recovered SAS tables is having the exact same issues. So perhaps I just need to rename the tables I am trying to create??? Very odd. I hate computer crashes.

[tmcrouse]

Well, never mind on the changing the table name. It tried to run but then came back with this compress error and reporting it to SAS representative. I have option compress as yes on this. I just think our IT folks did something wrong when saving these SAS tables and there are some I can access and are recognized and some that are not. They are trying the save again for me and this time not splitting my tables up between disks and sending it to me via Fedex tomm delivery. so I will try extracting them again and see what happens. It is just odd that tables that are not split I can query off of and tables that are split regardless of chnging the output table name I cannot.

[ballardw]

You haven't mentioned which OS you are using but very few will manage files split across multiple disks/directories. It could be that the backup process used a method that split files on the backup media but the restore wasn't told that or didn't recombine them correctly. The "not a SAS file" could come as a symptom of various forms of corruption but I'm not sure.

[tmcrouse]

Thanks. I am on Windows XP. I am sure it has to due with the way they split. They are supposedly redoing it by taking the subfolders from my main sas folder and splitting so they never have over 8gb. But if they run into instance where say a folder has 10gb and it would mean splitting a table they are not doing that. They will make sure no tables get split. So say 3gb of the 10gb is one table. That will go onto a disk so it is not split and it might end up only 7gb on a disk. I just told them do not split any SAS tables. period. I noticed I had a accdb Access table that was split as well 2-3 disk and it did not load right either. So it is not just SAS it is anything DB related that does not work.

[Patrick]

I could replicate your error by creating a corrupted SAS file. If the SAS tables now got corrupted as a result of something your IT department did wrong or as a result of your computer crash is another question.

I believe best would be to re-create all SAS tables from scratch or to get them restored from a backup copy if you have one and not from a recovered disk after a crash.

And here the code demonstrating that it's very likely about corrupted files.

%let TestDir=C:\temp;

/* prepare environment */
options xsync;
x "mkdir &TestDir";
x "del &TestDir\TestTable*";

/* create valid SAS test table "TestTable" */
libname test "&TestDir";

proc sql;
  create table test.TestTable as
    select *
    from sashelp.class
  ;
quit;

/* create corrupted SAS test table "TestTableCorrupt" by using TestTable as source */
/* and then removing the first byte in the first line when writing to target       */
data _null_;
  infile "&TestDir\TestTable.sas7bdat" lrecl=32000;
  file "&TestDir\TestTableCorrupt.sas7bdat" lrecl=32000;
  input;
  if _n_=1 then _infile_=substr(_infile_,2);
  put _infile_;
run;

/* use create table on a case where there is already a file with this name */
proc sql;
  create table test.TestTableCorrupt as
    select *
    from sashelp.class
  ;
quit;

ERROR: File TEST.TESTTABLECORRUPT.DATA is not a SAS data set.

Then I was asking myself why you still can see the corrupted table as entry in your library. It appears that this library list is simply built using file extensions (.sas7bdat) and not checking if it's actually a valid SAS table. But when trying to re-create the table SAS is somehow checking if it's a valid SAS table - and that's when the error occurs.

I assume that SAS must check table content when re-creating a table in order to check if the table is password protected, a generation dataset, has audit trails and so on.

%let TestDir=C:\temp;

/* prepare environment */
options xsync;
x "mkdir &TestDir";
x "del &TestDir\TestTable*";

data _null_;
  file "&TestDir\TestTableEmpty.sas7bdat";
  put ' ';
run;

libname test "&TestDir";
proc sql;
  create table test.TestTableEmpty as
    select *
    from sashelp.class
  ;
quit;

ERROR: File TEST.TESTTABLEEMPTY.DATA is not a SAS data set.