Not able to read an uncompressed file from work

[alepage]

Hello, 

I have tried two different uncompress command and based on the log file, it seems two work. But I am unable to read the .dat file from work.

how to solve that issue.

%let path=/dwh_actuariat/sasdata/Data_Retention/sas2002/be;
filename oscmd pipe "uncompress -c &path./bel.prod0110.se8030ac.m525.dat.Z > &path./bel.prod0110.se8030ac.m525.dat 2>&1";

data _null_;
infile oscmd;
input;
put _infile_;
run;

%let work=%sysfunc(pathname(work));
%put &=work;
filename oscmd pipe "uncompress -c &path./bel.prod0110.se8030ac.m525.dat.Z > &work./bel.prod0110.se8030ac.m525.dat 2>&1";

data _null_;
infile oscmd;
input;
put _infile_;
run;

NOTE: Writing HTML5(EGHTML) Body file: EGHTML
28
29 %let path=/dwh_actuariat/sasdata/Data_Retention/sas2002/be;
30 filename oscmd pipe "uncompress -c &path./bel.prod0110.se8030ac.m525.dat.Z > &path./bel.prod0110.se8030ac.m525.dat 2>&1";
31
32 data _null_;
33 infile oscmd;
34 input;
35 put _infile_;
36 run;

NOTE: The infile OSCMD is:

Pipe command="uncompress -c /dwh_actuariat/sasdata/Data_Retention/sas2002/be/bel.prod0110.se8030ac.m525.dat.Z >
/dwh_actuariat/sasdata/Data_Retention/sas2002/be/bel.prod0110.se8030ac.m525.dat 2>&1"

NOTE: 0 records were read from the infile OSCMD.
NOTE: DATA statement used (Total process time):
real time 1.48 seconds
cpu time 0.00 seconds

37
38 %let work=%sysfunc(pathname(work));
39 %put &=work;
WORK=/saswork2/ua_bicoe/SAS_work715900039C52_stha8p08j/SAS_work0FAD00039C52_stha8p08j
40 filename oscmd pipe "uncompress -c &path./bel.prod0110.se8030ac.m525.dat.Z > &work./bel.prod0110.se8030ac.m525.dat 2>&1";
41
42 data _null_;
43 infile oscmd;
44 input;
2 The SAS System 10:52 Thursday, September 26, 2024

45 put _infile_;
46 run;

NOTE: The infile OSCMD is:

Pipe command="uncompress -c /dwh_actuariat/sasdata/Data_Retention/sas2002/be/bel.prod0110.se8030ac.m525.dat.Z >
/saswork2/ua_bicoe/SAS_work715900039C52_stha8p08j/SAS_work0FAD00039C52_stha8p08j/bel.prod0110.se8030ac.m525.dat 2>&1"

NOTE: 0 records were read from the infile OSCMD.
NOTE: DATA statement used (Total process time):
real time 1.42 seconds
cpu time 0.00 seconds

But I am unable to import a .dat file in sas

PROC IMPORT DATAFILE = "&work./bel.prod0110.se8030ac.m525.dat"
OUT = np
DBMS = DLM REPLACE;
DELIMITER=" ";
RUN;

[Quentin]

Can you post the log from running your PROC IMPORT step, including all the messages?

I'm hoping it shows an informative message, like file does not exist.

If you look in the work directory, do you see that bel.prod0110.se8030ac.m525.dat exists?

[ballardw]

I would try uncompressing the file outside of SAS.

Open a command window and type that uncompress command with the actual path and not the SAS macro variable and see what happens. Or move to that location in the file system and skip the path.

You might get more feed back in a command window but may want to drop the 2>&1

If you have that file created from the uncompress then open it with a text editor and show us what the start of the file looks like by copy and pasting some of it into a text box on the forum.

[Patrick]

Not sure if it will work but worth trying:

%let path=/dwh_actuariat/sasdata/Data_Retention/sas2002/be;
filename oscmd pipe "uncompress -c &path./bel.prod0110.se8030ac.m525.dat.Z";

I believe if you direct standard out to a new file then you would need another filename statement for this new file from which you then read via a data step. I you omit this redirection then I believe your infile/input statement will directly read from stdout.

Also not sure but I believe 2>&1 would also redirect error messages to stdout and though become part of the data you're reading - which is likely not what you want.

[alepage]

Good Morning,

Those two scripts are working because there is no observation into the log and the file is uncompress on the server with the first command.  Does the work folder and work library are the same because if it is the case, there is nothing in the work library.

Also, when the file is uncompressed, how to read it with SAS?

%let path=/dwh_actuariat/sasdata/Data_Retention/sas2002/be;
filename oscmd pipe "uncompress -c &path./bel.prod0110.se8030ac.m525.dat.Z > &path./bel.prod0110.se8030ac.m525.dat 2>&1";

data _null_;
infile oscmd;
input;
put _infile_;
run;

%let work=%sysfunc(pathname(work));
%put &=work;
filename oscmd pipe "uncompress -c &path./bel.prod0110.se8030ac.m525.dat.Z > &work./bel.prod0110.se8030ac.m525.dat 2>&1";

data _null_;
infile oscmd;
input;
put _infile_;
run;

NOTE: Writing HTML5(EGHTML) Body file: EGHTML
28
29 %let path=/dwh_actuariat/sasdata/Data_Retention/sas2002/be;
30 filename oscmd pipe "uncompress -c &path./bel.prod0110.se8030ac.m525.dat.Z > &path./bel.prod0110.se8030ac.m525.dat 2>&1";
31
32 data _null_;
33 infile oscmd;
34 input;
35 put _infile_;
36 run;

NOTE: The infile OSCMD is:

Pipe command="uncompress -c /dwh_actuariat/sasdata/Data_Retention/sas2002/be/bel.prod0110.se8030ac.m525.dat.Z >
/dwh_actuariat/sasdata/Data_Retention/sas2002/be/bel.prod0110.se8030ac.m525.dat 2>&1"

NOTE: 0 records were read from the infile OSCMD.
NOTE: DATA statement used (Total process time):
real time 1.18 seconds
cpu time 0.00 seconds

37
38 %let work=%sysfunc(pathname(work));
39 %put &=work;
WORK=/saswork2/ua_bicoe/SAS_work29380001D84B_stha8p08j/SAS_work9F4A0001D84B_stha8p08j
40 filename oscmd pipe "uncompress -c &path./bel.prod0110.se8030ac.m525.dat.Z > &work./bel.prod0110.se8030ac.m525.dat 2>&1";
41
42 data _null_;
2 The SAS System 10:39 Friday, September 27, 2024

43 infile oscmd;
44 input;
45 put _infile_;
46 run;

NOTE: The infile OSCMD is:

Pipe command="uncompress -c /dwh_actuariat/sasdata/Data_Retention/sas2002/be/bel.prod0110.se8030ac.m525.dat.Z >
/saswork2/ua_bicoe/SAS_work29380001D84B_stha8p08j/SAS_work9F4A0001D84B_stha8p08j/bel.prod0110.se8030ac.m525.dat 2>&1"

NOTE: 0 records were read from the infile OSCMD.
NOTE: DATA statement used (Total process time):
real time 1.31 seconds
cpu time 0.00 seconds

libname temp "%sysfunc(pathname(work))";

PROC IMPORT DATAFILE = "temp.bel.prod0110.se8030ac.m525.dat"
OUT = np
DBMS = DLM REPLACE;
DELIMITER=" ";
RUN;

ERROR: Physical file does not exist, /opt/sas94/Config/Lev1/SASApp/temp.bel.prod0110.se8030ac.m525.dat.
ERROR: Import unsuccessful. See SAS Log for details.

[ballardw]

The error message here is why ANY time you reference a file external to SAS you start at a drive, whether a letter in Windows or mount point in Unix derived operating system. The first thing in the path should be the root of a drive.

libname temp "%sysfunc(pathname(work))";

PROC IMPORT DATAFILE = "temp.bel.prod0110.se8030ac.m525.dat"
OUT = np
DBMS = DLM REPLACE;
DELIMITER=" ";
RUN;

ERROR: Physical file does not exist, /opt/sas94/Config/Lev1/SASApp/temp.bel.prod0110.se8030ac.m525.dat.
ERROR: Import unsuccessful. See SAS Log for details.

The part starting at /opt is where you SAS session is currently executing. Since your Proc Import did not provide ANY path information with the datafile then SAS expects the path to be relative to where it executes.

If you think using LIBNAME.some_external_file_name is going to work, it doesn't. The LIbname.memname syntax is only for SAS files like data sets, views or catalogs.

It looks like the uncompress created this:

/saswork2/ua_bicoe/SAS_work29380001D84B_stha8p08j/SAS_work9F4A0001D84B_stha8p08j/bel.prod0110.se8030ac.m525.dat

That would be the Datafile= for Proc Import.

[Quentin]

---

This code is wrong:

libname temp "%sysfunc(pathname(work))";

PROC IMPORT DATAFILE = "temp.bel.prod0110.se8030ac.m525.dat"
OUT = np
DBMS = DLM REPLACE;
DELIMITER=" ";
RUN;

ERROR: Physical file does not exist, /opt/sas94/Config/Lev1/SASApp/temp.bel.prod0110.se8030ac.m525.dat.
ERROR: Import unsuccessful. See SAS Log for details.

---

The code in your question looks like it could work, if the unzip succeeded:

%let work=%sysfunc(pathname(work));
%put &=work;
PROC IMPORT DATAFILE = "&work./bel.prod0110.se8030ac.m525.dat"
OUT = np
DBMS = DLM REPLACE;
DELIMITER=" ";
RUN;

When you run that code, do you get the same file does not exist error? If so, then the unzip is not working.

Do you have a way to look at files in a directory without using SAS?  e.g. WinSCP or putty?  If you can look in the directory using non-SAS tools, that would be the easiest way to see if your unzip succeded, and and see if there is a file named bel.prod0110.se8030ac.m525.dat in the expected directory.

[Tom]

Do you really need to create a physical copy of the uncompressed file just to READ it?

Why not just read the output of your uncompress -c command directly.

Try looking at the first few lines of the file to make sure the uncompress is working and you know what the file has in it.

%let path=/dwh_actuariat/sasdata/Data_Retention/sas2002/be;
data _null_;
  infile "uncompress -c &path./bel.prod0110.se8030ac.m525.dat.Z" pipe obs=5;
  input;
  list;
run;

Once you can see the file then you can write a data step to read it.

data np;
  infile "uncompress -c &path./bel.prod0110.se8030ac.m525.dat.Z" pipe;
  length var1 $20 var2 8 .... varlast 8 ;
  input var1 -- varlast;
run;

If you must GUESS how to read the file then remember that PROC IMPORT is not able to read from a PIPE (why I have no idea).  So you could either use some other tool for GUESSING how to read the file such as : https://github.com/sasutils/macros/blob/master/csv2ds.sas 

Or just write your own data step to read it.

If you need to make a temporary permanent file then perhaps just use a data step to copy the lines.

filename tempdat temp;
data _null_;
  infile "uncompress -c &path./bel.prod0110.se8030ac.m525.dat.Z" ;
  file tempdat;
  input;
  put _infile_;
run;
proc import file=tempdat dbms=csv out=np replace;
run;

Also you appear to be trying to tell PROC IMPORT to read the file as if it was a CSV file that used space as the delimiter.  Normally files that use spaces as the delimiters are not really delimited files.  Instead they normally use the multiple spaces when values are short so that the actual values appear in columns.  So trying to read such a file as a delimited (csv) file will general cause mistakes.

If your code was supposed to be indicating that TAB (or some other invisible character) is the delimiter then use DELIMITER='09'x (or whatever hexcode is appropriate) instead of trying to figure out to insert and actual tab character inside quotes into your program file.  That is very likely to be misunderstood by humans.  And editors are likely to replace the tab with spaces any way.