topic Reading a flat file from Mainframe in SAS Procedures

Reading a flat file from Mainframe

park2039 — Wed, 31 Aug 2011 19:53:03 GMT

I am trying to read a text file generated in Mainframe. I used EBCDIC format with recfm=f option. However, SAS stops before it reaches the end-of-file. The input file has 8.2 mil records but SAS stop after it reads 6.2 mil. Does anyone have a solution to this problem?

Thanks.

Reading a flat file from Mainframe

art297 — Wed, 31 Aug 2011 19:59:22 GMT

Not sure if it would be relevant for EBCDIC, but have you tried the ignoredoseof option?

Reading a flat file from Mainframe

art297 — Wed, 31 Aug 2011 20:01:34 GMT

Actually, it may be quite relevant. Take a look at http://www.sascommunity.org/wiki/IGNOREDOSEOF_Option

Reading a flat file from Mainframe

park2039 — Wed, 31 Aug 2011 20:33:51 GMT

I tried but it produced the same result.

Reading a flat file from Mainframe

art297 — Wed, 31 Aug 2011 20:41:33 GMT

Take a look at: http://support.sas.com/techsup/technote/ts642.html

Reading a flat file from Mainframe

FriedEgg — Wed, 31 Aug 2011 20:57:57 GMT

Are you using any truncover or missover options? Have you checked to make sure that your layout is accurate to the file in question, especially in respect to the final length you a defining for a line? If the read is stopping on a consistent line have you inspected the line in question for some annomily?

Re: Reading a flat file from Mainframe

Tom — Wed, 31 Aug 2011 21:11:37 GMT

Is it 6.2 out of 8.2 lines in the flat file or observations in the output file?

How did you count the lines in the flat file?

How did you move the file from the mainframe?

Re: Reading a flat file from Mainframe

park2039 — Wed, 31 Aug 2011 21:15:26 GMT

It is observations. Each line contains one observation. The record counts were gievn in advance by the person who created the flat file. The file was moved using FTP.

Re: Reading a flat file from Mainframe

Tom — Wed, 31 Aug 2011 21:35:53 GMT

Can you verify that it was moved as BINARY rather then ASCII? Does it have any PIB or other fields that would contain binary data?

Was it originally fixed length records on the mainframe?

Do you get messages in the SAS log about SAS moving to another record to satisfy the input statements? Does it say that it read the same number of lines as the observations created. Does the data appear to get out of align with the specifications?

I like to use the LIST statement to look at the data.

You might try :

data _null_;

infile ....... ;

if _n_ <= 10 or 6200000 < _n_ ;

input ;

list;

run;

Where "62000000" is a small number of records before the point where it is stopping.

Re: Reading a flat file from Mainframe

art297 — Wed, 31 Aug 2011 21:39:05 GMT

Have you tried it including a truncover option? It is possible that all of the records are there, but that some are missing some data, thus reading from the next record.

Re: Reading a flat file from Mainframe

Peter_C — Wed, 31 Aug 2011 22:06:40 GMT

park2039

please could you provide the sas notes in the log of the step which failed to read all the data

peterC

Re: Reading a flat file from Mainframe

park2039 — Thu, 01 Sep 2011 00:46:04 GMT

The SAS finished the job w/o generating error message. So there is no error in the log. It is as if there were only 6.2 mil records in the input file. So if the record count was not given in advance there is no way of knowing the SAS finished before it reached the end of file.

Re: Reading a flat file from Mainframe

park2039 — Thu, 01 Sep 2011 00:47:26 GMT

I tried trunover option but it did not work.

Re: Reading a flat file from Mainframe

art297 — Thu, 01 Sep 2011 00:49:06 GMT

Did you try it with the truncover option?

Reading a flat file from Mainframe

ArpitSharma — Thu, 01 Sep 2011 01:52:17 GMT

1. I think you need to check your input file first. Please use the option "COUNT" to check the exact number of lines in the flat file..

this will give you the number of records in the text file.

After you know the number of records try reading only the first "character" using @ just to check if the file is complete or not.

After getting done with this

Try changing the output parameters

SPACE=(CYL,(500,100),RLSE),

RECFM=FB, LRECL= 250,BLKSIZE=0

Try increasing the assigned cylinders "500" till the point it works....

make sure u put BLKSIZE=0...

This should surely help you out of your problem....

Re: Reading a flat file from Mainframe

park2039 — Thu, 01 Sep 2011 04:13:48 GMT

I assume it was moved as binary. The file was created in a server then moved to mainframe from where I copied to PC. It had fixed length records from the start. 62000000 is a larger number of records at the point when it stopped.

Re: Reading a flat file from Mainframe

park2039 — Thu, 01 Sep 2011 04:15:57 GMT

The file was not created in the mainframe. I used the mainframe only as a placeholder. The file was created in another server.

Re: Reading a flat file from Mainframe

ArpitSharma — Thu, 01 Sep 2011 04:45:44 GMT

I understand your point.

But you need to narrow down your problem.

First try to figure out as to how many records in actual are there in Mainframe by using COUNT OPTION of MAINFRAME.

This is bcs sometimes it is possible that the complete file did not get FTP.

Once you are aware of the number of records in the flat file you know EXACTLY how many records are required to be read.

Then as others have specified CHECK use of MISSOVER/TRUNCOVER functions...

Now the problem narrows down to 2 things.

One some problem with SAS CODE.

TWO some problem with mainframe properties of output dataset.

ONE: Check for any missing ";" and also all code. Make sure SAS is not going into any indefinite loop.

TWO: I have encountered problems in mainframe while creating large datasets..In such case defining properties of the output dataset plays a vital role.You will have to go a little deep in to the mainframe commands like :

DISP=(XXXXX), UNIT=(XX,XX),SPACE,RECFM,LRECL,BLKSIZE etc....

I hope it helps

Re: Reading a flat file from Mainframe

vachuck — Thu, 01 Sep 2011 14:02:44 GMT

On your INFILE statement make certain you include the LRECL parameter. Michelle Burlew in her excellent book Reading External Data Files Using SAS notes that external files downloaded from EBCDIC system of fixed length do not contain End-of-Record (EOR) markers. As others have pointed out, you will need to determine the record length of the data set on the mainframe. Are you using $ebcdic informat for character variables and s370ff informat for numeric variables? As others have noted, check with the person who created the file to see if it contains binary or packed decimal data. I hope this solves your problem.

Charles

Reading a flat file from Mainframe

Doc_Duke — Thu, 01 Sep 2011 14:25:46 GMT

Park,

The fact that the data were not created on the mainframe may be key. If the step to upload the data to the mainframe is flawed, then it doesn't matter what you have done on download.

If the data started out as text (say on a unix box of some sort), then the user can upload it as text (rather than binary) and you can FTP it down as text. The FTP service will do the conversion from one coding schema to another.

Doc Muhlbaier

Duke