topic Re: Notes or Warning? in SAS Programming

Notes or Warning?

almmotamedi — Thu, 11 Feb 2016 22:25:22 GMT

I run my code (Data STEP: INFILE, INFORMAT, INPUT) to combine dozens of csv files without any error. However, I get a warning as below:

WARNING: Limit set by ERRORS= option reached. Further errors of this type will not be printed.

When I check my log, I see these NOTES:

(these are just 3 of 352 notes I get, but the NOTES are mostly similar)

NOTE: Invalid data for VARIABLE1 in line 2234110 133-136.
RULE: ----+----1----+----2----+----3----+----4----+----5----+----6----+----7----+----8----+----9----+----0
2234110 '13,'IN99,'000001,2/1/1999,,,,10/7/2014,1,1,X,3,3,,1,1,1,N,N,,G,0,0,15120,15120,15120,15120,15120,E,
101 ,,P,5382100000,1,3780,Floor,X,,,None,,,,,,,,,|,,,15120,15120,40,40,,40,40 173

NOTE: Invalid data errors for file '"/v_share/ali/SCOPES/sas_dataset/Oct-08-2015/ScopesCalculations_SecondaryConstruction_*"'

occurred outside the printed range.

NOTE: Increase available buffer lines with the INFILE n= option.

Could you please give me some hints?, where should I start to fix this?, should I be worried about this at all?

Thank you

Re: Notes or Warning?

Reeza — Thu, 11 Feb 2016 22:37:05 GMT

Check how you have specified the input for Variable1 and what is in that specific file. It's usually printed in the log so you can find it, since there's no input statement here or the rest of the log I can't comment beyond that.

It's usually a mismatch of types, ie trying to read variable1 in a specific format when it's actually in a different format. You'll need to modify your input statement and/or informat to fix it.

Re: Notes or Warning?

Tom — Thu, 11 Feb 2016 23:06:41 GMT

So the note you posted is saying the characters 'None' are not valid for Variable1. I assume because Variable1 is numeric. But I am confused why you would be reading columns 133-136 of a line with 173 columns in a variable that you are calling Variable1. Is it possible that you also got a note like :

NOTE: SAS went to a new line when INPUT statement reached past the end of a line.

Re: Notes or Warning?

almmotamedi — Fri, 12 Feb 2016 14:40:02 GMT

Here is the what I did: I imported only one (of dozens) file using DATA IMPORT WIZARD into SAS. Then, I checked how SAS specified numeric or character types to each variable (column). Next, I wrote the wildecard code and I applied the same type (character, or numeric) for my variables.

Since I had so many variables, I did this shortcut (if its a shortcut!) instead of using data dictionary.

Re: Notes or Warning?

Tom — Fri, 12 Feb 2016 14:52:57 GMT

PROC IMPORT will only guess at the types and lengths for the fields based on what it sees. The error you got could be caused by a character variable that was empty in the sample file you used and so SAS made the variable as character. The other possibility is that the files are not made consistently.

You can probably generate the data step faster and better from the data dictionary, especially if the dictionary is available as a dataset. The code the PROC IMPORT generates is very ugly. What you should generate is something that follows this pattern:

LENGTH or ATTRIB statements to define the variables.
INFORMAT statement for the variables that need them. Do NOT attach informats to character variables (unless you want to preserve leading spaces then used $CHAR informat) or ordinary numeric variables. Note that if you use ATTRIB statement you can add the INFORMAT setting there.
FORMAT statement for variables that need them. Do NOT attach formats to character variables .
Then you input statement can use a variable range so it becomes simply: INPUT firstvar -- lastvar ;

Re: Notes or Warning?

Reeza — Fri, 12 Feb 2016 14:56:33 GMT

Since SAS is identifying the variable that generates the error, it should be trivial to compare the input/informat for that variable to your data dictionary.

How are they specified in eacH?

Re: Notes or Warning?

almmotamedi — Fri, 12 Feb 2016 15:14:25 GMT

I think I found the cause of problem !! , it seems that every Note occures exactly for the first row of my new file (in the wildecard code). In fact, the code cannot recognize that the first row of each new file is a "header".

Asuuming that it is the issue, could you please advise how to modify the code?, how to revise my INFILE statement to recognize the headers?

Re: Notes or Warning?

Reeza — Fri, 12 Feb 2016 15:23:20 GMT

You change your input statement and add EOV to your infile statement.

Here's a writeup on the process, read the comments. To use the code,

replace the input statement with your input
add your informat
customize the infile statement to your wildcard path
add any additional options you used.

https://communities.sas.com/t5/SAS-Communities-Library/How-do-I-write-a-macro-to-import-multiple-text-files-that-have/ta-p/223627

Re: Notes or Warning?

almmotamedi — Fri, 12 Feb 2016 16:30:18 GMT

Nice! I ran a test code (with only couple of variables), with NO warning

infile "/v_share/ali/abcd/sas_dataset/Oct-08-2015/Calculations_Secondary_*" linesize = 300 firstobs = 2 eov = eov DSD ;

Informat
C_ST $3.0
C_DSTRB_AREA $5.0
N_RSK $7.
COT00503D_SRVY $10.0;

input @ ;

if eov then do ;
eov = 0 ;
delete ;
end ;

input
C_ST $
C_DSTRB_AREA $
N_RSK $
COT00503D_SRVY $;

run;

And I got excited! 🙂 However, when I add the other variables, I get the warning again. I am reading through the log again ...

Re: Notes or Warning?

almmotamedi — Fri, 12 Feb 2016 16:44:19 GMT

I think I found another source of problem. Please see my attached screen shot. It shows one of my csv files ..

Now, my question is how to ask SAS to DELETE any non-consistant row (with the specified data type/format) while running the code?