DATA Step, Macro, Functions and more

Notes or Warning?

Reply
Contributor
Posts: 72

Notes or Warning?

[ Edited ]

I run my code (Data STEP: INFILE, INFORMAT, INPUT) to combine dozens of csv files without any error. However, I get a warning as below:

 

WARNING: Limit set by ERRORS= option reached.  Further errors of this type will not be printed.

 

When I check my log, I see these NOTES:

(these are just 3 of 352 notes I get, but the NOTES are mostly similar)

 

NOTE: Invalid data for VARIABLE1 in line 2234110 133-136.
RULE: ----+----1----+----2----+----3----+----4----+----5----+----6----+----7----+----8----+----9----+----0
2234110 '13,'IN99,'000001,2/1/1999,,,,10/7/2014,1,1,X,3,3,,1,1,1,N,N,,G,0,0,15120,15120,15120,15120,15120,E,
101 ,,P,5382100000,1,3780,Floor,X,,,None,,,,,,,,,|,,,15120,15120,40,40,,40,40 173

 

NOTE: Invalid data errors for file '"/v_share/ali/SCOPES/sas_dataset/Oct-08-2015/ScopesCalculations_SecondaryConstruction_*"'

occurred outside the printed range.

 

NOTE: Increase available buffer lines with the INFILE n= option.

.

.

 

Could you please give me some hints?, where should I start to fix this?, should I be worried about this at all?

 

Thank you

Super User
Posts: 17,824

Re: Notes or Warning?

Check how you have specified the input for Variable1 and what is in that specific file. It's usually printed in the log so you can find it, since there's no input statement here or the rest of the log I can't comment beyond that.

 

It's usually a mismatch of types, ie trying to read variable1 in a specific format when it's actually in a different format. You'll need to modify your input statement and/or informat to fix it.

Contributor
Posts: 72

Re: Notes or Warning?

[ Edited ]

Here is the what I did:  I imported only one (of dozens) file using DATA IMPORT WIZARD into SAS. Then, I checked how SAS specified numeric or character types to each variable (column). Next, I wrote the wildecard code and I applied the same type (character, or numeric) for my variables. 

 

Since I had so many variables, I did this shortcut (if its a shortcut!) instead of using data dictionary.

Super User
Super User
Posts: 6,500

Re: Notes or Warning?

PROC IMPORT will only guess at the types and lengths for the fields based on what it sees.  The error you got could be caused by a character variable that was empty in the sample file you used and so SAS made the variable as character.  The other possibility is that the files are not made consistently.

 

You can probably generate the data step faster and better from the data dictionary, especially if the dictionary is available as a dataset.  The code the PROC IMPORT generates is very ugly.  What you should generate is something that follows this pattern:

  1. LENGTH or ATTRIB statements to define the variables.
  2. INFORMAT statement for the variables that need them.  Do NOT attach informats to character variables (unless you want to preserve leading spaces then used $CHAR informat) or ordinary numeric variables. Note that if you use ATTRIB statement you can add the INFORMAT setting there.
  3. FORMAT statement for variables that need them. Do NOT attach formats to character variables .
  4. Then you input statement can use a variable range so it becomes simply:  INPUT firstvar -- lastvar ;

 

Super User
Posts: 17,824

Re: Notes or Warning?

Since SAS is identifying the variable that generates the error, it should be trivial to compare the input/informat for that variable to your data dictionary.

 

How are they specified in eacH?

Contributor
Posts: 72

Re: Notes or Warning?

I think I found the cause of problem !! , it seems that every Note occures exactly for the first row of my new file (in the wildecard code). In fact, the code cannot recognize that the first row of each new file is a "header". 

 

Asuuming that it is the issue, could you please advise how to modify the code?, how to revise my INFILE statement to recognize the headers?

Super User
Posts: 17,824

Re: Notes or Warning?

You change your input statement and add EOV to your infile statement.

 

Here's a writeup on the process, read the comments. To use the code,

  1. replace the input statement with your input
  2. add your informat 
  3. customize the infile statement to your wildcard path
  4. add any additional options you used.

 

https://communities.sas.com/t5/SAS-Communities-Library/How-do-I-write-a-macro-to-import-multiple-tex...

Contributor
Posts: 72

Re: Notes or Warning?

[ Edited ]

Nice! I ran a test code (with only couple of variables), with NO warning

 

infile "/v_share/ali/abcd/sas_dataset/Oct-08-2015/Calculations_Secondary_*" linesize = 300 firstobs = 2 eov = eov  DSD ;

 

Informat
C_ST $3.0
C_DSTRB_AREA $5.0
N_RSK $7.
COT00503D_SRVY $10.0;


input @ ;

if eov then do ;
eov = 0 ;
delete ;
end ;

 
input
C_ST $
C_DSTRB_AREA $
N_RSK $
COT00503D_SRVY $;


run;

 

 

And I got excited! Smiley Happy However, when I add the other variables, I get the warning again. I am reading through the log again ...

Contributor
Posts: 72

Re: Notes or Warning?

[ Edited ]

I think I found another source of problem. Please see my attached screen shot. It shows one of my csv files .. 

 

Now, my question is how to ask SAS to DELETE any non-consistant row (with the specified data type/format) while running the code?


1.jpg
Super User
Super User
Posts: 6,500

Re: Notes or Warning?

[ Edited ]

So the note you posted is saying the characters 'None' are not valid for Variable1.  I assume because Variable1 is numeric.  But I am confused why you would be reading columns 133-136 of a line with 173 columns in a variable that you are calling Variable1.  Is it possible that you also got a note like :

NOTE: SAS went to a new line when INPUT statement reached past the end of a line.
Ask a Question
Discussion stats
  • 9 replies
  • 277 views
  • 0 likes
  • 3 in conversation