BookmarkSubscribeRSS Feed
almmotamedi
Obsidian | Level 7

I run my code (Data STEP: INFILE, INFORMAT, INPUT) to combine dozens of csv files without any error. However, I get a warning as below:

 

WARNING: Limit set by ERRORS= option reached.  Further errors of this type will not be printed.

 

When I check my log, I see these NOTES:

(these are just 3 of 352 notes I get, but the NOTES are mostly similar)

 

NOTE: Invalid data for VARIABLE1 in line 2234110 133-136.
RULE: ----+----1----+----2----+----3----+----4----+----5----+----6----+----7----+----8----+----9----+----0
2234110 '13,'IN99,'000001,2/1/1999,,,,10/7/2014,1,1,X,3,3,,1,1,1,N,N,,G,0,0,15120,15120,15120,15120,15120,E,
101 ,,P,5382100000,1,3780,Floor,X,,,None,,,,,,,,,|,,,15120,15120,40,40,,40,40 173

 

NOTE: Invalid data errors for file '"/v_share/ali/SCOPES/sas_dataset/Oct-08-2015/ScopesCalculations_SecondaryConstruction_*"'

occurred outside the printed range.

 

NOTE: Increase available buffer lines with the INFILE n= option.

.

.

 

Could you please give me some hints?, where should I start to fix this?, should I be worried about this at all?

 

Thank you

9 REPLIES 9
Reeza
Super User

Check how you have specified the input for Variable1 and what is in that specific file. It's usually printed in the log so you can find it, since there's no input statement here or the rest of the log I can't comment beyond that.

 

It's usually a mismatch of types, ie trying to read variable1 in a specific format when it's actually in a different format. You'll need to modify your input statement and/or informat to fix it.

almmotamedi
Obsidian | Level 7

Here is the what I did:  I imported only one (of dozens) file using DATA IMPORT WIZARD into SAS. Then, I checked how SAS specified numeric or character types to each variable (column). Next, I wrote the wildecard code and I applied the same type (character, or numeric) for my variables. 

 

Since I had so many variables, I did this shortcut (if its a shortcut!) instead of using data dictionary.

Tom
Super User Tom
Super User

PROC IMPORT will only guess at the types and lengths for the fields based on what it sees.  The error you got could be caused by a character variable that was empty in the sample file you used and so SAS made the variable as character.  The other possibility is that the files are not made consistently.

 

You can probably generate the data step faster and better from the data dictionary, especially if the dictionary is available as a dataset.  The code the PROC IMPORT generates is very ugly.  What you should generate is something that follows this pattern:

  1. LENGTH or ATTRIB statements to define the variables.
  2. INFORMAT statement for the variables that need them.  Do NOT attach informats to character variables (unless you want to preserve leading spaces then used $CHAR informat) or ordinary numeric variables. Note that if you use ATTRIB statement you can add the INFORMAT setting there.
  3. FORMAT statement for variables that need them. Do NOT attach formats to character variables .
  4. Then you input statement can use a variable range so it becomes simply:  INPUT firstvar -- lastvar ;

 

Reeza
Super User

Since SAS is identifying the variable that generates the error, it should be trivial to compare the input/informat for that variable to your data dictionary.

 

How are they specified in eacH?

almmotamedi
Obsidian | Level 7

I think I found the cause of problem !! , it seems that every Note occures exactly for the first row of my new file (in the wildecard code). In fact, the code cannot recognize that the first row of each new file is a "header". 

 

Asuuming that it is the issue, could you please advise how to modify the code?, how to revise my INFILE statement to recognize the headers?

Reeza
Super User

You change your input statement and add EOV to your infile statement.

 

Here's a writeup on the process, read the comments. To use the code,

  1. replace the input statement with your input
  2. add your informat 
  3. customize the infile statement to your wildcard path
  4. add any additional options you used.

 

https://communities.sas.com/t5/SAS-Communities-Library/How-do-I-write-a-macro-to-import-multiple-tex...

almmotamedi
Obsidian | Level 7

Nice! I ran a test code (with only couple of variables), with NO warning

 

infile "/v_share/ali/abcd/sas_dataset/Oct-08-2015/Calculations_Secondary_*" linesize = 300 firstobs = 2 eov = eov  DSD ;

 

Informat
C_ST $3.0
C_DSTRB_AREA $5.0
N_RSK $7.
COT00503D_SRVY $10.0;


input @ ;

if eov then do ;
eov = 0 ;
delete ;
end ;

 
input
C_ST $
C_DSTRB_AREA $
N_RSK $
COT00503D_SRVY $;


run;

 

 

And I got excited! 🙂 However, when I add the other variables, I get the warning again. I am reading through the log again ...

almmotamedi
Obsidian | Level 7

I think I found another source of problem. Please see my attached screen shot. It shows one of my csv files .. 

 

Now, my question is how to ask SAS to DELETE any non-consistant row (with the specified data type/format) while running the code?


1.jpg
Tom
Super User Tom
Super User

So the note you posted is saying the characters 'None' are not valid for Variable1.  I assume because Variable1 is numeric.  But I am confused why you would be reading columns 133-136 of a line with 173 columns in a variable that you are calling Variable1.  Is it possible that you also got a note like :

NOTE: SAS went to a new line when INPUT statement reached past the end of a line.

sas-innovate-2024.png

Join us for SAS Innovate April 16-19 at the Aria in Las Vegas. Bring the team and save big with our group pricing for a limited time only.

Pre-conference courses and tutorials are filling up fast and are always a sellout. Register today to reserve your seat.

 

Register now!

How to Concatenate Values

Learn how use the CAT functions in SAS to join values from multiple variables into a single value.

Find more tutorials on the SAS Users YouTube channel.

Click image to register for webinarClick image to register for webinar

Classroom Training Available!

Select SAS Training centers are offering in-person courses. View upcoming courses for:

View all other training opportunities.

Discussion stats
  • 9 replies
  • 1092 views
  • 0 likes
  • 3 in conversation