02-11-2016 05:24 PM - edited 02-11-2016 05:25 PM
I run my code (Data STEP: INFILE, INFORMAT, INPUT) to combine dozens of csv files without any error. However, I get a warning as below:
WARNING: Limit set by ERRORS= option reached. Further errors of this type will not be printed.
When I check my log, I see these NOTES:
(these are just 3 of 352 notes I get, but the NOTES are mostly similar)
NOTE: Invalid data for VARIABLE1 in line 2234110 133-136.
101 ,,P,5382100000,1,3780,Floor,X,,,None,,,,,,,,,|,,,15120,15120,40,40,,40,40 173
NOTE: Invalid data errors for file '"/v_share/ali/SCOPES/sas_dataset/Oct-08-2015/ScopesCalculations_SecondaryConstruction_*"'
occurred outside the printed range.
NOTE: Increase available buffer lines with the INFILE n= option.
Could you please give me some hints?, where should I start to fix this?, should I be worried about this at all?
02-11-2016 05:37 PM
Check how you have specified the input for Variable1 and what is in that specific file. It's usually printed in the log so you can find it, since there's no input statement here or the rest of the log I can't comment beyond that.
It's usually a mismatch of types, ie trying to read variable1 in a specific format when it's actually in a different format. You'll need to modify your input statement and/or informat to fix it.
02-12-2016 09:39 AM - edited 02-12-2016 09:40 AM
Here is the what I did: I imported only one (of dozens) file using DATA IMPORT WIZARD into SAS. Then, I checked how SAS specified numeric or character types to each variable (column). Next, I wrote the wildecard code and I applied the same type (character, or numeric) for my variables.
Since I had so many variables, I did this shortcut (if its a shortcut!) instead of using data dictionary.
02-12-2016 09:52 AM
PROC IMPORT will only guess at the types and lengths for the fields based on what it sees. The error you got could be caused by a character variable that was empty in the sample file you used and so SAS made the variable as character. The other possibility is that the files are not made consistently.
You can probably generate the data step faster and better from the data dictionary, especially if the dictionary is available as a dataset. The code the PROC IMPORT generates is very ugly. What you should generate is something that follows this pattern:
02-12-2016 09:56 AM
Since SAS is identifying the variable that generates the error, it should be trivial to compare the input/informat for that variable to your data dictionary.
How are they specified in eacH?
02-12-2016 10:14 AM
I think I found the cause of problem !! , it seems that every Note occures exactly for the first row of my new file (in the wildecard code). In fact, the code cannot recognize that the first row of each new file is a "header".
Asuuming that it is the issue, could you please advise how to modify the code?, how to revise my INFILE statement to recognize the headers?
02-12-2016 10:23 AM
You change your input statement and add EOV to your infile statement.
Here's a writeup on the process, read the comments. To use the code,
02-12-2016 11:27 AM - edited 02-12-2016 11:30 AM
Nice! I ran a test code (with only couple of variables), with NO warning
infile "/v_share/ali/abcd/sas_dataset/Oct-08-2015/Calculations_Secondary_*" linesize = 300 firstobs = 2 eov = eov DSD ;
input @ ;
if eov then do ;
eov = 0 ;
And I got excited! However, when I add the other variables, I get the warning again. I am reading through the log again ...
02-12-2016 11:43 AM - edited 02-12-2016 11:44 AM
I think I found another source of problem. Please see my attached screen shot. It shows one of my csv files ..
Now, my question is how to ask SAS to DELETE any non-consistant row (with the specified data type/format) while running the code?
02-11-2016 06:03 PM - edited 02-11-2016 06:06 PM
So the note you posted is saying the characters 'None' are not valid for Variable1. I assume because Variable1 is numeric. But I am confused why you would be reading columns 133-136 of a line with 173 columns in a variable that you are calling Variable1. Is it possible that you also got a note like :
NOTE: SAS went to a new line when INPUT statement reached past the end of a line.