topic Re: Error importing csv file in SAS Programming

Error importing csv file

xyxu — Fri, 27 Mar 2020 23:39:21 GMT

I am trying to import a large CSV file into SAS. I tried to use both

proc import datafile = "rawdata.csv" dbms = csv out = mydata replace; getnames = yes;  run;

and

data mydata;
	infile "rawdata.csv" delimiter = ',' dsd missover lrecl = 32767 firstobs = 2; ...; run;

These two commands appear to generate the same dataset. However, only the second command leads to error messages in log. The messages are as follows:

To investigate the issue, I open the raw file in MS Access and go to row 2322547. The issue seems to be that a character variable has weird values:

And in the imported dataset, this value becomes

I tried to use encoding='utf-8' but it does not help. How can I correctly import this dataset?

Re: Error importing csv file

Tom — Sat, 28 Mar 2020 01:12:54 GMT

Most likely you have poorly formed file. Perhaps it has some embedded line breaks in the middle of one the fields? This can cause the INPUT statement, either the one generated by PROC IMPORT or the one you wrote for your own data step, to get out of alignment with the values on the lines.

If that is your problem it is a common problem and has been answered on this forum many times. You might be able to read it if the line breaks in the fields just use single carriage return or single linefeed and the actual real end of line codes in the file use both a carriage return and a linefeed. If not then you need to pre-process the file to fix the issue by removing the extra line breaks.

PS Why did you paste photographs of text? You can just copy and paste the lines of text from the SAS log.

Re: Error importing csv file

xyxu — Sat, 28 Mar 2020 01:15:00 GMT

Thank, Tom. I found that the issue is due to hidden linebreakers in the string.

Re: Error importing csv file

ChrisNZ — Sun, 29 Mar 2020 02:50:55 GMT

Shame SAS decided not to address the issue.