topic Re: importing data stops SAS in the middle of a textfile without generating an error in SAS Programming

importing data stops SAS in the middle of a textfile without generating an error

fre — Wed, 03 Jun 2020 09:17:45 GMT

Dear community,

I'm using this code to import a specific textfile - which worked very well for years: (it's part of a macro)

		data WORK.ONE_FILE ;

		%let _EFIERR_ = 0; /* set the ERROR detection macro variable */
		infile "&path_in.&&bestand&ii."
		delimiter='09'x MISSOVER DSD lrecl=32767 firstobs=2 ;
	
		informat var1 best12. ;
		informat var2 best12.;
		
		/* informat var3, var4, etc... */
	
		format var1 best12.;
		format var2 best12. ;

		/* format var3, var4, etc... */
		
		input
		var1
		var2

		/* input var3, var4, etc... */
		;
		
		if _ERROR_ then call symputx('_EFIERR_',1);  /* set ERROR detection macro variable */

		run;

With my last textfile, there was a problem: SAS stopped importing after 3.000 lines, while there are 20.000 lines in the textfile, and SAS did not generate an error in the log.

After looking into the textfile, I saw the problem. There was a weird character in the textfile. I've made two screenshots (attached to this post). One taken from notepad and one taken from notepad++.

The weird character is visualized different in those 2 screenshots.

(i've saved the screenshot in a worddoc because this forum doesn't allow jpg-files)

My question is: is there a way that I can adapt my code so that SAS will import the whole textfile (and not stopping after 3000 lines), so basically that SAS ignores that weird character?

Or at least that it generates an error when it encounters such weird characters (and not stopping without warning)?

Re: importing data stops SAS in the middle of a textfile without generating an error

DavePrinsloo — Wed, 03 Jun 2020 11:09:51 GMT

What character set is SAS running? (You can see this from the SAS encoding option).
Is the file in the same character set? (Notepad ++ can help you to determine that)

I have an idea for a fix, but would like to test it before posting. What is the hex value of the offending character? (Notepad ++ can show that).

Re: importing data stops SAS in the middle of a textfile without generating an error

fre — Wed, 03 Jun 2020 12:02:15 GMT

Hi Dave, thank you for your reply.

In the SAS log, I can see ENCODING = UTF8

In Notepad++, I can see this matches (also UTF-8).

I didn't find the option to view the hex value, I've attached the piece of text that contains this weird character, hopefully you can see the hex value this way. It has to be an accent or something like that, I guess.

Re: importing data stops SAS in the middle of a textfile without generating an error

Tom — Wed, 03 Jun 2020 12:25:22 GMT

Try adding the IGNOREDOSEOF option to the INFILE statement.

https://communities.sas.com/t5/SAS-Programming/proc-import-reads-only-678-obs-from-csv-file/td-p/142741

Re: importing data stops SAS in the middle of a textfile without generating an error

fre — Wed, 03 Jun 2020 12:50:26 GMT

Indeed, the option IGNOREDOSEOF did the trick.

Thank you very much.