topic Re: How do I get rid of invalid characters error? in SAS Studio

How do I get rid of invalid characters error?

TaniaRTP — Thu, 06 Sep 2018 18:17:19 GMT

I'm extremely new to SAS Studio and SAS EG and I'm trying to takeover inherited programs. I've been using PC SAS for years. The program was written originally using SAS EG, however, I'm trying to switch to SS. If I submit the code in SAS EG, it works without issue.

When I submit the exact same code via SS, I get the following error, when the code is trying to write the WORK dataset out to a PDF using PROC REPORT.

ERROR: Invalid characters were present in the data.

ERROR: An error occurred while processing text data.

NOTE: The SAS System stopped processing this step because of errors.

There were not any issues reading the datasets from external source to WORK dataset. The error only popped up at the PROC REPORT.

proc report data=fjoin nowd headline headskip split='*' missing spacing=1
   style(header)=[font=('Arial',8pt)]
   style(column)=[font=('Arial',8pt)];
   column order1 order2 order3 matchx
      ("Data Management (Rave EDC as of &ravedt.)" dm_subno dm_begin dm_text dm_end dm_fromcopy)
      ("Drug Safety (Argus as of &argusdt)" ds_event ds_begindt ds_text ds_enddt ds_isclosed) stdyday color;
   by order1;
   define order1 / order noprint;
   define order2 / order noprint;
   define order3 / order noprint;
   define matchx      / display width=1 format=matchx. center ' ' style(column)=[font=('Symbol',8pt)];
   define dm_subno    / display width=6 format=z6. center 'Subject*Number' style(column)=[cellwidth=0.6in];
   define dm_begin    / display width=6 format=$20. center 'Onset*Date' style(column)=[cellwidth=0.9in];
   define dm_text     / display width=35 left flow 'Preferred Term/*Reported Term/*Causality' style(column)=[cellwidth=1.95in];
   define dm_end      / display width=6 format=$20. center flow 'Resolved*Date' style(column)=[cellwidth=0.9in];
   define dm_fromcopy / display width=8 format=1. center 'CRF Is*Copy?' style(column)=[cellwidth=0.5in];
   define ds_event    / display width=15 format=$20. left flow 'Case*Number' style(column)=[cellwidth=0.6in];
   define ds_begindt / display width=6 format=$11. center 'Onset*Date' style(column)=[cellwidth=0.9in];
   define ds_text     / display width=35 left flow 'Preferred Term/*Reported Term /*Causality' style(column)=[cellwidth=1.95in];
   define ds_enddt    / display width=6 format=$11. center 'Resolved*Date' style(column)=[cellwidth=0.9in];
   define ds_isclosed / display width=8 format=$yesno. center 'Case Is*Closed?' style(column)=[cellwidth=0.5in];
   define stdyday     / display width=5 format=best8. center 'Study*Day' style(column)=[cellwidth=0.5in];
   define color       / display noprint;
   compute color;
      if color='Red' then call define(_row_, "style", "style=[background=lightRed]");
      else if color='Yellow' then call define(_row_, "style", "style=[background=Yellow]");
   else if color='Green' then call define(_row_, "style", "style=[background=lightGreen]");
      else if color='White' then call define(_row_, "style", "style=[background=White]");
   endcomp;
   break after order1 / skip;
run;

Can somebody please help by pointing me to what option(s) I need to play around with? Many thanks in advance.

Re: How do I get rid of invalid characters error?

Reeza — Thu, 06 Sep 2018 18:26:55 GMT

Can you post the exact log? It usually indicates where the error originates so we don't have to read every line very specifically.

Did you by any chance move from a Unix to Windows system at all here?

Or did you copy and paste any code from different systems? Sometimes that can introduce sometimes invisible characters that mess things up.

@TaniaRTP wrote:

I'm extremely new to SAS Studio and SAS EG and I'm trying to takeover inherited programs. I've been using PC SAS for years. The program was written originally using SAS EG, however, I'm trying to switch to SS. If I submit the code in SAS EG, it works without issue.

When I submit the exact same code via SS, I get the following error, when the code is trying to write the WORK dataset out to a PDF using PROC REPORT.

ERROR: Invalid characters were present in the data.

ERROR: Invalid characters were present in the data.

ERROR: An error occurred while processing text data.

NOTE: The SAS System stopped processing this step because of errors.

There were not any issues reading the datasets from external source to WORK dataset. The error only popped up at the PROC REPORT.

proc report data=fjoin nowd headline headskip split='*' missing spacing=1
   style(header)=[font=('Arial',8pt)]
   style(column)=[font=('Arial',8pt)];
   column order1 order2 order3 matchx
      ("Data Management (Rave EDC as of &ravedt.)" dm_subno dm_begin dm_text dm_end dm_fromcopy)
      ("Drug Safety (Argus as of &argusdt)" ds_event ds_begindt ds_text ds_enddt ds_isclosed) stdyday color;
   by order1;
   define order1 / order noprint;
   define order2 / order noprint;
   define order3 / order noprint;
   define matchx      / display width=1 format=matchx. center ' ' style(column)=[font=('Symbol',8pt)];
   define dm_subno    / display width=6 format=z6. center 'Subject*Number' style(column)=[cellwidth=0.6in];
   define dm_begin    / display width=6 format=$20. center 'Onset*Date' style(column)=[cellwidth=0.9in];
   define dm_text     / display width=35 left flow 'Preferred Term/*Reported Term/*Causality' style(column)=[cellwidth=1.95in];
   define dm_end      / display width=6 format=$20. center flow 'Resolved*Date' style(column)=[cellwidth=0.9in];
   define dm_fromcopy / display width=8 format=1. center 'CRF Is*Copy?' style(column)=[cellwidth=0.5in];
   define ds_event    / display width=15 format=$20. left flow 'Case*Number' style(column)=[cellwidth=0.6in];
   define ds_begindt / display width=6 format=$11. center 'Onset*Date' style(column)=[cellwidth=0.9in];
   define ds_text     / display width=35 left flow 'Preferred Term/*Reported Term /*Causality' style(column)=[cellwidth=1.95in];
   define ds_enddt    / display width=6 format=$11. center 'Resolved*Date' style(column)=[cellwidth=0.9in];
   define ds_isclosed / display width=8 format=$yesno. center 'Case Is*Closed?' style(column)=[cellwidth=0.5in];
   define stdyday     / display width=5 format=best8. center 'Study*Day' style(column)=[cellwidth=0.5in];
   define color       / display noprint;
   compute color;
      if color='Red' then call define(_row_, "style", "style=[background=lightRed]");
      else if color='Yellow' then call define(_row_, "style", "style=[background=Yellow]");
   else if color='Green' then call define(_row_, "style", "style=[background=lightGreen]");
      else if color='White' then call define(_row_, "style", "style=[background=White]");
   endcomp;
   break after order1 / skip;
run;

Can somebody please help by pointing me to what option(s) I need to play around with? Many thanks in advance.

Re: How do I get rid of invalid characters error?

TaniaRTP — Thu, 06 Sep 2018 18:56:21 GMT

Yes, I've attached the log.

No, the programs did not move from Unix to Windows. The way it's setup here, we have access to both EG & SS. We connect remotely to SAS servers, so even though the program was built in EG, when I open it using SS, I'm accessing the EG program from the same remote location/servers. I did not have to copy/paste.

Thanks again!

Re: How do I get rid of invalid characters error?

Reeza — Thu, 06 Sep 2018 19:05:33 GMT

It's going to be a character variable.

Start by reducing proc report and removing half the character variables and running it. If it runs without error it's in the other half.

Repeat until you know which variable is causing the issue.

Then repeat again with say different time periods or such.

Once you've found the 'offending' record so to speak, then we can help you clean it up. From a quick scan of your code it looks like you may have free text fields which are difficult to enforce types ~~so it's possible its a data issue that may come up regardless of what machine you're running it on~~. (Sorry, you already said it run's fine on EG).

Maybe someone else will have a better solution, especially since it works in EG and not SS. I would also consider reporting it to SAS Tech Support once you isolate the issue because errors in EG vs SAS Studio like this should not happen IMO. You could also just contact them directly and they would help you work through this, though I suspect taking the approach above will be significantly faster.

Re: How do I get rid of invalid characters error?

data_null__ — Thu, 06 Sep 2018 19:24:50 GMT

This may have something to do with it. Set MSGLEVEL as indicated and see additional information you get.

 NOTE: One or more variables were converted because the data type is not supported by the V9 engine. For more details, run with 
       options MSGLEVEL=I.
 NOTE: The import data set has 2628 observations and 43 variables.

Re: How do I get rid of invalid characters error?

ballardw — Thu, 06 Sep 2018 20:25:46 GMT

Repeated use of option encoding or inencoding on the LIBNAME statements indicates that possibly someone was aware that the data may have encoding other than the current default system encoding.

 data fjoin (encoding=asciiany) ;

You may want to examine what your session encoding is

proc options option=encoding;    
run;

You may need to specify the encoding option in the places that you haven't, such as proc report.

Re: How do I get rid of invalid characters error?

Patrick — Sat, 08 Sep 2018 02:44:04 GMT

@TaniaRTP

Hmm... Same code but different outcome when run out of EG as compared to SS. Both EG and SS are the clients. The actual code and the error encountered is server side. So something environmental must be different.

1. Are you connecting to the same SAS machine and SAS application server with both clients?

2. After connecting to the application server have you run as very first step a Proc contents (or queried the dictionary tables) to check if there are differences in Option settings?

3. With EG (not sure about SS) one can set-up the project to execute additional code which eventually could change the environment. Have you already checked if that's the case?

And then it's eventually also worth to actually understand what these hurting characters are. I'd expect they come from the Excel file. You could write a step after the import and check all character variables for non-printable characters (just use an array _character_ and then loop over the vars to search for such characters) - Line Feeds (LF) in Excel cells can be "fun" for example.

Re: How do I get rid of invalid characters error?

Tom — Sat, 08 Sep 2018 15:37:02 GMT

Usually this is caused by having values in character variables that cannot be trancoded.

When you are running in with a single byte character encoding then what is actually in the character string doesn't matter much. But if you try to treat the same string of bytes as if it was UTF-8 then you could have a sequence of bytes that do not represent any known unicode character.

Re: How do I get rid of invalid characters error?

Cynthia_sas — Sat, 08 Sep 2018 18:21:08 GMT

Hi, even though the PROGRAMS did not work from Unix to SAS, the encoding on the server might have changed between when the data files were originally created and now. Was SAS on the server recently updated?
Cynthia

Re: How do I get rid of invalid characters error?

TaniaRTP — Mon, 17 Sep 2018 15:40:56 GMT

Thank you.

I have checked my session encoding (UTF-8). I have also specified the encoding option everywhere

--- PROC REPORT DATA=xxx(encoding=yyyy)

--- all my work datasets

I tried each of these encoding options: UTF-8, asciiany and "session".

Additionally, I tried to confirm if it was a particular observation by deleting observations various ways, but still get the error:

ERROR: Invalid characters were present in the data.

ERROR: An error occurred while processing text data.

I switched to Enterprise Guide & ran it with encoding=utf8, asciiany and they both ran without error, producing a PDF. Any other thoughts?

Re: How do I get rid of invalid characters error?

Reeza — Mon, 17 Sep 2018 15:43:46 GMT

Were you able to identify which fields are causing the issue?

Re: How do I get rid of invalid characters error?

TaniaRTP — Mon, 17 Sep 2018 15:44:08 GMT

I'm unaware if SAS was recently updated.

Re: How do I get rid of invalid characters error?

TaniaRTP — Mon, 17 Sep 2018 16:01:45 GMT

Thank you Reeza for the reminder. I hadn't looked at variables causing the issue, only observations.

It was the variable MATCHX. There was a format being applied (don't know why). I commented out the format and it works perfect!

value matchx 0='B4'x 1='D6'x;

Re: How do I get rid of invalid characters error?

TaniaRTP — Mon, 17 Sep 2018 16:03:26 GMT

Thank you Reeza for the reminder. I hadn't looked at variables causing the issue, only observations.

It was the variable MATCHX. There was a format being applied (don't know why). I commented out the format and it works perfect!

Re: How do I get rid of invalid characters error?

Reeza — Mon, 17 Sep 2018 16:04:51 GMT

I wonder if the format was corrupted somehow when switching versions....

Definitely compare outputs to make sure they're identical and glad you got the problem solved!

Re: How do I get rid of invalid characters error?

TaniaRTP — Mon, 17 Sep 2018 16:13:55 GMT

The format was created within the program so it wasn't being read from an external file.

I looked at old PDFs, and I can't ascertain what the purpose of the format was. The previous output displayed as 0 and 1 (the raw values), not special characters, so I think omitting the format is fine. I truly thank you for your replies & assistance.