Viewing invalid value conflicts during PROC IMPORT after installing hotfix I9R004

[zgassoumis]

Hotfix I9R004 was very helpful, to eliminate the dqstring lines that would appear when there were large numbers of invalid value conflicts during PROC IMPORT. These were especially problematic when working with large datasets, so I am very happy to not have them clog up my log file anymore. However, I now have no way of knowing if there are invalid value conflicts or how many conflicts existed. Is there any way to track the number of invalid value conflicts that occur during PROC IMPORT now that the dqstring lines are suppressed?

For reference on the original dqstring issue solved by hotfix I9R004, see https://support.sas.com/kb/66/835.html

[Tom]

I don't understand. The Knowledge Base article you linked is talking about extra lines written to the log by the GUESSING process of PROC IMPORT.  Not the messages written to the log when the actual data step that PROC IMPORT generated runs and finds values that to not match the type that PROC IMPORT guessed for a variable.

If you have large text files to convert into SAS datasets you should probably not be using PROC IMPORT anyway.  You should just write the data step to read them yourself and eliminate the guessing completely.

[ballardw]

If you are reading multiple files that should contain similar data in terms of variable names, order, and type (numeric or character) you really really should not be using Proc Import.  A major reason is that each call to proc import can result in different variable types and lengths. Which will make combining what is supposed to be similar data a nightmare with errors and/or truncated (i.e. LOST) data.

Not only that but anything related to "invalid data" you can add specific checks and much more meaningful information written to a report file at the time it is read.

One example is to use custom informats to read data that should be from a known list of values. Using the Proc format Other=_error_ option on an Invalue then any value that makes into your source data is treated as invalid data, writes an error message to the log and sets the value to missing. I use this frequently as I have a data source that constantly added SITE codes. This behavior warned me when a file included a new site so that I could contact the source to get all of the other metadata that would be required to make valid reports that included that site.

I would also say that routinely receiving lots of "invalid value conflicts" means that Proc Import may not be guessing the type of variable correctly to begin with and possibly means you are not using the correct options in Proc Import to begin with.