How to Standardize Text Values with Text Miner in SAS Enterprise Miner?

[ertr]

Hello everybody,

I need to do some analysis with Text Miner in SAS Enterprise Miner. I have a Text variable which has a 500 character length and this variable has 30000 distinct values, I want to standartize this variable by excluding +/- signs or conjuctions like “or”, “with” and etc. I also want to transform this values from complicated value to pure value, let’s pretend that I have a value as below;

Papers *?

PAPER

paper

pAPER

Paper

pape

PAPE

Pape

Paper with

Paper or book

I want to see the above values only Paper, how I can do it with Text Miner? Can somebody hepl me to resolve this, please

Thanks

[RussAlbright]

Hi,

This sounds a little more like a data cleansing-fuzzy matching type of task. Take a look at something like this for sas functions and programs to help you standardize the input. https://www.lexjansen.com/sesug/2018/SESUG2018_Paper-143_Final_PDF.pdf

Text Miner is based on how terms tend to cooccur together within documents. The learning occurs across the collection based on how these patterns of cooccurrences exist. In your example, where you mostly have a single term per document, there is no cooccurrence going on and so Text Miner is not the best tool for this kind of task.