https://communities.sas.com/t5/SAS-Data-Science/Text-Length-Population-Stability-Index-and-Language-detection/m-p/832893#M10303 <P>I'm training a language detection model using:</P><UL><LI><P>a training set, classified between English and not English sentences or small paragraps, where the length of the sentences can vary</P></LI><LI><P>a score set that is derived from web scraping activities, which length is usually more extensive than the length of the training set (a blog entry as example)</P></LI></UL><P>Now, the language identification model work with bigrams (combos of all the sequence of two characters extracted from text each set), which then being continuous variable, are re-classified into bins using deciles.</P><P>The upper and lower limits defining each bin and derived from the training set are then used to re-classify the score set' bigrams into bins too.</P><P>Now, being above the reclassification process, if in the training set I would extract each sequence independently, I would end up having much shorter text lengths than the average score set' entry, meaning that:</P><UL><LI><P>the BINS extracted from the training set would have a different distribution in the training set (skewed to left, as the lengths are shorter, therefore corresponding to lower frequencies of the bigrams)</P></LI><LI><P>than in the score set (skewed to right, as being longer text lengths, the bigrams would end up being all re-classified in the higher bins)</P></LI></UL><P>Now, this would generate a not stable Population Stability Index, and the training model therefore would be impacted by that.</P><P>So, considering a scenario where the training text dataset available is different in length of sentences than the scoring text dataset, in order to have a stable PSI, how should I proceed?</P><OL><LI><P>Combine together the sentences/paragraphs in the training set in order to reach an average text length similar to the average text length in the score set?</P></LI><LI><P>Any other suggestion/material which would explain how to best approach the subject?</P></LI></OL> Mon, 12 Sep 2022 13:29:10 GMT dcortell 2022-09-12T13:29:10Z https://communities.sas.com/t5/SAS-Data-Science/Text-Length-Population-Stability-Index-and-Language-detection/m-p/832893#M10303 <P>I'm training a language detection model using:</P><UL><LI><P>a training set, classified between English and not English sentences or small paragraps, where the length of the sentences can vary</P></LI><LI><P>a score set that is derived from web scraping activities, which length is usually more extensive than the length of the training set (a blog entry as example)</P></LI></UL><P>Now, the language identification model work with bigrams (combos of all the sequence of two characters extracted from text each set), which then being continuous variable, are re-classified into bins using deciles.</P><P>The upper and lower limits defining each bin and derived from the training set are then used to re-classify the score set' bigrams into bins too.</P><P>Now, being above the reclassification process, if in the training set I would extract each sequence independently, I would end up having much shorter text lengths than the average score set' entry, meaning that:</P><UL><LI><P>the BINS extracted from the training set would have a different distribution in the training set (skewed to left, as the lengths are shorter, therefore corresponding to lower frequencies of the bigrams)</P></LI><LI><P>than in the score set (skewed to right, as being longer text lengths, the bigrams would end up being all re-classified in the higher bins)</P></LI></UL><P>Now, this would generate a not stable Population Stability Index, and the training model therefore would be impacted by that.</P><P>So, considering a scenario where the training text dataset available is different in length of sentences than the scoring text dataset, in order to have a stable PSI, how should I proceed?</P><OL><LI><P>Combine together the sentences/paragraphs in the training set in order to reach an average text length similar to the average text length in the score set?</P></LI><LI><P>Any other suggestion/material which would explain how to best approach the subject?</P></LI></OL> Mon, 12 Sep 2022 13:29:10 GMT https://communities.sas.com/t5/SAS-Data-Science/Text-Length-Population-Stability-Index-and-Language-detection/m-p/832893#M10303 dcortell 2022-09-12T13:29:10Z