topic Re: Classifing new records using already classified observations with SAS in SAS Programming

Classifing new records using already classified observations with SAS

wlierman — Thu, 08 Jul 2021 23:03:32 GMT

I have a dataset of 35,000 observations. These are self-reported places of employment from a contact-tracing database of persons identified as having covid and completing treatment / quarantine or presumed to have been exposed to someone with covid and quarantine. The self-reported value is text like XYZ Drug Store, State University, Central Manufacturing, Safeway, Bob's Honda and so on.

I am classifying the place of employment according to the BLS NAICS Industrial Classification System. I have completed 1,279 obs. My idea is to use those completed records as sort of a training set and then apply SAS techniques to loop through the remaining records with identified keywords so that the basic Goods / Services designation can be made. Then from there hopefully identify the sub_type of firm. The schematic would look like this

NAICS / Industry-Sector Goods Services

NAICS subsector Natural Resources, Mining Trade, Transportation, Utilities

Construction Information

Manufacturing Financial Activities

Professional & Business Services

Education_Health Services

Leisure & Hospitality

Other

In the 1,279 records I have completed manually I have hit every one of these sub-sectors many time over.

I think (hope) SAS can help here but I don't know the best method to use. Are there technique/s in SAS so that identified keywords e.g., Bank, Restaurant, Bar, Food, School etc can be used to identify similar keywords in the remaining 33,000 plus rows of Places of Employment so that further identification can be automated.

I realize the matching won't obviously be close to 100% but even a match rate of 50 or 60% would be great.

Thank you for your help.

wklierman

Re: Classifing new records using already classified observations with SAS

ChrisNZ — Fri, 09 Jul 2021 00:44:26 GMT

Are you asking for Machine Learning techniques?

Re: Classifing new records using already classified observations with SAS

SASKiwi — Fri, 09 Jul 2021 02:40:49 GMT

"I have completed 1,279 obs" - What does this mean? If you have built some code that handles the first 1,279 rows in your data what is the problem with running it over the full 30,000 rows?

Re: Classifing new records using already classified observations with SAS

wlierman — Fri, 09 Jul 2021 03:59:36 GMT

That was done manually on an xlsx worksheet. To do 33,000 additional records will be oh, I'd say two weeks. That's why I hope there is some coding alternative with SAS.
Thanks.

Re: Classifing new records using already classified observations with SAS

wlierman — Fri, 09 Jul 2021 04:01:36 GMT

Hello,
Anything (Python, SAS, even R). The 1,299 obs was done manually on an xlsx worksheet and to complete the other 33,000 that's a good half-month work. And of course the data base increases every week.

Re: Classifing new records using already classified observations with SAS

SASKiwi — Fri, 09 Jul 2021 05:04:03 GMT

An initial exploration could be done by pulling the NAICS list(s) into a SAS dataset then using that to search your data. Depending on how similar the data sources are you might be surprised how good the match is. Start by importing the NAICS data into SAS and post an example of it here along with some rows from your data. Then we can provide a search method for you.

Re: Classifing new records using already classified observations with SAS

wlierman — Fri, 09 Jul 2021 13:31:11 GMT

Thanks. I will work to construct the NAICS.

wklierman

Re: Classifing new records using already classified observations with SAS

Reeza — Fri, 09 Jul 2021 15:30:26 GMT

Not sure if this is helpful or not, but you can buy this data as well....it is pricey though.

Re: Classifing new records using already classified observations with SAS

wlierman — Fri, 09 Jul 2021 16:18:28 GMT

I appreciate your input. The self-identified responses are pretty unique so to preserve that I think it'll be a combination of SAS search methods and more manual input.
Data...you got to love it...

Thanks

Re: Classifing new records using already classified observations with SAS

wlierman — Fri, 09 Jul 2021 23:57:14 GMT

Okay. I have transcribed the NAICS codes into a SAS Data Set. I have also included the place of employment field from my data that i manually transcribed.

I have attached the two files - both sas datasets. The arias_economics_occupations is the data after I did the manual classification on the 1,290 some records. I eliminated lines with no response to place of employment N/A, NA, n/a,na etc. So there are about 29,500 or so total obs.

I hope this is useful. Even being able to make a little progress on classification will be a big time saver.

Thank you.

wklierman

Re: Classifing new records using already classified observations with SAS

wlierman — Tue, 13 Jul 2021 19:10:39 GMT

Any ideas on the matching/classification question?

I have come across some potentially good SAS papers. Nothing looks straightforward but some combination of Select statements and maybe If/Then coding.

Thanks for looking into the problem.

wklierman