BookmarkSubscribeRSS Feed
deleted_user
Not applicable
I would like to implement with SAS text Miner a co word analysis on a 30 000 scientific articles corpus.

My very first issue is to obtain the word count matrice (not reduced in vectors).

in line every start word
and in column ecery start word
and as value, their count of their respective co occurences.

Is anyone familiar with the co word analysis ?
Is any one willing to help me ?
2 REPLIES 2
RussAlbright
SAS Employee
Text Miner uses a compressed representation of the term-by-doc frequency matrix. You will find an OUT data set in the project data directory of your text miner run. Its label will include the string "OUT" in it. Since a 30,000 document collection will have as many as 500,000 to a million distinct terms, be sure to restrict your terms of interest with a start list. I give an example of creating the cooccurrence matrix with the following code which expands the compressed version to an uncompressed version and then computes the co-occurrence count with proc corr and the sscp option.

Thanks.
Russ

data myOUT;
input term doc count;
datalines;
1 1 1
1 3 1
1 4 1
2 2 1
2 3 2
3 1 2
3 3 2
3 4 1
4 2 2
4 4 1
5 3 2
;
run;

proc sort data=myOUT;
by doc term;
run;

data docbyterm;
set myOUT;
by doc;
array t{5};
retain t;
if first.doc then do;
do i=1 to 5;
t{i}=0;
end;
end;
t{term}=count;
if last.doc then do;
output;
end;
run;


proc corr data=docbyterm cov outp=cooccur sscp;
var t1-t5;
run;

Register today and join us virtually on June 16!
sasglobalforum.com | #SASGF

View now: on-demand content for SAS users

deleted_user
Not applicable
many thanks,

Charles

sas-innovate-2024.png

Join us for SAS Innovate April 16-19 at the Aria in Las Vegas. Bring the team and save big with our group pricing for a limited time only.

Pre-conference courses and tutorials are filling up fast and are always a sellout. Register today to reserve your seat.

 

Register now!

How to choose a machine learning algorithm

Use this tutorial as a handy guide to weigh the pros and cons of these commonly used machine learning algorithms.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 2 replies
  • 2069 views
  • 0 likes
  • 2 in conversation