Building models with SAS Enterprise Miner, SAS Factory Miner, SAS Visual Data Mining and Machine Learning or just with programming

Co Word Analysis with SAS

Reply
N/A
Posts: 0

Co Word Analysis with SAS

I would like to implement with SAS text Miner a co word analysis on a 30 000 scientific articles corpus.

My very first issue is to obtain the word count matrice (not reduced in vectors).

in line every start word
and in column ecery start word
and as value, their count of their respective co occurences.

Is anyone familiar with the co word analysis ?
Is any one willing to help me ?
SAS Employee
Posts: 32

Re: Co Word Analysis with SAS

Posted in reply to deleted_user
Text Miner uses a compressed representation of the term-by-doc frequency matrix. You will find an OUT data set in the project data directory of your text miner run. Its label will include the string "OUT" in it. Since a 30,000 document collection will have as many as 500,000 to a million distinct terms, be sure to restrict your terms of interest with a start list. I give an example of creating the cooccurrence matrix with the following code which expands the compressed version to an uncompressed version and then computes the co-occurrence count with proc corr and the sscp option.

Thanks.
Russ

data myOUT;
input term doc count;
datalines;
1 1 1
1 3 1
1 4 1
2 2 1
2 3 2
3 1 2
3 3 2
3 4 1
4 2 2
4 4 1
5 3 2
;
run;

proc sort data=myOUT;
by doc term;
run;

data docbyterm;
set myOUT;
by doc;
array t{5};
retain t;
if first.doc then do;
do i=1 to 5;
t{i}=0;
end;
end;
t{term}=count;
if last.doc then do;
output;
end;
run;


proc corr data=docbyterm cov outp=cooccur sscp;
var t1-t5;
run;
N/A
Posts: 0

Re: Co Word Analysis with SAS

Posted in reply to RussAlbright
many thanks,

Charles
Ask a Question
Discussion stats
  • 2 replies
  • 484 views
  • 0 likes
  • 2 in conversation