topic Re: Co Word Analysis with SAS in SAS Data Science

Co Word Analysis with SAS

deleted_user — Wed, 01 Jul 2009 08:54:44 GMT

I would like to implement with SAS text Miner a co word analysis on a 30 000 scientific articles corpus.

My very first issue is to obtain the word count matrice (not reduced in vectors).

in line every start word
and in column ecery start word
and as value, their count of their respective co occurences.

Is anyone familiar with the co word analysis ?
Is any one willing to help me ?

Re: Co Word Analysis with SAS

RussAlbright — Fri, 17 Jul 2009 13:26:40 GMT

Text Miner uses a compressed representation of the term-by-doc frequency matrix. You will find an OUT data set in the project data directory of your text miner run. Its label will include the string "OUT" in it. Since a 30,000 document collection will have as many as 500,000 to a million distinct terms, be sure to restrict your terms of interest with a start list. I give an example of creating the cooccurrence matrix with the following code which expands the compressed version to an uncompressed version and then computes the co-occurrence count with proc corr and the sscp option.

Thanks.
Russ

data myOUT;
input term doc count;
datalines;
1 1 1
1 3 1
1 4 1
2 2 1
2 3 2
3 1 2
3 3 2
3 4 1
4 2 2
4 4 1
5 3 2
;
run;

proc sort data=myOUT;
by doc term;
run;

data docbyterm;
set myOUT;
by doc;
array t{5};
retain t;
if first.doc then do;
do i=1 to 5;
t{i}=0;
end;
end;
t{term}=count;
if last.doc then do;
output;
end;
run;

proc corr data=docbyterm cov outp=cooccur sscp;
var t1-t5;
run;

Re: Co Word Analysis with SAS

deleted_user — Fri, 17 Jul 2009 16:55:23 GMT

many thanks,

Charles