topic Re: Pairwise cosine similarity of a large dataset in SAS Programming https://communities.sas.com/t5/SAS-Programming/Pairwise-cosine-similarity-of-a-large-dataset/m-p/609915#M177581 <P>Calling&nbsp;<a href="https://communities.sas.com/t5/user/viewprofilepage/user-id/13684">@Rick_SAS</a>&nbsp;</P> Fri, 06 Dec 2019 11:43:32 GMT Ksharp 2019-12-06T11:43:32Z Pairwise cosine similarity of a large dataset https://communities.sas.com/t5/SAS-Programming/Pairwise-cosine-similarity-of-a-large-dataset/m-p/609851#M177562 <P>Hi,</P><P>&nbsp;</P><P>I have a table (matrix) like this:</P><P>&nbsp;</P><TABLE border="0" cellspacing="0" cellpadding="0"><TBODY><TR><TD>id</TD><TD>year</TD><TD>var1</TD><TD>var2</TD><TD>var3</TD><TD>….</TD><TD>var300</TD></TR><TR><TD>1</TD><TD>1997</TD><TD>3</TD><TD>4</TD><TD>5</TD><TD>&nbsp;</TD><TD>6</TD></TR><TR><TD>1</TD><TD>1998</TD><TD>5</TD><TD>2</TD><TD>1</TD><TD>&nbsp;</TD><TD>3</TD></TR><TR><TD>……</TD><TD>……</TD><TD>……</TD><TD>……</TD><TD>……</TD><TD>……</TD><TD>……</TD></TR><TR><TD>1</TD><TD>2007</TD><TD>5</TD><TD>3</TD><TD>6</TD><TD>&nbsp;</TD><TD>2</TD></TR><TR><TD>2</TD><TD>1997</TD><TD>1</TD><TD>1</TD><TD>2</TD><TD>&nbsp;</TD><TD>0</TD></TR><TR><TD>……</TD><TD>……</TD><TD>……</TD><TD>……</TD><TD>……</TD><TD>……</TD><TD>……</TD></TR><TR><TD>2</TD><TD>2007</TD><TD>3</TD><TD>1</TD><TD>6</TD><TD>&nbsp;</TD><TD>0</TD></TR><TR><TD>3</TD><TD>1997</TD><TD>2</TD><TD>4</TD><TD>5</TD><TD>&nbsp;</TD><TD>4</TD></TR><TR><TD>&nbsp;</TD><TD>……</TD><TD>……</TD><TD>……</TD><TD>……</TD><TD>……</TD><TD>……</TD></TR><TR><TD>3</TD><TD>2006</TD><TD>0</TD><TD>4</TD><TD>3</TD><TD>&nbsp;</TD><TD>4</TD></TR><TR><TD>&nbsp;</TD><TD>……</TD><TD>……</TD><TD>……</TD><TD>……</TD><TD>……</TD><TD>……</TD></TR><TR><TD>5000</TD><TD>1997</TD><TD>0</TD><TD>0</TD><TD>2</TD><TD>&nbsp;</TD><TD>6</TD></TR><TR><TD>……</TD><TD>……</TD><TD>……</TD><TD>……</TD><TD>……</TD><TD>……</TD><TD>……</TD></TR><TR><TD>5000</TD><TD>2006</TD><TD>3</TD><TD>1</TD><TD>2</TD><TD>&nbsp;</TD><TD>6</TD></TR></TBODY></TABLE><P>That said, I have a lot of observations and variables.</P><P>&nbsp;</P><P>Ideally, I want to calculate pairwise cosine similarity between two observations and output like this:</P><TABLE border="0" cellspacing="0" cellpadding="0"><TBODY><TR><TD>d1</TD><TD>id2</TD><TD>year</TD><TD>distance</TD></TR><TR><TD>1</TD><TD>2</TD><TD>1997</TD><TD>xx</TD></TR><TR><TD>1</TD><TD>3</TD><TD>1997</TD><TD>xx</TD></TR><TR><TD>…</TD><TD>…</TD><TD>&nbsp;</TD><TD>…</TD></TR><TR><TD>1</TD><TD>5000</TD><TD>2006</TD><TD>xx</TD></TR><TR><TD>2</TD><TD>1</TD><TD>1997</TD><TD>xx</TD></TR><TR><TD>…</TD><TD>…</TD><TD>&nbsp;</TD><TD>…</TD></TR><TR><TD>2</TD><TD>5000</TD><TD>2006</TD><TD>xx</TD></TR><TR><TD>…</TD><TD>…</TD><TD>&nbsp;</TD><TD>…</TD></TR></TBODY></TABLE><P>&nbsp;</P><P>&nbsp;</P><P>I am exploring proc distance and proc iml but have not figured it out yet.&nbsp; I will appreciate it very much if someone can help me out here.</P><P>&nbsp;</P><P>Thanks!</P> Fri, 06 Dec 2019 16:53:44 GMT https://communities.sas.com/t5/SAS-Programming/Pairwise-cosine-similarity-of-a-large-dataset/m-p/609851#M177562 daradanye 2019-12-06T16:53:44Z Re: Pairwise cosine similarity of a large dataset https://communities.sas.com/t5/SAS-Programming/Pairwise-cosine-similarity-of-a-large-dataset/m-p/609901#M177575 Maybe you find some useful information in this blog: <A href="https://blogs.sas.com/content/iml/2019/09/03/cosine-similarity.html" target="_blank">https://blogs.sas.com/content/iml/2019/09/03/cosine-similarity.html</A> Fri, 06 Dec 2019 09:49:55 GMT https://communities.sas.com/t5/SAS-Programming/Pairwise-cosine-similarity-of-a-large-dataset/m-p/609901#M177575 JosvanderVelden 2019-12-06T09:49:55Z Re: Pairwise cosine similarity of a large dataset https://communities.sas.com/t5/SAS-Programming/Pairwise-cosine-similarity-of-a-large-dataset/m-p/609915#M177581 <P>Calling&nbsp;<a href="https://communities.sas.com/t5/user/viewprofilepage/user-id/13684">@Rick_SAS</a>&nbsp;</P> Fri, 06 Dec 2019 11:43:32 GMT https://communities.sas.com/t5/SAS-Programming/Pairwise-cosine-similarity-of-a-large-dataset/m-p/609915#M177581 Ksharp 2019-12-06T11:43:32Z