Pairwise cosine similarity of a large dataset

[daradanye]

Hi,

I have a table (matrix) like this:

id	year	var1	var2	var3	….	var300
1	1997	3	4	5		6
1	1998	5	2	1		3
……	……	……	……	……	……	……
1	2007	5	3	6		2
2	1997	1	1	2		0
……	……	……	……	……	……	……
2	2007	3	1	6		0
3	1997	2	4	5		4
	……	……	……	……	……	……
3	2006	0	4	3		4
	……	……	……	……	……	……
5000	1997	0	0	2		6
……	……	……	……	……	……	……
5000	2006	3	1	2		6

That said, I have a lot of observations and variables.

Ideally, I want to calculate pairwise cosine similarity between two observations and output like this:

d1	id2	year	distance
1	2	1997	xx
1	3	1997	xx
…	…		…
1	5000	2006	xx
2	1	1997	xx
…	…		…
2	5000	2006	xx
…	…		…

I am exploring proc distance and proc iml but have not figured it out yet.  I will appreciate it very much if someone can help me out here.

Thanks!

[JosvanderVelden]

Maybe you find some useful information in this blog: https://blogs.sas.com/content/iml/2019/09/03/cosine-similarity.html

[Ksharp]

Calling @Rick_SAS