Here's the tweaked version, attached. The main changes are:
1. ordered member site set count table generated early on.
2. Only sites that were visited by more than 6 members in the data set were retained for analysis.
I should have thought of this first, as there is the guideline that for logit models (and chi-square tests of association) the smallest effect crossing (cell) should have at leat 5 events for the model (test) to be valid. This reduces the number of sites, N, below 255, by eliminating the long tail of monosite visitors that are useless for what you want anyway. You should probably hard-coded that! This is important because the output of the last (and important) glmmod step will generate 1+N*N/2 pseudo-coded variables into the first big data set, and the limit for variables in a SAS data set is 256*256/2! Because this 1st large data set is generated by a proceedure and not proc sql or a data step, we have no control over the byte length of the pseudo-coded indicator numeric variables. Hence the data set easily grows large, even with the above subsetting in place. Not observing that above 255 limit also means that some macro variables concatenating sitenames et c. would oterwise exceed SAS macro variable limits.
3. Large data sets are deleted when no longer needed.
I estimate you will be OK if you have access to storage resource of the order of 150 GB, prefereably local but NAS or LAN server storage will do.
I have been running it as a test on my research platform and it has got past the 'hump' stage (the last glmmod step) which took over 5 hours real time. This could probably be reduced to 30 mins or less if you had sufficient local or very high bandwidth NAS.
Unfortunately there are a 4 data/proc sql steps to perform on evolutions of said large file, and some then the modelling has to run, so even under the best conditions this might take a few hours or more.
Further enhancements you could consider if you are going to run this regularly:
1. a better way to determine weights and freqs for member site pairs is to calculate the min(site1,site2) for every member pair visited, and use that rather than the min(of sites) as at present. Slighly more complicated but arguably more accurate / less assuming.
2. automate the final conditional probability calculation.
3. add a length statement in the first data step that processes the large data for all the binary indicator variables before the set statement, setting length of the big range to 3. That would speed up subsequent processing, including the modelling, a bit.
4. seriously rework the post-glmmod processing of the big data prior to modelling. I'm sure it can be done more efficiently in fewer steps with more elegant sql, aybe in 2 or 3 steps.
Let me know how you get on.
. should i consider truncating the data before the ? if you're interested in having a look at the full data-set i've attached it here as a text file.
