PROC NETFLOW size constraints

[deleted_user]

I am looking to calculate minimum paths in a large sparse network using PROC NETFLOW, and am striking runtime issues given the size of the network I want to analyse.

NETFLOW works pretty well for small(ish) networks, but seems to hit a wall at a certain point. Observed runtimes for differing sized networks I have tested appear as follows (note k = 000):

NNODES +++++ NARCS +++++ Runtime (secs)
8k +++++++++ 300k +++++++++ 2.5 sec
30k +++++++++ 2,300k +++++++++ 14.4 sec
42k +++++++++ 7,500k +++++++++ 6,331.0 sec !!!!

Is there just a limit to the size of the network NETFLOW can cope with?

**** Sample of the PROC statement I used to test the 14.1 second result *******
169 proc netflow
170 MEMREP
171 ARC_SINGLE_OBS
172 ARCS_ONLY_ARCDATA
173 NNODES = 30000
174 NARCS = 2300000
175 NNAS = 0
176 DEFCOST = 1
177 MAXIT1 = 100000
178 shortpath
179 sourcenode = P1028050
180 sinknode = P1026752
181 ARCDATA = arcs
182 arcout = spath (where =( _flow_ =1));
183 tail P1;
184 head P2;
185 run;

[Ed_Hughes_sas]

It's possible that, for your largest network, you are encountering SAS memory limits that can be adjusted via use of general SAS options such as MEMSIZE=. On what operating system are you running? What does the log from your SAS call for the 7.5M arc network look like? I'm interested specifically in the log messages produced by the MEMREP option that you're already using. These messages will tell us about the memory usage and about any paging that NETFLOW is doing.

If you like, we could take a look at your data and code and advise you--contact me at Ed.Hughes@sas.com.

Ed Hughes
Product Manager, SAS/OR

[Matthew_Galati]

Hi - can you send me your data? I'd like to try this out on a new product I am working on - PROC OPTGRAPH. For finding shortest paths, it should be very efficient - even for very large scale problems.

Thanks,
Matt - matthew.galati@sas.com