turn on suggestions

Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.

Showing results for

Find a Community

- Home
- /
- SAS Programming
- /
- Base SAS Programming
- /
- shortest distance

Topic Options

- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Email to a Friend
- Report Inappropriate Content

01-23-2017 10:05 PM

Hi All,

I have a dataset with around 100000 records. I want to know only those three records that have shortest distance with a particular record. If I run the distance procedure then the program generates the output of matrix of 100000 variables giving distance of a variable with all other variables. It is actually not required and unecessarily it eats up my system time and sometimes evenhangs my system. If there is any other procedure that gives me the variables with just the shortest distance, please share.

Thanks in advance.....

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Email to a Friend
- Report Inappropriate Content

Posted in reply to deega

01-23-2017 11:17 PM

Hi,

What do you mean by a particular record?

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Email to a Friend
- Report Inappropriate Content

Posted in reply to stat_sas

01-24-2017 12:09 AM

@stat_sas

by particular record, i mean for each record, something like below

Record No. Nearest Record 1 Nearest Record 2 Nearest Record 3

1 5 6 7

2 1 2 3

3 2 1 6

4 5 1 6

5 : : :

6 : : :

7 : : :

8 : : :

9 : : :

10 : : :

:

:

by particular record, i mean for each record, something like below

Record No. Nearest Record 1 Nearest Record 2 Nearest Record 3

1 5 6 7

2 1 2 3

3 2 1 6

4 5 1 6

5 : : :

6 : : :

7 : : :

8 : : :

9 : : :

10 : : :

:

:

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Email to a Friend
- Report Inappropriate Content

Posted in reply to deega

01-24-2017 12:42 AM

Not sure but one way to get that is using k means clustering. Make a reference value and use that in proc fastclus in instat option to get the distances with regard to reference value. Then sort the output data set to flag the top 3 closest values.

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Email to a Friend
- Report Inappropriate Content

Posted in reply to deega

01-24-2017 05:43 AM

Deega:

1 5 6 7

2 1 2 3

3 2 1 6

4 5 1 6

In this sample data set, you say 5, 6, 7 are the records that are closer to Record 1. You show a small example as how you got 5, 6, and 7?

What variable(s) (not shown here) are used for this decision?

Provide an example and illustrate how you find them. Finding algorithm to do fast is simpler once your your example is clear.

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Email to a Friend
- Report Inappropriate Content

Posted in reply to deega

01-23-2017 11:49 PM

Is there any other way besides calculating all distances to identify the shortest 3 distances?

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Email to a Friend
- Report Inappropriate Content

Posted in reply to Reeza

01-24-2017 12:02 AM

@Reeza This is what my question is... Few days back I used SPSS Modeler, using its KNN Node, we can quickly find the nearest record ...

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Email to a Friend
- Report Inappropriate Content

Posted in reply to deega

01-24-2017 03:32 PM

It was a logical question. You can't find the nearest neighbour without calculating all values first. Perhaps the KNN process in SPSS precalculates the distance and then obtains the nearest neighbour when you request it. This means it isn't calculating the distance matrix every time.

You need to provide more details regarding what is the issue in your current process. Ideally, you can provide sample input, your code and a message with which step is inefficient. Then we'll be able to suggest alternatives for your.

PROC DISTANCE is relatively fast, but if you don't need it run each time and can cache the results somehow that's a better process.

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Email to a Friend
- Report Inappropriate Content

Posted in reply to deega

01-24-2017 08:25 AM

I assume you want the Euclidean distance.

If you know the "particular record", then this is an easy problem that you can complete in the DATA step.

1. Put the particular record first in the data set (or hard-code it into an array).

2. Use the EUCLID function to compute the Euclidean distance between the particular record and the others.

3. Sort the data by distance and use the first 3 records.

For example, the following DATA step computes the Euclidean distances between the numerical measurements for the first observation ("Alfred") and the other observations. The result shows that Mary, William, and Janet are the students whose numerical measurements are closest to Alfred, as measured by the Euclidean distance function:

```
data Dist(drop=i);
array refPt [3] _temporary_; /* automatically RETAINed */
array diff [3] _temporary_;
set sashelp.class;
array Pt [*] Height Weight Age;
/* initialize refPt, or use
array refPt[3] (val1 val2 val3); */
if _N_ = 1 then
do i = 1 to dim(Pt);
refPt[i] = Pt[i];
end;
/* compute difference between Pt and refPt */
do i = 1 to dim(pt);
diff[i] = refPt[i] - Pt[i];
end;
dist = euclid(of diff[*]);
if _N_>1;
run;
proc sort data=Dist;
by dist;
run;
proc print data=Dist(obs=3);
run;
```

You can generalize this problem. Instead of finding the points nearest to a single reference point, you can find the points in one group that are nearest to points in another group. See the article "Distances between observations in two groups."

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Email to a Friend
- Report Inappropriate Content

Posted in reply to Rick_SAS

01-24-2017 10:09 PM

@Rick_SAS

Thanks for the reply. I have not used arrays much and trying to understand your program. Could you please tell what is _N_ here ???

Thanks for the reply. I have not used arrays much and trying to understand your program. Could you please tell what is _N_ here ???

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Email to a Friend
- Report Inappropriate Content

Posted in reply to deega

01-24-2017 10:36 PM

_n_ is an automatic variable that counts the number of boundary steps. It is typically used as a pseudo row counter.

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Email to a Friend
- Report Inappropriate Content

Posted in reply to Rick_SAS

01-25-2017 12:07 AM

@Rick_SAS If I have around 1000000 records and for each record I need at least three nearest records,( means KNN with K=3) will this work? The Distance proc is not working because the output is 1000000 * 1000000 matrix.

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Email to a Friend
- Report Inappropriate Content

Posted in reply to deega

01-25-2017 05:47 AM

Please clarify two issue:

1. How many variables in this problem?

2. Do you have ONE reference point, and you want the three closest from among 1M observations? Or do you have 1M points and for each of those you want to find the three nearest neighbors? The first case requires computing 1M distances, The second case requires computing 1E12 distances.

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Email to a Friend
- Report Inappropriate Content

Posted in reply to Rick_SAS

01-25-2017 08:21 PM

The shape of my dataset is as follows:

S. No. | Var1 | Var2 | Var3 | Var4 | Var5:: | Var30 |

A1 | 1 | 2 | 1 | 3 | 1 | 5 |

A2 | 2 | 1 | 1 | 3 | 1 | 2 |

A3 | 1 | 1 | 2 | 1 | 2 | 1 |

A4 | 2 | 3 | 3 | 1 | 2 | 1 |

A5 | 1 | 2 | 1 | 2 | 1 | 2 |

A6 | 3 | 1 | 3 | 3 | 1 | 1 |

A7 | 2 | 1 | 2 | 1 | 2 | 1 |

A8 | 1 | 2 | 1 | 2 | 2 | 1 |

A9 | 1 | 1 | 3 | 5 | 2 | 2 |

A10:: | 2 | 2 | 1 | 0 | 0 | 0 |

A1000000 | 1 | 5 | 2 | 0 | 3 | 4 |

I want KNN (K=3) for each observation from A1 to A1000000. Actually I had 200M observations, first I made clusters using fastclus and thought of using distance procedure for calculating distance and sort and take the top 3 records. Some clusters have less records but some still have high number of records. I was able to use distance procedure till 100,000 records but it is not responding above this limit. When I tried KNN Node in SPSS it responded to higher number of records too, so I thought of customising program for distance instead of using Distance proc.

Please advise me something for this situation.

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Email to a Friend
- Report Inappropriate Content

Posted in reply to deega

01-25-2017 08:45 PM

Customizing only makes sense if you have some other manner of filtering your calculations. For example, if this was spatial you might limit it to neighbouring provinces/states.