Ha, Bart: Vox populi, vox Dei! Thanks 1E6 for sharing and the good news. Kind regards Paul D. ... View more

Hi @SASKiwi: Have you read any of my papers on the subject? For example: https://support.sas.com/resources/papers/proceedings09/010-2009.pdf I'm not sure this was the latest but it should suffice.   One real-life use case (from my own experience) is described on page 17. As I recall, in that case using the APP function to move the data in blocks instead of one array item at a time resulted in the reduction of CPU time usage (which mattered since it was on the mainframe, and CPU time usage meant a lot of moolah) by about two orders of magnitude [sic].   I'd also add that using these functions is quite paramount in "character" bitmap search from the standpoint of performance if one wants to use every bit of the bitmap to reduce memory footprint (with a "numeric" bitmap, only the mantissa of each numeric array item can be used, i.e. 53 or 56 out of 64 bits, depending on the OS). If you have time and inspiration, you can find why here: https://support.sas.com/resources/papers/proceedings19/3101-2019.pdf on page 13 and on.   Kind regards Paul D.    ... View more

Quentin: Yes, it is, particularly for yours truly. And for quite a few folks who will find out that their production programs no longer work, especially my clients, both  former and current. So, now the once famous SAS backward compatibility is out the window - following the cadre who had made SAS what it was. I wonder what SAS flowers this mower will cut next under the pretext of cutting weeds.     Best Paul D.   ... View more

>But probably the most important advice is to resist the atavistic urge to use PROC SQL for a purpose that it is totally unsuited for.< Totally, Mark. There may be some perverse utility (like developing extra flexibility) in scratching the left ear with the right foot while standing on the left leg, but scratching it with the left hand is a tad saner. Or, one might be able to perform a heart surgery with an axe and cut a tree with a scalpel, but why do it when both tools are available. Kind regards Paul D. ... View more

Hi Naveen, it's nice to hear from you. I'm alive and kicking, just not active in this space too much. Hope WI treats you well! ... View more

Nice to hear from you, too, Art. I remember our interactions, both online and face to face, rather fondly. Best regards Paul ... View more

Art,  Just need to point out that this method was invented by Richard A. DeVenezia. In fact, the whole idea of storing hash object references in another hash object has come from his SAS-L post following SUGI in Montreal where Koen Vyverman and I did the first user-created SUGI presentation on the SAS hash object unveiled by SAS in 2003. During the aftertalk Q&A someone (sorry, don't remember who) asked if a hash object could contain other hash objects as data. I did not think so - and replied accordingly. A few days after the conference Richard posted a message on SAS-L where he demonstrated that not only that HOH (hash-of-hashes) was possible but how to use it in conjunction with the hash iterator to split an unsorted file. Basically, with this post alone, Richard laid the foundation of HOH and a practical example sufficient for anyone with a couple of hash crevices to expand its usage way beyond the task of file splitting. Later on, HOH has become a major part of the hash object book Don Henderson and I published in 2018.      ... View more

Right on, Bart! ... View more

Cześć Bartku,   Thanks for the plug.  Key-indexed search / aggregation always wins speed-wise if there is enough memory for it.  If not, an array-based hash table is the next fastest option coupled with a drastic memory reduction. I'd just use the simplest collision resolution policy, the linear probing, and set the hash table size at approximately twice the number of distinct keys. Code can be adapted almost verbatim from our (with Richard DeVenezia) last year's SESUG paper "Hashes From the Ashes", Section 2: https://www.lexjansen.com/sesug/2023/SESUG2023_Paper_203_Final_PDF.pdf   The hash object is a great tool (it would be funny if I thought otherwise) with gobs of functionality compared to array-based hashing but the latter can be faster/lighter at specialized tasks. You know, at times one needs a dedicated blade instead of those included in a Swiss Army knife.   Gorące pozdrowienia Paul D. ... View more

Hi, KachiM:   I understand that but you're sorting only a few elements. What happens if you try: call qsort (k, 1, 10000) ;  or many more than 10000?   Best regards Paul D.      ... View more

Cześć Bartku,   Frankly, I had no problems with my recursive FCMP implementation at any N, except having to use the quick sort all the way without switching to the straight insertion sort at some M => 9 point (you can find it in the SAS-L archives; or alternatively I can send it to you if you're interested).   Any purely SAS "external" array sort implementation I've tried runs about twice or more the time compared to SORTN - not surprising since it's internal, even though based on the heap sort, not quite the fastest sorting scheme. For the sake of curiosity, I've implemented the heap sort "externally" (i.e. in DATA step code using the LINK routine for heap-rebuilding recursion), and it's about 4 times slower than SORTN. However, PROTO C quick sort coupled with FCMP (Matt Kastin posted it on SAS-L circa 2017-18, IIRC) is almost on par with SORTN speed-wise (not surprising, either).       OTOH, external array-sorting routines are much more functionally flexible than SORTN/C in terms of ascending/descending, handling dupes, sorting parallel arrays, etc. All these options are the simplest to implement using the hash object, particularly when more than one array is used as the key, even though it comes at the expense of losing some speed compared to the hard-core implementations like quick sort.      Serdeczne pozdrowienia, Pavlo   ... View more

Hi, @KachiM : Have you tested this recursive implementation against even a modestly sizeable array - let's say, N=10000? For me, it hits a fatal exception both under Linux 64 and Windows 64 already at N=5000. Frankly, I can quite understand how this can work for teh quick sort algorithm since I don't see the fundamental quick sort "burn the candle from both ends" loop in it. Typically, it would look like so: p = a[l] ; i = l ; j = h + 1 ; do until (i >= j) ; do until (a[i] ge p) ; i ++ 1 ; end ; do until (p ge a[j]) ; j +- 1 ; end ; if i < j then call swap (a, i, j) ; end ; call swap (a, l, j) ;  Also, there is no provision against the quick sort's worst case scenario - i.e., its O(N) time when the array in question is already sorted.    I do have proper FCMP quick sort implementations, both recursive and iterative (shared circa 2018 on SAS-L). However, the straight iterative implementation used in my old QSORT macro works somewhat faster than either - most likely, because it avoids the overhead of internally buiding up and unrolling the recursion. It would be interesting to see how recursion implemented via the LINK routine would fare compared to the rest (just occurred to me, will try, time permitting).    Kind regards Paul D.   ... View more

Hey Peter, Thanks 1E6 for your generous plug.  😀 Kind regards Paul D. ... View more

Hi, Anders: You can copy the entire QSORT macro directly from the pdf paper Peter Clemmensen has linked, then use it as described in the (pretty extensive) comments in the header. Another advantages of using qsort or another method not based on SORTN/C is that you can also sort descending and control what to do with duplicates.  If you don't like to use QSORT for any reason, you can use the hash object to do what you need:   data _null_ ; retain lb 3 hb 8 order "a" ; * use d for descending ; array arr [10] (2 10 444 333 111 888 777 666 4 5) ; if _n_ = 1 then do ; dcl hash h (ordered:order, multidata:"y") ; h.definekey ("k") ; h.definedone () ; dcl hiter hi ("h") ; end ; do _n_ = lb to hb ; k = arr[_n_] ; h.add() ; end ; do _n_ = lb by 1 while (hi.next() = 0) ; arr[_n_] = k ; end ; put arr(*) ; run ; Alternatively, you can use CALL SORTN against an auxiliary array with the required range and then copy the sorted items back into the array in question, for example:     %let range = 3:8 ; data _null_ ; array arr [10] (2 10 444 333 111 888 777 666 4 5) ; array aux [&range] _temporary_ ; do _n_ = lbound (aux) to hbound (aux) ; aux [_n_] = arr [_n_] ; end ; call sortn (of aux[*]) ; do _n_ = lbound (aux) to hbound (aux) ; arr [_n_] = aux [_n_] ; end ; put arr(*) ; run ; As far as KSharp's bubble sort is concerned, it's okay for a few items but will get painfully slow even for a thousand since its running time is O(N**2), while all of the above approaches can handle millions of array items in a second or so.    Kind regards Paul D.   ... View more

Understood. Well, in this case you already have your answers and code samples.  ... View more