Macro for loop hangs exactly at 27th cycle

[Edoedoedo]

I have this code:

/* Mock dataset */
data DaCensire;
    do i=1 to 4000;
        id=cats('z',put(rand('integer',1,999999),z6.));
        Newgroup='XXX';
        output;
    end;
run;

data DaCensire;
    set DaCensire;
    call symputx('NCensimenti',_N_);
    call symputx(cats('censimento',_N_), id);
    call symputx(cats('gruppo',_N_), Newgroup);
run;


%macro assegnazioneGruppo;
    
    %do h=1 %to &NCensimenti;
        %put Inserisco utenza &h/&NCensimenti e &&censimento&h nel gruppo &&gruppo&h;
            proc http url="https://xxx/identities/groups/&&gruppo&h/userMembers/&&censimento&h"
                method="PUT"
                OAUTH_BEARER=SAS_SERVICES;
                debug level = 1;
            run;
        %put Fine inserimento utenza &h/&NCensimenti e &&censimento&h nel gruppo &&gruppo&h;
    %end;

%mend;

%assegnazioneGruppo;

The macrovar NCensimenti is about 4.000.

This code works exactly fine, but a really weird thing happens: the 26th iteration runs to the end (I see in the log the last %put), but the 27th iteration never starts (I don't see the first %put) and the code freezes forever.

I taught there may be something with the 27th value, but when I run the whole code again the list is randomized, so the 27th value changes every time but still, exactly at the 27th iteration, it never starts.

What may be happening?

Thanks

Regards

[tsap]

Can you provide some logic to generate dummy data that represents the data on 'DaCensire'? That would be helpful in understanding the logic better. Thanks

[Edoedoedo]

Sure, I've just edited the code with a mock dataset. Just to clarify, the dataset 'DaCensire' contains a list of user id and a SASViya group they should belong to. Then, with the macro %assegnazioneGruppo, an http call is executed against Viya API to add the i-th user to its group.

[ballardw]

When something regarding a web site or service quits working at a fixed number of iterations regardless of the data content I start to suspect some sort of setting at the web site. You might be triggering a defense against denial of service attacks, web crawlers or any number of things hitting the same site numerous times in a short period of time.

You did not attach any LOG to inspect.

[Edoedoedo]

Because there is nothing to inspect in the log, it just freezes... The web site is the standard SASViya REST APIs for handling identities.

I add here the log from the 25th to the end of the log. I've added some red marks (*) to comment that line after:

.......
Inserisco utenza 25/2083 e z123456 nel gruppo XXX (*1)

> PUT /identities/groups/XXX/userMembers/z123456 HTTP/1.1

> User-Agent: SAS/9

> Host: sasviya

> Accept: */*

> Authorization: Bearer

> Connection: Keep-Alive

> Content-Length: 0

> Content-Type: application/octet-stream

>

< HTTP/1.1 201

< Cache-Control: no-cache, no-store, max-age=0, must-revalidate

< Content-Security-Policy: default-src 'self'; script-src 'self' 'unsafe-inline' 'unsafe-eval'; img-src 'self' *.sas.com blob:
data:; style-src 'self' 'unsafe-inline'; child-src 'self' blob: data: mailto:;

< Date: Tue, 25 Jun 2019 11:35:23 GMT

< Expires: 0

< Pragma: no-cache

< Server: Apache/2.4

< Strict-Transport-Security: max-age=31536000 ; includeSubDomains

< Vary: User-Agent

< X-Content-Type-Options: nosniff

< X-Frame-Options: SAMEORIGIN

< X-XSS-Protection: 1; mode=block

< Content-Length: 0

< Connection: keep-alive

<

NOTE: PROCEDURE HTTP ha utilizzato (tempo totale di elaborazione): (*2)

      real time           0.21 seconds
      cpu time            0.02 seconds


Fine inserimento utenza 25/2083 e z123456 nel gruppo XXX (*3)

Inserisco utenza 26/2083 e z123457 nel gruppo XXX (*4)

> PUT /identities/groups/XXX/userMembers/z123457 HTTP/1.1

> User-Agent: SAS/9

> Host: sasviya

> Accept: */*

> Authorization: Bearer

> Connection: Keep-Alive

> Content-Length: 0

> Content-Type: application/octet-stream

>

< HTTP/1.1 201

< Cache-Control: no-cache, no-store, max-age=0, must-revalidate

< Content-Security-Policy: default-src 'self'; script-src 'self' 'unsafe-inline' 'unsafe-eval'; img-src 'self' *.sas.com blob:
data:; style-src 'self' 'unsafe-inline'; child-src 'self' blob: data: mailto:;

< Date: Tue, 25 Jun 2019 11:35:33 GMT

< Expires: 0

< Pragma: no-cache

< Server: Apache/2.4

< Strict-Transport-Security: max-age=31536000 ; includeSubDomains

< Vary: User-Agent

< X-Content-Type-Options: nosniff

< X-Frame-Options: SAMEORIGIN

< X-XSS-Protection: 1; mode=block

< Content-Length: 0

< Connection: keep-alive

<

NOTE: PROCEDURE HTTP ha utilizzato (tempo totale di elaborazione): (*5)
      real time           0.04 seconds
      cpu time            0.00 seconds


Fine inserimento utenza 26/2083 e z123457 nel gruppo XXX (*6)
(*7)

Comments:

1. It corresponds to the very first %put of the cycle, so the 24th iteration has just started
2. The proc http called with the 24th user ended and is successful
3. It corresponds to the very last %put of the cycle, so the 24th iteration has just completely ended
4. It corresponds to the very first %put of the cycle, so the 25th iteration has just started
5. The proc http called with the 25th user ended and is successful
6. It corresponds to the very last %put of the cycle, so the 25th iteration has just completely ended
7. !!! It doesn't even start the next cycle with the first %put! If the problem was the proc http, I would expect that at least the first %put of the 27th iteration would have been printed! But it just stops doing anything, the code remains running forever, and nothing else happens.

[FriedEgg]

What interface are you using to submit the SAS code?

[Edoedoedo]

At first with Sas Studio V. Then directly with a scheduled Job (so without an interface, like command line) redirecting the log to a file.

In both the cases the behavior is the same as I described above.

Thank you

Regards

[FriedEgg]

Ok, the best next step I can think of is to try to get some additional logging, from when it hangs.

When you submit from the command line, I want you to specify the following command line option:

-LOGPARM "WRITE=IMMEDIATE"

In your SAS program, let's also increase the log verbosity by adding the following line of code at the top of the program

OPTIONS MLOGIC MLOGICNEST MPRINT MPRINTNEST MSGLEVEL=I;

In your HTTP Procedure call you use the debug statement. Let's also increase the level

PROC HTTP [...];
  DEBUG LEVEL = 3;
RUN;

This all will produce a significantly larger log file, but specifically, what we care about most is if any additional log information is produced when the loop gets 'stuck' and the process appears to hang.  As well as any potential logging produced when you, I assume, terminate the process.

[Edoedoedo]

Thanks!

I'm having difficulties setting

-LOGPARM "WRITE=IMMEDIATE"

because the program is launched as a job in the SASJobExecution (so it's like a batch but there isn't a real command line where I can put parameters) and inside I redirect the log to a file with proc printto. How to set the LOGPARM using SASJobExecution?