ERROR: Resource is write-locked by another thread

[dcortell]

Dear experts

I have a code iteratively scraping a list of URLs using the following code:

MPRINT(PROCHTTP_CHECK_RETURN):   filename src temp;
MPRINT(PROCHTTP_CHECK_RETURN):   proc http url="communities.sas.com/t5/forums/recentpostspage/user-id/407567" out=src;
MPRINT(PROCHTTP_CHECK_RETURN):   run;

I get during the execution the following error:

ERROR: Resource is write-locked by another thread.  File =/batchtmp/SAS_work2268000075CE_miseiddvp1/#LN00021.
ERROR: File is in use, /batchtmp/SAS_work2268000075CE_miseiddvp1/#LN00021.
ERROR: Resource is write-locked by another thread.  File =/batchtmp/SAS_work2268000075CE_miseiddvp1/#LN00021.
ERROR: File is in use, /batchtmp/SAS_work2268000075CE_miseiddvp1/#LN00021.
ERROR: Resource is write-locked by another thread.  File =/batchtmp/SAS_work2268000075CE_miseiddvp1/#LN00021.
ERROR: File is in use, /batchtmp/SAS_work2268000075CE_miseiddvp1/#LN00021.

There is any way I can work to ensure that the write-locking of the file do not persist?

[s_lassen]

I tried your code in SAS Studio, and did not get any errors.

I am not sure, but one explanation that you get this error could be that PROC HTTP tries to run several threads - if that's the case, you could try using

option nothreads;

in your macro before calling PROC HTTP.

[dcortell]

hanks. I will try that.

Also, to provide more details I add the extended log, as the error is persisting:

As I see the error triggered on the 15/09, and there was no other version of the code running in parallel on that day, I can also exclude that the error is generated by a parallel session running. so no idea at all why this is happening.

MPRINT(PROCHTTP_CHECK_RETURN):   filename src temp;
MPRINT(PROCHTTP_CHECK_RETURN):   proc http url="vle.sas.com/mod/page/view.php?id=139728" out=src;
MPRINT(PROCHTTP_CHECK_RETURN):   run;

ERROR: Resource is write-locked by another thread.  File =/batchtmp/SAS_workB9F50001F7E4_miseiddvp1/#LN00021.
ERROR: File is in use, /batchtmp/SAS_workB9F50001F7E4_miseiddvp1/#LN00021.
ERROR: Resource is write-locked by another thread.  File =/batchtmp/SAS_workB9F50001F7E4_miseiddvp1/#LN00021.
ERROR: File is in use, /batchtmp/SAS_workB9F50001F7E4_miseiddvp1/#LN00021.
ERROR: Resource is write-locked by another thread.  File =/batchtmp/SAS_workB9F50001F7E4_miseiddvp1/#LN00021.
ERROR: File is in use, /batchtmp/SAS_workB9F50001F7E4_miseiddvp1/#LN00021.
ERROR: Resource is write-locked by another thread.  File =/batchtmp/SAS_workB9F50001F7E4_miseiddvp1/#LN00021.
ERROR: File is in use, /batchtmp/SAS_workB9F50001F7E4_miseiddvp1/#LN00021.
ERROR: Resource is write-locked by another thread.  File =/batchtmp/SAS_workB9F50001F7E4_miseiddvp1/#LN00021.
ERROR: File is in use, /batchtmp/SAS_workB9F50001F7E4_miseiddvp1/#LN00021.
ERROR: Resource is write-locked by another thread.  File =/batchtmp/SAS_workB9F50001F7E4_miseiddvp1/#LN00021.
ERROR: File is in use, /batchtmp/SAS_workB9F50001F7E4_miseiddvp1/#LN00021.
NOTE: 200 OK
ERROR: Generic HTTP Client Error
NOTE: PROCEDURE HTTP used (Total process time):
      real time           0.22 seconds
      user cpu time       0.00 seconds
      system cpu time     0.00 seconds
      memory              487.37k
      OS Memory           579068.00k
      Timestamp           09/15/2023 07:31:55 PM
      Step Count                        3758064  Switch Count  0
      Page Faults                       0
      Page Reclaims                     68
      Page Swaps                        0
      Voluntary Context Switches        13
      Involuntary Context Switches      0
      Block Input Operations            0
      Block Output Operations           0
      
NOTE: The SAS System stopped processing this step because of errors.

[yabwon]

I think it's not the "threads" option:

1    options threads;
2    filename src temp;
3    proc http url="communities.sas.com/t5/forums/recentpostspage/user-id/407567" out=src;
4    run;

NOTE: 200 OK
NOTE: PROCEDURE HTTP used (Total process time):
      real time           0.74 seconds
      cpu time            0.00 seconds

I would rather consider network issue (maybe VPN is blocking access).

You can try filename URL approach too:

1
2    filename src temp lrecl=1 recfm=n;;
3    filename in URL "https://communities.sas.com/t5/forums/recentpostspage/user-id/407567" lrecl=1 recfm=n;
4
5    data _null_;
6      rc = fcopy("in", "src");
7      rctxt = sysmsg();
8      if rc then put rctxt ;
9    run;

NOTE: DATA statement used (Total process time):
      real time           10.40 seconds
      cpu time            0.28 seconds

Bart

[ballardw · Accepted solution]

Partial code is such a pain.

Since you are "iteratively scraping " where are you CLEARING the filename SRC assignment?

I suspect that you need a

Filename SRC CLEAR;

before attempting the subsequent "iteration". But where it may go can't tell.