topic Re: proc http cannot return large GZIP Files in Developers

proc http cannot return large GZIP Files

daRocha — Thu, 15 Dec 2022 21:16:27 GMT

Hi all

My question is related with the issue reported here:

https://communities.sas.com/t5/Developers/proc-http-can-not-return-GZIP-file-gt-64-KB/td-p/645154

I am facing the exact same issue and in the original thread, even though the issue is marked as solved, the last comment made by @SimonDawson says:
"This will be looked at and addressed in a future release."

Has any progress been done or is there a workaround?

Thanks,

daRocha

Re: proc http cannot return large GZIP Files

andreas_lds — Fri, 16 Dec 2022 07:37:36 GMT

Are you using the latest sas release with hotfixes applied?

Re: proc http cannot return large GZIP Files

daRocha — Fri, 16 Dec 2022 16:16:28 GMT

All updates are in place. I am currently running my code on SAS 9.4. In particular: 9.04.01M7P080520

Re: proc http cannot return large GZIP Files

andreas_lds — Mon, 19 Dec 2022 11:23:49 GMT

Open a track with sas tech support, they should know if the bug will be fixed with m8 or if the plans to fix the issue have been abandoned.

Re: proc http cannot return large GZIP Files

joeFurbee — Wed, 21 Dec 2022 02:52:42 GMT

Hi @daRocha,

I checked with SAS R&D and found out:

The fix is in 9.4m7 and beyond and in Viya.

Thanks,

Joe

Re: proc http cannot return large GZIP Files

daRocha — Wed, 21 Dec 2022 12:53:51 GMT

Hi @joeFurbee

With the tests I ran it does not seem to be working with my version (9.04.01M7P080520)

For example, this code works for small files:

/*API Endpoint */
%let urlLink=https://reference-data-api.kaiko.io/v1/exchanges;

/*Temp work location*/
%let dataLoc=%sysfunc(getoption(work))\kkExhange;
/*Create location and assign lib*/
options dlcreatedir;
libname kkExchan "&dataLoc.";
options nodlcreatedir;

/* Results file*/
filename KaikoEx "&dataLoc.\metaKaiko.json.gz";

proc http 
  url="&urlLink."
    method='get'
    out=KaikoEx;
  headers
    'Accept' = 'application/json'
    'Accept-Encoding'= 'gzip, deflate, br';
run;

/*Filename poiting to the results file as gzip*/
filename KaikoEx zip "&dataLoc.\metaKaiko.json.gz" gzip;
filename KaikMap temp;

/*Json libname */
libname KaikoExc json fileref=KaikoEx map=KaikMap automap=create;

/*List json file contents*/
proc contents data=kaikoExc._all_;
run;

but when I use the same code for a larger gzip file it does not work. For large files, I am using the API Endpoint to test:
https://reference-data-api.kaiko.io/v1/instruments

Is there something wrong with my code or do I need to update my SAS version?

Thank you,

daRocha

Re: proc http cannot return large GZIP Files

joeFurbee — Thu, 22 Dec 2022 15:05:26 GMT

Hi @daRocha,

Your SAS version should have the patch. We may have to open a tech support ticket to investigate. Before we do that, the developer I spoke with asked the following:

Can he provide the debug logs for the run? Setting logger App.tk.HTTPC to debug would be good to see what is going on.

Can you provide the logs for us here?

Re: proc http cannot return large GZIP Files

joeFurbee — Thu, 22 Dec 2022 20:18:01 GMT

Hi @daRocha,

The developer took another look at this and while the error seems to be suppressed in his testing the file is not processed. IOW, we still have an issue. I'll follow the progress of the fix and will update you when it's fixed and in which version of SAS 9.

Thanks,

Joe