topic Re: Quick Export sas dataset into .txt for a big dataset in SAS Programming

Quick Export sas dataset into .txt for a big dataset

LineMoon — Wed, 14 Dec 2016 21:58:30 GMT

Hello experts,

I have used the solution below given from the esteemed advisor Tom, it works well, but I have a matter with a big data, the exporting take more than 15 minutes for 3000,000 observations, I would like to have less than 2 minutes, is it possible ?

Thank to @Tom and @ll other members

https://communities.sas.com/t5/Base-SAS-Programming/Exporting-a-sas-data-set-into-txt/m-p/314405#M68468

data _null_;
  file log ;
  set test(obs=1) test;
  length __name $32 __length 8 __value $200 ;
  do while (1=1);
    call vnext(__name);
    if lowcase(__name)='__name' then leave;
    if _n_=1 then __value = __name ;
    else __value = quote(strip(vvaluex(__name)));
    __length=lengthn(__value);
    put __value $varying200. __length ';' @ ;
  end;
  put;
run;

Re: Quick Export sas dataset into .txt for a big dataset

Kurt_Bremser — Wed, 14 Dec 2016 22:24:38 GMT

Run it with options fullstimer. If real time is considerably longer than CPU time, you are I/O bound and need to work on storage throughput.

Re: Quick Export sas dataset into .txt for a big dataset

Ksharp — Thu, 15 Dec 2016 07:33:38 GMT

You should not use VVALUEX(). PUT these variable names directly .

put x1 x2 .........;

Re: Quick Export sas dataset into .txt for a big dataset

LineMoon — Mon, 19 Dec 2016 13:02:06 GMT

@Ksharp: thank you

put x1 x2 .........;

put the variables names do not reply to my need because the name can change from x1 x2...toto Wi or Yi ...so on

Re: Quick Export sas dataset into .txt for a big dataset

LineMoon — Mon, 19 Dec 2016 13:07:10 GMT

@Kurt_Bremser : thank you

The real time still higher ? I am still open for the optimization of the time

Re: Quick Export sas dataset into .txt for a big dataset

Kurt_Bremser — Mon, 19 Dec 2016 13:16:38 GMT

If real time exceeds CPU time significantly (not just a few percent), you either

- have to share CPU power with other processes. Run the necessary system tools (Task Manager on Windows, nmon or topas on UNIX) to determine the processes competing for CPU.

- run into I/O bottlenecks

Possible avenues for I/O tuning:

- separate disks that are being read from those that are being written (keep source and target libraries on physically separate disks)

- set up disk arrays, so that more than one disk handles a certain I/O load; this is called striping

- migrate to SSDs