topic Conditionally judgement whether append a new data or output in SAS Programming

Conditionally judgement whether append a new data or output

cczzzl — Mon, 25 Dec 2017 08:36:10 GMT

Hi,

I have a number of daily trading data set, which are named as t20161201, t20161202, etc.

I want to compute a cumulative trading volume (e.g. volume + size), and once the cumulative volume exceeds a specified threshold, then stop the computing and output this single obs (since I want to know how long to get this volume).

The problem is that sometimes the threshold is very large, and it may take several days for a stock to reach this trading volume. So I have to judge whether I need to append next day's data to compute the volume until it meets the threshold.

One possible logic is to combine these daily data set together and compute the volume directly, but the size of these data set could be very large ( 4g per data set), so it may be inefficient to compute it in this way.

In this case, I have to judge whether I can get the volume within one day, if not, then I need to append next day' data and compute again, then I have to judge again and compute again...until the volume meets the threshold.

I am struggling to write the code.

Any helps is very appreciated!!

Re: Conditionally judgement whether append a new data or output

Shmuel — Mon, 25 Dec 2017 09:53:40 GMT

Do you know the "volume + size" of each of your datasets?

I assume that appending two datasets will result in the sum of those two.

You can simulate the appending result manually and decide what period of datasets

can be combined.

Re: Conditionally judgement whether append a new data or output

cczzzl — Mon, 25 Dec 2017 11:08:12 GMT

Sorry I didn't clarify the question.
The size is a variable in these data set.
The volume is a new variable computed based on the size.

My goal is trying to decide automatically by program otherwise it will be very time-consuming.

Re: Conditionally judgement whether append a new data or output

Shmuel — Mon, 25 Dec 2017 13:05:35 GMT

You can use a macro program, compute the volume of one day dataset.

Keep few global macro variables: dataset volume, accumulated volume, max volume, flag to check is max reached.

If <accumulated> + <new volume> LE <max volume> then append data set to the accumulated dataset.

Skilton of program should look like:

%let accumulated_volume = 0;
%let max_volume = ...;
%global new_volume;
%let flag = 0;
proc datasets lib=<library>; delete acc_dataset; quit;

%macro check(dsname);
  %if flag ne 0 %then %goto exit;
   data _null_;
       set &dsname end=eof;
           ... compute volume by size...
          if eof then call sympy('new_volume', <computed volume>);
   run;
   %if %eval(&accumulated_volume + &new_volume) LE &max_volume
   %then %do;
          %let accumulated_volume = %eval(&accumulated_volume + &new_volume);
          proc append base=acc_dataset data=&dsn; run;
    %end;
   %else %do;
       %put Max Volume reached;
       %let flag = 1;
   %end;
%exit:       /* <<< line added */
%mend check;
%check(<day1 dataset>);
%check(<day2 dataset>0;
...
%check(<last day dataset>);

Re: Conditionally judgement whether append a new data or output

Astounding — Mon, 25 Dec 2017 14:42:59 GMT

From what you describe, this should be an easy problem. For example:

data want;

set t20161201 t20161202 t20161203 t20161204 t230161205;

volume +size;

if volume > 100000 then do;

output;

stop;

end;

run;

You decide how many data sets you want to add to the SET statement, and what the cutoff should be. But SAS doesn't need to read any extra observations with this structure to the program.

Re: Conditionally judgement whether append a new data or output

mkeintz — Tue, 26 Dec 2017 22:57:27 GMT

Make a list of ALL the daily datasets, and SET them with "open=defer" which tells sas to reused the same buffer for each dataset, rather than create a buffer for each. Then once the threshold has been reached, output and stop. No need to conditionally add datasets. It's easier to conditionally stop reading.

This example reads all the December 2012 datasets, which you have named very usefully, allowing use of the name prefix for all the daily datasets of interest.

data want;

set t 201612: open=defer;

volume+size;

if volume>=threshold;

output;

stop;

run;