Conditionally judgement whether append a new data or output

Accepted Solution Solved
Reply
Occasional Contributor
Posts: 10
Accepted Solution

Conditionally judgement whether append a new data or output

Hi,

 

I have a number of daily trading data set, which are named as t20161201, t20161202, etc. 

I want to compute a cumulative trading volume (e.g. volume + size), and once the cumulative volume exceeds a specified threshold, then stop the computing and output this single obs (since I want to know how long to get this volume). 

The problem is that sometimes the threshold is very large, and it may take several days for a stock to reach this trading volume. So I have to judge whether I need to append next day's data  to compute the volume until it meets the threshold. 

One possible logic is to combine these daily data set together and compute the volume directly, but the size of these data set could be very large ( 4g per data set), so it may be inefficient to compute it in this way. 

In this case, I have to judge whether I can get the volume within one day, if not, then I need to append next day' data and compute again, then I have to judge again and compute again...until the volume meets the threshold.

I am struggling to write the code. 

Any helps is very appreciated!! 


Accepted Solutions
Solution
‎12-27-2017 10:03 PM
Trusted Advisor
Posts: 1,148

Re: Conditionally judgement whether append a new data or output

Make a list of ALL the daily datasets, and SET them with "open=defer" which tells sas to reused the same buffer for each dataset, rather than create a buffer  for each. Then once the threshold has been reached, output and stop.   No need to conditionally add datasets.  It's easier to conditionally stop reading.

 

This example reads all the December 2012 datasets, which you have named very usefully, allowing use of the name prefix for all the daily datasets of interest.

 

data want;

   set t 201612: open=defer;

   volume+size;

   if volume>=threshold;

   output;

   stop;

run;

View solution in original post


All Replies
Trusted Advisor
Posts: 1,683

Re: Conditionally judgement whether append a new data or output

Do you know the "volume + size" of each of your datasets?

I assume that appending two datasets will result in the sum of those two.

You can simulate the appending result manually  and decide what period of datasets

can be combined.

Occasional Contributor
Posts: 10

Re: Conditionally judgement whether append a new data or output

Sorry I didn't clarify the question.
The size is a variable in these data set.
The volume is a new variable computed based on the size.

My goal is trying to decide automatically by program otherwise it will be very time-consuming.
Trusted Advisor
Posts: 1,683

Re: Conditionally judgement whether append a new data or output

[ Edited ]

You can use a macro program, compute the volume of one day dataset.

Keep few global macro variables:  dataset volume, accumulated volume, max volume, flag to check is max reached.

If <accumulated> + <new volume> LE <max volume> then append data set to the accumulated dataset.

Skilton of program should look like:

%let accumulated_volume = 0;
%let max_volume = ...;
%global new_volume;
%let flag = 0;
proc datasets lib=<library>; delete acc_dataset; quit;

%macro check(dsname);
  %if flag ne 0 %then %goto exit;
   data _null_;
       set &dsname end=eof;
           ... compute volume by size...
          if eof then call sympy('new_volume', <computed volume>);
   run;
   %if %eval(&accumulated_volume + &new_volume) LE &max_volume
   %then %do;
          %let accumulated_volume = %eval(&accumulated_volume + &new_volume);
          proc append base=acc_dataset data=&dsn; run;
    %end;
   %else %do;
       %put Max Volume reached;
       %let flag = 1;
   %end;
%exit: /* <<< line added */ %mend check; %check(<day1 dataset>);
%check(<day2 dataset>0;
...
%check(<last day dataset>);
Super User
Posts: 5,987

Re: Conditionally judgement whether append a new data or output

From what you describe, this should be an easy problem.  For example:

 

data want;

set t20161201 t20161202 t20161203 t20161204 t230161205;

volume +size;

if volume > 100000 then do;

   output;

   stop;

end;

run;

 

You decide how many data sets you want to add to the SET statement, and what the cutoff should be.  But SAS doesn't need to read any extra observations with this structure to the program.

Solution
‎12-27-2017 10:03 PM
Trusted Advisor
Posts: 1,148

Re: Conditionally judgement whether append a new data or output

Make a list of ALL the daily datasets, and SET them with "open=defer" which tells sas to reused the same buffer for each dataset, rather than create a buffer  for each. Then once the threshold has been reached, output and stop.   No need to conditionally add datasets.  It's easier to conditionally stop reading.

 

This example reads all the December 2012 datasets, which you have named very usefully, allowing use of the name prefix for all the daily datasets of interest.

 

data want;

   set t 201612: open=defer;

   volume+size;

   if volume>=threshold;

   output;

   stop;

run;

☑ This topic is solved.

Need further help from the community? Please ask a new question.

Discussion stats
  • 5 replies
  • 200 views
  • 0 likes
  • 4 in conversation