BookmarkSubscribeRSS Feed
🔒 This topic is solved and locked. Need further help from the community? Please sign in and ask a new question.
cczzzl
Calcite | Level 5

Hi,

 

I have a number of daily trading data set, which are named as t20161201, t20161202, etc. 

I want to compute a cumulative trading volume (e.g. volume + size), and once the cumulative volume exceeds a specified threshold, then stop the computing and output this single obs (since I want to know how long to get this volume). 

The problem is that sometimes the threshold is very large, and it may take several days for a stock to reach this trading volume. So I have to judge whether I need to append next day's data  to compute the volume until it meets the threshold. 

One possible logic is to combine these daily data set together and compute the volume directly, but the size of these data set could be very large ( 4g per data set), so it may be inefficient to compute it in this way. 

In this case, I have to judge whether I can get the volume within one day, if not, then I need to append next day' data and compute again, then I have to judge again and compute again...until the volume meets the threshold.

I am struggling to write the code. 

Any helps is very appreciated!! 

1 ACCEPTED SOLUTION

Accepted Solutions
mkeintz
PROC Star

Make a list of ALL the daily datasets, and SET them with "open=defer" which tells sas to reused the same buffer for each dataset, rather than create a buffer  for each. Then once the threshold has been reached, output and stop.   No need to conditionally add datasets.  It's easier to conditionally stop reading.

 

This example reads all the December 2012 datasets, which you have named very usefully, allowing use of the name prefix for all the daily datasets of interest.

 

data want;

   set t 201612: open=defer;

   volume+size;

   if volume>=threshold;

   output;

   stop;

run;

--------------------------
The hash OUTPUT method will overwrite a SAS data set, but not append. That can be costly. Consider voting for Add a HASH object method which would append a hash object to an existing SAS data set

Would enabling PROC SORT to simultaneously output multiple datasets be useful? Then vote for
Allow PROC SORT to output multiple datasets

--------------------------

View solution in original post

5 REPLIES 5
Shmuel
Garnet | Level 18

Do you know the "volume + size" of each of your datasets?

I assume that appending two datasets will result in the sum of those two.

You can simulate the appending result manually  and decide what period of datasets

can be combined.

cczzzl
Calcite | Level 5
Sorry I didn't clarify the question.
The size is a variable in these data set.
The volume is a new variable computed based on the size.

My goal is trying to decide automatically by program otherwise it will be very time-consuming.
Shmuel
Garnet | Level 18

You can use a macro program, compute the volume of one day dataset.

Keep few global macro variables:  dataset volume, accumulated volume, max volume, flag to check is max reached.

If <accumulated> + <new volume> LE <max volume> then append data set to the accumulated dataset.

Skilton of program should look like:

%let accumulated_volume = 0;
%let max_volume = ...;
%global new_volume;
%let flag = 0;
proc datasets lib=<library>; delete acc_dataset; quit;

%macro check(dsname);
  %if flag ne 0 %then %goto exit;
   data _null_;
       set &dsname end=eof;
           ... compute volume by size...
          if eof then call sympy('new_volume', <computed volume>);
   run;
   %if %eval(&accumulated_volume + &new_volume) LE &max_volume
   %then %do;
          %let accumulated_volume = %eval(&accumulated_volume + &new_volume);
          proc append base=acc_dataset data=&dsn; run;
    %end;
   %else %do;
       %put Max Volume reached;
       %let flag = 1;
   %end;
%exit: /* <<< line added */ %mend check; %check(<day1 dataset>);
%check(<day2 dataset>0;
...
%check(<last day dataset>);
Astounding
PROC Star

From what you describe, this should be an easy problem.  For example:

 

data want;

set t20161201 t20161202 t20161203 t20161204 t230161205;

volume +size;

if volume > 100000 then do;

   output;

   stop;

end;

run;

 

You decide how many data sets you want to add to the SET statement, and what the cutoff should be.  But SAS doesn't need to read any extra observations with this structure to the program.

mkeintz
PROC Star

Make a list of ALL the daily datasets, and SET them with "open=defer" which tells sas to reused the same buffer for each dataset, rather than create a buffer  for each. Then once the threshold has been reached, output and stop.   No need to conditionally add datasets.  It's easier to conditionally stop reading.

 

This example reads all the December 2012 datasets, which you have named very usefully, allowing use of the name prefix for all the daily datasets of interest.

 

data want;

   set t 201612: open=defer;

   volume+size;

   if volume>=threshold;

   output;

   stop;

run;

--------------------------
The hash OUTPUT method will overwrite a SAS data set, but not append. That can be costly. Consider voting for Add a HASH object method which would append a hash object to an existing SAS data set

Would enabling PROC SORT to simultaneously output multiple datasets be useful? Then vote for
Allow PROC SORT to output multiple datasets

--------------------------

sas-innovate-2024.png

Join us for SAS Innovate April 16-19 at the Aria in Las Vegas. Bring the team and save big with our group pricing for a limited time only.

Pre-conference courses and tutorials are filling up fast and are always a sellout. Register today to reserve your seat.

 

Register now!

How to Concatenate Values

Learn how use the CAT functions in SAS to join values from multiple variables into a single value.

Find more tutorials on the SAS Users YouTube channel.

Click image to register for webinarClick image to register for webinar

Classroom Training Available!

Select SAS Training centers are offering in-person courses. View upcoming courses for:

View all other training opportunities.

Discussion stats
  • 5 replies
  • 1104 views
  • 0 likes
  • 4 in conversation