topic Re: Sorting issue, missing observations in SAS Enterprise Guide

Sorting issue, missing observations

matt23 — Thu, 28 Jun 2018 15:24:56 GMT

Hi, I run this PROC REG to get results for every hour and I get this error and I miss a result for hour 23.

ERROR: Data set WORK.MATT is not sorted in ascending sequence. The current BY group has Hour = 23 and
       the next BY group has Hour = 0.

This is my code:

proc sort data=matt (where=(Month in(6,7,8)));
by Month Hour;
run;

proc reg data=matt (where=(Month in(6,7,8)));
model Load=Temperature;
by Hour;
label Load="Load (MW)" Temperature="Temperature (degrees F)";
title 'Load vs Temperature (Summer)';
run;

Also, I should be getting results for about 900 observation for every hour and only get about 250. What is wrong with my code?

Re: Sorting issue, missing observations

Reeza — Thu, 28 Jun 2018 15:30:09 GMT

@matt23 wrote:

Hi, I run this PROC REG to get results for every hour and I get this error and I miss a result for hour 23.
ERROR: Data set WORK.MATT is not sorted in ascending sequence. The current BY group has Hour = 23 and
       the next BY group has Hour = 0.
This is my code:
proc sort data=matt (where=(Month in(6,7,8)));
by Month Hour;
run;

proc reg data=matt (where=(Month in(6,7,8)));
model Load=Temperature;
by Hour;
label Load="Load (MW)" Temperature="Temperature (degrees F)";
title 'Load vs Temperature (Summer)';
run;
Also, I should be getting results for about 900 observation for every hour and only get about 250. What is wrong with my code?

There are only 24 hours so specifying "BY hour;" would give you 24 estimates.

It means your BY group isn't really hour, it's probably day and hour.

If you're using HOUR it expects all hour=0 to be the same group which is not what you want.

Re: Sorting issue, missing observations

matt23 — Thu, 28 Jun 2018 15:33:14 GMT

How would I fix this? How would I specify that I want all data from months 6,7,8 and that I want it separately for every hour?

Re: Sorting issue, missing observations

Reeza — Thu, 28 Jun 2018 15:34:57 GMT

You need to uniquely identify each group that you want somehow. Which variables in your data set uniquely identify the group? If you don't have a day, what happens if the data is out of order? How can you be sure which hour goes with which day? Relying solely on order is not a good idea.

And note that if you only have a single measurement per record (ie per hour/day) then you would get no results.

@matt23 wrote:
How would I fix this? How would I specify that I want all data from months 6,7,8 and that I want it separately for every hour?

Re: Sorting issue, missing observations

matt23 — Thu, 28 Jun 2018 15:39:43 GMT

Should I just do this for every hour then ?

proc reg data=matt (where=(Month in(6,7,8)) AND (Hour in(1)));

Re: Sorting issue, missing observations

Astounding — Thu, 28 Jun 2018 15:43:35 GMT

You switch your BY statements. Sorting uses BY MONTH HOUR, but regression uses BY HOUR.

They need to be the same. I'm not sure which one should be used, since that depends on the output that you want. But the BY statement should be the same for both sorting and regression.

Re: Sorting issue, missing observations

matt23 — Thu, 28 Jun 2018 15:47:52 GMT

This solved it. Thank you so much !!

Re: Sorting issue, missing observations

Reeza — Thu, 28 Jun 2018 15:54:04 GMT

I'm sure you know but your data violates the assumptions for linear regression because there's likely serial correlation.