topic Re: Trouble with Group By, when row is missing in SAS Programming

Trouble with Group By, when row is missing

Banker337 — Wed, 04 Aug 2021 21:37:58 GMT

Good afternoon !

Currently got stuck with Proc SQL code. When I use group by operation, some missing rows are spoiling the output result.

According to SQL logic, I should use aggregation function (e.g. sum/avg) to collapse rows, but there is no aggregation function to exclude missing row from a group.

Example:

Input data

ID	Customer	Date	Sum	Field
ID1	C1	05.05.2021	15	AAA
ID2	C1	05.05.2021	16
ID3	C1	05.05.2021	21	AAA
ID4	C2	21.05.2021	30	BBB

I use this code:

proc sql noprint; 
  create table output as 
    select Customer, Date, Field
      ,sum(SUM) as Sum 
      ,count(*) as count
  from work.data_test
  group by Customer, Date, Field
;quit;

And get this:

Customer	Date	SUM	Field	count
C1	05.05.2021	36	AAA	2
C1	05.05.2021	16		1
C2	21.05.2021	30	BBB	1

Instead I need to get this:

Customer	Date	Sum	Field	count
C1	05.05.2021	52	AAA	3
C2	21.05.2021	30	BBB	1

Is there any possible solution ?

Thanks

Re: Trouble with Group By, when row is missing

sbxkoenk — Wed, 04 Aug 2021 21:42:21 GMT

The missing value for FIELD is of course a valid level, just like 'AAA' or 'BBB'.

If you don't want this, you have to impute the missing values.

You can for example do a Last Observation Carry Forward (LOCF = Forward Filling) to impute the missing value with a non-missing level.

Regards,

Koen

Re: Trouble with Group By, when row is missing

Reeza — Wed, 04 Aug 2021 22:30:54 GMT

Actually, you CAN use aggregation on character fields in SQL but not if you're aggregating by that variable as well. So do you actually need to aggregate by FIELD? How do you know what value it should be?

Re: Trouble with Group By, when row is missing

Ksharp — Thu, 05 Aug 2021 15:55:51 GMT

proc sql noprint;
create table output as
select Customer, Date, max(Field) as Field
,sum(SUM) as Sum
,count(*) as count
from work.data_test
group by Customer, Date
;quit;

Re: Trouble with Group By, when row is missing

Tom — Thu, 05 Aug 2021 16:03:34 GMT

What is the logic that says the missing FIELD should be replace by AAA in your example?

Why not include it as FIELD='BBB' or 'CCC'?

Re: Trouble with Group By, when row is missing

Banker337 — Fri, 06 Aug 2021 19:47:07 GMT

Same customer and date