topic Re: Histogram Challenge in SAS Programming

Histogram Challenge

NKormanik — Mon, 23 Aug 2021 00:01:18 GMT

Two data sets, A and B. Continuous variables. X in A. Y in B.

Using Proc Univariate (or other) we can create histograms of the single variable in each dataset.

Imagine plotting the two histograms one above the other:

Objective: Sum the corresponding bins. (And then histogram that.)

Perhaps the same result can be obtained using potentially generated kernel data. Summing bin kernel data, then, and histogramming that would yield the same conclusions.

There might be other approaches as well.

Your thoughts appreciated....

Nicholas Kormanik

(As has been the case so many times before, my question above was not asked in the best of ways. Sorry. It evolves below....)

Re: Histogram Challenge

Reeza — Tue, 17 Aug 2021 00:43:16 GMT

I’m not sure what the problem is … if you combine the data sets univariate allows you to specify the bins interval and widths via the options and the histogram statement in proc sgplot also allows you to control the binwidths and intervals but not density overlays.

No procedure works with multiple data sets as the main data source AFAIK (mapping excluded).

Re: Histogram Challenge

NKormanik — Tue, 17 Aug 2021 05:11:20 GMT

No "problem", per se. Just was interested in your thoughts.

A priori, I haven't a clue what the bins distribution or sizes might be, until Proc Univariate, Histogram, shows me.

And, as one might anticipate, this case is just one of many. Say, 400 or so.

I am particularly interested in LOW areas, and especially areas where there is as little 'overlap' as possible between the two histograms.

"No procedure works with multiple data sets as the main data source..."

Important to be aware of this. Thus, some combining of the two datasets will be necessary.

There are options to output potentially created datasets: outhistogram and outkernel. But not sure if that would be the way to go in present case. Perhaps it is.

Re: Histogram Challenge

ballardw — Tue, 17 Aug 2021 11:24:58 GMT

My basic approach to what I think you are asking if it is to create a histogram is to combine data sets adding an identifier for which set each record comes from. Then use Proc SGplot with a histogram and the identifier as a Group variable

/* first make a couple of data sets with common variables*/
data a;
   set sashelp.cars;
   where type='Sedan';
run;

data b;
   set sashelp.cars;
   where type='SUV';
run;
/* combine*/

data combined;
   set a b indsname=dsn;
   source=dsn;
run;

Proc sgplot data=combined;
   histogram wheelbase /group=source transparency=.5;
run;

If you haven't seen it before the INDSNAME option creates a temporary variable that holds the name of the data set contributing the current observation. The Source=dsn; then makes a variable that stays in the data set with that information.

Then the Histogram statement overlays the two data sources with different color. The transparency allows seeing a "short" bar hidden by taller one with similar values of the variable. Options in the Histogram statement allow you to control the size of bins with options like BINWIDTH, which sets the range of values represented by bars Binwidth=5 would mean bins represent multiples of 5; NBINS where you specify the number of bins and SAS calculates the widths based on that.

Proc sgplot data=combined;
   histogram wheelbase /group=source transparency=.5
                        binwidth=5
   ;
run;


Proc sgplot data=combined;
   histogram wheelbase /group=source transparency=.5
                        nbins=25
   ;
run;

There are more options available. Read the documentation.

Re: Histogram Challenge

Ksharp — Tue, 17 Aug 2021 13:17:19 GMT

data a;
   set sashelp.cars;
   where type='Sedan';
run;

data b;
   set sashelp.cars;
   where type='SUV';
run;
/* combine*/

data combined;
   set a b indsname=dsn;
   source=dsn;
run;

Proc sgpanel data=combined;
   panelby source/layout=rowlattice novarname;
   histogram wheelbase /datalabel scale=count dataskin=pressed ;
   density wheelbase/type=kernal;
   density wheelbase/type=normal;
run;

Re: Histogram Challenge

NKormanik — Thu, 19 Aug 2021 00:51:08 GMT

Further explanation....

The histograms have been generated through 'machine learning' (Proc HPSplit).

The goal was to find a 'sweet spot' for some process.

The top histogram says the 'sweet spot' is GREATER THAN the parameter values indicated.

The lower histogram says the 'sweet spot' is LESS THAN the parameter values indicated.

Where there is a lot of overlap in parameter estimation, well, that's not good. 'Sweet spot' can't be GT and LT some value simultaneously.

Thus, the notion of adding the two histograms, and looking for the low area. Not at all perfect.

Bear in mind that there are over 400 such sets of histograms. And the objective is to find out which sets represents the best 'sweet spots' possible.

Presently the approach I'm using is to eyeball it.

Re: Histogram Challenge

ChrisNZ — Thu, 19 Aug 2021 07:26:09 GMT

Like this?

data a;
   set sashelp.cars;
   where type='Sedan';
run;

data b;
   set sashelp.cars;
   where type='SUV' ;
   wheelbase=wheelbase*1.4;
run;

/* combine*/
data combined;
   set a b indsname=dsn;
   source=dsn;
run;

proc sql noprint; select wheelbase into : lineval from (select wheelbase, count(*) as freq from combined group by wheelbase) having freq=min(freq);

proc sgpanel data=combined;
   panelby source/layout=rowlattice novarname;
   histogram wheelbase /datalabel scale=count dataskin=pressed ;lineparm X=&lineval Y=150 slope=. ;
   density wheelbase/type=kernal;
   density wheelbase/type=normal;
run;

Re: Histogram Challenge

NKormanik — Fri, 20 Aug 2021 07:23:51 GMT

@ChrisNZ @Ksharp @ballardw @Reeza All appreciated examples of what one might do. Good to have them here.

My original question probably wasn't exactly clear enough.

Using an analogy I'll try to illustrate a bit more:

Imagine a river. Suppose there are two banks (sides, rocks, soil). Call one a "Left Bank" (Greater Than) and the other "Right Bank" (Less Than). There is water in the middle between the Left Bank and Right Bank.

I'm trying to find the 'water' between the banks. I.e., a flat area between the histograms.

Below is another image that shows too much water between the banks:

I.e., huge open area between the histograms. Not helpful.

Looking for a Goldilocks-sized river.

Re: Histogram Challenge

ballardw — Fri, 20 Aug 2021 09:04:37 GMT

Still clear as mud.

Re: Histogram Challenge

ChrisNZ — Fri, 20 Aug 2021 10:04:55 GMT

Love the analogy and the pics! 🙂

What do you look for when you eyeball the charts by hand? What's the pattern you look for that could be coded?

Re: Histogram Challenge

Reeza — Fri, 20 Aug 2021 15:41:32 GMT

Plateau - you're looking for the plateaus.

Re: Histogram Challenge

NKormanik — Sat, 21 Aug 2021 00:20:00 GMT

Plateau? No, the opposite. A valley. And a nice medium-sized river running through the valley.

Re: Histogram Challenge

ChrisNZ — Sun, 22 Aug 2021 02:46:32 GMT

> nice medium-sized river running through the valley

This does not describe an algorithm. My simple logic looks for the lowest point and plots it. What would the logic to implement be?

Re: Histogram Challenge

Ksharp — Sat, 21 Aug 2021 09:38:36 GMT

So you want overlay these two histogram ?

data a;
   set sashelp.heart;
   where sex='Male';
run;

data b;
   set sashelp.heart;
   where sex='Female';
run;
/* combine*/

data combined;
   set a b indsname=dsn;
   source=dsn;
run;

Proc sgplot data=combined;
   histogram height /group=source datalabel scale=count  transparency=0.5;
   density height/type=kernel group=source;
run;

Re: Histogram Challenge

NKormanik — Sun, 22 Aug 2021 02:21:24 GMT

@ballardw Apt and clever response. That's what we see in the histograms shown at top.

Below is one that's even worse:

Clear as rocky banks and mud.

Re: Histogram Challenge

NKormanik — Sun, 22 Aug 2021 02:37:37 GMT

Remember, the 'machine learning' has attempted to show a 'sweet spot' -- some level along a continuous range of values that supposedly results in a best chance of outcome.

The top histogram is for Greater Than values. The lower histogram is for Less Than values.

Of the nearly 400 such sets of histograms, NONE appear to be perfect -- i.e., a reasonable clear distance between GT and LT. So, the objective is to find the 'best' of the 400.

Two more examples of "obviously nuts":

See the overlap? Like saying your 'sweet spot' is greater than 3, but less than 2.

Here's another:

So, on a scale of -71 to 71, the 'sweet spot' lies between -71 and 71.

How helpful. (NOT!)

Re: Histogram Challenge

Reeza — Sun, 22 Aug 2021 02:41:07 GMT

Do all intervals possible and calculate the metric of interest.
Then isolate the value of interest.

Re: Histogram Challenge

ballardw — Sun, 22 Aug 2021 06:41:28 GMT

@NKormanik wrote:

@ballardw Apt and clever response. That's what we see in the histograms shown at top.

Below is one that's even worse:

Clear as rocky banks and mud.

Can you add something to point out what you are finding of interest? Or mention coordinates?

I think we have, at least partially, one of those things where you have become so involved with your data that you expect others to know it without actually describing it clearly. I see lots of "low" spot. But haven't a clue what makes this "even worse" than the first one.

If you actually want to "sum" the two then the first part is to create the discrete bins, without actual data that might be interesting challenge by itself but with ODS graphics you can get usually get the graphed data using ODS output. If the two sets have x axis values/increments in common then combining two output sets with the counts and summing is a pretty trivial Proc freq or Proc means/summary step.

But starting data would be needed and likely the specific options you are /want to use for binning the data.Sgplot where you set bidwidths and and a starting offset might work best.

Re: Histogram Challenge

Ksharp — Sun, 22 Aug 2021 09:47:46 GMT

I think it is more like a OR problem ，not histogram graph.

Calling @Rick_SAS @RobPratt

Re: Histogram Challenge

NKormanik — Sun, 22 Aug 2021 09:56:56 GMT

By the way, what would the ideal set of histograms look like?

Glad you asked....

So, the programming objective is to find among the actual 400 sets of histograms, the ones that are as close as possible to the ideal.

(Actually, I already know that NONE even come close. But which one is 'best'? Are any even usable?)

(This was, you know, a well-funded Government study. Tax payers want.... well, maybe not.... At any rate.)