An Exception has been encountered.

[DavidsM]

Hello community,

I am trying to save a very big data set into smaller data set so that I can draw a graph but am encountering this error "An Exception has been encountered".. I request for advise on how to solve this problem.

I also have 8GB ram but am wondering why drawing graphs beyond 1,000,000 observations ends into an error ' Insufficient memory', How can I increase my memory utilization?

Below is the procedure i have been running

ods select none;
proc kde data=Calibration;
bivar x y / out=Want;
run;
ods select all;

I will be glad to be assisted.

[Reeza]

What exactly is your question? How to increase memory SAS uses? How to use all points in your graph? How to split data set?

What SAS version are you using?

[DavidsM]

How to increase memory usage in SAS will be of good help for the start and I see if its memory problem. 

I think all the 3 above can solve the problem am encountering. I dont know exactly what the problem is because the data is extremity big more than 8,000,000,000 observations. 

If possible help me with method to increase memory and plotting all points.

Thank you for the quick response and assistance.

[Reeza]

What SAS version are you using? Visualizing 8 million points isn’t usually useful, is there a summarization that would make sense?

[DavidsM]

I am using SAS 9.4 University Edition and I will take any advise you give me so that I can solve the problem.

I also think visualizing reduced number of observation would be best and I was trying to reduce the data using proc KDE and save the output data but this error didn't allow.

Thank for responding.

[Reeza]

That's what I was afraid of... You can increase the RAM in the VM box settings, but the reality is that SAS UE has limitations and you're running into it. You can increase your RAM which will help I think the default is one or two GB.
You can also redirect your library by creating a user library, but this only works when you're doing data step and sort processes, I don't think it will help with graphics.
You cannot change the other settings.

Make sure to create the temp folder to hold data sets here if you do this and remember to periodically delete them, because SAS does not when you do this.
*create a user library;
libname user '/folders/myfolders/mywork';

[DavidsM]

The data sets are directed to the external hard drive and I have them read but plotting them is the problem. But I have not understood exactly how to increase my memsize. Which procedure do i use?

Thanks.

[Reeza]

It’s not a SASoption, it’s an option on Virtual Box, that you’re using to run SAS UE. Do not change the number of codes past 2.

[RW9]

As @Reeza has mentioned there are limits to what can be processed.  Remember SAS UE is a learning tool, its not a full fledged product for standard use, and thus has other constraints.  

You should never be plotting millions of observations, that is not at all useful.  Consider an 800*600 pixel graph could only hold 480k points, you are just wasting memory and processing.  Look at your data and summarise it.  I can't tell you how as you have not provided any, but some procedures which jump to mind are proc means - for sum, mean, stddev etc. summation.  It may also be that you need to group the data, or process it further to get a small set of data ready for plotting.  For instance we collect lab paramters, and these get processed to find changes from baseline values, then means then output to a separate plot per parameter.  Exact processing of course is down to what you want and what data you have, neither of which I know.  

[Kurt_Bremser]

---

@DavidsM wrote:

But I have not understood exactly how to increase my memsize.

---

You can't. MEMSIZE is set at SAS startup, and the configuration files for that are stored in a part of the UE VM that you can't reach for editing.

As the others said, UE is a learning tool designed for example data, not production-size datasets.

[Ksharp]

Could you try this ?

 https://communities.sas.com/t5/Graphics-Programming/RE-PLOTTING-GRAPH-FOR-DATA-CONTAINING-MANY-OBSER...

ods _all_ close;
proc kde data=Calibration;
bivar x y / out=Want;
run;
ods html;