Desktop productivity for business analysts and programmers

sometimes limits observations in data view

Reply
Contributor
Posts: 29

sometimes limits observations in data view

when no limit set. is this by design?
Trusted Advisor
Posts: 2,114

Re: sometimes limits observations in data view

Please be more specific. What data source? What version of EGuide?

I have noticed that SAS datasets come into the data view in 'chunks'. I suspect that is both for performance (to view the beginning of the data quickly) and memory utilization (EGuide puts data into memory whereas SAS processes one record at a time).
Contributor
Posts: 29

Re: sometimes limits observations in data view

sure thing. I'm running eg 4.2 and these are native sas data sets on our unix server. sometimes, for no reason, sas decides to limit the number of observations to view. what's even more frustrating is that right clicking the data set to view properties -> rows shows the filtered count. I have to slog thru the log to find out the real number of rows

thanks for your help here!
Frequent Contributor
Posts: 91

Re: sometimes limits observations in data view

Hi,

I have noticed the same problem. I am now using 4.3 and have not noticed this quirk so far.

Nick
Contributor
Posts: 29

Re: sometimes limits observations in data view

fingers crossed. thanks
Community Manager
Posts: 2,891

Re: sometimes limits observations in data view

The reason is that EG cannot determine the number of records that would be yielded by a VIEW. The information is not part of the VIEW definition (as it is for SAS data sets), and it could be quite expensive to count, since it requires a query to process it.

So the behavior you see is by design. Note that it affects just the data viewer and properties windows. When you run a task or query, the SAS session doing the work has access to the full set of records that are contained in the VIEW.

Chris
Contributor
Posts: 29

Re: sometimes limits observations in data view

allow me to clarify. sas will intermittently choose for me a row limit of 10,000. Checking the properties will show 10,000. I have no such limit set

thanks
Community Manager
Posts: 2,891

Re: sometimes limits observations in data view

Yes, EG does set that OBS= limit when viewing DBMS tables. It's done to improve performance, and it can make a big difference.

The OBS= is in effect only for the data viewer: any tasks or queries are run against the full data table.

There is an option in Tools->Options->Data Performance that you can adjust to change this behavior: specify a different limit or remove the limit altogether.

Chris
Contributor
Posts: 29

Re: sometimes limits observations in data view

by DBMS do you mean non SAS data sets?
Community Manager
Posts: 2,891

Re: sometimes limits observations in data view

Yes -- such as Oracle tables, DB2, Teradata, etc. Anything that you access via a libname with SAS/Access.

Chris
Contributor
Posts: 29

Re: sometimes limits observations in data view

Understood. Problem is that SAS exhibits said behavior on SAS data sets....not DBMS tables
Super Contributor
Posts: 356

Re: sometimes limits observations in data view

We also see the same behaviour on SAS data sets.
Community Manager
Posts: 2,891

Re: sometimes limits observations in data view

Do you know whether SAS is accessing the data via the META libname engine (instead of using the BASE or V9 engine directly)? That might have an effect on how EG interprets the style of access for this data source.

If you access the data via the META engine, that's a layer of indirection that sort of "hides" the location and data mechanism from your SAS process, and thus from EG.

See http://support.sas.com/techsup/technote/ts788.pdf for more info.

Chris
Contributor
Posts: 29

Re: sometimes limits observations in data view

sas is accessing the same library at all times. this behavior is intermittent. Sometimes sas exposes all 82 million records, sometimes it decides to limit to 10k. no changes in settings or libraries.
Super Contributor
Posts: 356

Re: sometimes limits observations in data view

In our case it is seen (but not limited to) some sas datasets in WORK so shouldn't be a meta lib. And as with other comments in this thread is intermittent.

Barry
Ask a Question
Discussion stats
  • 15 replies
  • 315 views
  • 0 likes
  • 6 in conversation