Largest Data Set I Have Used

[Langbert]

I have used SAS in an elementary way for many years, but I am now working on a project that is bigger than what I'm used to.  I am extracting about five million observations from a government data source that has about 15 columns of data. 

What kind of hardware is required for a data set of that size?  Can I used my usual laptop, or do I need to get access to a mainframe?  Also, my college, CUNY, has a VMware-accessed version of SAS in the cloud. Will that work?  

Thanks for any advice, and please forgive my naivete. 

--Mitchell

[Reeza · Accepted solution]

5 million rows can be processed on a standard desktop in my experience. You may need to ensure you clean up so you don't have three 5GB files laying around that you're working with, but in general most things should work fine with a standard laptop (assuming 8GBRAM, 1TB drive). A cloud install via VM could mean accessing a SAS server (likely equivalent to what you're thinking of with a mainframe) or could mean accessing a limited box so that's not enough information to answer your question. Mainframes aren't really used anymore, I suspect you mean a Server version instead?

[jimbarbour]

---

@Reeza wrote:
 Mainframes aren't really used anymore

---

Interesting statement.  🙂  In a lot of large corporations, mainframes are still used for OLTP (think ATM networks, airline reservations, etc.).  A large university might have one even if only to run legacy systems on.  

In analytical environments, I think mainframe users are few and far between unless perhaps the mainframe is there for some other reason.

Jim

[Langbert]

The guy I'm working with is retired and probably is thinking about five or ten years ago. I'm not a computer guy, but I get that I haven't heard the word "mainframe" in years.

[Reeza]

---

@jimbarbour wrote:

---

@Reeza wrote:
 Mainframes aren't really used anymore

---

Interesting statement.  🙂  In a lot of large corporations, mainframes are still used for OLTP (think ATM networks, airline reservations, etc.).  A large university might have one even if only to run legacy systems on.  

In analytical environments, I think mainframe users are few and far between unless perhaps the mainframe is there for some other reason.

Jim

---

Definitely speaking from an analytical standpoint! I've worked with some really old data sources in my 15 years but have managed to avoid a mainframe system to date. 

[Langbert]

I'll check. I'm working with someone who knows more computerese than I. 

[Langbert]

Yeah, I think I meant server. I just arrived from the 1970s.

[jimbarbour]

Probably your laptop will work. I would certainly try it before going to another platform.  5 million isn't that big, but of course size is dependent on how many columns and the size of the columns.

Make sure you have OPTIONS COMPRESS=YES in effect if your columns are mainly character or COMPRESS=BINARY if your columns are mostly numeric -- at least that's the standard advice.  In actual practice I've found that sometimes binary compression is more space efficient than character compression even if there are a lot of character columns present.  To really know, one has to experiment, but for now, just making sure that you are using compression should go a long way towards efficient space allocation.

Jim