Desktop productivity for business analysts and programmers

Insufficient Space Error while Importing Raw Data in EG

Accepted Solution Solved
Reply
Frequent Contributor
Posts: 85
Accepted Solution

Insufficient Space Error while Importing Raw Data in EG

Hi,

I am trying to import a raw data file (flat file in txt format) into SAS EG. The details of the file and data size are given below. I am getting 'Insufficient Space Error' which is pasted below.

Data size - 1.05 GB

file format - TXT file

No. of Rows - 4.5 Million

No. of Columns - 46

IMPORTERROR.png

I have checked my server space as well which has 45 GB of space. I tried importing in WORK library and also I attempted to import to one of the SAS registered permanently library however, I keep getting this error that the space is insufficient. Any suggestions would be greatly appreciated. Thanks in advance.


Accepted Solutions
Solution
‎02-27-2014 01:50 AM
Esteemed Advisor
Posts: 6,669

Re: Insufficient Space Error while Importing Raw Data in EG

1. What DBailey said, it depends a lot on the definition of your columns.

2. Anytime you have a SAS data set with long(er) char columns that are mostly filled with blanks, use the compress=yes data set option. The compute overhead is rather small and always offset by the reduced I/O load. With long text fields, the compression ratio can reach > 95% :smileyalert:

---------------------------------------------------------------------------------------------
Maxims of Maximally Efficient SAS Programmers

View solution in original post


All Replies
Super Contributor
Posts: 578

Re: Insufficient Space Error while Importing Raw Data in EG

When you say server space...you do mean free space and not total, right?

Some systems have user quotas so that any one user can't consume the entire disk.  Sometimes libraries map to different file systems.  Assuming your sys admin tells you there are no quotas, you might look at the path to the WORK library and get the free space available in that path. 

Frequent Contributor
Posts: 85

Re: Insufficient Space Error while Importing Raw Data in EG

even tried with WORK. the SAS server space is of 45 GB and its taking up the entire space and the import was not even 40% complete. Every SAS user in the organization has 15 GB of space. However, why will the data size explodes to such a size in first place? If it goes smooth, then the data size will reach close to 120 GB which all started with 1GB data? Kindly assist.

Super Contributor
Posts: 578

Re: Insufficient Space Error while Importing Raw Data in EG

How are you defining the columns on the import?  SAS does not have variable character length, so if you have an incoming file with 100 records and one column in each record with column width of 10 (1000 bytes or so) but you define the input column as char(100), the resulting SAS dataset will be 10,000 (100 records x 100 bytes per record).

Valued Guide
Posts: 2,111

Re: Insufficient Space Error while Importing Raw Data in EG

The space issue is with the bsl_data library, not work.  The could be on the same server share, but should not be.  In *nix, it is common for SysAdmins to limit the space for users or data shares.

Limit you import to a few thousand lines and see what it uses; then you can extrapolate your needs or modify your input.

Super Contributor
Posts: 578

Re: Insufficient Space Error while Importing Raw Data in EG

I was just trying to suggest he ask SAS to let him know how much free space it saw in each of the available libraries.  I think that even if WORK is on the same filesystem, it can have different quotas.

Solution
‎02-27-2014 01:50 AM
Esteemed Advisor
Posts: 6,669

Re: Insufficient Space Error while Importing Raw Data in EG

1. What DBailey said, it depends a lot on the definition of your columns.

2. Anytime you have a SAS data set with long(er) char columns that are mostly filled with blanks, use the compress=yes data set option. The compute overhead is rather small and always offset by the reduced I/O load. With long text fields, the compression ratio can reach > 95% :smileyalert:

---------------------------------------------------------------------------------------------
Maxims of Maximally Efficient SAS Programmers
Respected Advisor
Posts: 3,775

Re: Insufficient Space Error while Importing Raw Data in EG

Sounds like this has been mostly diagnosed but I will add this.  While I can't tell from your screen shot exactly what you are using the read the file, some sort of wizard or PROC IMPORT perhaps.  If that is the case I would abandon that and write a data step where you have control over how the variables are defined (no guessing) and the rest.  Plus using COMPRESS as suggested and you can probably get it all to work nicely.

Valued Guide
Posts: 3,206

Re: Insufficient Space Error while Importing Raw Data in EG

The Server side is a Unix system.
There is no problem in importing the data that it has put on temporary "\saswork"

When it should go to the real end location it bloops writing to BSL_DATA.SPSALES dataset. It is that location having issues of shortage in space.

You cannot check the space on Unix in the same way as WIndows. The HFS can be set up with many file-sytems each having his own limit.

The ulimit options on the account (inherited from teh object spawner)  should not be a problem as the data was cerated in work and being copied to another location

Check the location of BSL_DATA with the:  "df -g ."  command.

You could do that when having XCMD access using piping mentioning the real directory location instead of the "." .

You are running by the key satya on machine "sasdemo" is it a eductional purpose?

---->-- ja karman --<-----
Frequent Contributor
Posts: 85

Re: Insufficient Space Error while Importing Raw Data in EG

Guys, thanks very much for all your replies, it helped me to fix the issue. It was the char. length issue due to which the size was expanding. Now I have got my data set into SAS with the same size of the source file with no issue.

Esteemed Advisor
Posts: 6,669

Re: Insufficient Space Error while Importing Raw Data in EG

Just one thing: you might consider to mark one of the answers as correct so that your initial question ist listed as "answered". Likewise for helpful posts.

---------------------------------------------------------------------------------------------
Maxims of Maximally Efficient SAS Programmers
☑ This topic is SOLVED.

Need further help from the community? Please ask a new question.

Discussion stats
  • 10 replies
  • 1162 views
  • 10 likes
  • 6 in conversation