Newbie needs help with working with large files with FTP program

Reply
Frequent Contributor
Posts: 101

Newbie needs help with working with large files with FTP program

Hello,

I am relatively proficient in SAS, but have recently been asked to work with very large data that is stored on a server that I have to log into.  All files on the server can be downloaded in .sas7bdat format.  I am currently using Fetch as my FTP program and I am working on a Mac (OSX 10.6.8).  I have essentially two questions:

1) Is there a way that I can directly reference the database file structure from my SAS code if I am logged into the server? 

2) I have tried to download small parts of the data only my hard drive in order to get used to working with it, and I cannot get the INFILE statement to work in SAS, so I cannot import data stored in .sas7bdat format into SAS. 

If you were me, how would you go about getting the large amounts of data from the server, into SAS for analysis, and then output something usable?

Any help would be greatly appreciated,

John

Frequent Contributor
Posts: 83

Re: Newbie needs help with working with large files with FTP program

Posted in reply to mahler_ji

Try pulling in the data using filename: SAS(R) 9.2 Language Reference: Dictionary, Fourth Edition

Super User
Posts: 5,431

Re: Newbie needs help with working with large files with FTP program

Posted in reply to mahler_ji

You are using Mac, and I guess that you are sing a local SAS installation i parallels, or in a virtual machine?

.sas7bdat is usually platform independent, but there could be some restrictions, depending on platforms involved (operating system, 32/64-bit), SAS versions, locales used in the source environment and in the client environment.

The ftp program shouldn't be an issue, be sure to transfer your SAS data in binary mode.

A .sas7bdat file is accessed by issuing a libname statement that point to the directory of the file(s), not the file itself.

Data never sleeps
Super User
Posts: 5,431

Re: Newbie needs help with working with large files with FTP program

Posted in reply to mahler_ji

On question one, not if your login is via ftp.

If you have the possibility to map the the location as local network drive, then yes. Contact the responsible person for the source server to see if this is feasible.

Data never sleeps
Frequent Contributor
Posts: 101

Re: Newbie needs help with working with large files with FTP program

Linus,

Thank you for your help.  Specifically, I am not running SAS natively on my machine, I am running it from a server using X11.  And the data that I am working with is on the same server, and I am accessing the data through an FTP.  It is not feasible to get the data on my machine natively because the files are so large.  I am not able to get the LIBNAME statement to work.  Here is an excerpt from my code:

LIBNAME sasdata 'sftp://jmahler:@equity.mit.edu/.johnmahler/';

SET sasdata.Data1;

run;

The log says that the library cannot be created.

Am I unable to access data from using the FTP to the other server?

Frequent Contributor
Posts: 101

Re: Newbie needs help with working with large files with FTP program

Posted in reply to mahler_ji

As another piece of information, I changed the nickname that I am assigning to the library to "datalib" and that gave me different results, though it says something interesting now.  This is an excerpt from my log:

NOTE: Libname DATALIB refers to the same physical library as SASDATA.

WARNING: Library DATALIB does not exist.

NOTE: Libref DATALIB was successfully assigned as follows:

      Engine:        V9

      Physical Name: /mnt/nfs6/jmahler/sftp://jmahler:@equity.mit.edu/

So on one line it says that the library doesn't exist, but on the next one it says that it was successfully assigned???

Super Contributor
Posts: 358

Re: Newbie needs help with working with large files with FTP program

Posted in reply to mahler_ji

Hi:

If the SAS system that you are accessing and the data are on the same server, then you do not need to use FTP at all.

Just allocate the file to the SAS session with a libname statement and you should be good to go.    The FTP should only be needed to transfer the data to a different machine.

Frequent Contributor
Posts: 101

Re: Newbie needs help with working with large files with FTP program

Thank you all for your help, though I still cannot get it to work.

1) I cannot use the simple location of the file on the server, that has not worked.

2) I cannot figure out how to make the LIBNAME statement work, either when calling a file local to my computer, or a file that can be seen on the FTP interface that I have.  (I am using Fetch)

Can someone please show me some sample code that both creates a library from a native file, as well as a file that is seen on my FTP interface?

Thank you

Super Contributor
Posts: 644

Re: Newbie needs help with working with large files with FTP program

Posted in reply to mahler_ji

@mahler.ji

As far as I know there is no SAS implementation of a library engine that would support access to data through FTP.

If, as you say, the data is on the same server as the instance of SAS you are using, or on any attached server that you have normal (login) access to, then you should be able allocate a library using the fully referenced name of the folder that hoses the SAS data files:

libname ABC "//servername/level1/level2/level3/.../sasdata" ;

If for some reason you are forced to use FTP then use it outside SAS to copy the data to a folder on the server that you can assign with a standard libname.

Richard

Ask a Question
Discussion stats
  • 8 replies
  • 454 views
  • 6 likes
  • 5 in conversation