topic Re: SAS 9.4 incredibly slow to import/sort Stata data file in SAS Programming

SAS 9.4 incredibly slow to import/sort Stata data file

jsmall — Thu, 10 May 2018 15:06:56 GMT

Hi there,

My computer was recently upgraded to windows 10, so I changed over to SAS 9.4. Originally I had Windows 7 and was using SAS 9.3. I'm trying to import and sort a 2GB stata dataset that used to load in about 1-2 minutes with Windows7/SAS 9.3, Now it takes over 8 minutes. And then after it's imported, trying to merge this with any other dataset takes significantly more time.

I'm using data steps, not proc SQL since I'm old school and haven't converted over yet 🙂 Could this be the problem? Any suggestions?

Thank you!

Jen

Re: SAS 9.4 incredibly slow

Rick_SAS — Thu, 10 May 2018 14:46:53 GMT

Strange. I think we need more details. Posting your code would be good. For example, are you sorting by numerical variables or character? Are the characters variables encoded in WLATIN1 or something else?

Re: SAS 9.4 incredibly slow

ChrisHemedinger — Thu, 10 May 2018 15:06:23 GMT

The log would help too. I assume you're using PROC IMPORT with DBMS=dta.

Your SAS session encoding might be a factor -- if SAS has to transcode from Latin1 to UTF-8. Check the encoding with

PROC OPTIONS group=languagecontrol; run;

And is it the Import step or the Sort step that takes longer -- or both?

Re: SAS 9.4 incredibly slow to import/sort Stata data file

ballardw — Thu, 10 May 2018 16:00:17 GMT

Are you reloading Stata data that you had imported previously or is this a new data set?

I am asking because it almost sounds like you might be reading the same data source repeatedly which sounds inefficient.

Other things about your environment might have an impact. When you upgraded the SAS version did you move to a server version from stand-alone (or vice versa).

How much free memory do you have? I might be that your Windows10 set up isn't allowing as much memory for SAS if you upgraded OS software but no hardware.

Is network connectivity involved any where in the process?

Re: SAS 9.4 incredibly slow to import/sort Stata data file

jsmall — Tue, 15 May 2018 16:57:35 GMT

Thanks everyone, something was going on with Windows 10, it was storing my temporary programs in my C drive, which used up all my memory. I didn't encounter these problems at all with Windows 7, so instead of working with IT who know nothing about SAS, I just decided to go back to Windows 7-seemed like the quickest, easiest solution for now.

Re: SAS 9.4 incredibly slow to import/sort Stata data file

ChrisHemedinger — Tue, 15 May 2018 17:23:02 GMT

Thankfully you didn't have to revert to Windows XP 😉

In future, one thing to check is the TEMP and/or TMP environment variable, which you can control in System Properties, Environment Variables settings.

Most apps, including SAS, take their cue for the "temp space" root from one of those environment variables.

Your upgrade to Windows 10 might have affect that, or it might have changed the %USERPROFILE% location which in turn affects the Temp space.

Re: SAS 9.4 incredibly slow to import/sort Stata data file

jsmall — Tue, 15 May 2018 17:29:37 GMT

Yes! I'm having issues with the user profile! I didn't realize I was getting these error messages- any idea what's going on?:
NOTE: SASUSER.PROFILE.CATALOG does not exist and is being created from
NOTE: SASUSER.PROFBAK.CATALOG.

NOTE: SASUSER.PROFBAK.CATALOG has been copied to SASUSER.PROFILE.CATALOG.

Re: SAS 9.4 incredibly slow to import/sort Stata data file

ChrisHemedinger — Tue, 15 May 2018 17:34:52 GMT

That's different and might not be related. The SASUSER.PROFILE contains your SAS settings for things like key mappings, window layout, etc.

If it's not present or corrupted, SAS creates a new one. See the doc about how that works.