BookmarkSubscribeRSS Feed
shiv999
Calcite | Level 5
Hello SAS people
I have question I am using a append to load the data in odbc SQL server

I was using the libname statement as below to append some 8.5 million data with 100 columns which is working fine but taking around 1 hour to complete the load.

Do you think it is there a way to optimize it?
Shall I increase the INSERTBUFF to 100,000? Or any bulkload things please suggest.

Libname mylib ODBC DSN=XXX schema= xxx UID=XXX PWD=XXX IGNORE_READ_ONLY_COLUMNS= YES INSERTBUFF= 10000
4 REPLIES 4
CarmineVerrell
SAS Employee

It's hard to say if increasing the buff size would help, it could just be that you are dealing with a large dataset that takes time to access. This type of question requires lots of benchmarking. The best way to do this would be to provide us with a sample dataset and code so we can test.  You can just use SAS Data instead of SQL Server and recreate your append so we can test ourselves. 

 

If you decide that is too much work to provide a sample and would like to do your own benchmarking then you can use this SAS option to help.

Fullstimer sas option

FULLSTIMER SAS Option

 

You could also benchmark other programs in the future and collect performance metrics using a program that would read the log produced by your program to store those metrics. That program is called Logparse, I have included a link for your information. 

219-30: Programmatically Measure SAS® Application Performance on Any Computer Platform with the New ...

 

Hope this helps.

 

Carmine

andreas_lds
Jade | Level 19

Please post proc contents of the data you try to append, so that we see how large it really is.

 

LinusH
Tourmaline | Level 20

Bulk load?

I mean that is the obvious thing to try.

Data never sleeps
SASKiwi
PROC Star

Another option to try is DBCOMMIT which is set to 1000 by default. You would probably want to keep this value similar to INSERTBUFF.

 

I've found from experience that experimenting with different values of both INSERTBUFF and DBCOMMIT can make a big difference in table insert times.

Ready to join fellow brilliant minds for the SAS Hackathon?

Build your skills. Make connections. Enjoy creative freedom. Maybe change the world. Registration is now open through August 30th. Visit the SAS Hackathon homepage.

Register today!
How to Concatenate Values

Learn how use the CAT functions in SAS to join values from multiple variables into a single value.

Find more tutorials on the SAS Users YouTube channel.

Click image to register for webinarClick image to register for webinar

Classroom Training Available!

Select SAS Training centers are offering in-person courses. View upcoming courses for:

View all other training opportunities.

Discussion stats
  • 4 replies
  • 732 views
  • 8 likes
  • 5 in conversation