Database append

[shiv999]

Hello SAS people
I have question I am using a append to load the data in odbc SQL server

I was using the libname statement as below to append some 8.5 million data with 100 columns which is working fine but taking around 1 hour to complete the load.

Do you think it is there a way to optimize it?
Shall I increase the INSERTBUFF to 100,000? Or any bulkload things please suggest.

Libname mylib ODBC DSN=XXX schema= xxx UID=XXX PWD=XXX IGNORE_READ_ONLY_COLUMNS= YES INSERTBUFF= 10000

[CarmineVerrell]

It's hard to say if increasing the buff size would help, it could just be that you are dealing with a large dataset that takes time to access. This type of question requires lots of benchmarking. The best way to do this would be to provide us with a sample dataset and code so we can test.  You can just use SAS Data instead of SQL Server and recreate your append so we can test ourselves. 

If you decide that is too much work to provide a sample and would like to do your own benchmarking then you can use this SAS option to help.

Fullstimer sas option

FULLSTIMER SAS Option

You could also benchmark other programs in the future and collect performance metrics using a program that would read the log produced by your program to store those metrics. That program is called Logparse, I have included a link for your information. 

219-30: Programmatically Measure SAS® Application Performance on Any Computer Platform with the New ...

Hope this helps.

Carmine

[andreas_lds]

Please post proc contents of the data you try to append, so that we see how large it really is.

[LinusH]

Bulk load?

I mean that is the obvious thing to try.

[SASKiwi]

Another option to try is DBCOMMIT which is set to 1000 by default. You would probably want to keep this value similar to INSERTBUFF.

I've found from experience that experimenting with different values of both INSERTBUFF and DBCOMMIT can make a big difference in table insert times.