topic Re: Database append in SAS Programming

Database append

shiv999 — Thu, 11 Feb 2021 12:55:20 GMT

Hello SAS people
I have question I am using a append to load the data in odbc SQL server

I was using the libname statement as below to append some 8.5 million data with 100 columns which is working fine but taking around 1 hour to complete the load.

Do you think it is there a way to optimize it?
Shall I increase the INSERTBUFF to 100,000? Or any bulkload things please suggest.

Libname mylib ODBC DSN=XXX schema= xxx UID=XXX PWD=XXX IGNORE_READ_ONLY_COLUMNS= YES INSERTBUFF= 10000

Re: Database append

CarmineVerrell — Thu, 11 Feb 2021 14:26:00 GMT

It's hard to say if increasing the buff size would help, it could just be that you are dealing with a large dataset that takes time to access. This type of question requires lots of benchmarking. The best way to do this would be to provide us with a sample dataset and code so we can test. You can just use SAS Data instead of SQL Server and recreate your append so we can test ourselves.

If you decide that is too much work to provide a sample and would like to do your own benchmarking then you can use this SAS option to help.

Fullstimer sas option

FULLSTIMER SAS Option

You could also benchmark other programs in the future and collect performance metrics using a program that would read the log produced by your program to store those metrics. That program is called Logparse, I have included a link for your information.

219-30: Programmatically Measure SAS® Application Performance on Any Computer Platform with the New LOGPARSE SAS Macro

Hope this helps.

Carmine

Re: Database append

andreas_lds — Thu, 11 Feb 2021 14:41:31 GMT

Please post proc contents of the data you try to append, so that we see how large it really is.

Re: Database append

LinusH — Thu, 11 Feb 2021 15:56:31 GMT

Bulk load?

I mean that is the obvious thing to try.

Re: Database append

SASKiwi — Thu, 11 Feb 2021 20:26:37 GMT

Another option to try is DBCOMMIT which is set to 1000 by default. You would probably want to keep this value similar to INSERTBUFF.

I've found from experience that experimenting with different values of both INSERTBUFF and DBCOMMIT can make a big difference in table insert times.