Upload to Databricks via ODBC is very slow

[js5 · Accepted solution]

Since we did not have SAS/Access to Spark licensed, I ended up implementing it myself. Performance is great in comparison. In brief:

1. create an empty table using implicit sql passthrough
2. export csv, formatting datetime and date to default spark formats
3. upload csv to S3
4. execute copy into statement on databricks warehouse using explicit sql passthrough