Speed comparison for Python, using Pandas vs SWAT

[hamcap]

Has anyone done any speed comparisons between the two?

I have only had access to the trial environment, so the testing I've done have been on very small datasets.

These results have shown no whatsoever faster speed using SWAT, compared to Pandas. When manipulating data, they were both around the same speed, but Pandas has even been a couple of thousand times faster when doing grouping reports.

Anyone managed to do testing with large datasets and/or any other sort of testing?

[joeFurbee]

Hi hamcap

I spoke with a developer and he indicated that:

When doing small data sets, pandas is always going to win.  That's comparing an in-memory local operation (pandas) against a network service over possibly multiple servers being combined into a single result (CAS / SWAT).  However, try running analytics on a few gigabytes of data in pandas and CAS.  Pandas will probably just run out of memory and quit; CAS will be done in a few seconds.

[hamcap]

Hi Joe,

Thanks for your reply!

This was my thoughts too, that's why I wanted to know if anyone has ever tried comparing operations on large files (preferably >10GB).

I've never tried SAS for this, but pandas handles around 10GB with no problem. Though it takes a few minutes. I'm sure SAS might be faster here, but it would be nice to see comparisons on a few different operations (and a list of specs on the hardware), in the exactly same datasets.

[joeFurbee]

I'll dig deeper into this to see if I can find any metrics that have been published or any general guidelines on performance. I'll post anything that I find here.

[hamcap]

Hi Joe,

I was wondering whether you've had the chance to find any metrics yet?

KR