topic Re: Speed comparison for Python, using Pandas vs SWAT in Developers

Speed comparison for Python, using Pandas vs SWAT

hamcap — Wed, 05 Sep 2018 14:40:18 GMT

Has anyone done any speed comparisons between the two?

I have only had access to the trial environment, so the testing I've done have been on very small datasets.

These results have shown no whatsoever faster speed using SWAT, compared to Pandas. When manipulating data, they were both around the same speed, but Pandas has even been a couple of thousand times faster when doing grouping reports.

Anyone managed to do testing with large datasets and/or any other sort of testing?

Re: Speed comparison for Python, using Pandas vs SWAT

joeFurbee — Wed, 05 Sep 2018 19:40:35 GMT

Hi hamcap

I spoke with a developer and he indicated that:

When doing small data sets, pandas is always going to win. That's comparing an in-memory local operation (pandas) against a network service over possibly multiple servers being combined into a single result (CAS / SWAT). However, try running analytics on a few gigabytes of data in pandas and CAS. Pandas will probably just run out of memory and quit; CAS will be done in a few seconds.

Re: Speed comparison for Python, using Pandas vs SWAT

hamcap — Thu, 06 Sep 2018 07:33:58 GMT

Hi Joe,

Thanks for your reply!

This was my thoughts too, that's why I wanted to know if anyone has ever tried comparing operations on large files (preferably >10GB).

I've never tried SAS for this, but pandas handles around 10GB with no problem. Though it takes a few minutes. I'm sure SAS might be faster here, but it would be nice to see comparisons on a few different operations (and a list of specs on the hardware), in the exactly same datasets.

Re: Speed comparison for Python, using Pandas vs SWAT

joeFurbee — Thu, 06 Sep 2018 12:23:07 GMT

I'll dig deeper into this to see if I can find any metrics that have been published or any general guidelines on performance. I'll post anything that I find here.

Re: Speed comparison for Python, using Pandas vs SWAT

hamcap — Thu, 29 Nov 2018 16:11:11 GMT

Hi Joe,

I was wondering whether you've had the chance to find any metrics yet?