Excel has to pull all the data to the client to perform the aggregations on the tabular data and then display the results in the PivotTable. So, the most likely explanation of the delay is the time it takes to get the data to the client. That would also explain why it is much faster if you first open the data into a worksheet. It also takes Excel some time to perform the aggregations -- obviously longer for more data.
For better PivotTable performance on larger data, we typically recommend using OLAP cubes/server, so processing takes place on the server instead of the client.
Casey
ps- You can test Excel's PivotTable performance on your data outside of the SAS Add-in by manually creating a PivotTable in Excel using an external data source and the appropriate data provider (SAS IOM provider for server data, SAS Local Data provider for local data, SAS OLAP provider for OLAP cubes). It should be equivalent to the performance you experience in the SAS Addin.