We've encountered that behavior in the past, especially when the table used to populate the list comes from a database. The delay is the time it's taking to connect to the database, since it doesn't maintain a connection behind the scenes, and in some environments on some databases that can take time to establish.
As a general practice, I advise my team to use SAS datasets for lists like that specifically to avoid this issue. Most of the time these lists are pretty static, so they can be created and maintained in EG, for example. If they really are data driven from a database, then a recurring ETL job to keep the dataset current is the easiest way to decouple the list from the database connection.