System Performance – The conversation you really want to have

3 Likes

In my experience one of the toughest issues I’ve faced on any implementation was the discussion we would have with the customer around the final performance of the system we were implementing. Regardless of when we had the discussion it took a lot of thought and usually more than one conversation to get the customer to understand the complexity involved in tackling performance. As we gathered requirements for the implementation, it was inevitable that a customer would say something like, “…and we want all of the queries to run in less than 5 seconds!” or something like that. Having been a customer myself, I can understand this kind of desire.

Most project managers and consultants will recognize this request as a non-functional requirement. Nonfunctional requirements address performance, security, data encryption, sign-on, administration, etc. Unlike most other nonfunctional requirements, what makes a performance requirement unique is that you won’t get to determine if you achieved it for certain until the end of the project when the system goes into production. To make matters more “interesting” from a SAS perspective, we don’t get to control all the parts of the system that contribute to the customer’s perception of performance. Keep in mind, it is the customer’s perception of performance that will be critical to the project’s success with this issue. So how should we deal with that??

It is a major part of a SAS project manager’s job to make sure the customer’s perception of all aspects of the project, including performance, are reasonable and realistic. This point supports one of my beliefs that project management is largely a communications job. As a project manager, I always want to have a discussion about performance as early in the project lifecycle as possible so that we can “design in” the performance characteristics for the parts we do control, help the customer understand the issue of performance with the implementation, and look for ways to measure the performance of the design we implement. While these discussions are going on, project managers need to work with a technical lead or system architect and identify every part of the system to see where the border is between our responsibility and the customer’s responsibility.

Performance – A joint challenge

Implementations today almost always involve transmitting data and/or transactions from a local computer, device, or browser across a network as shown below. Sometime that network hop is a short one but most often there can be many hops across a customer’s global network or the public internet. There can be many routers and other network appliances in between the source and destination of the transaction. The server or storage unit at the end of the transmission will usually have many other transactions hitting it at the same time.

The server may have other applications running simultaneously. The network connection or “pipeline” between the source and destination may have segments that are faster or slower than other segments.

All of these factors, and more, can affect a customer’s perception of system performance. The bottom line of all of this is that since SAS cannot control all of the aspects of the environment in which we do an implementation, attaining acceptable performance cannot be a SAS responsibility alone. Dealing with system performance is a joint responsibility between SAS, the customer, and possibly their internet service provider. Our customers will expect us to understand how to get the best performance even though we don’t control all aspects of the issue.

The approach I have found most successful involves the following:

During requirements gathering, the SAS and customer project teams should make sure to discuss all non-functional requirements (performance, security, administration, etc.) and any system performance expectations a customer may have.
The SAS team will need to find out what systems a customer has in place to monitor system and network performance. Many customers will have environment monitoring systems like Tivoli, CA Unified Infrastructure Management, HP Network Management Center, Nagios, ManageEngine, or others. These could be useful to track down performance issues on the client side and set up alerts to monitor performance in the future. If for some reason they don’t have one, I’d recommend they consider getting one implemented. If cost is an issue, Nagios is open source and free to download. It runs on Linux or other Unix variants. These systems will be key to monitoring performance going forward and giving early warnings when storage or system capacity is getting near set limits.
Customers should work with their SAS technical architect and project manager to discuss and set expectations about the various aspects of the system architecture that affect performance and note the areas SAS cannot control. Then discuss what systems or procedures they have in place to monitor and manage those parts of the environment.
Both parties should make sure to note the areas SAS cannot control in the project statement of work or charter. These should be documented in an architecture diagram and communicated with the customer stakeholders including the customer IT organization. This is a key step to set everyone’s expectations and should be communicated in writing.
The SAS team can provide support to the customer IT organization, where needed, to help them understand the impact the SAS system will have on their infrastructure. Customers can then, if needed, perform the system and network upgrades that will ensure a good environment. At least, they will understand the reasons why the SAS application performance is where it is.
Customers can work with their SAS technical architect to determine the best implementation design to meet the customer’s performance needs and make sure the performance standards are communicated to the rest of the project team.
If possible, the project team leadership should build in to the project schedule and technical approach, times where the team can do incremental performance testing as the system is being built. The SAS technical lead will work with a customer and the customer IT organization to figure out the best way to do incremental testing of performance.
Your SAS representative should be using the services of the SAS Enterprise Excellence Center (EEC) to obtain hardware and system recommendations as early as possible. That will help to ensure the hardware and system are sized for the application(s) you wish to use. The sizing recommendation may need to be revisited as new information about the requirements is discovered.
Hardware sizing is never an exact science particularly when you consider the potential these days for database and user population growth. So when you receive a sizing from SAS consider going over it with your SAS representative and confirm whether the sizing takes into account your best estimates for storage and usage growth in the near term and the foreseeable future. All customers should have plans in place to manage application growth.

Performance – The on-going challenge

It is very important to make sure that a customer understands that great system performance is built-in and managed from the beginning and not added after system implementation is completed. SAS customers have a critical role to play during and after implementation to make sure the implementation stays at peak performance. The performance of any system will not remain at its peak over time. Some of the reasons for this can include a large growth in database size, changes in database queries, increased network traffic, increased server load, and much more. The more a customer understands these effects, the better they can manage the system over time and the happier they will be with their SAS system.

As you know, SAS has a built in interest in making sure our partnership with customers is a productive and positive one for both parties. We want to also make them understand that we will be there to offer help if there is a performance drop off. They can lock in this help in advance with a Customer Care Agreement (CCA) or other forms of pre-arranged support services if they wish.

System performance is without doubt a tough issue to tackle and manage over the long term but with the right project management approach, tools, and good teamwork between the project manager, technical lead, technical architect, and the customer it can be successfully managed over time.