Analytical thinking in performance analysis

Many performance job requirements list analytical skills as one of the requirements. For people who love analysis this is a interesting job.

Lot of analysis is done using mental models or mathematical models for the scenario. Without analysis the test results are just numbers.

Major limitation of performance testing is that you cannot test all possible scenarios and use-cases. Testing and modeling complement each other. One cannot expect major re-engineering decision to be taken based on performance flaws pointed out by analytical reasoning only, they have to be backed by actual measurements.

Good modeling requires a solid understanding of how things work. When you keep looking at data from various perspectives and keep asking questions interesting insights emerge.

Below are some of the scenarios where modeling and analysis can complement testing and measurements.

Modeling and pre-analysis helps in design of right tests

For example from the understanding of architecture, when you expect synchronization issues ,you will design the test so that saturation even while CPU resource is available is clearly established

Modeling and analysis can even provide bounds on  expected results

For example, In case where all concurrent transactions serialize on a shared data structure, we know that time spent in synchronized block is serial. So if 50 microseconds are being spent inside synchronized block then there can be at most 20000 transactions per second beyond which the system will saturate.This limit is independent of number of threads executing the transactions in the server and number of available cores. If synchronization is the primary bottleneck by reducing the time spent in synchronized block to half we can double the throughput.

Modeling and analysis help in identifying the bottleneck

Simple model above has given a upper limit on throughput which can be validated by testing. if the solution saturates before that we know that primary bottleneck is somewhere else. Quite often people justify the observed results just on hunches like “Oh! solution saturated using 50% of CPU,it must be a synchronization issue” and accept the results.Without a right quantitative approach this can hide the actual bottleneck. For example in the above example if the saturation is much before 20000 transactions per seconds asking the poor developer to optimize the synchronization will not really solve the problem as the bottleneck is somewhere else

Modeling and analysis can help in answering the what if scenarios.

For example you have tested using 10 G network and someone asks what are your projections if the customer runs on a 1 G network? What are the capacity projections for the solution in this scenario?

What do you think we need higher per core capacity or larger number of cores for the deployment of the solution? if we double the available RAM what kind of improvements can we expect?
If we double the number of available cores what is the expected capacity of the system?

Modeling and analysis help in evaluation of hypothesis

For example someone thinks a particular performance problem is due to OS scheduling, this can be analytically evaluated. For example we had a scenario in which 600 threads were scheduled on 8 core server. When there were performance issues developer felt problem should be from thread contending to get scheduled. When we created the model of how CPU was being used it was very obvious the issue was not of scheduling.By accepting a false hypothesis the real cause of issues remain hidden. 

Modeling and analysis enhances our understanding of the system

When models dont match the measurements there are new learnings as we revisit our assumptions

Modeling and analysis helps in extrapolation of results

We might want to extrapolate the results due to various reasons , there can be practical limitations like you dont have enough hardware to simulate the required number of users.

We will discuss how we can model various performance scenarios in future posts.

Related Post :

Software bottlenecks:- Understanding performance of software systems with examples from road traffic

Time scale of system latencies

In my previous post we were discussing why providing low latency is more difficult than providing scalability. Below is a scale showing latency cost of various operations.

Event	Latency	Scaled
1 CPU Cycle	0.3 ns	1 s
Level 1 cache access	0.9 ns	3 s
Level 2 cache access	2.8 ns	9 s
Level 3 Cache access	12.9 ns	43 s
Main memory access (DRAM, from CPU)	120 ns	6  min
Solid-state disk I/O (flash memory)	50-150 µs	2-6 days
Rotational disk I/O	1-10 ms	1-12 months
Internet: San Francisco to New York	40 ms	4 years
Internet: San Francisco to UK	81 ms	8 years
Internet: San Francisco to Australia	183 ms	19 years
TCP packet retransmit	1-3 s	105-317 years

Unit	Abbreviation	Fraction of 1 sec
Millisecond	ms	0.001
Microsecond	µs	0.000001
Nanosecond	ns	0.000000001

Data is from the book Systems Performance: Enterprise and the Cloud by Brendan Gregg
First table gives the cost of operations and the second table explains the units of time. The third column of first table converts the cost to time scale that we understand, time scale of seconds, minutes, days, months and years. Here the cost of operations is scaled assuming if the cost of CPU cycle is 1 second hypothetically than what will be the cost of other operations.

How to use knowledge of above latencies?

•  It can help in understanding the lower bound of latency or response time for a certain scenario
• For example if the user is in Australia and server is in San Francisco than time between request and response will be greater than 2 network hops or greater than 366 ms (183 * 2 )
•   Performance objectives cant be met using a certain technology/operation
• For example if you want response in microseconds than the critical path of code execution cannot have a disk read/write. 
• If your algorithm has more than 10 random memory operation to do some work it will at least take more than 1.2 microseconds (120ns * 10).
• Overall latencies can be estimated by knowing the cost of basic operations 
• Finding reasons for performance jitters
• For example if response time jumped  from microseconds to milliseconds. What could be the possible reasons? We know disk reads can take milliseconds so page fault can be a possible reason. GC in Java could be a reason as GC jitters are from few to 100s of milliseconds, but we should not suspect networking at 1 GBPS LAN as it will not cause delays in milliseconds especially if utilization is low. Knowing the system latencies help you in narrowing your search for the culprit.

Related previous posts:

• Low Latency Arms Race
• Why providing low latency is more challenging than providing scalability 

Why providing low latency is more challenging than providing scalability

Response time and capacity are the two fundamental indicators of software performance.In industry like finance low latency has a lot of value for the traders. High frequency trading which drives large percentage of trades in many markets requires very fast trading software that can react to market depth in microseconds or nanoseconds. Read about low latency challenges in finance industry here.

Performance challenges in the web technologies is more on scalability side.In trading software where we are looking at milliseconds, microseconds in the web world even 250 milliseconds is good enough as response time for end users. Even in the web world people want low service time at server side as high service time indirectly means low capacity but they wont care much if 50-100 milliseconds are added by Internet.

Why providing low latency is more challenging ?
Problem of scalability was solved with load balancing and horizontal scaling. Although there are technical challenges but a load of million of users can be distributed on thousands of serves and with proper design of software, load balancing and capacity planning.

Latency is the sum of time spent in all activities required to do some work.The time may be spent on CPU or in waits. Basic service time is the latency in which the request does not have to wait in the queue before being serviced. At very high capacity utilization the response time may be 10-100 times the basic service time if the utilization is high due to queuing. So the first important step is to have good capacity planning to manage the response time.If you get this wrong than even achieving a very low basic service time latency will be practically wasted.

Lets say you are able to achieve it and still want lower latency what do you do? Now that you know you have right load balancing and appropriate capacity utilization so that  response time is around basic service time mostly except for bursts.We now focus on reducing the basic service time.

Two basic things that can reduce the basic service time are

• Better algorithms
• Parallelism

here again not many problems have high parallelism. If you want even lower latency what do you do? To understand this we need to discuss what are the sources of latency and how industry is addressing these sources. I will discuss them in my posts in future. At a very basic level you need reduce the number of steps to complete the task or find a way to reduce the cost of steps. There are fundamental limits to reduction in number of steps or reduction of cost of a single step. Even if we look at hardware for example RAM, the increase in size and bandwidth over time has been significant but reduction in latencies is much less comparatively.

Comparing performance optimization and road travel
If you compare performance optimization to the road travel. you can achieve higher capacity by adding more lanes to the road. Here basic service time is the time to travel when road is completely empty. Your latency or time to travel increases if there is lot of traffic on the road which is like high latency due to high capacity utilization.Now if you want to reduce the time from the basic service time or time to travel on empty road, what can you do. Either you can find a shorter path or you need faster driving speed. There are limits to both of them. you cannot reduce the distance between two points beyond the shortest distance. You can not keep increasing the speed beyond a point. Similarly when reducing latency beyond a point it becomes more and more difficult. Industry is hacking into OS, libraries,network, switches hardware to reduce the sources of latencies but reducing it further is becoming more and more difficult and expensive. There are market gateways that can provide market update from wire to app memory in 1 microseconds. a single memory fetch from RAM can take 40 to 100 nano seconds so you can imagine you will exceed this latency even if you do 10 memory fetches.

Low Latency Arms Race

Electronic exchanges and algorithmic trading have made low latency a big business. At all levels servers, switches, network, OS, applications etc there is a race to achieve lower and lower latencies.  A 1-millisecond advantage in trading applications can be worth $100 million a year to a major brokerage firm(ref). This was an estimate in 2007. In 2012 people are talking in terms of microseconds and nanoseconds.

For same trading strategy, market and instrument, competing algotrading software in two trading firms, the firm that gets the market data first and executes its order first in the market will beat the other firm in trading profits. 

Not all financial IT products require ultra low latencies, for example back office products for fees calculations etc no one cares about latencies, while front office products like market gateways have very "low latency" reqirements. The latency requirements for a product is derived from the business context.

Even for the front office products requirements for institutional investors and high frequency trading are quite different. For long term investment  to create a large position institutional invester may choose to use algorithmic trading to break large orders into small trades but here instead of market orders, limit orders will be used to buy at desired price. Order is late or fast does not matter much.

In the middle are short term manual traders. Minimum human reaction time to read a market depth and click a trade is between 100 - 250 milliseconds. So a normal trader will not try to aggress market opportunities that vanish in milliseconds but will trade on strategies that will be from minute to hours to a day to multiple days. So 1 millisecond here or there is not a big deal 

At the other extreme high frequency trading(HFT) is all about speed. HFT employs computerized algorithms to analyze incoming market data and implement proprietary trading strategies. High-frequency traders compete on a basis of speed with other high-frequency traders. By 2010 high-frequency trading accounted for over 70% of equity trades in the US (ref). One of the HFT strategy is to exploit arbitraging price discrepancies in some particular security trading simultaneously in different markets. This requires ultra low latencies and between the competing firms one with the lowest latency will win.

"Trading mostly with their owners money,they scoop up hundreds or thousands of shares in one transaction,only to offload them less than a second later before buying more.They can move millions of shares around in minutes,earning a 10th of a penny off each share.As a group,they earned $12.9 billion in profit in the past two years,according to the TABB Group,a specialist on the markets.TABB expects their earnings to slow this year as Wall Streets big brokerage firms fight back with their own faster computerised trading" NY Times 10 Oct 2011

With these kind of stakes there are big budgets and many buyers for low latency products.

Two important latencies are market data latency and order to market latency. The market data latency includes time from event in exchange till data is available to algotrading software or visible to human on trading GUI. This will include latencies on wire(exchange to trading center), latencies of internet gateway, latencies of switches/hub, latencies of router. At software level latencies of middleware, OS context switches, protocol libraries like FIX, TCP latencies etc will come into picture. 

Distance between algorithmic trading server and market exchange is a very important factor. There is roughly a delay of 5 microseconds per KM in fiber optic cable. If trading center is 200 km away 1 millisecond delay will be added to latencies
Co-Location is used to solve this problem.Trading firms can place computers containing their trading algorithms in special data centers close to or in the exchange. CME Co-Location Services is one example of Co-Location provided by exchange. 

Providing lower latency is a much bigger challenge than providing higher throughput. Over past few decades the bandwidth and size of RAM has increased exponentially but the latencies of RAM have decreased linearly. Even for very high end servers and fast RAM the latencies of memory access is 30 to 100 nanoseconds.Low latency requirements are nearly approaching hardware limits.

Quoting Kevin Slavin

"It takes you 500,000 microseconds just to click a mouse. But if you’re a Wall Street algorithm and you’re five microseconds behind, you’re a loser."