8PP or “The art of performance tuning/analysis/testing”

I believe that performance tuning/analysis/testing is one of the most complex tasks in the IT world. I read articles from well known IT people who have been known for well founded statements but when they did performance tuning/analysis/testing they have been proven wrong.

I am planing a series of performance tuning/analysis/testing posts about storage performance with linux file systems, Docker, MySQL, MS SQL Server and more. So to avoid the common mistakes I searched the INTERNET for some scientific approaches.

Andrew Pruski wrote a nice article and Raj Jan wrote a book  about this topic. Even this presentation shows the common mistakes. I decided to use these approaches and hopefully provide well founded posts.

I added some points to Andrews approach. I will call it 8PP (8 Phases of Performance tuning/analysis/testing) now, because I will often reference this approach.

AND Don’t forget: “Performance tuning/analysis/testing is a continues process“. What you consider to be optimal for your workload today may not be optimal tomorrow.

I will link a real example showing the 8PP soon.

8PP – The 8 Phases of Performance tuning/analysis/testing (Draft 1.3)

Phase 1 – Observation

  • 1.1 Understand the problem/issue
    • Talk to all responsible people if possible
    • Is the problem/issue based on a real workload?
    • Is the evaluation technique appropriate?
  • 1.2 Define your universe
    • If possible isolate the system as much as you can
    • Make sure to write down exactly how your system/environment is build
      • Firmware, OS, driver, application versions, etc…
  • 1.3 Define and run basic baseline tests (CPU,MEM,NET,STORAGE)
    • Define the basic tests and run them while the application is stopped
    • Document the basic baseline tests
    • Compare to older basic baseline tests if any are available
  • 1.4 Describe the problem/issue in detail
    • Document the symptoms of the problem/issue
    • Document the system behavior (CPU,MEM,NETWORK,Storage) while the problem/issue arise

Phase 2 – Declaration of the end goal or issue

  • Official declare the goal or issue
  • Agree with all participants on this goal or issue

Phase 3 – Forming a hypothesis

  • Based on observation and declaration form a hypothesis

Phase 4 – Define an appropriated method to test the hypothesis

  • 4.1 don’t define too complex methods
  • 4.2 choose … for testing the hypothesis
    • the right workload
    • the right metrics
    • some metrics as key metrics
    • the right level of details
    • an efficient approach in terms of time and results
    • a tool you fully understand
  • 4.3 document the defined method and setup a test plan

Phase 5 – Testing the hypothesis

  • 5.1 Run the test plan

    • avoid or don’t test if other workloads are running
    • run the test at least two times
  • 5.2 save the results

Phase 6 – Analysis of results

  • 6.2 Read and interpret all metrics
    • understand all metrics
    • compare metrics to basic/advanced baseline metrics
    • is the result statistical correct?
    • has sensitivity analysis been done?
    • concentrate on key metrics
  • 6.3 Visualize your data
  • 6.4 “Strange” results means you need to go back to “Phase 4.2 or 1.1”
  • 6.5 Present understandable graphics for your audience

Phase 7 – Conclusion

Is the goal or issue well defined? If not go back to  “Phase 1.1”

  • 7.1 Form a conclusion if and how the hypothesis achieved the goal or solved the issue!
  • 7.2 Next Step
    • Is the hypothesis true?
      • if goal/issue is not achieved/solved, form a new hypothesis.
    • Is the hypothesis false?
      • form a new hypothesis
    • Is there a dependency to something else?
      • form a new hypothesis
    • If the goal is achieved or issue solved
      • Document everything! (You will need it in the future)

Phase 8 – Further research

  • 8.1 If needed form a new goal/issue
  • 8.2 Define and run advanced baseline tests for future analysis
  • 8.3 If possible implement a continues approach to monitor the key metrics

The 8PP itself will change from time to time because performance tuning/analysis/testing will evolve.

Go docker Kitematic!

Leave a Reply

Your email address will not be published.