Site performance tuning using jMeter and jVisualVm

I recently had to work on a requirement where I had to support (response time  less than 5 seconds) for   500 concurrent users, with 10 million users in database and a single server. I tried going through many forums and open source tools, from which I finalized on jMeter for performance testing and jVisualVm for finding bottlenecks.

This was my approach:

1. Make sure that the server (hardware can be as per the staging/production requirements) has no other installations that can affect the performance. 
2. For setting up the users in DB, a procedure can be used and can be called as a part of jmeter test plan.
3. Install jmeter on a separate machine, so that jmeter won't affect the performance. 
4. Create a test plan in jmeter (as shown in the figure 1) for all the uri's, with response checking and timer based requests.  
5. Take the initial benchmark, using jmeter.
6. Check for the low performance uri's. These are the points to expect for bottlenecks.
7. Try different options for performance improvement, but focus on only one bottleneck at a time. Some of them can be 
1. Database index. Use the DB server logging to find the slow queries, use explain plan and improve the sql statements.
2. Check the configuration parameters for any third party tools.
3. Check if there are any dead locks (can be checked with jVisualVM as shown in Figure 2).
4. Profiling the application using jVisualVm(can be checked with jVisualVM as shown in Figure 3).
5. Data structures, algorithms and loops.
6. UI components, try advices from YSlow. Try streaming, caching, etc.
7. Read the hardware configuration and try for any options like cpu cache, jvm tuning, paging, etc. This step does not include hardware scaling.
8. Check for memory leaks using jVisualVm.
8. Try any one fix from step 6 and then take an benchmark. If there is any improvement commit the changes and repeat from step 5. Otherwise revert and try for any other options from step 6.
9. The next step would be to use load balancing, hardware scaling, clustering, etc. This may include some physical setup and hardware/software cost. Give the results with the scalability options.

jMeter test plan

Figure 1: jMeter test plan

jVisualVM run for a sample deadlock program.

Figure 2: jVisualVM shows Thread 1 and 2 waiting for the monitor

Figure 3: jVisualVm showing profile results for memory

Results:

In the end it is better to prepare a complete report from the benchmarks and finally define the performance with details of the hardware, software, maximum db load, maximum number of concurrent users supported(users active every 2 seconds), maximum number of sessions, changes made and the improvement achieved.  Any unimplemented changes can be suggested for improvement, so that the effort to find the bottleneck is improved.