Benchmarking Appserver remotely - ab, nohup, ec2, and more ..

This time we wanted to benchmark WSO2 Stratos Application Server instances running on two amazon ec2 instances. Both of the instances are proxied by WSO2 Load Balancer. Here we have used appserver.cloud1.wso2.com which has only the load balancing, and appserver.cloud2.wso2.com with load balancing and auto-scaling.

Now we picked Java Bench (A clone of Apache Bench - ab) for the purpose to benchmark the application server running over the two amazon ec2 instances. It was a long running test from another ec2 instance. This test lasted for more than 30 hours. We were able to benchmark and also notice a few interesting stuff too.

Here even after closing the connection to the remote instance where the test is running, the test should keep running.

Now we are running the test using the load of web service calls on a web service that is deployed on Application Server. The service is a simple adder service.

The below is stored as request.xml

From two directories cloud1 and cloud2, from the instance,

nohup ab -p request.xml -n 10000 -c 200 -k -H "SOAPAction: multiply" -T "application/soap+xml; charset=UTF-8" http://appserver.cloud1.wso2.com/services/HelloService &

nohup ab -p request.xml -n 10000 -c 200 -k -H "SOAPAction: multiply" -T "application/soap+xml; charset=UTF-8" http://appserver.cloud2.wso2.com/services/HelloService &

Here nohup comes handy. Even after you terminate the connection to the remote host, the test will continue, unless you find the process and kill it using kill -9.

You can view the nohup output from the above directories cloud1 and cloud2 above using tail -f nohup.out 

But since ab itself can't handle this much of a bulky requests at once, it may fail as below, timing ab out for the specific requests.

Benchmarking appserver.cloud1.wso2.com (be patient)
apr_poll: The timeout specified has expired (70007)

So let's create a shell script instead, to prevent firing all the requests to ab at once, like a million of requests. 

while true; do ab -p request4.xml -n 4000 -c 200 -k -H "SOAPAction: multiply" -T "application/soap+xml; charset=UTF-8" http://appserver.cloud1.wso2.com/services/HelloService; sleep 3; ab -p request4.xml -n 4000 -c 200 -k -H "SOAPAction: multiply" -T "application/soap+xml; charset=UTF-8" http://appserver.cloud2.wso2.com/services/HelloService; sleep 3; done

Save the above script (test.sh). Now by running 

nohup sh test.sh &

you can start the load test for both the instances.

ubuntu@ip-10-120-61-234:~$ nohup sh dual.sh &
[1] 6512
ubuntu@ip-10-120-61-234:~$ nohup: ignoring input and appending output to `nohup.out'

Here 6512 above indicates the pid of dual.sh, which is the benchmarking process of ab.

You can see the nohup output for the above script with the results from both the instances alternating, with a gap of 3 seconds. Here the time interval between two chunks of requests on the same instance is 6 seconds.

This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking appserver.cloud1.wso2.com (be patient)
Completed 400 requests
Completed 800 requests
Completed 1200 requests
Completed 1600 requests
Completed 2000 requests
Completed 2400 requests
Completed 2800 requests
Completed 3200 requests
Completed 3600 requests
Completed 4000 requests
Finished 4000 requests


Server Software:        WSO2
Server Hostname:        appserver.cloud1.wso2.com
Server Port:            80

Document Path:          /services/HelloService
Document Length:        260 bytes

Concurrency Level:      200
Time taken for tests:   25.772 seconds
Complete requests:      4000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    0
Total transferred:      1652000 bytes
Total POSTed:           2188000
HTML transferred:       1040000 bytes
Requests per second:    155.20 [#/sec] (mean)
Time per request:       1288.623 [ms] (mean)
Time per request:       6.443 [ms] (mean, across all concurrent requests)
Transfer rate:          62.60 [Kbytes/sec] received
                        82.91 kb/s sent
                        145.50 kb/s total

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        1    2   5.3      1      81
Processing:     7 1256 711.9   1290    4351
Waiting:        6 1256 711.9   1290    4351
Total:          7 1258 711.7   1293    4363

Percentage of the requests served within a certain time (ms)
  50%   1293
  66%   1696
  75%   1883
  80%   1957
  90%   2165
  95%   2292
  98%   2416
  99%   2447
 100%   4363 (longest request)
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking appserver.cloud2.wso2.com (be patient)
Completed 400 requests
Completed 800 requests
Completed 1200 requests
Completed 1600 requests
Completed 2000 requests
Completed 2400 requests
Completed 2800 requests
Completed 3200 requests
Completed 3600 requests
Completed 4000 requests
Finished 4000 requests


Server Software:        WSO2
Server Hostname:        appserver.cloud2.wso2.com
Server Port:            80

Document Path:          /services/HelloService
Document Length:        260 bytes

Concurrency Level:      200
Time taken for tests:   20.210 seconds
Complete requests:      4000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    0
Total transferred:      1652000 bytes
Total POSTed:           2188000
HTML transferred:       1040000 bytes
Requests per second:    197.92 [#/sec] (mean)
Time per request:       1010.486 [ms] (mean)
Time per request:       5.052 [ms] (mean, across all concurrent requests)
Transfer rate:          79.83 [Kbytes/sec] received
                        105.73 kb/s sent
                        185.55 kb/s total

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        1    1   1.1      1      14
Processing:     6  987 577.5    990    3700
Waiting:        6  987 577.5    990    3700
Total:          6  988 577.4    990    3700

Percentage of the requests served within a certain time (ms)
  50%    990
  66%   1292
  75%   1468
  80%   1570
  90%   1782
  95%   1907
  98%   1986
  99%   2013
 100%   3700 (longest request)
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

This nohup.out contains the useful load testing results.