lunes, 26 de enero de 2015

... Java / JBoss AS tuning

When you want to move your application to production environment, you always wonder how good it will perform, or if you will have any time of problem. It is a good practice to perform some characteristics testing of your application befor going into this stage. For being able to suceed in a characteritics and tuning process you need to continually gather information on your system, not only on the stress tests that you may execute but for the normal lifetime of your application, as it will be more close to reality, and you’ll see if the application/system is able to recover and continue it’s operation after a burst of load, or if a long period of inactivity affects in some way your development.
People usually overlook the process of understanding one’s application, as well as the operating system, the JVM and the architecture where the application is running, as it can similar or more impact than the development itself.
In a Java application, the most important parts to monitor, at least from the beginning are:
  • Process (cpu, load, threads,…​)
  • IO (disk and network)
  • JVM (memory and garbage collection)
  • Application specifics (connection pools, messaging resources length, )
  • Business related metrics (Request/response timings)
In this article I will focus on the first 3, as they are general purpose.

JVM Tuning

One of the things that you’ll need to look at is at the JVM. To do that, I oftleny use Java Mission Control.

Configure Java Mission Control to connect to a remote JBoss AS instance

To enable Java Mission Control for remote connection to a JBoss AS instance you should:
  • Edit the "$JAVA_HOME/bin/jmc.ini" file which contains the JVM and JAVA_OPTS related information in this file somewhere at the end users can add the "-Xbootclasspath/a" option to include the "$JBOSS_HOME/bin/client/jboss-client.jar" as following:
-Xbootclasspath/a:/jboss-eap-6.2.0/bin/client/jboss-client.jar   <--------NOTICE HERE------>
  • Once the "jmc.ini" file is edited as above then Java Mission Control (jmc) can be started as following:
  • Then you’ll be able to connect with:

Configure JVM to enable Flight Recording

To use Oracle JDK’s Java Flight Recorder you’ll need to run JBoss AS with the following options:
-XX:+UnlockCommercialFeatures -XX:+FlightRecorder
These can be used with JAVA_OPTS on the shell, or editing either the $JBOSS_HOME/bin/standalone.conf, $JBOSS_HOME/domain/configuration/domain.xml or $JBOSS_HOME/domain/configuration/host*.xml depending on where the jvm parameters are set for your standalone or domain instance.
You can start a Flight recording right when you start your JVM, or you can do it programatically, using jcmd, so once JBoss starts with both of the above options, jcmd is the application needed to start, check, dump and stop JFR.
  • $JAVA_HOME/bin/jcmd $JBOSS_PID JFR.start: Starts JFR recording. It returns the recording number, which is used to dump the results to disk
  • $JAVA_HOME/bin/jcmd $JBOSS_PID JFR.check: Lists the number of recordings running
  • $JAVA_HOME/bin/jcmd $JBOSS_PID JFR.dump recording=<recording_number> filename=<path/file>: Dumps the JFR recording to disk
  • $JAVA_HOME/bin/jcmd $JBOSS_PID JFR.stop recording=<recording_number>: Stops the specified JFR recording. This should be done after dumping and not prior.
  • $JAVA_HOME/bin/jcmd $JBOSS_PID: help and any of the commands above can be used to list the available options for each command.
To limit the duration of the recording, to for 2 hours for example, use run:
$JAVA_HOME/bin/jcmd $JBOSS_PID duration=2h name=MyRecording filename=/tmp/myrecording.jfr
A JFR recording should occur when the system starts displaying the problematic behaviour on a production environment. On a development or testing environment, recording can start as soon as JBoss comes up and stop after the testing is done.

Scripting all these commands is the best way to make it work, so you can start a recording before doing some tests, and dump the recording when done, so you can analyze it.
Marcus Hirt has written a good tutorial on JMC.

Another JVM Tuning tools

Another useful tool that I oftenly use for scripting is Swiss Java Knife - jvm-tools. It has command line tools that enhance some of the out of the box commands/tools that come with the JVM like jps, jmap, jtop,…​

Monitoring the OS

When you monitor the OS you need to pay special attention to the cpu, load, threads as well as io, disk and network.
I’ve seen times where a poor networking configuration or architecture made applications behave very badly, so you need to constantly track every disk and network adapter you are using, and for the network adapters the type of traffic they will be managing (tcp, udp,…​).
Also, any other process that runs on the machine being used for testing needs to be identified, and it’s impact need to be minimized as much as possible, so the information being gathered is as related to your real application as possible.

sysstat - sar/ksar

The sysstat utilities are a collection of performance monitoring tools for Linux. These include sar, sadf, mpstat, iostat, nfsiostat-sysstat, cifsiostat, pidstat and sa tools.
Can monitor a huge number of different metrics:
  • Input / Output and transfer rate statistics (global, per device, per partition, per network filesystem and per Linux task / PID).
  • CPU statistics (global, per CPU and per Linux task / PID), including support for virtualization architectures.
  • Memory, hugepages and swap space utilization statistics.
  • Virtual memory, paging and fault statistics.
  • Per-task (per-PID) memory and page fault statistics.
  • Global CPU and page fault statistics for tasks and all their children.
  • Process creation activity.
  • Interrupt statistics (global, per CPU and per interrupt, including potential APIC interrupt sources, hardware and software interrupts).
  • Extensive network statistics: network interface activity (number of packets and kB received and transmitted per second, etc.) including failures from network devices; network traffic statistics for IP, TCP, ICMP and UDP protocols based on SNMPv2 standards; support for IPv6-related protocols.
  • NFS server and client activity.
  • Socket statistics.
  • Run queue and system load statistics.
  • Kernel internal tables utilization statistics.
  • System and per Linux task switching activity.
  • Swapping statistics.
  • TTY device activity.
  • Power management statistics (instantaneous and average CPU clock frequency, fans speed, devices temperature, voltage inputs, USB devices plugged into the system).
  • Filesystems utilization (inodes and blocks).

Of course, it is quite easy to integrate sysstat with Kibana, Graphana, or any other plotting tool for metrics.
It seems there are some alternatives to sysstat, like DStat

2 comentarios:

Arun Kumar dijo...
Este comentario ha sido eliminado por un administrador del blog.
Arun Kumar dijo...

I really appreciate information shared above. It’s of great help. If someone want to learn Online (Virtual) instructor lead live training in JBoss.kindly contact us
MaxMunus Offer World Class Virtual Instructor led training on JBoss. We have industry expert trainer. We provide Training Material and Software Support. MaxMunus has successfully conducted 100000+ trainings in India, USA, UK, Australlia, Switzerland, Qatar, Saudi Arabia, Bangladesh, Bahrain and UAE etc.

For Free Demo Contact us:
Name : Arunkumar U
Email :
Skype id: training_maxmunus
Contact No.-+91-9738507310
Company Website –