How to monitor a process over time (eg apache, postgresql)
Being able to monitor server metrics such as CPU, RAM, DISK IO and get alerted when things go bad is crucial for any service operating on the internet, but sometimes a deeper level of monitoring is necessary. One might be in need to monitor specific processes and see how they behave over time, whether this is an Apache server, a MySQL database or any other process. Using collectd's process plugin we can get this information easily.
First edit file /opt/mistio-collectd/collectd.conf on your server and add a new section:
ProcessMatch "mysqld" "mysqld"
You can add a ProcessMatch line for every process you need to monitor. In our case we will monitor mysqld on a server that operates a busy site. The first entry after ProcessMatch is an arbitrary naming we provide to distinguish the metrics (can be anything that suits us).and the second entry is the name of the process to monitor.
Now restart ClearGLASS collectd
root@server:~# /opt/mistio-collectd/collectd.sh restart
and visit ClearGLASS page for this server. Press 'add graph' and select processes, then the process you are monitoring (in our case mysqld)
here are many available metrics that can be enabled for us, for example
ps_rss shows the resident segment size memory (the amount of memory the process has in RAM),
ps_vm shows the memory the process has in total (including swap),
ps_count will show number of processes and threads our process occupies in total,
pc_cputime will show the cpu usage of the process,
pg_pagefault shows the number of major and minor page faults.
these are very useful metrics to measure the performance of a service and figure out deficiencies and the time they occur.
Get alerted when a process stops
We can also set rules and take some action once a condition occurs. Example, when ps_rss or ps_vm raises a limit, and we are afraid that the system will soon become unstable, we might choose to reload or restart the service that leaks memory.
As an example, we want to get alerted when mysql process stops. We'll select 'ADD GRAPH' and then processes –> mysqld –> ps_count –> processes.
We now add a rule and specify “Processes mysqld ps_count processes” and enter a condition of < 1.
As soon as mysql stops running, we will receive an email informing us.
But there's more than just getting alerted. We can specify a command to run when this rule is triggered, example we will start mysql once it exits. Instead of alert, select *command* and set the command you'd like to run (/etc/init.d/mysqld start in our case)
As soon as mysql stops now, it will get started back and we'll receive an email