Developers Documentation



301 error for file:

User Tools

Site Tools

Module Building for ClearOS 5.x Requirements

Ever have the feeling “ClearOS is good piece of software…maybe even great. I just wish it had …”.

All around the world, in 22 different languages, I'm sure the ending to that sentence has been completed with an equally broad range of functions - from running a media server used for streaming music and movies to your TV to cooking up a corn-beef on rye. For those that are interested, the former is coming soon while the latter…well, maybe never and definitely not from me…I'm not a big fan of the salted beef.

At any rate, I had this exact feeling over the weekend when I started wondering how I managed to fill up a 140 GB hard disk drive on my ClearOS server at home. Seems the combination of having two girls under the age of 5 has a rather exponential effect of disk space usage with:

a) the number of digital photos of the two of them growing up (Doh! I had to buy the 15.1 megapixel model)

b) the astounding number of princess movies one can downlo….ahem, collect

But I digress, and I'm only going to get myself into copyright trouble. I know about Bash's “du” using a wack of parameters and possibly having to resort to piping the output to yet more cryptic bash commands…

but what if you didn't. Even in Microsoft, you're one “Right Click –> Properties” away from a purple and blue pie graph showing disk usage. Can't we have something similar…or better…in ClearOS!

And so, like most searches for a solution to a new-found problem, I turned to Google with the terms “linux find used disk space web-based”, which brought me here as the number one ranked page. Google, you never cease to amaze. According to the summary, Philesight's website says:

“[Philesight]…is designed to run on a remote server without graphical user interface”

Sounded promising. The next will document how to go about creating a module for ClearOS.

The following is for ClearOS 5.x. The module building procedure in ClearOS 6.0 will be significantly different!

Installing 3rd Party Binaries

In Part 1, we found an ideal solution in Philesight for determining where our disk space usage is being consumed. In part 2, we'll get this software installed on our ClearOS server and functioning. I'm going to have to assume a base level of technical knowledge. Accessing the Unix command line, basic shell commands and using a text editor falls into that 'required knowledge' category. If you're really new to Linux and have yet to jump into to the command line, you'll find some useful resources here, here and here.

Most software for Linux will have dependencies. A dependency means the software you are intent on installing requires (or is dependent on) other software having been installed correctly, and of a compatible version. Rather than re-writing a library to perform some task that's already been written, developers are naturally lazy beings and will jump through hoops not to create work for themselves. Hence, they rely on other software libraries, applications or even languages to be installed, in order for their 'widget' to function properly.

Installing software on a Linux distribution can really be a quick turn-off to a new user coming from the Windows world when one gets into dependency hell. In part 4 of this series, our goal will be to make installing this module simple (i.e. one click on a page in Webconfig). For now, because we're breaking new ground on this module, we need to follow the installation requirements listed on the Philesight website so that it will function correctly.

The Philesight webpage indicates the following recommendation for a RedHat/CentOS (and hence, ClearOS) Linux distro (note - I don't actually recommend following the instructions quoted directly below since there are alternatives to adding repositories, which I do detail further along in the post):

“I've been told the following should work on RedHat/Centos”

# yum -y update # cd /etc/yum.repos.d/ # wget # wget # yum -y –enablerepo=kbs-CentOS-Testing install ruby bitmap-fonts ruby-bdb ruby-cairo

The first four commands above essentially add repositories (software equivalent of your public library) that contain the packages required by the Philesight program.

A package is an efficient way of 'wrapping up' a software module. In fact, our ultimate goal in this blog series is to create a package for ClearOS that will remove all the brain-damage we're about to go through to get Philesight installed on ClearOS.

As ClearOS is based on CentOS (no room for confusion there, eh?), and CentOS is based from source code from RedHat, all three distributions share the same file format for package management, the RPM. The fifth and final line of the instructions indicates four packages (or RPMs) are required:

  • ruby
  • bitmap-fonts
  • ruby-bdb
  • ruby-cairo

ClearOS has a significant number of packages mirrored from the CentOS project - but not all, since it's a designated server platform, with no interest in desktop functions. Our first goal is to see which ones we can install immediately from our native repository. YUM is a package manager…it makes installing modules much easier because it has the AI to figure out which packages have dependencies - and which of those have dependencies and so on, and so on. You can see how easily you could slip into dependency hell doing this manually. So, using YUM, let's try installing ruby:

   yum install ruby
   Loading "protect-packages" plugin
   Loading "kmod" plugin
   base-kernels                                        |  951 B      00:00 
   Total download size: 1.9 M
   Is this ok [y/N]: y
   Downloading Packages:
   (1/2): ruby-libs-1.8.5-5.el5_4.8.i386.rpm           | 1.6 MB      00:08
   (2/2): ruby-1.8.5-5.el5_4.8.i386.rpm                | 274 kB      00:01
   Running rpm_check_debug
   Running Transaction Test
   Finished Transaction Test
   Transaction Test Succeeded
   Running Transaction
   Installing: ruby-libs                    #########################  [1/2]
   Installing: ruby                         #########################  [2/2]
   Installed: ruby.i386 0:1.8.5-5.el5_4.8
   Dependency Installed: ruby-libs.i386 0:1.8.5-5.el5_4.8

Now we'll try the bitmap-fonts requirement.

   yum install bitmap-fonts
   Loading "protect-packages" plugin
   Loading "kmod" plugin
   Setting up Install Process
   Parsing package install arguments
   (1/1): bitmap-fonts-0.3-5.1.1.noarch.rpm         | 906 kB      00:10
   Running rpm_check_debug
   Running Transaction Test
   Finished Transaction Test
   Transaction Test Succeeded
   Running Transaction
   Installing: bitmap-fonts                 #########################  [1/1]
   Installed: bitmap-fonts.noarch 0:0.3-5.1.1

Too easy. Alas, our reality check hits with the last two RPMs which we'll need to do a bit of work for. True, we could follow the instructions exactly and add the repository, but since we're going to eventually add any required packages to the ClearOS default (or at the very least, 'extras', repository), we'll try downloading and installing these packages manually.

The package ruby-cairo was part of the CentOS-DennouRuby repository. I downloaded and installed the RPM by doing the following:

  cd /tmp
  rpm -ivh ruby-cairo-1.4.1-1dc.i386.rpm

Similarly, the package ruby-bdb was part of the kbsingh-CentOS-Extras repository, so I ran:

  rpm -ivh ruby-bdb-0.6.0-1.el5.kb.i386.rpm

In theory, those two RPM's could have had dependencies which we'd need to go track down and install, but fortunately, it wasn't the case. The end result means we'll need to submit two packages (ruby-cairo and ruby-bdb) for inclusion into the ClearOS repository in order make this module a one click install.

With the dependencies out of the way, we can now get to the business of installing Philesight. Staying in the /tmp directory, we grab the tgz (compressed archive) file and extract its contents in our temporary directory:

  tar -xzvf philesight-20091214.tgz

Almost time for a beer - we've got some celebrating to do. Reading the README file, let's try creating a database file of our directory starting at the root path using the philesight tool:

  cd /tmp/philesight-20091214
  ./philesight --db test.db --index /

I peeked at the configuration lines in philesight.cgi, so I know it's looking for a database file in /tmp/ps.db. Let's move our newly created database file there:

  mv test.db /tmp/ps.db

Now to view it…we know we're going to want to use Webconfig's Apache engine since our ultimate goal is to display the tool in Webconfig. Let's copy the required Philesight files to Webconfig's CGI directory:

  cp philesight.* /var/webconfig/htdocs/cgi/

The moment of truth…a drum roll please (note, your 'system.lan' might vary, depending on your server configuration - use the internal IP, if necessary) while we add the following URL into our browser address bar:


Whoa! Forget a beer…better make that a two-four.

When I've recovered from my hang over…part 3 of Princess and the ClearOS - integrating what we've accomplished so far into Webconfig.

Webconfig Integration

In the section above, we managed to get get Philesight installed and displaying the disk usage graph for our ClearOS server.

This section's goal is relatively straightforward…we're going to create all the necessary files to have Philesight update the summary database once per day and display as a menu item in ClearOS's webconfig interface.

First, let's create the admin pages…There will be three files that need generating and are common to all modules:

 1. Menu file
 2. PHP admin page
 3. PHP admin language template

File Location: /var/webconfig/htdocs/menu/diskusage.en_US

Let's make life easier on ourselves and pick a module in the current navigation that we think the Philesight module would best fit in with. Under “System –> Resources” there's the “[System] Processes” modules that looks like a good fit. We'll copy the English version of that menu file and just make some required edits.

  cp /var/webconfig/htdocs/menu/processes.en_US /var/webconfig/htdocs/menu/diskusage.en_US

We then edit the file, changing the menu title and filename so it looks like this:

  System|Resources|Disk Usage|diskusage.php|configuration|5300

Note, the filename uses the “en_US” locale…if you are developing on webconfig where you use a different locale (i.e. fr_FR, pt_BR etc.), you would change the extension of the filename to match or set your system to English while developing the module.

Simple enough! Let's move on to the admin page.

PHP admin page

File Location: /var/webconfig/htdocs/admin/diskusage.php

This page is going to be very simple, but, of course, would vary depending on the complexity of information it contains. Here too, we can copy another page to use as the template.

  cp /var/webconfig/htdocs/admin/hardware-report.php /var/webconfig/htdocs/admin/diskusage.php

Editing the new diskusage.php, we'll get rid of any non-relevant code and stick a single Iframe to pull in the CGI page that Philesight generates.

  // Copyright 2010 ClearFoundation.
  // This program is free software; you can redistribute it and/or
  // modify it under the terms of the GNU General Public License
  // as published by the Free Software Foundation; either version 2
  // of the License, or (at your option) any later version.
  // This program is distributed in the hope that it will be useful,
  // but WITHOUT ANY WARRANTY; without even the implied warranty of
  // GNU General Public License for more details.
  // You should have received a copy of the GNU General Public License
  // along with this program; if not, write to the Free Software
  // Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.
  // Main
  WebDialogIntro(WEB_LANG_PAGE_TITLE, "/images/icon-status.png", WEB_LANG_PAGE_INTRO);
  // F U N C T I O N S
  // DisplayUsage
  function DisplayUsage()
      echo "";

A brief summary of the above, although it should be quite self-evident. First, we have our copyright and licensing comments.

Next, we pull some required classes and the language template.

We then call a couple of functions to force authentication to access the page and to create webconfig's header. The header abstracts away from the developer things like the menu, style etc. We'll then add the standard bubble that describes what the module does and call a very simple function that creates the Iframe.

Calling a standard footer finishes things off.

PHP admin language template

File Location: /var/webconfig/htdocs/admin/lang/diskusage.en_US

Finally, we're going to create our webconfig language file (same note applies regarding locale settings for webconfig as discussed in the menu section above) for the few tags we've used.

  // Generated by export.php.
  define("WEB_LANG_PAGE_INTRO", "This page displays a snap shot of disk usage as of midnight on your system.  Click on the areas of the graph to zoom into folders for a more detailed analysis.  Click on 'cd..' to return to the previous folder (zoom out).");
  define("WEB_LANG_PAGE_TITLE", "Disk Usage");
  define("WEB_LANG_IFRAME_NOT_SUPPORTED", "Sorry...your browser does not support Iframes.");
  // vi: syntax=php ts=4

ClearOS currently uses a database submission utility which we'll use in part 4 of this series when we're packaging things up. For now, having the one language file (and menu file), as long as it matches our webconfig locale setting should suffice.

If you've been playing along at home, at this point, you should be able to point your browser at the new page in webconfig and see the summary graph, as well as navigate through folders.

A couple of things have to be done to clean it up.

1. Put an icon in place for the information bubble. We'll just copy one from another page.

  cd /var/webconfig/htdocs/images/icons/32x32
  cp icon-hardware-report.png icon-diskusage.png

2. Make some changes to the configuration settings in /var/webconfig/htdocs/cgi/philesight.cgi:

  $path_db = "/usr/webconfig/tmp/ps.db"
  $img_size = 650
  $show_list = false

The first puts the database file in a more appropriate location than the /tmp directory. The next scales down down the image to fit the real-estate limits within the webconfig template. Finally, we hide the file/folder list for simplicity.

3. Move the database file (ps.db) that we created in Part 2 when we originally installed and ran Philesight that creates the database:

  mv /tmp/ps.db /usr/webconfig/tmp/

4. Remove the background color of the colorwheel so it becomes transparent. Line 186 of /var/webconfig/htdocs/cgi/philesight.cgi becomes:

  puts '          body {color:black;text-align:center;}'

5. Copy the Philesight executable over to our script directory and create a symbolic link to the required ruby file.

  cp /tmp/philesight-20091214/philesight /var/webconfig/scripts/
  cd /var/webconfig/scripts
  ln -s /var/webconfig/htdocs/cgi/philesight.rb .

6.Create a cron job that runs at midnight to update the disk usage database automatically.

  echo '0 0 * * * root /var/webconfig/scripts/philesight >/dev/null 2>&1' > /etc/cron.d/app-diskusage
  /etc/rc.d/init.d/crond reload

That ought to do it. In the next section we'll package things up and add the new RPM's to ClearFoundation's repositories so you can simply install the module through webconfig or by running:

  yum install app-diskusage

RPM Packaging

In this section, we're going to package our work. The RPM packaging instructions are available here:

Source Files and Patches

Let's quickly download our sources again (you may have already done this if you've been following the series, but it won't hurt to grab the source again)

  cd rpms/SOURCES

We're going to extract the contents of the tar file and create a patch using 'diff'. Yes, we could have just tarred up our changes that were necessary to display in webconfig (see end of Part 3) and tarred the file up, but creating a patch is the preferred and correct way.

  tar -xzvf philesight-20081120.tgz 

Here's how we create the patch between the original CGI file and ours (note, this will only work if you followed along in Part 3).

  diff -uN philesight-20091214/philesight.cgi /var/webconfig/htdocs/cgi/philesight.cgi > clear_webconfig.patch

Now we have our patch in the SOURCES directory, we can remove the extracted folder and keep our source directory clean:

  rm -rf philesight-20091214

Creating the SPEC File

Now that we have our source material, time to create our spec file…The spec file is the 'blueprint' for creating an RPM…it defines the version, dependencies, files, permissions, scripts to be executed on install, remove, upgrade etc. Move over to the RPM SPECS directory:

  cd ../SPECS

It's impossible to go through the logic of each part in this blog, but here are some highlights and at the end of this blog are links to additional references that would help to make sense of the details:

  • We define our version information and description
  • We add our package dependencies (see Part 2) to pull in automatically - ruby, ruby-cairo, cairo, ruby-bdb and bitmap-fonts
  • We define our patch to customize the size, location of database file and background colors
  • We define where to install this Philesight (/usr/webconfig/include/philesight is as good a place as any)
  • We define what happens pre and post install

Hopefully, if you find yourself having to build a SPEC file from scratch for software that nobody has created an RPM for, you can find something similar and just make the necessary changes.

  Name: philesight
  Version:  20091214
  Release:  1.0.clearos
  Group: Applications/Modules
  Packager: ClearFoundation
  Summary: Philesight is a tool to browse your filesystem and see where
  the diskspace is being used at a glance.
  License: GPL
  Source: philesight-%{version}.tgz
  Requires: ruby ruby-cairo ruby-bdb cairo bitmap-fonts
  BuildRoot: /tmp/%{name}-build
  Patch100:  clear_webconfig.patch
  Philesight is a tool to browse your filesystem and see where the
  diskspace is being used at a glance. It is implemented as a simple
  command line program that generates PNG files; a wrapper CGI script
  is supplied to allow navigating through the filesystem. Philesight
  is designed to run on a remote server without graphical user
  # B U I L D
  %setup -n philesight-%{version}
  %patch100 -p1
  # I N S T A L L  F I L E S
  [ "$RPM_BUILD_ROOT" != "/" ] && rm -rf $RPM_BUILD_ROOT
  mkdir -p -m 755 $RPM_BUILD_ROOT/usr/webconfig/include/philesight
  cp -r * $RPM_BUILD_ROOT/usr/webconfig/include/philesight
  logger -p local6.notice -t installer "philesight - installing"
  exit 0
  # U N I N S T A L L  S C R I P T
  if [ "$1" = "0" ]; then 
          logger -p local6.notice -t installer "philesight - uninstalling"
  # C L E A N  U P
  [ "$RPM_BUILD_ROOT" != "/" ] && rm -rf $RPM_BUILD_ROOT

Building an RPM

While still in the SPECS directory, let's build the RPM:

  rpmbuild -ba philesight.spec

If the build was successful, (you get a clean exit code and no errors [warnings are usually OK]), you'll now have two additional files.

1. Your RPM located in:


2. Your SOURCE RPM located in:


We now need to build the RPM for the disk usage module in webconfig. For this, we do things slightly differently.

The first thing we need to do is checkout the source code for webconfig. The details are explained here.

You don't need write access to SVN if you're new to building ClearOS modules…in fact, you probably won't get past our 'gatekeeper' until one or more contributions are sent in for peer review. You can check in any directory…I'll assume you're working as 'devel' and you're in the home directory.

  svn co svn://

This should checkout a whole wack of files…essentially, every file that makes up the code-base for all ClearOS modules.

We're going to build this RPM for the 5.1 branch (and we'll have to add it to the trunk as well so it becomes included in future releases), so the directory we're interested in is:

  cd clearos/legacy/modules/branches/5.1

If you do an 'ls' and see the directory names, you'll get a feel for how things are organized.

  ls -l
  drwxr-xr-x 3 devel devel 4096 Mar  1 17:21 app-amavisd-new
  drwxr-xr-x 3 devel devel 4096 May 10 15:04 app-archive
  drwxr-xr-x 3 devel devel 4096 Mar  1 17:21 app-autofs
  drwxr-xr-x 3 devel devel 4096 Mar  1 17:21 app-awstats
  drwxr-xr-x 3 devel devel 4096 Mar  1 17:21 app-backuprestore
  drwxr-xr-x 3 devel devel 4096 Mar  1 17:21 app-wireless
  drwxr-xr-x 3 devel devel 4096 Mar  1 17:21 BACKUP
  drwxr-xr-x 3 devel devel 4096 May 10 16:23 BUILD
  -rwxr-xr-x 1 devel devel  172 Mar  1 17:21 IGNORE.enterprise
  -rw-r--r-- 1 devel devel 1065 Mar  1 17:21 Makefile

Let's copy one of the folders as a template and start are modifications.

  cp -R app-roundcube app-diskusage
  cd app-diskusage
  rm .svn -rf

Rename the spec file template:


We don't need the spec file…it will get generated automatically from the Makefile using the template.

  rm app-roundcube.spec

Now, we edit the MANIFEST file to add the webconfig files that are to be included:

  menu htdocs/menus/diskusage
  locale diskusage htdocs/admin/lang
  include htdocs/admin/diskusage.php
  include htdocs/images/icons/32x32/icon-diskusage.png
  include ./app-diskusage.cron

You'll notice a one file tacked on to the end…its the cron file that will execute the philesight database once per day. Let's make that file (app-diskusage.cron) now, adding the following contents:

  0 0 * * * root './philesight --db /usr/webconfig/tmp/ps.db --index /' >/dev/null 2>&1

Finally, we'll edit the spec file. Hopefully, you'll see some common themes from the previous spec file we made for philesight. The highlights:

  • Note the version information matches the version we're building for
  • The release number can be used for minor updates (think bug fixes)
  • Note the dependency on the 'philesight' module we've built…this will pull in the philesight as a dependency automatically
  • We create links from the philesight scripts to our webconfig CGI directory
  • Post install, we run the philesight script so we'll have instant data for webconfig
  # P A C K A G E  I N F O
  Name: app-diskusage
  Version: 5.1
  Release: 1
  Summary: Disk usage module
  License: GPL
  Group: Applications/Modules
  Source: %{name}-%{version}.tar.gz
  Vendor: ClearFoundation
  Packager: ClearFoundation
  Requires: app-setup >= 5.1
  Requires: app-webconfig >= 5.1
  Requires: philesight
  BuildRoot: %_tmppath/%name-%version-buildroot
  Disk usage module
  # B U I L D
  # I N S T A L L  F I L E S
  [ "$RPM_BUILD_ROOT" != "/" ] && rm -rf $RPM_BUILD_ROOT
  mkdir -p -m 755 $RPM_BUILD_ROOT/etc/cron.d
  mkdir -p -m 755 $RPM_BUILD_ROOT/var/webconfig
  mkdir -p -m 755 $RPM_BUILD_ROOT/var/webconfig/htdocs/cgi
  install -m 0644 app-diskusage.cron $RPM_BUILD_ROOT/etc/cron.d/app-diskusage
  cp -r webconfig/* $RPM_BUILD_ROOT/var/webconfig/
  # I N S T A L L  S C R I P T
  logger -p local6.notice -t installer "app-diskusage - installing"
  ln /usr/webconfig/include/philesight/philesight.cgi /var/webconfig/htdocs/cgi/philesight.cgi
  ln /usr/webconfig/include/philesight/philesight.rb /var/webconfig/htdocs/cgi/philesight.rb
  # Create the first database
  echo 'Creating disk space utilization database.  Please be patient...this may take a while.'
  cd /usr/webconfig/include/philesight
  ./philesight --db /usr/webconfig/tmp/ps.db --index /
  exit 0
  # U N I N S T A L L  S C R I P T
  if [ "$1" = "0" ]; then
      logger -p local6.notice -t installer "app-diskusage - uninstalling"
      rm /var/webconfig/htdocs/cgi/philesight.cgi
      rm /var/webconfig/htdocs/cgi/philesight.rb
      rm /usr/webconfig/tmp/ps.db
  # C L E A N  U P
  [ "$RPM_BUILD_ROOT" != "/" ] && rm -rf $RPM_BUILD_ROOT
  # F I L E S

Let's drop down a directory and try to build it…But before we do, we need to have a look at two environment variables that control the build behavior.

  env | grep ROOT

The two variables should be set (either manually or in your .bashrc files) - if they aren't set, you need to set them. They dictate where files are found. With the above settings, are build woul fail because our webconfig files aren't in the SVN tree yet…they are only in our actually webconfig root directory (/var/webconfig) because we've been developing and testing directly in the live webconfig.

Let's set WEBCONFIG_ROOT to use the actual webconfig directory:

  export WEBCONFIG_ROOT=/var/webconfig

The MODULES_ROOT is the root directory of where we build modules…in our case, /home/devel/clearos/legacy/modules/trunk is wrong because we're working on the 5.1 branch, not trunk.

  export MODULES_ROOT=/home/devel/clearos/legacy/modules/branches/5.1

Make sure you're in the module root before executing the Make file:

  cd /home/devel/clearos/legacy/modules/branches/5.1
  make app-diskusage

You'll be prompted for some default parameters…defaults below are displayed in the square brackets:

  [devel@clearos 5.1]$ make app-diskusage
  rm -rf /var/tmp/app-buildroot-devel
  rm -f BUILD/Makefile.spec
  find -type l | xargs rm -f
  Vendor [ClearFoundation]:
  Product [enterprise]:
  Version [5.1]:
  Release [1]:

If all goes well and the build completes, we'll have a friend for our philesight package in our RPM and SRPM directories:

  ls -l /home/devel/rpms/RPMS/i386/
  -rw-r--r-- 1 devel devel 11592 May 10 22:37 app-diskusage-5.1-1.i386.rpm
  -rw-r--r-- 1 devel devel 12289 May 10 17:48 philesight-20091214-1.0.clearos.i386.rpm
  ls -l /home/devel/rpms/SRPMS/
  -rw-r--r-- 1 devel devel  9296 May 10 22:37 app-diskusage-5.1-1.src.rpm
  -rw-r--r-- 1 devel devel 12518 May 10 17:48 philesight-20091214-1.0.clearos.src.rpm

Whoa! Quite a workout for just a couple of files. But, with these two RPMs we can then submit them to the ClearOS package manager to be included in ClearFoundation's repository (incubation is usually done in the 'Extras' repository) so an end user can install with the following:

  yum install --enablerepo=base-extras app-diskusage

Additional Reading

content/en_us/kb_howtos_creating_a_clearos_module.txt · Last modified: 2015/03/01 11:26 (external edit)