Community Dashboard

12
May

Princess and the ClearOS Module - Part 4 of 4

Posted by on in ClearFoundation
  • Font size: Larger Smaller
  • Hits: 9380
  • 10 Comments
  • Print

Apologies for the delay in bringing you Part 4! Too much to do...too little time.

For those joining late or who have troubles (like me) remembering what they did yesterday, let's do a quick recap.

In Part 1, our focus was to find a solution to the problem "Where has all my disk space gone". We settled on some code written by a software-engineer in the Netherlands who wrote Philesight - a web-based graphical chart of disk usage on a Linux server.

The goal in Part 2 was to get Philesight installed in our environment (ClearOS's webconfig), with a bit of an introduction to packages and RPM dependencies.

We then graduated to embedding the output Philesight provides (i.e. the graph) right into webconfig. Also in Part 3, we created the necessary localization files and took a quick peek into cron jobs and scripts.

In Part 4, we're going to package our work. Ugh....packaging. Gotta do it though, otherwise, we'll have everyone becoming experts in the command line environment and see hourly wages for Linux admins/developers plummet...can't have that on my conscience! Grab a Red Bull...we could be in for a long haul.

 

Creating an RPM Build Environment

To start we'll need some tools to build RPM's with.

yum install rpm-build

We're not going to need a C compiler, but if you did, this would come in handy (read essential):

yum install group "Development Tools"

 

Creating a User

Packaging as 'root' is widely considered a really bad idea. It's just too easy to stomp on files you really would rather not have stomped on. So, let's create a user called 'devel', give this user shell access, then switch over to this user (i.e. out of 'root'):

adduser devel -s /bin/bash
su - devel

The only problem with the above is that if you ever reboot the server, you'll have to re-create the 'devel' user. To make it permanent, you'll need to add the following option to webconfig's environment: /etc/system/webconfig shell_access = 1 And then create the 'devel' user via webconfig, giving them the /bin/bash shell.

 

RPM Build Directory Structure

As your non-root user (i.e. devel), execute the following:

cd ~
mkdir -p --verbose ~/rpms/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
mkdir -p --verbose ~/rpms/RPMS/{athlon,i386,i486,i586,i686,noarch}
echo "%_topdir ~/rpms" > ~/.rpmmacros

The above just creates the appropriate directories and tells the 'rpmbuild' command where the top directory of our rpm folder is located.

 

Source Files and Patches

Let's quickly download our sources again (you may have already done this if you've been following the series, but it won't hurt to grab the source again)

cd rpms/SOURCES
wget http://zevv.nl/play/code/philesight/philesight-20081120.tgz

We're going to extract the contents of the tar file and create a patch using 'diff'. Yes, we could have just tarred up our changes that were necessary to display in webconfig (see end of Part 3) and tarred the file up, but creating a patch is the preferred and correct way.

tar -xzvf philesight-20081120.tgz

Here's how we create the patch between the original CGI file and ours (note, this will only work if you followed along in Part 3).

diff -uN philesight-20091214/philesight.cgi /var/webconfig/htdocs/cgi/philesight.cgi > clear_webconfig.patch

Now we have our patch in the SOURCES directory, we can remove the extracted folder and keep our source directory clean:

rm -rf philesight-20091214

 

Creating the SPEC File

Now that we have our source material, time to create our spec file...The spec file is the 'blueprint' for creating an RPM...it defines the version, dependencies, files, permissions, scripts to be executed on install, remove, upgrade etc. Move over to the RPM SPECS directory:

cd ../SPECS

You can view/download a copy of our Philesight spec file here. It's impossible to go through the logic of each part in this blog, but here are some highlights and at the end of this blog are links to additional references that would help to make sense of the details:

  • We define our version information and description
  • We add our package dependencies (see Part 2) to pull in automatically - ruby, ruby-cairo,  cairo, ruby-bdb and bitmap-fonts
  • We define our patch to customize the size, location of database file and background colors
  • We define where to install this Philesight (/usr/webconfig/include/philesight is as good a place as any)
  • We define what happens pre and post install

Hopefully, if you find yourself having to build a SPEC file from scratch for software that nobody has created an RPM for, you can find something similar and just make the necessary changes.

 

RPM Packaging

Let 'er rip!  While still in the SPECS directory, let's build the RPM:

rpmbuild -ba philesight.spec

If the build was successful, (you get a clean exit code and no errors [warnings are usually OK]), you'll now have two additional files.

1.  Your RPM located in:

/home/devel/rpms/RPMS/i386/philesight-20091214-1.0.clearos.i386.rpm

2.  Your SOURCE RPM located in:

/home/devel/rpms/SRPMS/philesight-20091214-1.0.clearos.src.rpm

Half way home...we now need to build the RPM for the disk usage module in webconfig.  For this, we do things slightly differently.

The first thing we need to do is checkout the source code for webconfig.  The details are explained here.

You don't need write access to SVN if you're new to building ClearOS modules...in fact, you probably won't get past our 'gatekeeper' until one or more contributions are sent in for peer review.  You can check in any directory...I'll assume you're working as 'devel' and you're in the home directory.

cd
svn co svn://svn.clearfoundation.com/clearos

This should checkout a whole wack of files...essentially, every file that makes up the code-base for all ClearOS modules.

We're going to build this RPM for the 5.1 branch (and we'll have to add it to the trunk as well so it becomes included in future releases), so the directory we're interested in is:

cd clearos/legacy/modules/branches/5.1

If you do an 'ls' and see the directory names, you'll get a feel for how things are organized.

ls -l
drwxr-xr-x 3 devel devel 4096 Mar  1 17:21 app-amavisd-new
drwxr-xr-x 3 devel devel 4096 May 10 15:04 app-archive
drwxr-xr-x 3 devel devel 4096 Mar  1 17:21 app-autofs
drwxr-xr-x 3 devel devel 4096 Mar  1 17:21 app-awstats
drwxr-xr-x 3 devel devel 4096 Mar  1 17:21 app-backuprestore
...
..
.
drwxr-xr-x 3 devel devel 4096 Mar  1 17:21 app-wireless
drwxr-xr-x 3 devel devel 4096 Mar  1 17:21 BACKUP
drwxr-xr-x 3 devel devel 4096 May 10 16:23 BUILD
-rwxr-xr-x 1 devel devel  172 Mar  1 17:21 IGNORE.enterprise
-rw-r--r-- 1 devel devel 1065 Mar  1 17:21 Makefile

Let's copy one of the folders as a template and start are modifications.

cp -R app-roundcube app-diskusage
cd app-diskusage
rm .svn -rf

Rename the spec file template:

mv app-roundcube.spec.in app-diskusage.spec.in

We don't need the spec file...it will get generated automatically from the Makefile using the .spec.in template.

rm app-roundcube.spec

Now, we edit the MANIFEST file to add the webconfig files that are to be included:

menu htdocs/menus/diskusage
locale diskusage htdocs/admin/lang
include htdocs/admin/diskusage.php
include htdocs/images/icons/32x32/icon-diskusage.png
include ./app-diskusage.cron

You'll notice a one file tacked on to the end...its the cron file that will execute the philesight database once per day.  Let's make that file (app-diskusage.cron) now, adding the following contents:

SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
HOME=/usr/webconfig/include/philesight
0 0 * * * root './philesight --db /usr/webconfig/tmp/ps.db --index /' >/dev/null 2>&1

Finally, we'll edit the spec file.  Hopefully, you'll see some common themes from the previous spec file we made for philesight.  The highlights:

  • Note the version information matches the version we're building for
  • The release number can be used for minor updates (think bug fixes)
  • Note the dependency on the 'philesight' module we've built...this will pull in the philesight as a dependency automatically
  • We create links from the philesight scripts to our webconfig CGI directory
  • Post install, we run the philesight script so we'll have instant data for webconfig

Let's drop down a directory and try to build it...But before we do, we need to have a look at two environment variables that control the build behavior.

env | grep ROOT
WEBCONFIG_ROOT=/home/devel/clearos/legacy/webconfig/trunk
MODULES_ROOT=/home/devel/clearos/legacy/modules/trunk

The two variables should be set (either manually or in your .bashrc files) - if they aren't set, you need to set them.  They dictate where files are found.  With the above settings, are build woul fail because our webconfig files aren't in the SVN tree yet...they are only in our actually webconfig root directory (/var/webconfig) because we've been developing and testing directly in the live webconfig.

Let's set WEBCONFIG_ROOT to use the actual webconfig directory:

export WEBCONFIG_ROOT=/var/webconfig

The MODULES_ROOT is the root directory of where we build modules...in our case, /home/devel/clearos/legacy/modules/trunk is wrong because we're working on the 5.1 branch, not trunk. 

export MODULES_ROOT=/home/devel/clearos/legacy/modules/branches/5.1

Make sure you're in the module root before executing the Make file:

cd /home/devel/clearos/legacy/modules/branches/5.1
make app-diskusage

You'll be prompted for some default parameters...defaults below are displayed in the square brackets:

[devel@clearos 5.1]$ make app-diskusage
rm -rf /var/tmp/app-buildroot-devel
rm -f BUILD/Makefile.spec
find -type l | xargs rm -f
Vendor [ClearFoundation]:
Product [enterprise]:
Version [5.1]:
Release [1]:
...
..
.

If all goes well and the build completes, we'll have a friend for our philesight package in our RPM and SRPM directories:

ls -l /home/devel/rpms/RPMS/i386/
-rw-r--r-- 1 devel devel 11592 May 10 22:37 app-diskusage-5.1-1.i386.rpm
-rw-r--r-- 1 devel devel 12289 May 10 17:48 philesight-20091214-1.0.clearos.i386.rpm
ls -l /home/devel/rpms/SRPMS/
-rw-r--r-- 1 devel devel  9296 May 10 22:37 app-diskusage-5.1-1.src.rpm
-rw-r--r-- 1 devel devel 12518 May 10 17:48 philesight-20091214-1.0.clearos.src.rpm


Whoa!  Quite a workout for just a couple of files.  But, with these two RPMs we can then submit them to the ClearOS package manager to be included in ClearFoundation's repository (incubation is usually done in the 'Extras' repository) so an end user can install with the following:

yum install --enablerepo=base-extras app-diskusage

I've gone ahead and added these packages to the repository so go ahead and give it a try.

I hope this series has in some small way achieved the desired objective - to lower the bar for potential developers to contribute to ClearOS and help us improve and expand the functionality.  While it's true this module is very simple in scope and we haven't touched on some important concepts (the powerful PHP classes and API is the most prominent example) I can personally attest to the competence and desire of myself (OK, in my case, just desire) and guys like Peter, Darryl, David and John to offer assistance to new developers who are interested in joining the ClearFoundation team and work together in a strong sense of community.

 

Additional Reading

http://bradthemad.org/tech/notes/patching_rpms.php
http://www.ibm.com/developerworks/library/l-rpm1/
https://fedoraproject.org/wiki/How_to_create_an_RPM_package
http://freshrpms.net/docs/fight/

Thanks for reading...happy packaging!

Ben

0
Tagged in:
Df43e790f3b1e7a2e9fcde4024b123ce Medium
Ben Chambers has not set their biography yet
Message Author | Add as friend | View author's profile | Show more posts from author | Subscribe to updates from author

Achievements

  • Bookworm
  • The Voice
Author's recent posts

Overall Rating (0)

0 out of 5 stars

People in this conversation

Comments (10)

  • If like me you have some mounted drives you don't want to be indexed as it tends to skew the picture, you can exclude all mount points by modifying the crontab and adding the flag --one-file-system

    Mine now looks like (also adjusted the time to 2am)
    0 2 * * * root /var/webconfig/scripts/philesight --one-file-system --db /usr/webconfig/tmp/ps.db --index / >/dev/null 2>&1

    You can also skip specific paths by adding --skip

    0 Like

  • Ah...mis-understanding...I thought you were installing using the 'yum install...' command and seeing errors. You were actually following along with the devel process in Part 3.

    Yes...there were some errors in Part 3 and if you install the module that was finally built, you'll see some cleanup and fixes in the way we call the script.

    Ben

    0 Like

  • Quick fix for the path missing issue if you only run the install disk app
    Create a database
    ./philesight --db test.db --index /

    #Move it over to the temp webconfig directory
    mv test.db /usr/webconfig/tmp/ps.db

    #Modify
    vi /var/webconfig/htdocs/cgi/philesight.cgi

    #Make same changes
    #Include the path name
    $path_db = "/usr/webconfig/tmp/ps.db"

    #Make the image windows better size
    $img_size = 650

    #Hide the clutter
    $show_list = false

    #Save Changes

    #The use the cron job as described
    echo '0 0 * * * root /var/webconfig/scripts/philesight >/dev/null 2>&1' > /etc/cron.d/app-diskusage
    /etc/rc.d/init.d/crond reload

    ##########

    This got my vm working without any issues. Thanks for putting this together hope its included in the 6.0 build

    0 Like

  • Thanks Ben and Peter! I realise things are a bit of a moving target at the moment - keep us posted, looking forward to the 6.x release

    0 Like

  • Hi Tim,

    We're still getting up to speed on the new build system. As I'm getting trained on the system, I'm taking notes along the way. Once we're comfortable with the process, I'll write a developer packaging guide on the topic. It will be similar to what you see for SME Server @ http://wiki.contribs.org/Package_Modification

    Also of note: the web framework is going to be changed quite a bit (read: a lot) in version 6.0. The API will stay basically the same, but everything else will look quite different!

    0 Like

  • Hi Tim.

    RE: philesight.spec

    Good catch...in fact, I think Shad and Pete are working on that part of the build environment that takes upstream packages an incorporates them into both SVN (or CVS?) and Shad's automated build system. Let that shake out (I think they're aim is to use it for the 5.2 ISO), and then we'll shed light that process.

    For now, I summed up the Blog series in a howto:

    [url]http://www.clearfoundation.com/docs/howtos/creating_a_clearos_module[/url]

    and posted the philesight.spec for reference.

    RE: Package Manager

    That's be [url]http://www.clearfoundation.com/Community/Profile/pbaldwin.html[/url]. I know he holds a lot of respect for what you've done Tim, so I'm sure a request for write access to the webconfig source would be granted.

    B.

    0 Like

  • Excellent! many thanks Ben - it proved very useful reading, particularly the section on building webconfig packages

    One thing I noticed your first link to the philesight.spec (under creation of the spec file) actually points to the app-diskusage spec?

    Also, who is the ClearOS package manager referred to? and whats the process for contributing to the API if we wish to add other functions

    0 Like

  • That was meant to say '/' Path not found in Database

    0 Like

  • Cheers for putting this together ben, just a side not but I ran yum install --enablerepo=base-extras app-diskusage on a spare vm which resulted in "/ not found in database" so it can find the path

    That aside got philesight running without a hitch on my box at home

    0 Like

  • If you're not a developer and want to try this module out, try:

    yum install --enablerepo=base-extras app-diskusage

    B.

    0 Like