macdiggs.com

Just a quick follow-up to my yesterday's entry..

I spent the whole bloody day yesterday trying to make things work the way they should, and only got Seg faults on my application server as the result. Well, today I just re-installed everything I need from source, all the latest versions of the software - graphics & text processing libs, ImageMagic, Apache, PHP, Ruby and Rails of course with a slew of gems I use (including the latest rmagick, of course) - and now everything works together perfectly.

The whole process took me slightly less than 2 hours, and that includes reinstalling OS from the scratch, using Dell's installation disks! How about that huh?..

Talking about Dell-supplied install stuff… It is not bad, but come on - after default OS install is complete, you can not even remove packages you don't need without getting an error (" you need to install blah-blah-blah packages in order to remove bah-bah-bah packages " huh?!?)

Oh well. There's still that problem with gettext gem throwing errors in production mode which I solved somehow, long time ago - need to re-fix it and I'm ready to move everything from the old weaker server to the new powerful 2-server system.

I've been working on setting up two RedHat-based Dell servers at work today.  We have RedHat 4.5 on both machines, with RedHat subscription on one of them (the other one is for DB serving only with extremely limited access to it, so no need to spend money for updated software or patches)

The problem is that I needed to install and configure the environment for my Ruby on Rails based portal systems hosting. New servers, relatively new OS (there's no RHEL 5 for Japanese users yet :/ ), software subscription allowing to install stuff with a single mouse click (well.. literally, that is). What could possibly go wrong?

Well.. RHEL 4.5 went wrong. I just don't get it WHY the software which is piped to the server thru up2date program is so hopelessly, horribly outdated. Apache is at version 2.0.xx, PHP is stuck at 4.x, ImageMagick is at 3.0x, etc, etc…

And I need Apache 2.2 in order to use load balancing feature, RMagick for Rails doesn't want to work with ImageMagick < 3.6.x, PHP 4 just plain sucks (come on! We're at 5.2x series already! Not stable enough huh?). And of course everything gets installed by randomly spreading files over the disk (well.. that's how RHEL works, but I like to have everything/most stuff inside my precious /usr/local directory)

So I had to  reinstall half of the software which was supposed to work out of the box, installed/updated the other half using RedHat Network, and at the end of the day, when I though I have everything working, and moved one of my sites to test on the new system, I started to get seg faults here and there, and the system went pretty unstable.

So, tomorrow I'll just reinstall the whole OS, leaving most the RedHat-supplied stuff off - and will get a 100% working and controllable system with 100% probability.

What I don't understand is - is it me who just doesn't know how to build stuff on top of pre-built stuff, or is it RedHat's approach to installs, oriented to "average user" sort of system admin? Oh well.. one day of work lost, but lessons were learned :)

I have no idea why you would need them, but I just spent the best half of day trying to compile Subversion from the source of some ancient RHL 7.3 machine - the operation which usually takes a few minutes, but I was getting errors no matter what. Luckily, I found RPM repository which contained everything I needed to install Subversion of that ancient RedHat machine.Here's the link:http://summersoft.fay.ar.us/pub/subversion/1.1.4/redhat-7.x/bin/ 

.. is a new server of course :)

Though I'm not much of a full-fledged SA, and just have to do it because nobody else in the company where I work at, can.. Still, there was a very warm and fuzzy feeling when I was unpacking our new Dell PowerEdge 1900 servers (well, only one of them because I'll do setup next month).

These actually are the first "real" servers which I'll be setting up and administering myself (there were lots of built-it-yourself servers before.. I wonder how they match against the "real" ones)

I'll be setting up separate application and database servers.. The specs are as follows:

Application: Intel Xeon Quad 2.33MHz, 8Gb RAM, RAID10 (6×250Gb), DRAC5 remote access card, Intel's Dual 1Gb Ethernet, RedHat 4.5

Database: Intel Xeon Quad 1.86MHz, 4Gb RAM, RAID10 (6×250Gb), DRAC5, Intel Dual 1Gbit Ethernet, RedHat 4.5

Well we don't yet even have a server room, closet, or anything.. Lots of things to setup, even more stuff to learn.

If you haven't noticed already, Rails-based application's logs can grow up pretty fast with default settings even in production mode. I just have one system which generated about 500Megs of production.log in just about a month. This is not a very good situation when you're on a shared host with limited hard disk space. So unless you are totally anal about what's going when on your production server (and well.. I don't know why you should be, since stuff is supposed to be at least partially stable when you actually deploy it) - you might want to limit logging, or eliminate it completely. Here's how.By default, logging level in production mode is set to :info, meaning every request will be logged, and that's not your Apache log. A single entry is something like that:

Processing FrontController#news (for 124.110.11.17 at 2007-11-09 17:55:11) [GET]  Session ID: f14f25c56d59f278194146955357c198  Parameters: {"action"=>"news", "id"=>"249", "controller"=>"front"}Cookie set: visitor_id=177; path=/; expires=Sat, 08 Nov 2008 14:55:11 GMTRendering  within layouts/kyoroman/frontRendering front/newsCompleted in 0.56727 (1 reqs/sec) | Rendering: 0.29647 (52%) | DB: 0.00000 (0%) | 200 OK [http://kyo-roman.com/news/249]  

Pretty long for a single request huh? And do you really really need this info in your logs?If not, just open your config/production.rb file and add the following line:

 config.log_level = :error  

This way only errors will be logged. You can use :fatal instead of :error and have even fewer stuff being logged.But still, even with :fatal settings, SOME stuff will be logged, and this stuff are really fatal dumps like:

ActionController::UnknownAction (No action responded to unknown_request):    /usr/local/lib/ruby/gems/1.8/gems/actionpack-1.13.2/lib/action_controller/filters.rb:632:in `call_filter'    /usr/local/lib/ruby/gems/1.8/gems/actionpack-1.13.2/lib/action_controller/filters.rb:638:in `call_filter'    /usr/local/lib/ruby/gems/1.8/gems/actionpack-1.13.2/lib/action_controller/filters.rb:438:in `call'    /usr/local/lib/ruby/gems/1.8/gems/actionpack-1.13.2/lib/action_controller/filters.rb:637:in `call_filter'    /usr/local/lib/ruby/gems/1.8/gems/actionpack-1.13.2/lib/action_controller/filters.rb:638:in `call_filter'  

…………………. many many lines ……………… 

So you might want to just disable the whole logging totally. In a hasta-la-vista-baby approach. In order to do that, just add the following 2 lines to the end of your environment.rb file:

ActiveRecord::Base.logger = nil

ActionController::Base.logger = nil 

 And that's it. You'll get zero-sized logs. Just remember - ALL logs will be empty, development, production and testing :)  Now.. I think there might be a better way to control logging, so if anyone knows a better way please share it in the comments. 

There's one simple thing every system administrator should take into account, when settings up cron jobs, and which I - to my shame - didn't know until now.

When you setup a cron job, especially for a root user, I guess, your command runs in a very limited environment (basically, not paths and other variables set). So if you have a command in /usr/local/bin directory and can "test" it executes just fine from command line - it doesn't mean the same exact command will be executed successfully when running as a cron job!

Same goes for setting up and "testing" cron jobs with Webmin. You can click the "Run Now" button in Webmin's cron jobs manager, and the command will run just fine, generating output you expect - but when running as a cron tab job, your command might NOT work at all.

The problem, as outlined above, is that not all paths you expect to be set are actually set for cron jobs.

How to confirm?

You can just read root's email (or cron job owner's email). In terminal just type "mail" and then enter a number of message you want to read. Here's the output for a problematic setup:

& 5000
Message 5000:
From root@………  Thu Nov  1 00:02:01 2007
Date: Thu, 1 Nov 2007 00:02:01 +0900
From: root@……….. (Cron Daemon)
To: root@……….
Subject: Cron <root@www> monit validate  #relaunches monit if needed
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>

/bin/sh: monit: command not found

The solution? Just specify full path to the command!

In my case, monit is located in  /usr/local/bin/ directory, which clearly is not defined in the PATH. So, it's /usr/local/bin/monit validate and you're set.

So, use full paths to your commands, and inside shell scrips (or set paths), and don't really trust Webmin' Cron Run Now command.

I do alot of development in Rails recently, and it all goes fine and nice, and Rails is fast and easy to use, but sometimes I'm better off using already existing PHP scripts. Just place them into Rails project, and they just work (think of PHP-based web forms etc).

How do you setup your Apache & Mongrel cluster based Rails project to support execution of PHP scripts inside the Rails project structure? Very easy, actually. (more…)

Well.. Sometimes I'm a smart (or you might call it lazy) folk, so I do reuse my code. And the code is usually is stored inside a Subversion system, and several projects are always checked out to my local computer and therefore have these invisible .svn sub-directories all-over the place.
Being lazy (or you call it smart? :) to checkout stuff I need to be duplicated from a previous project into a current one, I usually just copy, say, "tiny_mce" folder from an old project to a new one.

The problem that appears when I try to do that and add the newly duplicated folder to the new project, is that svn complains the folder is already being under version control. Well of course, because all these .svn dirs were copied along with the "meat" :) What's the easiest way to remove .svn directories from that directory? If you're on a Mac or some other Unix-based system, that is.

Look no further than for the code below!

cd /path/to/your/dir

rm -rf `find . -type -d -name ".svn"`

Voila. After some complaining about some .svn dirs couldn't be found (and you just safely ignore them) - you have a clean directory structure which you can than safely add to version control using everyone's favourite "svn add tiny_mce" (or whatever).

Mostly a note to myself, but hopefully it'll save somebody some time some day :)

Oh and of course you can just as easily remove dirs or files with a different file names, just bu replacing the ".svn" with, say, "Icon.icn" or something like that.

I've been installing everything that is needed to run Ruby on Rails application onto a host (which is CentOS 4.x), which didn't even have Ruby installed, mySQL was too old and Apache didn't have required modules for running Mongrel clusters. So I had to install everything from the scratch, and the whole process was documented so I could repeat it in the future :) But may be this will helpful to someone else too, so here it is, right from my Yojimbo's note :)
Just one thing to notice. CentOS already has some of the required libs installed (zlib, libpng, libjpeg and freetype), so if you don't have them installed, do it before proceeding. I think there are plenty of info on this matter so I'll omit it.

Now, to the business.

RUBY

wget ftp://ftp.ruby-lang.org/pub/ruby/ruby-1.8.5.tar.gz
tar zxvf ruby-1.8.5.tar.gz
cd ruby-1.8.5
./configure
make
sudo make install

Warning! If you're on Mac, you have to use the following string to configure Ruby install:

./configure –prefix=/usr/local –enable-pthread

On Mac OS X, once you have installed Ruby, you have to fix default paths for your default shell, because Ruby installs itself to /usr/local/bin by default, and Apple's default binaries location is in /usr/bin. Do this in Terminal:

pico ~/.bash_login

# and paste the following text:
export PATH="/usr/local/bin:/usr/local/sbin:$PATH"
# save the file (Control-W) and relogin, or just re-open terminal session before proceeding.

RAILS : GEMS

wget http://rubyforge.rubyuser.de/rubygems/rubygems-0.9.2.tgz
tar zxvf rubygems-0.9.2.tgz
cd rubygems-0.9.2
ruby setup.rb

RAILS: CORE (possible to use –include-dependencies, but i was getting some 'not found' error with this option)

#install the specific version, which happens to be 1.2.2 in this case.
gem install -v=1.2.2 rails

RAILS: STANDARD STUFF (possible to use –include-dependencies)

gem install capistrano
gem install mongrel
gem install mongrel_cluster
gem install gettext
gem install unicode

APACHE 2 with load balancer (for Mongrel cluster)

wget http://ftp.kddilabs.jp/infosystems/apache/httpd/httpd-2.2.4.tar.gz
tar zxvf httpd-2.2.4.tar.gz
cd httpd-2.2.4
./configure –enable-rewrite –enable-proxy –enable-load-balancer –enable-mods-shared=ALL
make
sudo make install

GHOSTSCRIPT & FONTS

# we need to install/update ghostscript and install ghostscript-fonts in order to get rid of the "`get_type_metrics': unable to read font `(null)' (Magick::ImageMagickError)" error:

wget http://jaist.dl.sourceforge.net/sourceforge/ghostscript/ghostscript-8.56.tar.gz
tar zxvf ghostscript-8.56.tar.gz
cd ghostscript-8.56
./configure
make
sudo make install

wget http://jaist.dl.sourceforge.net/sourceforge/gs-fonts/ghostscript-fonts-std-8.11.tar.gz
tar zxvf ghostscript-fonts-std-8.11.tar.gz
mv fonts /usr/local/share/ghostscript/8.56/

IMAGEMAGICK (required by RMagick)

wget ftp://ftp.u-aizu.ac.jp/pub/graphics/image/ImageMagick/imagemagick.org/ImageMagick-6.3.3-10.tar.gz
cd ImageMagick-6.3.3
./configure
make
sudo make install

RAILS: RMAGICK (manual install - I have always had multi-systems compatibility problems with the gem install)

wget http://files.rubyforge.mmmultiworks.com/rmagick/RMagick-1.15.4.tar.gz
tar zxvf RMagick-1.15.4.tar.gz
cd RMagick-1.15.4
./configure
make
sudo make install

MYSQL

wget http://ftp.iij.ad.jp/pub/db/mysql/Downloads/MySQL-5.0/mysql-5.0.37.tar.gz
tar zxvf mysql-5.0.37.tar.gz
cd mysql-5.0.37
./configure –with-extra-charsets=all –prefix=/usr/local/mysql
make
sudo make install

#in /etc/my.cnf
[mysqld]
set-variable = max_allowed_packet=32M # I upload large images into DBs, therefore need a bigger allowed packet size

(also need to change encodings to utf8)
default-character-set=utf8

SUBVERSION (required to capistrano deployment to work)

wget http://subversion.tigris.org/downloads/subversion-1.4.3.tar.gz
wget http://subversion.tigris.org/downloads/subversion-deps-1.4.3.tar.gz
tar zxvf subversion-1.4.3.tar.gz
tar zxvf subversion-deps-1.4.3.tar.gz
cd subversion-1.4.3
./configure
make
sudo make install

That all folks :) Hope this was helpful.

Today I have been restoring automatic backup system for the company I work at. The idea is to take backups of our clients' web sites (both web/html files and database snapshots). So I have decided why not to share? This script was dutifully working for me for almost a year and though it might be not the best way to do stuff, it proven itself pretty robust and problem-free.

Before pasting the script code here, here's how it works.

The script logs into remote server and takes backup of remote html files and database files. One important thing is to notice that database files backup is takes "as is", as I have found out not in the most pleasant way that taking backups of databases via mysqldump command can lead to problems in case tables in the SQL database use different encodings - so binary dump is much safer.

One more thing to notice is that you need to generate an ssh key-pair in order to be able to log into a remote server without entering a password (that's why there is no mention of passwords in the script's settings). You can read about logging into SSH without using passwords here.

Since I use rsync to fetch files from server, only files which were changes between backup sessions are being transferred - which makes backups much faster in case of successive backups.

Usage: php backup.php <backup_type>

you can specify type of backup, so it will be placed into a separate directory. Default is "nightly". Files will be backed up into directories inside the $backupDir you specified, in form of: "<sitename>_<backup_type>_<web or db>".

Once the script is deployed and configured, you might want to specify cron jobs which would make nightly, weekly and monthly backups for you. This way you will have at least 3 versions to choose from in case something shitty happens :)

And of course I would like to notice that the script goes "as-is" without any warranties that it won't destroy your data and nullify you hard disk (although if configured sanely, it would only save you from problems of loosing data).

Have fun with the script, modify it if you want. Hopefully it'll save you some nerve cells if ever a problem occurs on your remote servers.

The script follows:

(more…)