Archive for the 'Servers' Category

Making your sendmail to respond on both 25 and 587 ports the easy way

Wednesday, March 5th, 2008

Yes, you can do it very easy and fast if you have Webmin installed on your server.

Go to Servers -> Sendmail Configuration -> Sendmail Options and make sure that SMTP port options parameters are set to:

Name=MTA
Port=587, Name=MSA, M=E

Just like on the screenshot below:

Click "Save and Apply" button and you're done. Now your mail server will accept mail on both 25 and 587 ports.

Rails mailer's receive and sendmail procmail "Service unavailable" problem

Wednesday, February 13th, 2008

I have migrated all my Ruby on Rails system from Fedora Core 6 to RedHat 4.5 recently, and everything was fine, except for that the feature of blogging by email was broken (got a phone call today about that).

I use standard sendmail/procmail integration feature to receive mail to a designated address,  and then pipe it into Rails Mailer's receive function.

So, it looks like that in my /etc/mail/virtusertable:

m@example.com        examplemobile

this way all email which is coming to m@example.com address will be forwarded to system user called examplemobile.

In order to process the mail further and pipe it into my Rails script, I have the following setup in my /etc/aliases file :

examplemobile: "|/usr/local/bin/ruby /var/www/html/examplesite/current/script/runner -e production 'Mailman.receive STDIN.read'"

This setup worked perfectly on Fedora, but broke under RedHat. What's the problem? I was getting the following errors in my maillog:

Feb 13 11:45:33 appserv1 sendmail[32000]: m1D2jXJw032000: from=<xxxxxx@c.vodafone.ne.jp>, size=397, class=0, nrcpts=1, msgid=<20080213114719020972.1b6f@0016E68C4270>, proto=SMTP, daemon=MTA, relay=mmrts035p01c.softbank.ne.jp [123.108.236.87]
Feb 13 11:45:33 appserv1 smrsh: uid 8: attempt to use "ruby /var/www/html/examplesite/current/script/runner -e production 'Mailman.receive STDIN.read'" (stat failed)
Feb 13 11:45:33 appserv1 sendmail[32001]: m1D2jXJw032000: to="|/usr/local/bin/ruby /var/www/html/examplesite/current/script/runner -e production 'Mailman.receive STDIN.read'", ctladdr=<m@examplesite.com> (8/0), delay=00:00:00, xdelay=00:00:00, mailer=prog, pri=30623, dsn=5.0.0, stat=Service unavailable

The problem turned out to be in sendmail setup under RedHat was more secure than the one I had in Fedora. Specifically, this has to do with smrsh (or Sendmail Restricted Shell) thingy. More info about it here. Basically, sendmail only allows piping to programs which (or aliases to them) are present in /etc/smrsh directory.

Since I only use ruby to pipe to, I have added a symbolic link to it inside the /etc/smrsh directory:

[mike@appserv1 smrsh]$ pwd
/etc/smrsh
[mike@appserv1 smrsh]$ ls -l
lrwxrwxrwx  1 root root 19 Feb 13 11:50 ruby -> /usr/local/bin/ruby

Everything works as it should now, and our mobile-blogging users are supposedly happy again :)

One cool way to check your email routing

Monday, February 4th, 2008

I've been getting a strange "User unknown" error from sendmail today, when everything should have worked fine and even though users did exist. Here's some info on the problem's source and solution.

We have a domain called "talentnavi.biz", and I was asked to add a new subdomain to that domain ("kansai.talentnavi.biz"), and add new email addresses for several users who will be using that domain. I have setup everything properly on the server site by adding users and updating local domains and virtusers lists, however, when I was trying to test the delivery of the messages, I was getting "User unknown" errors and my emails to the newly-created mailboxes were bounced back.

I checked my mail server's logs, but there were of no help at all. After some googling, I found a very nice solution! In order to see how an email to a particular email address gets delivered, you should run the following command as root:

sendmail -d60.5 -d27.2 -bv info@example.com

So, in my case the command and output were as follows:

[root@me-ja mike]# sendmail -d60.5 -d27.2 -bv info@kansai.talentnavi.biz
map_lookup(dequote, mike, %0=mike) => NOT FOUND (0)
map_lookup(host, kansai.talentnavi.biz, %0=kansai.talentnavi.biz) => talentnavi.biz. (0)
map_lookup(dequote, info, %0=info) => NOT FOUND (0)
map_lookup(virtuser, info@talentnavi.biz, %0=info@talentnavi.biz, %1=info) => NOT FOUND (0)
map_lookup(virtuser, @talentnavi.biz, %0=@talentnavi.biz, %1=info) => NOT FOUND (0)
alias(info)
info@kansai.talentnavi.biz… User unknown

So, turns out that I have incorrectly configured my DNS records. They were set up like the following:

talentnavi.biz A 123.123.123.123

talentnavi.biz MX talentnavi.biz

kansai.talentnavi.biz CNAME talentnavi.biz

So, by default, the kansai.talentnavi.biz was aliased to talentnavi.biz, and since I didn't have a separate MX record for kansai.talentnavi.biz, the talentnavi.biz was used for mail delivery.

mike-meja:~ mike$ host -t mx kansai.talentnavi.biz
kansai.talentnavi.biz is an alias for talentnavi.biz.
talentnavi.biz mail is handled by 20 talentnavi.biz.

Therefore, the all email which was sent to "info@kansai.talentnavi.biz", was actually attempted to be delivered to "info@talentnavi.biz", because there was no separate MX record for the kansai. subdomain.

The solution is of course to add :

kansai.talentnavi.biz MX kansai.talentnavi.biz

Record to DNS server, and everything should work fine.

New "server room"

Thursday, January 31st, 2008

We have finally set up something which might be called a server room, at the company I work at.

We now have 4 servers, two of which are pretty powerful Dell PowerEdge 1900 servers I blogged about some time ago, one is our old server (which is basically just a custom-built stock computer), and one Mac which only serves a simple MySQL and FileMaker databases.

Now, my job for the next severals weeks is to gradually migrate stuff from old server to the new ones, which will become a PITA for sure, but I just hope it won't turn into MAJOR PITA :)

Dell servers RedHat troubles follow-up

Tuesday, January 29th, 2008

Just a quick follow-up to my yesterday's entry..

I spent the whole bloody day yesterday trying to make things work the way they should, and only got Seg faults on my application server as the result. Well, today I just re-installed everything I need from source, all the latest versions of the software - graphics & text processing libs, ImageMagic, Apache, PHP, Ruby and Rails of course with a slew of gems I use (including the latest rmagick, of course) - and now everything works together perfectly.

The whole process took me slightly less than 2 hours, and that includes reinstalling OS from the scratch, using Dell's installation disks! How about that huh?..

Talking about Dell-supplied install stuff… It is not bad, but come on - after default OS install is complete, you can not even remove packages you don't need without getting an error (" you need to install blah-blah-blah packages in order to remove bah-bah-bah packages " huh?!?)

Oh well. There's still that problem with gettext gem throwing errors in production mode which I solved somehow, long time ago - need to re-fix it and I'm ready to move everything from the old weaker server to the new powerful 2-server system.

RedHat troubles..

Monday, January 28th, 2008

I've been working on setting up two RedHat-based Dell servers at work today.  We have RedHat 4.5 on both machines, with RedHat subscription on one of them (the other one is for DB serving only with extremely limited access to it, so no need to spend money for updated software or patches)

The problem is that I needed to install and configure the environment for my Ruby on Rails based portal systems hosting. New servers, relatively new OS (there's no RHEL 5 for Japanese users yet :/ ), software subscription allowing to install stuff with a single mouse click (well.. literally, that is). What could possibly go wrong?

Well.. RHEL 4.5 went wrong. I just don't get it WHY the software which is piped to the server thru up2date program is so hopelessly, horribly outdated. Apache is at version 2.0.xx, PHP is stuck at 4.x, ImageMagick is at 3.0x, etc, etc…

And I need Apache 2.2 in order to use load balancing feature, RMagick for Rails doesn't want to work with ImageMagick < 3.6.x, PHP 4 just plain sucks (come on! We're at 5.2x series already! Not stable enough huh?). And of course everything gets installed by randomly spreading files over the disk (well.. that's how RHEL works, but I like to have everything/most stuff inside my precious /usr/local directory)

So I had to  reinstall half of the software which was supposed to work out of the box, installed/updated the other half using RedHat Network, and at the end of the day, when I though I have everything working, and moved one of my sites to test on the new system, I started to get seg faults here and there, and the system went pretty unstable.

So, tomorrow I'll just reinstall the whole OS, leaving most the RedHat-supplied stuff off - and will get a 100% working and controllable system with 100% probability.

What I don't understand is - is it me who just doesn't know how to build stuff on top of pre-built stuff, or is it RedHat's approach to installs, oriented to "average user" sort of system admin? Oh well.. one day of work lost, but lessons were learned :)

Subversion RPMs for RedHat Linux 7.3

Tuesday, January 22nd, 2008

I have no idea why you would need them, but I just spent the best half of day trying to compile Subversion from the source of some ancient RHL 7.3 machine - the operation which usually takes a few minutes, but I was getting errors no matter what. Luckily, I found RPM repository which contained everything I needed to install Subversion of that ancient RedHat machine.Here's the link:http://summersoft.fay.ar.us/pub/subversion/1.1.4/redhat-7.x/bin/ 

SA's best X'Mas present

Friday, December 21st, 2007

.. is a new server of course :)

Though I'm not much of a full-fledged SA, and just have to do it because nobody else in the company where I work at, can.. Still, there was a very warm and fuzzy feeling when I was unpacking our new Dell PowerEdge 1900 servers (well, only one of them because I'll do setup next month).

These actually are the first "real" servers which I'll be setting up and administering myself (there were lots of built-it-yourself servers before.. I wonder how they match against the "real" ones)

I'll be setting up separate application and database servers.. The specs are as follows:

Application: Intel Xeon Quad 2.33MHz, 8Gb RAM, RAID10 (6×250Gb), DRAC5 remote access card, Intel's Dual 1Gb Ethernet, RedHat 4.5

Database: Intel Xeon Quad 1.86MHz, 4Gb RAM, RAID10 (6×250Gb), DRAC5, Intel Dual 1Gbit Ethernet, RedHat 4.5

 

Well we don't yet even have a server room, closet, or anything.. Lots of things to setup, even more stuff to learn.

Turn off / tune logger level in Rails

Friday, November 9th, 2007

If you haven't noticed already, Rails-based application's logs can grow up pretty fast with default settings even in production mode. I just have one system which generated about 500Megs of production.log in just about a month. This is not a very good situation when you're on a shared host with limited hard disk space. So unless you are totally anal about what's going when on your production server (and well.. I don't know why you should be, since stuff is supposed to be at least partially stable when you actually deploy it) - you might want to limit logging, or eliminate it completely. Here's how.By default, logging level in production mode is set to :info, meaning every request will be logged, and that's not your Apache log. A single entry is something like that:

Processing FrontController#news (for 124.110.11.17 at 2007-11-09 17:55:11) [GET]  Session ID: f14f25c56d59f278194146955357c198  Parameters: {"action"=>"news", "id"=>"249", "controller"=>"front"}Cookie set: visitor_id=177; path=/; expires=Sat, 08 Nov 2008 14:55:11 GMTRendering  within layouts/kyoroman/frontRendering front/newsCompleted in 0.56727 (1 reqs/sec) | Rendering: 0.29647 (52%) | DB: 0.00000 (0%) | 200 OK [http://kyo-roman.com/news/249]  

Pretty long for a single request huh? And do you really really need this info in your logs?If not, just open your config/production.rb file and add the following line:

 config.log_level = :error  

This way only errors will be logged. You can use :fatal instead of :error and have even fewer stuff being logged.But still, even with :fatal settings, SOME stuff will be logged, and this stuff are really fatal dumps like:

ActionController::UnknownAction (No action responded to unknown_request):    /usr/local/lib/ruby/gems/1.8/gems/actionpack-1.13.2/lib/action_controller/filters.rb:632:in `call_filter'    /usr/local/lib/ruby/gems/1.8/gems/actionpack-1.13.2/lib/action_controller/filters.rb:638:in `call_filter'    /usr/local/lib/ruby/gems/1.8/gems/actionpack-1.13.2/lib/action_controller/filters.rb:438:in `call'    /usr/local/lib/ruby/gems/1.8/gems/actionpack-1.13.2/lib/action_controller/filters.rb:637:in `call_filter'    /usr/local/lib/ruby/gems/1.8/gems/actionpack-1.13.2/lib/action_controller/filters.rb:638:in `call_filter'  

…………………. many many lines ……………… 

So you might want to just disable the whole logging totally. In a hasta-la-vista-baby approach. In order to do that, just add the following 2 lines to the end of your environment.rb file:

ActiveRecord::Base.logger = nil

ActionController::Base.logger = nil 

 

 And that's it. You'll get zero-sized logs. Just remember - ALL logs will be empty, development, production and testing :)  Now.. I think there might be a better way to control logging, so if anyone knows a better way please share it in the comments. 

Why some cron jobs fails to execute

Friday, November 2nd, 2007

There's one simple thing every system administrator should take into account, when settings up cron jobs, and which I - to my shame - didn't know until now.

When you setup a cron job, especially for a root user, I guess, your command runs in a very limited environment (basically, not paths and other variables set). So if you have a command in /usr/local/bin directory and can "test" it executes just fine from command line - it doesn't mean the same exact command will be executed successfully when running as a cron job!

Same goes for setting up and "testing" cron jobs with Webmin. You can click the "Run Now" button in Webmin's cron jobs manager, and the command will run just fine, generating output you expect - but when running as a cron tab job, your command might NOT work at all.

The problem, as outlined above, is that not all paths you expect to be set are actually set for cron jobs.

How to confirm?

You can just read root's email (or cron job owner's email). In terminal just type "mail" and then enter a number of message you want to read. Here's the output for a problematic setup:

& 5000
Message 5000:
From root@………  Thu Nov  1 00:02:01 2007
Date: Thu, 1 Nov 2007 00:02:01 +0900
From: root@……….. (Cron Daemon)
To: root@……….
Subject: Cron <root@www> monit validate  #relaunches monit if needed
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>

/bin/sh: monit: command not found

The solution? Just specify full path to the command!

In my case, monit is located in  /usr/local/bin/ directory, which clearly is not defined in the PATH. So, it's /usr/local/bin/monit validate and you're set.

So, use full paths to your commands, and inside shell scrips (or set paths), and don't really trust Webmin' Cron Run Now command.