macdiggs.com

Was working on a remote machine today…

Thankfully the machine was smart enough to reboot itself :)

I've been doing some maintenance of my server, and wanted to do some spring clearning, deleting all spam files inside users' directories. These files are automatically created by spamassassin software. Also, wanted to get rid of Rails production.log files.

Doing everything manually is no fun, and I have to admit, I completely suck at shell scripting.. But if you never try - you'll never learn, so that's what I came up with.

Calculating their size

First, I wanted to find out how much space exactly files called spam inside all directories inside the /home directory take. That's the command which I came up with ( of course, I first had to cd /home ):

[root@me-ja home]# find * -name spam -type f -exec echo {} \; | xargs du -ks | awk '{total += $1} END {print total}'

2952880

A little explanation. I use 3 commands each piping its output to the next one (and the last outputs everything to standard output which is the screen. (more…)

Ok so today I have noticed that ranking ramp-up for a specific person still continues. I have blocked one device, but the other found another one, it seems..

Both devices are mobile phones of the same carrier - KDDI. Looks like some manager (the site is a talents' directory site) decided to rank his favorite pet up, no matter what, but it also looks like that person is only doing this from mobile phone, and his carrier stands unchanged. So, I decided to tighten block logic (if you can call that logic) on that specific person's profile.

Before, I was only blocking a specific User-Agent:

RewriteCond %{HTTP_USER_AGENT} "KDDI\-KC35 UP\.Browser/6\.2\.0\.5" [NC]

RewriteRule ^.*$ - [F,L]

On one of my servers, I was under some weird "ranking up!" attack, which basically was just a loop of requests to one user's profile (making that users access count higher, and therefore ranking higher in the access top)

Here's what my Apache logs were telling me:

218.25.251.170 - - [17/Oct/2008:11:02:26 +0900] "GET /ono/profile HTTP/1.1" 200 2363 "-" "KDDI-KC35 UP.Browser/6.2.0.5 (GUI) MMP/2.0"
218.25.251.170 - - [17/Oct/2008:11:02:30 +0900] "GET /ono/profile HTTP/1.1" 200 2363 "-" "KDDI-KC35 UP.Browser/6.2.0.5 (GUI) MMP/2.0"
218.25.251.170 - - [17/Oct/2008:11:02:34 +0900] "GET /ono/profile HTTP/1.1" 200 2363 "-" "KDDI-KC35 UP.Browser/6.2.0.5 (GUI) MMP/2.0"

SetEnvIfNoCase User-Agent Mozilla getout

<Directory "/var/www/html/myserver">

Order allow,deny

Allow from all

Deny from env=getout

</Directory>

I've been setting up a very simple backup of one site's MySQL database to another server today.

What I needed to be done is to have the MySQL database files to get archived, compressed and transferred to my other server, and be named after the actual backup date and time. And that operation should happen every night.

Here's the full code for those who are in hurry (a cool one-liner heh :), and more detailed explanation of steps taken will follow.

ssh root@remoteserver.example.com 'mysqladmin flush-tables –socket=/tmp/mysql.sock; rm -f remote_db.tar* && tar cf remote_db.tar /var/lib/mysql && bzip2 remote_db.tar' && scp root@remoteserver.example.com:~/remote_db.tar.bz2 /home/mike/remote-backups/`date +%y%m%d_%H%M%S`.tar.bz2

And the explanation.

First, the server I'm backup up from is "remoteserver.example.com" and the forementioned command is invoked from the server I am backing up to. Also, I have my public ssh key installed for root account on the remote server, so I don't need to enter password to log into the remote server. You can read more about setting up SSH keys here.

First step is to log into the remote server, which is achieved simply by running

ssh root@remoteserver.example.com

However, actually instead of logging into the server, I only need to execute some commands on the remote server. This can be accomplished by giving a string of commands to execute as a second (last parameter to ssh command), for example:

ssh root@remoteserver.example.com 'ls -lh'

will give you a listing of files of remote server root's home directory.

So, now that we are connected to the remote server, we actually just need to prepare backup files which we will later transfer from remote to local server. Though the command is actually a one-liner, I'll split the lines for easier understanding, and give them numbers. Also, please note that the command concatenation with the && symbol does the following - it runs next command in chain only of previous command executed successfully.

1. mysqladmin flush-tables –socket=/tmp/mysql.sock
2. rm -f remote_db.tar*
3. tar cf remote_db.tar /var/lib/mysql
4. bzip2 remote_db.tar

#1 flushes mysql tables (so everything that is possibly in memory cache is writted to disk).

#2 removes any previous backup files (that's easy)

#3 this archives the whole MySQL data directory (this path can differ from mine, depending on your installaion parameters!). Also, please note that this approach is NOT SAFE! You can easily get a corrupted backup if any data changes during the archival process. The reason why I'm doing it myself is because I have 100% guarantee that the database will not be updated (the backup is for some inter-corporate system, and I have 100% guarantee that nobody is accessing the database at 3 o'clock in the morning when my backup task is running). You might want to lock tables during backup and unlock them once it is complete. Also, I'm backing everything up this way because I need a drop-in backup - anything happens, and I can just drop the backed up DB in place of the old one.

#4 and the final step is just to compress the backup archive to lessen the time required for transfer. You can compress it in one step actually, using 'tar cjf yourfile.tar ….'. The reason I'm running it in two steps is in order to lessen the time require for creating a database snapshot, so there's even less probability of database being modified during archivation process (tar takes ~5 seconds, tar with bzipping takes almost a minute).

Now, we have the backup file prepared on remote server, and all we need to do is to transfer it to our local server. That's an easy task.

Using the scp command for that task (you can read a little more about it here)

scp root@remoteserver.example.com:~/remote_db.tar.bz2 /home/mike/remote-backups/`date +%y%m%d_%H%M%S`.tar.bz2

There's a little trick here though! I am naming backup files after the date and time I copied them to the local server.

Well that's all. Hopefully somebody find info here helpful :)

PS: Also notice, there is no error checking and notifications if something goes wrong in this script! So feel free to enchance the functionality yourself.

PPS: Oh and I almost forgot! Of course in order to do daily (or hourly, or any periodic backups for that matter), you need to add this one-liner to crontab on your backup server!

Sometimes you might want to run a long action on a remote server, like dump a few gigs DB, or rsync a few thousands of files between two not very fast computers.

You want to take next action as soon as that long operation completes but don't want to baby sit your servers.

How about getting email notification to your.. say.. iPhone.. when the action completes? It's pretty easy! Have a look!

mysqldump -u user -p mydatabase && echo "Dump finished!" | mail -s "Wake up!" mike@example.com

Once the command mysqldump finishes executing,  you'll get mail with subject "Wake up!" and body "Dump finished!"

You can chain more commands, of course, by adding && between them ( && executes next chained command only if the previous one completed without errors)

Pretty cool to have such a simple but useful trick under your belt, isn't it? :)

Yes, you can do it very easy and fast if you have Webmin installed on your server.

Go to Servers -> Sendmail Configuration -> Sendmail Options and make sure that SMTP port options parameters are set to:

Name=MTA
Port=587, Name=MSA, M=E

Just like on the screenshot below:

Click "Save and Apply" button and you're done. Now your mail server will accept mail on both 25 and 587 ports.

I have migrated all my Ruby on Rails system from Fedora Core 6 to RedHat 4.5 recently, and everything was fine, except for that the feature of blogging by email was broken (got a phone call today about that).

I use standard sendmail/procmail integration feature to receive mail to a designated address,  and then pipe it into Rails Mailer's receive function.

So, it looks like that in my /etc/mail/virtusertable:

m@example.com        examplemobile

this way all email which is coming to m@example.com address will be forwarded to system user called examplemobile.

In order to process the mail further and pipe it into my Rails script, I have the following setup in my /etc/aliases file :

examplemobile: "|/usr/local/bin/ruby /var/www/html/examplesite/current/script/runner -e production 'Mailman.receive STDIN.read'"

This setup worked perfectly on Fedora, but broke under RedHat. What's the problem? I was getting the following errors in my maillog:

Feb 13 11:45:33 appserv1 sendmail[32000]: m1D2jXJw032000: from=<xxxxxx@c.vodafone.ne.jp>, size=397, class=0, nrcpts=1, msgid=<20080213114719020972.1b6f@0016E68C4270>, proto=SMTP, daemon=MTA, relay=mmrts035p01c.softbank.ne.jp [123.108.236.87]
Feb 13 11:45:33 appserv1 smrsh: uid 8: attempt to use "ruby /var/www/html/examplesite/current/script/runner -e production 'Mailman.receive STDIN.read'" (stat failed)
Feb 13 11:45:33 appserv1 sendmail[32001]: m1D2jXJw032000: to="|/usr/local/bin/ruby /var/www/html/examplesite/current/script/runner -e production 'Mailman.receive STDIN.read'", ctladdr=<m@examplesite.com> (8/0), delay=00:00:00, xdelay=00:00:00, mailer=prog, pri=30623, dsn=5.0.0, stat=Service unavailable

The problem turned out to be in sendmail setup under RedHat was more secure than the one I had in Fedora. Specifically, this has to do with smrsh (or Sendmail Restricted Shell) thingy. More info about it here. Basically, sendmail only allows piping to programs which (or aliases to them) are present in /etc/smrsh directory.

Since I only use ruby to pipe to, I have added a symbolic link to it inside the /etc/smrsh directory:

[mike@appserv1 smrsh]$ pwd
/etc/smrsh
[mike@appserv1 smrsh]$ ls -l
lrwxrwxrwx  1 root root 19 Feb 13 11:50 ruby -> /usr/local/bin/ruby

Everything works as it should now, and our mobile-blogging users are supposedly happy again :)

I've been getting a strange "User unknown" error from sendmail today, when everything should have worked fine and even though users did exist. Here's some info on the problem's source and solution.

We have a domain called "talentnavi.biz", and I was asked to add a new subdomain to that domain ("kansai.talentnavi.biz"), and add new email addresses for several users who will be using that domain. I have setup everything properly on the server site by adding users and updating local domains and virtusers lists, however, when I was trying to test the delivery of the messages, I was getting "User unknown" errors and my emails to the newly-created mailboxes were bounced back.

I checked my mail server's logs, but there were of no help at all. After some googling, I found a very nice solution! In order to see how an email to a particular email address gets delivered, you should run the following command as root:

sendmail -d60.5 -d27.2 -bv info@example.com

So, in my case the command and output were as follows:

[root@me-ja mike]# sendmail -d60.5 -d27.2 -bv info@kansai.talentnavi.biz
map_lookup(dequote, mike, %0=mike) => NOT FOUND (0)
map_lookup(host, kansai.talentnavi.biz, %0=kansai.talentnavi.biz) => talentnavi.biz. (0)
map_lookup(dequote, info, %0=info) => NOT FOUND (0)
map_lookup(virtuser, info@talentnavi.biz, %0=info@talentnavi.biz, %1=info) => NOT FOUND (0)
map_lookup(virtuser, @talentnavi.biz, %0=@talentnavi.biz, %1=info) => NOT FOUND (0)
alias(info)
info@kansai.talentnavi.biz… User unknown

So, turns out that I have incorrectly configured my DNS records. They were set up like the following:

talentnavi.biz A 123.123.123.123

talentnavi.biz MX talentnavi.biz

kansai.talentnavi.biz CNAME talentnavi.biz

So, by default, the kansai.talentnavi.biz was aliased to talentnavi.biz, and since I didn't have a separate MX record for kansai.talentnavi.biz, the talentnavi.biz was used for mail delivery.

mike-meja:~ mike$ host -t mx kansai.talentnavi.biz
kansai.talentnavi.biz is an alias for talentnavi.biz.
talentnavi.biz mail is handled by 20 talentnavi.biz.

Therefore, the all email which was sent to "info@kansai.talentnavi.biz", was actually attempted to be delivered to "info@talentnavi.biz", because there was no separate MX record for the kansai. subdomain.

The solution is of course to add :

kansai.talentnavi.biz MX kansai.talentnavi.biz

Record to DNS server, and everything should work fine.

We have finally set up something which might be called a server room, at the company I work at.

We now have 4 servers, two of which are pretty powerful Dell PowerEdge 1900 servers I blogged about some time ago, one is our old server (which is basically just a custom-built stock computer), and one Mac which only serves a simple MySQL and FileMaker databases.

Now, my job for the next severals weeks is to gradually migrate stuff from old server to the new ones, which will become a PITA for sure, but I just hope it won't turn into MAJOR PITA :)