macdiggs.com

Was working on a remote machine today…

Thankfully the machine was smart enough to reboot itself :)

I've been doing some maintenance of my server, and wanted to do some spring clearning, deleting all spam files inside users' directories. These files are automatically created by spamassassin software. Also, wanted to get rid of Rails production.log files.

Doing everything manually is no fun, and I have to admit, I completely suck at shell scripting.. But if you never try - you'll never learn, so that's what I came up with.

Calculating their size

First, I wanted to find out how much space exactly files called spam inside all directories inside the /home directory take. That's the command which I came up with ( of course, I first had to cd /home ):

[root@me-ja home]# find * -name spam -type f -exec echo {} \; | xargs du -ks | awk '{total += $1} END {print total}'

2952880

A little explanation. I use 3 commands each piping its output to the next one (and the last outputs everything to standard output which is the screen. (more…)

I spent half an hour surfing the net for an answer and luckily found this page: Problems with IE7 Sessions Not Saved in Rails or PHP

So basically if session cookies (or any cookies for that matter) work just fine on Safari/Firefox, but just get silently ignored on Internet Explorer - please check if there are any underscores in sub-domain you are accessing.

For example, I was using domain called 'amtes_shop.local' for my Rails application, and I could never "shop" for anything because cart was using sessions, and sessions were silently ignored by IE 6/7 (so I was in the latest stage of development when one tests everything on broken browsers ;) Renamed amtes_shop.local to amtesshop.local - and everything magically fixed itself! Bingo!

Ok so today I have noticed that ranking ramp-up for a specific person still continues. I have blocked one device, but the other found another one, it seems..

Both devices are mobile phones of the same carrier - KDDI. Looks like some manager (the site is a talents' directory site) decided to rank his favorite pet up, no matter what, but it also looks like that person is only doing this from mobile phone, and his carrier stands unchanged. So, I decided to tighten block logic (if you can call that logic) on that specific person's profile.

Before, I was only blocking a specific User-Agent:

RewriteCond %{HTTP_USER_AGENT} "KDDI\-KC35 UP\.Browser/6\.2\.0\.5" [NC]

RewriteRule ^.*$ - [F,L]

On one of my servers, I was under some weird "ranking up!" attack, which basically was just a loop of requests to one user's profile (making that users access count higher, and therefore ranking higher in the access top)

Here's what my Apache logs were telling me:

218.25.251.170 - - [17/Oct/2008:11:02:26 +0900] "GET /ono/profile HTTP/1.1" 200 2363 "-" "KDDI-KC35 UP.Browser/6.2.0.5 (GUI) MMP/2.0"
218.25.251.170 - - [17/Oct/2008:11:02:30 +0900] "GET /ono/profile HTTP/1.1" 200 2363 "-" "KDDI-KC35 UP.Browser/6.2.0.5 (GUI) MMP/2.0"
218.25.251.170 - - [17/Oct/2008:11:02:34 +0900] "GET /ono/profile HTTP/1.1" 200 2363 "-" "KDDI-KC35 UP.Browser/6.2.0.5 (GUI) MMP/2.0"

SetEnvIfNoCase User-Agent Mozilla getout

<Directory "/var/www/html/myserver">

Order allow,deny

Allow from all

Deny from env=getout

</Directory>

I have some very weird problems going on with some Ruby on Rails-driven sites on my server.. Basically, everything works perfect after my mongrel clusters just launched, but as time passes (and it could be hours or even days), "serialize" part of my apps start returning absolutely wrong values, don't return them at all, or just don't store values correctly.

It is all fixed by restarting Mongrels, but I wonder if anybody knows what's up with this problem? I tried different versions of ruby (currently running 1.8.6p287), and don't want to upgrade to 1.8.7 yet (have too many systems, some running on Rails 1.2…)

WTF is going on?…

We had a call from a client for whom I have built my first "commercial" Rails application. The client was telling the pages on blog part of their site started loading pretty slow.

After some quick investigation, I found out that it was somehow related to the access logging part of the application. Monitoring the SQL requests, I noticed requirest like "select * from accesses where accesses.site_id=1" taking huge time to get processes (well of course..). And I don't remember I was writing selections for all access records for a particular site anyways.

So who (or what?) was doing these huge selections? Turned out it was a pretty harmful code inside the Post class (which represents a blog post):

def log_access()
oneAccess = Access.new()
self.accesses << oneAccess
self.site.accesses << oneAccess
end

This just created a new access record and then attaches it to site, and to the post's accessses list. Well that's fine, and attaching access to the post's accesses list is actually pretty fast, but when it comes to the code:

self.site.accesses << oneAccess

things start getting hairy.. Basically, a post site's accesses are being completely read into memory, and THEN a new access record is attached to the list. And that's what generated the forementioned "select * from accesses where accesses.site_id=1" queries.

Fix is pretty easy of course, now that I know Rails better than few years ago ;)

def log_access()
oneAccess = Access.create(:post_id => self.id, :site_id => self.site_id)
end

So, the lesson is  - always be careful about making things the "smart" way, unless you understand what actually is going under the hood…

I've been setting up a very simple backup of one site's MySQL database to another server today.

What I needed to be done is to have the MySQL database files to get archived, compressed and transferred to my other server, and be named after the actual backup date and time. And that operation should happen every night.

Here's the full code for those who are in hurry (a cool one-liner heh :), and more detailed explanation of steps taken will follow.

ssh root@remoteserver.example.com 'mysqladmin flush-tables –socket=/tmp/mysql.sock; rm -f remote_db.tar* && tar cf remote_db.tar /var/lib/mysql && bzip2 remote_db.tar' && scp root@remoteserver.example.com:~/remote_db.tar.bz2 /home/mike/remote-backups/`date +%y%m%d_%H%M%S`.tar.bz2

And the explanation.

First, the server I'm backup up from is "remoteserver.example.com" and the forementioned command is invoked from the server I am backing up to. Also, I have my public ssh key installed for root account on the remote server, so I don't need to enter password to log into the remote server. You can read more about setting up SSH keys here.

First step is to log into the remote server, which is achieved simply by running

ssh root@remoteserver.example.com

However, actually instead of logging into the server, I only need to execute some commands on the remote server. This can be accomplished by giving a string of commands to execute as a second (last parameter to ssh command), for example:

ssh root@remoteserver.example.com 'ls -lh'

will give you a listing of files of remote server root's home directory.

So, now that we are connected to the remote server, we actually just need to prepare backup files which we will later transfer from remote to local server. Though the command is actually a one-liner, I'll split the lines for easier understanding, and give them numbers. Also, please note that the command concatenation with the && symbol does the following - it runs next command in chain only of previous command executed successfully.

1. mysqladmin flush-tables –socket=/tmp/mysql.sock
2. rm -f remote_db.tar*
3. tar cf remote_db.tar /var/lib/mysql
4. bzip2 remote_db.tar

#1 flushes mysql tables (so everything that is possibly in memory cache is writted to disk).

#2 removes any previous backup files (that's easy)

#3 this archives the whole MySQL data directory (this path can differ from mine, depending on your installaion parameters!). Also, please note that this approach is NOT SAFE! You can easily get a corrupted backup if any data changes during the archival process. The reason why I'm doing it myself is because I have 100% guarantee that the database will not be updated (the backup is for some inter-corporate system, and I have 100% guarantee that nobody is accessing the database at 3 o'clock in the morning when my backup task is running). You might want to lock tables during backup and unlock them once it is complete. Also, I'm backing everything up this way because I need a drop-in backup - anything happens, and I can just drop the backed up DB in place of the old one.

#4 and the final step is just to compress the backup archive to lessen the time required for transfer. You can compress it in one step actually, using 'tar cjf yourfile.tar ….'. The reason I'm running it in two steps is in order to lessen the time require for creating a database snapshot, so there's even less probability of database being modified during archivation process (tar takes ~5 seconds, tar with bzipping takes almost a minute).

Now, we have the backup file prepared on remote server, and all we need to do is to transfer it to our local server. That's an easy task.

Using the scp command for that task (you can read a little more about it here)

scp root@remoteserver.example.com:~/remote_db.tar.bz2 /home/mike/remote-backups/`date +%y%m%d_%H%M%S`.tar.bz2

There's a little trick here though! I am naming backup files after the date and time I copied them to the local server.

Well that's all. Hopefully somebody find info here helpful :)

PS: Also notice, there is no error checking and notifications if something goes wrong in this script! So feel free to enchance the functionality yourself.

PPS: Oh and I almost forgot! Of course in order to do daily (or hourly, or any periodic backups for that matter), you need to add this one-liner to crontab on your backup server!

… and publishing this article using iPhone's WordPress application! Pretty cool huh:)

Sometimes you might want to run a long action on a remote server, like dump a few gigs DB, or rsync a few thousands of files between two not very fast computers.

You want to take next action as soon as that long operation completes but don't want to baby sit your servers.

How about getting email notification to your.. say.. iPhone.. when the action completes? It's pretty easy! Have a look!

mysqldump -u user -p mydatabase && echo "Dump finished!" | mail -s "Wake up!" mike@example.com

Once the command mysqldump finishes executing,  you'll get mail with subject "Wake up!" and body "Dump finished!"

You can chain more commands, of course, by adding && between them ( && executes next chained command only if the previous one completed without errors)

Pretty cool to have such a simple but useful trick under your belt, isn't it? :)