Archive for the 'Tech blog' Category

Blocking fight continues!

Tuesday, October 21st, 2008

Ok so today I have noticed that ranking ramp-up for a specific person still continues. I have blocked one device, but the other found another one, it seems..

Both devices are mobile phones of the same carrier - KDDI. Looks like some manager (the site is a talents' directory site) decided to rank his favorite pet up, no matter what, but it also looks like that person is only doing this from mobile phone, and his carrier stands unchanged. So, I decided to tighten block logic (if you can call that logic) on that specific person's profile.

Before, I was only blocking a specific User-Agent:

RewriteCond %{HTTP_USER_AGENT} "KDDI\-KC35 UP\.Browser/6\.2\.0\.5" [NC]
RewriteRule ^.*$ - [F,L]
The new approach is to block all mobile phones of the carrier in question, which try to access the profile of the person in question, so the rewritecond was changed to the following:
RewriteCond %{HTTP_USER_AGENT} "KDDI" [NC]
RewriteCond %{REQUEST_URI} "/ono/profile" [NC]
RewriteRule ^.*$ - [F,L]
So basically, in the first line I set on of rewrite conditions to be true when User-Agent of the device accessing the site contains the "KDDI" string (the name of the carrier), and the second line make the block more specific, telling only to apply the first rule when access is coming to the "/ono/profile" URI.
If both of these conditions are satisfied, the rewrite rule on the third line denies access to the page in question.
OK. Lets see how the smarty pants manager deals with that ;) 

Block Apache users based on their User-Agent

Friday, October 17th, 2008

On one of my servers, I was under some weird "ranking up!" attack, which basically was just a loop of requests to one user's profile (making that users access count higher, and therefore ranking higher in the access top)

Here's what my Apache logs were telling me:

218.25.251.170 - - [17/Oct/2008:11:02:26 +0900] "GET /ono/profile HTTP/1.1" 200 2363 "-" "KDDI-KC35 UP.Browser/6.2.0.5 (GUI) MMP/2.0"
218.25.251.170 - - [17/Oct/2008:11:02:30 +0900] "GET /ono/profile HTTP/1.1" 200 2363 "-" "KDDI-KC35 UP.Browser/6.2.0.5 (GUI) MMP/2.0"
218.25.251.170 - - [17/Oct/2008:11:02:34 +0900] "GET /ono/profile HTTP/1.1" 200 2363 "-" "KDDI-KC35 UP.Browser/6.2.0.5 (GUI) MMP/2.0"

So, the flood of accesses was originating from some user who was using a KDDI-flavour browser (it is a mobile phone browser used in Japanese AU operator's phones).
Luckily, though we do support mobile browsers to a degree, that's not the main feature of the site, so I have decided I can block that specific browser without affecting too many users (if any).
There are actually at least two ways to block a user from visiting you site, based on User-Agent. First is setting server environment variable and then denying users for which that variable has been set:
for example:
SetEnvIfNoCase User-Agent Mozilla getout
<Directory "/var/www/html/myserver">
Order allow,deny
Allow from all
Deny from env=getout
</Directory>
will deny all users who user Mozilla-based browsers (this includes Safari as well, as it has the "Mozilla" substring in its user agent).
However, with the site in question was running on Rails, and being served by a Mongrel cluster (via proxy balancer), the directives above didn't work for me somehow..
I had to add the following just below the RewriteEngine On directive to achieve the same blocking effect (now, targeted specifically to the offender's browser in question):
RewriteCond %{HTTP_USER_AGENT} "KDDI\-KC35 UP\.Browser/6\.2\.0\.5" [NC]
RewriteRule ^.*$ - [F,L]
Restarted Apache, and all the flood of accesses just stopped. A user was started to get access denied errors on his/her site.
Sure it wouldn't be as easy if you have flood accesses from more popular browsers (I guess in that case you'll have to block by both user agent and, say, user's subnetwork). But it worked in my limited case. Hopefully will have somebody else to fight flooders, as well :)

Rails/Ruby's serialize rot after some time?..

Friday, September 19th, 2008

I have some very weird problems going on with some Ruby on Rails-driven sites on my server.. Basically, everything works perfect after my mongrel clusters just launched, but as time passes (and it could be hours or even days), "serialize" part of my apps start returning absolutely wrong values, don't return them at all, or just don't store values correctly.

It is all fixed by restarting Mongrels, but I wonder if anybody knows what's up with this problem? I tried different versions of ruby (currently running 1.8.6p287), and don't want to upgrade to 1.8.7 yet (have too many systems, some running on Rails 1.2…)

WTF is going on?…

Do it the simple way..

Thursday, September 18th, 2008

We had a call from a client for whom I have built my first "commercial" Rails application. The client was telling the pages on blog part of their site started loading pretty slow.

After some quick investigation, I found out that it was somehow related to the access logging part of the application. Monitoring the SQL requests, I noticed requirest like "select * from accesses where accesses.site_id=1" taking huge time to get processes (well of course..). And I don't remember I was writing selections for all access records for a particular site anyways.

So who (or what?) was doing these huge selections? Turned out it was a pretty harmful code inside the Post class (which represents a blog post):

def log_access()
oneAccess = Access.new()
self.accesses << oneAccess
self.site.accesses << oneAccess
end

This just created a new access record and then attaches it to site, and to the post's accessses list. Well that's fine, and attaching access to the post's accesses list is actually pretty fast, but when it comes to the code:

self.site.accesses << oneAccess

things start getting hairy.. Basically, a post site's accesses are being completely read into memory, and THEN a new access record is attached to the list. And that's what generated the forementioned "select * from accesses where accesses.site_id=1" queries.

Fix is pretty easy of course, now that I know Rails better than few years ago ;)

def log_access()
oneAccess = Access.create(:post_id => self.id, :site_id => self.site_id)
end

So, the lesson is  - always be careful about making things the "smart" way, unless you understand what actually is going under the hood…

A very simple offsite backup over ssh/scp

Wednesday, August 27th, 2008

I've been setting up a very simple backup of one site's MySQL database to another server today.

What I needed to be done is to have the MySQL database files to get archived, compressed and transferred to my other server, and be named after the actual backup date and time. And that operation should happen every night.

Here's the full code for those who are in hurry (a cool one-liner heh :), and more detailed explanation of steps taken will follow.

ssh root@remoteserver.example.com 'mysqladmin flush-tables –socket=/tmp/mysql.sock; rm -f remote_db.tar* && tar cf remote_db.tar /var/lib/mysql && bzip2 remote_db.tar' && scp root@remoteserver.example.com:~/remote_db.tar.bz2 /home/mike/remote-backups/`date +%y%m%d_%H%M%S`.tar.bz2

And the explanation.

First, the server I'm backup up from is "remoteserver.example.com" and the forementioned command is invoked from the server I am backing up to. Also, I have my public ssh key installed for root account on the remote server, so I don't need to enter password to log into the remote server. You can read more about setting up SSH keys here.

First step is to log into the remote server, which is achieved simply by running

ssh root@remoteserver.example.com

However, actually instead of logging into the server, I only need to execute some commands on the remote server. This can be accomplished by giving a string of commands to execute as a second (last parameter to ssh command), for example:

ssh root@remoteserver.example.com 'ls -lh'

will give you a listing of files of remote server root's home directory.

So, now that we are connected to the remote server, we actually just need to prepare backup files which we will later transfer from remote to local server. Though the command is actually a one-liner, I'll split the lines for easier understanding, and give them numbers. Also, please note that the command concatenation with the && symbol does the following - it runs next command in chain only of previous command executed successfully.

1. mysqladmin flush-tables –socket=/tmp/mysql.sock
2. rm -f remote_db.tar*
3. tar cf remote_db.tar /var/lib/mysql
4. bzip2 remote_db.tar

#1 flushes mysql tables (so everything that is possibly in memory cache is writted to disk).

#2 removes any previous backup files (that's easy)

#3 this archives the whole MySQL data directory (this path can differ from mine, depending on your installaion parameters!). Also, please note that this approach is NOT SAFE! You can easily get a corrupted backup if any data changes during the archival process. The reason why I'm doing it myself is because I have 100% guarantee that the database will not be updated (the backup is for some inter-corporate system, and I have 100% guarantee that nobody is accessing the database at 3 o'clock in the morning when my backup task is running). You might want to lock tables during backup and unlock them once it is complete. Also, I'm backing everything up this way because I need a drop-in backup - anything happens, and I can just drop the backed up DB in place of the old one.

#4 and the final step is just to compress the backup archive to lessen the time required for transfer. You can compress it in one step actually, using 'tar cjf yourfile.tar ….'. The reason I'm running it in two steps is in order to lessen the time require for creating a database snapshot, so there's even less probability of database being modified during archivation process (tar takes ~5 seconds, tar with bzipping takes almost a minute).

Now, we have the backup file prepared on remote server, and all we need to do is to transfer it to our local server. That's an easy task.

Using the scp command for that task (you can read a little more about it here)

scp root@remoteserver.example.com:~/remote_db.tar.bz2 /home/mike/remote-backups/`date +%y%m%d_%H%M%S`.tar.bz2

There's a little trick here though! I am naming backup files after the date and time I copied them to the local server.

Well that's all. Hopefully somebody find info here helpful :)

PS: Also notice, there is no error checking and notifications if something goes wrong in this script! So feel free to enchance the functionality yourself.

PPS: Oh and I almost forgot! Of course in order to do daily (or hourly, or any periodic backups for that matter), you need to add this one-liner to crontab on your backup server!

Upgraded to wordpress 2.6

Friday, July 25th, 2008

… and publishing this article using iPhone's WordPress application! Pretty cool huh:)

photo

Want to get notified when a long action on remote server completes?

Sunday, July 20th, 2008

Sometimes you might want to run a long action on a remote server, like dump a few gigs DB, or rsync a few thousands of files between two not very fast computers.

You want to take next action as soon as that long operation completes but don't want to baby sit your servers.

How about getting email notification to your.. say.. iPhone.. when the action completes? It's pretty easy! Have a look!

mysqldump -u user -p mydatabase && echo "Dump finished!" | mail -s "Wake up!" mike@example.com

Once the command mysqldump finishes executing,  you'll get mail with subject "Wake up!" and body "Dump finished!"

You can chain more commands, of course, by adding && between them ( && executes next chained command only if the previous one completed without errors)

Pretty cool to have such a simple but useful trick under your belt, isn't it? :)

Hold on with upgrading to ruby 1.8.6-p230!!

Thursday, June 26th, 2008

I started getting weird

(eval):2:in `protect_against_forgery?'

error from one of my Rails2-based system today. Mongrels were seg-faulting as well. The problem was quite unexpected as I have not touched the code for a while!

Turned out the problem was with the latest upgrade to ruby 1.8.6 patch level 230 (which is recommended by the Ruby on Rails site). And looks like this was the only system re-deployed (and restarted under new ruby version) since I have upgraded.

Now I'm back to p111, "vulnerable", but at least working version until problems sort themselves out.

Until they do, do not upgrade!

UPDATE:

Looks like the is a patch which addresses vulnerabilities which patch 230 addressed in turn. Check here for details. However, I'll just stick here for a couple of days for a real fix from the Ruby team.

Using RMagick to create cool photos

Wednesday, June 18th, 2008

Having something like this

Wouldn't it be cool if you could creating something like this?

All in real-time, in your Ruby on Rails project, using RMagick.

Hold on because I'll show you how. (more…)

Temporarily disable Rails timestamp magic columns updates

Friday, April 18th, 2008

I recently faced a problem when I didn't want Rails' timestamp "magic" columns to be updated during update of records in database. I have a site where recently updated pages are displayed on front page, and I use updated_at timestamp to identify the most recently updated ones. However, the system also has a feature to reorder pages, where only the 'position' attribute is updated, and not the actual page content. The reorder operation can't be counted as a real update. However, any saves or update_attribute operations on a model fire up the magic columns update.

However, there's a simple way to permanently turn this feature off. Here's how I implemented it:

ActiveRecord::Base.record_timestamps = false #temporarily turn off magic column updates

@page.update_attribute(:position, pos)

ActiveRecord::Base.record_timestamps = true #turning updates back on