macdiggs.com

Well finally after 2 weeks of caffeine-powered crazy coding, I have a nice new project running locally on my computer, waiting to be released into the wilderness :)

The project is a full-fledged CMS/blogging system, based on my previous, already released Rails-based skinnable blog system. I have used the same basis (and the basis was multi-user blog system), and have added pages-based CMS system into the mix - so now it is a site with multiple-users blog.

There are several types of pages a user can add to the site, including normal page with editing based on TinyMCE with some custom plugins I have wrote (notably the plugin for easier images uploads). There are also special pages for products categories and products page, as well as special page with handling links to Google maps, implemented in the same way my WordPress Inline Google Maps plugin works. You can also upload different banners for site's front page, as well as all custom pages of the site. There's a ping going to Google when a blog is updated, and Google sitemap, which is generated automatically. Overall, I have a built a system I can be proud of (to a degree :)
Rails definitely rules and I can't imagine even in my happiest dreams to create such feature set in just 2 small weeks (that includes layout coding!).

Now, what is even more exciting (at least for me) is that the system is actually designed to support multiple sites running on different domains, but using the exact same Rails codebase. What I mean is that same servers, same Mongrel clusters will be serving different sites depending on the URL by which a user accesses the system (meaning different content, different design, different everything, without duplicating scripts and clusters!!). Now, that being set, I didn't try to setup the whole multiple-sites system yet - the testing is performed against single site.

If things go well, the site will go online next month. Let's keep fingers crossed though :)

Some pics..

First of all, the site (keep in mind that the design overall is copyrighted, but some images used as placeholders were taken from devianart.com).

And here are some pics of admin interface.

Phew..

Yesterday I have discovered a pretty annoying mistake in my own projects design.

As I have wrote previousely (simple caching in Rails) , I had simple front page caching implemented for my site's front page. The caching code was defined in ApplicationController, which in turn was responsible for generating the front page content.

To put it simple, it looked like this:

class ApplicationController < ActionController::Base

caches_action :index
before_filter :index_cache_expirer, :only => [:index]

The code caches (and expires) action named :index. But, there is a may be invisible at first glance, but very big problem with this code.

The problem is that "caches_action :index" directive will propagate to all other pages on the site, effectively making all :index actions in all controllers to cache their content in the same way as the front page.

Even pages which must be avoided to be cached (like /admin area on multiuser sites). Here's a simple example - a multiuser blog with /admin/posts action, showing a list of blog posts for each user, depending on who is logged in. So, say, user "mike" logs in, and watches his posts list (which gets caches, since the caches_action :index gets propagated from ApplicationController to Admin::PostsController). Soon after that, a user "jane" logs into the same admin/posts trying to have a look at her posts list - only to get served a cached page containing lists of posts of user "mike"! The page will expire eventually and "jane" will get her list of posts, but think about the next user - he or she will get Jane's posts list instead, and so on, and so on.

One of the biggest probs is that this scenario doesn't work if you are working in development mode, because caching is only active in production mode by default! So it looks like that - you happily debug your code in development mode, deploy to public server, and the caching problems begin to appear.

Now, I didn't tell why exactly this gets propagated. But that's very easy to notice if you just look at, say, PostsController definition:

class Admin::PostsController < ApplicationController

See how PostsController inherits from ApplicationController, which in turn has caching to action :index implemented? Therefore, PostsController will have the same exact set of functionality as ApplicationController, including caching of its own :index action!

So, the point is - putting too much stuff into ApplicationController is definitely not a good idea, as well as it seems to be now very wise to use ApplicationController to render site's front page (in case this page has alot of custom functionality, caching and stuff like that).

Solution? It's pretty simple, at least for the caching problem I've been having. You just have to make a separate controller for front page only, and define all these front page specific stuff inside of that controller.

So, watch out for the "hidden" inheritance problems. And happy coding :)

Well, after 2 days of weird server behavious, after which I have installed the Monit services monitor, it turned out that Mongrel was not actually the biggest source of the problem.

Yesterday, server started behaving very weirdly - restarting itself every 10-20 minutes. No trace of errors in logs, no notifications about restart to be performed even. I though that may be, just may be it is a software problem, although everything looked like a hardware problem from at the first glance. So I have checked cron (oh well, who know may be there's some process restarting the server?), even tried to turn off Monit as it was the only software I have installed recently. All this just to find the server restarting in next 10 minutes :)

At this point, I finally decided to attach a monitor to the server - yeah it was headless before that heh :) And once it was done, I could witness the exact way the server restarts itself - by just.. turning itself off and on, like somebody just pressed the Reset button. So.. since then it definitely started to look like hardware problems so I took the server offline completely and booted up from memtest CD. Just to find out that server restarts itself before the test progress even reaches 2%.

Long story short - got a new memory module, installed, started server - and all is fine since then. But anyway, all these manipulations with setting up Monit were not without use. Mongrel instances still crash several times a day. But are quickly relaunched by Monit.

So, hopefully now more midnight server fixes :)

Mongrel, which, along with Apache 2.2 & load balancer & mongrel cluster, I run for hosting my Rails site - is a very nice piece of software. But there is a problem. It crashes.

I found about it the hard way - by a phone call at 0:30 in the morning, saying "the server seems to be down". "Doh", I though - and got out of my comfy bad.. Got to my computer, logged in to the server, restarted Mongrel cluster and got back to sleep (you can always check the logs later in the morning, right? :) (more…)

My first Rails project finally went online today! Imported over 500Mb of blog data from the old blog, set up a simple caching on the front page, and made a prayer ;) Let's see how it will hold up :D

Check it out: MediaJapan Gakuen blog

And here's the old version of the blog: old version of Gakuen blog

I've been setting up some Rails caching code today and found a pretty simple and easy way to implement caching for index page which was taking too much time loading due to some pretty expensive DB queries (accesses, polls counts etc etc).

First of all, there is a very nice article on the subject written by Robert Evans - "Rails Caching" - and you can read about the ways to implement caching of pages, actions and also find a link to a very cool cached.rb gem for model-level caching.

The problem in my case was that Robert's approach (and this approach of of course Rails-recommended) for expiring cached actions is to install a cache sweeper, and in my case, index page was using data from several controllers and installing cache sweepers for all of them is needless overkill. What I wanted is just to add time-expiring action_cache for my site's top page's :index action.

So, here's the bits of code I had to add to my ApplicationController (inside application.rb) in order to make it done:

class ApplicationController < ActionController::Base

@@indexLastCached = Time.now #initializing the last-cached timer
before_filter :index_cache_expirer, :only => [:index] #installing cache-expirer action
caches_action :index  #turning on action cache for action :index

#….. controller code and stuff like that ….. #
def index_cache_expirer
if Time.now-@@indexLastCached>60 # expiring index page every 60 seconds
expire_action :action => :index
@@indexLastCached = Time.now
end
end

Well, now, I'm sure there are much better approaches to caching actions with complicated content (tell me about it?), but the above code works just fine for me and dutifully expires cached action every 60 seconds. Thanks to that, my requests-per-second count increased from 0.61 req/sec to value near 36 req/sec. Pretty cool huh?

I had a plenty of "good" time today as I have found out that all of the subscribe-by-sending an-empy-mail systems I maintain just stopped working a few days ago. Here I will describe sympthoms and solutions for this problem.
The forementioned systems work so that a user sends an empty email message to a designated email address, and gets added to a corresponding mailing list. Simple and easy to use system.

Most users (like say, 90%) are Japanese users who add themselves into mailing lists using their mobile phones. Therefore using Japanese mobile operator's email servers, such as DoCoMo's, Vodafone/Softbank and others.

The problem was that although the system was working without any problems for over a year now, a couple of days ago it just stopped working. I turned out that email messages which were sent from DoCoMo and Vodafone/Softbank mobile phones were not reaching the processing scripts on my server.

After a little digging, it turned out that these emails were classified as spam by SpamAssassin which I have of course running of the server. Now.. everything worked just a few days ago, and suddenly stopped to. Why?

After a little more digging, I found the following information added to spam messages by the SpamAssassin:

Content analysis details: (5.3 points, 3.0 required)

pts rule name description
—- ———————- ————————————————–
0.6 NO_REAL_NAME From: does not include a real name
0.5 DNS_FROM_RFC_ABUSE RBL: Envelope sender in abuse.rfc-ignorant.org
1.4 DNS_FROM_RFC_POST RBL: Envelope sender in
postmaster.rfc-ignorant.org
1.3 MISSING_SUBJECT Missing Subject: header
1.5 EMPTY_MESSAGE Message appears to be empty with no Subject: text

As the subscription emails by their nature were always sent without any subject and content, and without real name in most cases (and my SpamAssassin was set to filter out all messages with spam level of 3.0 or higher), it looks like the reason for the mail to suddenly became spam were the lines:

0.5 DNS_FROM_RFC_ABUSE RBL: Envelope sender in abuse.rfc-ignorant.org
1.4 DNS_FROM_RFC_POST RBL: Envelope sender in postmaster.rfc-ignorant.org

A quick look at rfc-ignorant.org's blacklist turned out the following:

So, both operators are black-listed because they don't conform to requirements of having postmaster@… and abuse@… email addresses. I'm not really sure that such non-conformance can be classified as a possible source of spam, but oh well.. whatever. How to fix the bloody thing?

Actually, pretty easy. You just need to edit a single config file for SpamAssassin and set the spam level of both DNS_FROM_RFC_ABUSE and DNS_FROM_RFC_POST messages to lower level (or to zero if you wish). The file is located at : /etc/mail/spamassassin/local.cf

And you just need to add the following two lines to the file and save it. Changes will be activated immediately:

score DNS_FROM_RFC_POST 0.0
score DNS_FROM_RFC_ABUSE 0.0

That's it! We just ignore the stupid false-spam messages and everything works fine again, as it was doing for a long-long time before. Just one question remains - WHY, though both operators' mail servers were black-listed more than 2 years ago, my local SpamAssassin started to pay attention to it just a few days ago?.. I don't remember updating any of server software lately, so.. a mistery? :)