AJAX and Japanese. Check your encodings.

I'm starting to use AJAX more in more in CMS I develop. And that exactly what I was doing today. Had to "optimize" some site for speed. I have created the CMS for the site, using AJAX in parts, but was a little bit too lazy to add AJAX everywhere. So the pages we loading.. well.. fast, but not fast for the client, so I was told to make some optimizations.

Basically, site's functionality is one big gallery. There are galleries from weddings, galleries of wedding dresses etc.. The company has 5 filials, so there are multiple galleries for each filials are possible. Before today, although loading photos, navigating thru thumbnails pages and galleries pages didn't require any page reloads (AJAX magic baby :), switching between galleries did require a complete page refresh. And because the was pretty big pulldown menu in all these pages, loading time was..well.. several seconds, which didn't go well with the customer (and yeah, I understand him). So, the solution I went was to go the way of "complete AJAX" - that is, make the page not to reload at all if you don't change filials (meaning swiching galleries don't require any page reloads).
The implementation was actually pretty easy and I could test everything on my Mac without any problems. Problems appeared when I tried to upload everything to public server. The public server, just like some public servers are, was not the same configuration as my development server.

Specifically, there was different PHP settings for mb_string encodings there. By default, I use Japanese ShiftJIS encoding (and that's the encoding of almost all Japanese sites). On my server, SjiftJIS is used thru the whole processing of PHP scripts, from html_input to html_output. As well as internal_encoding parameter is also set to SJIS.

The public server, on the other hand, had internal_encoding set to another flavour of Japanese encoding, named EUC. Not that it's a bad encoding, but it differed from the whole "pipe" I used to test my code, and results of all AJAX calls, after insertion into page code, produced unreadable text.

There were 2 ways to solve the problem, and I tried them both and they both worked. First way was just to synchronize the php.ini settings between both servers, but at the time I was trying to apply this solution it was not possible due to some configuration override limits on public server (oh.. uncontrolled hosting solutions..).

So I went another way. The garbled text I got inserted into HTML was "looking" like it was having UTF-8 encoding. I was thinking.. "ok. here we have UTF, but we need SJIS, so how about some server-side conversion before returning results from AJAX call?".
The solution was to use mb_convert_encoding function to convert encoding from UTF-8 to SJIS. And it worked!

The thing it that it worked for just 20 minutes :) And after that, I got the server configuration override permissions fixed on the hosting side and was able just to synchronize settings of both public and development servers. Talking about time wasted ;)

So, the idea is that (and it applies not only Japanese I guess) - if you have AJAX calls returning garbled text, check settings of php.ini so that text's encoding is not altered during processing. And if it do gets changed, convert it to the one you need just before returning result. For minimum impact into your code's logic, the recommended solution is just output everything into output buffer, get it's contents and convert them. The just return the converted result back to AJAX call.

Happy AJAX coding :)

Leave a Reply