Shaakunthala's Portal

Hi dear readers! First of all, I WISH YOU A HAPPY AND PROSPEROUS NEW YEAR WITH LOT OF ACHIEVEMENTS, GLORY AND JOYNESS!! Anyway, 2012 is also approaching.. :D (just kidding)

After a long long time, I've got an interesting problem to solve. I'm not an expert. I'm just writing my own way that I followed in the situation.

This happened during the Christmas days in 2009, and after all, I feel it like a Christmas gift, seriously! :-) I wrote two blog posts in my native language, I you can read, just visit the following links. You'll find it more interesting than this one if you can read. :)

1. http://blog.shaakunthala.com/2009/12/hacker.html
2. http://blog.shaakunthala.com/2009/12/bash.html
   

Alright, then... I'm responsible for the administration of several websites. As I feel, an administrator's job is very much similar to the job of a sea captain. He has to look after the system, like his own... be vigilant of the attacks and other problems,.. and many more work.

Recently, I've been notified that one of my sites is down. An empty page with an error message is displayed when the site is visited, and according to that error message, there's an error on index.php, line 38 and the character < is the cause. This is the way how PHP shows error messages. As my immediate actions, I logged on to the FTP server where our website is hosted and opened the index.php

The website was developed using a CMS. The code looked some kind of strange for me because it had no corresponding ?> tag for its beginning <php tag. An unknown HTML/ Javascript code snippet has appended at the end of the file.

Yes, that's the cause of that error. Somebody has injected a malicious code snippet at the end of the index document, and the PHP engine on the server side has tried to interpret it as PHP. As this has caused a syntax error in PHP, the whole site has gone down as the final result.

Here's the structure of index.php :

<?php
/* PHP
codings of
CMS */

<script> // The foreign Javascript code snippet </script>

What I did is, just copy-pasted the code into a separate text file (for analyzing), and cleaned index.php. Then everything looked normal, but sooner I got to know, actually it is not.

I've never experienced such a situation before. I didn't know where to start, and what to do. But I wanted to find out what the code says. It was some kind of scary and big JavaScript code in a single line. However, it's not so scary!

Okey,... a closer look...

Right... and this is our troublemaker...

Just see carefully,.. you don't need to be a JavaScript guru. :-)

It's not a big deal to identify such big codes. Vigilance is what matters here. They have used the replace () method in JavaScript. See... strategically hiding text by just randomly mixing punctuation, retrieving the original text at run time. Wow!

Finally, it's this:

Looks like it has been created for phishing purposes... but I'm not sure exactly. However, this URL points to an empty page. What I expected was a JavaScript code, but this resulted nothing... I don't know a reason. :-/

Within few hours after fixing the issue, I got to know that our website is down again. The same thing has happened, same style, but the malicious code resulted a different URL. I fixed it again, and started thinking... what on earth could be happened here? :-O

Whoever the attacker has done is injecting some malicious code into the index document, and letting it execute at the client's (browser) end. However, as the code has blindly appended at the end of the file despite the structure of it, I came to a conclusion -- definitely this is done by a bot / script or some other automated mechanism.

So, I did the same operation as before for cure, and then tried to find some solution. Yes, it's gonna be a new experience. To prevent further attacks, I put the following line at the end of the index.php file. It prevented interpreting any code below the line. When I say die!, no further interpretation of code at all. Hence, the site is safe from being down, but the risk is still their till I find where the attack comes from and where the security hole is.

die ();

I tried to find any clue on site logs,.. but no luck. If this attack was carried out through HTTP, site logs (not the CMS logs) should indicate that. What I suspect is, somebody has gained access to the server, and executed a script. By adjusting file permissions on the index document, I found out that the malicious script on the server (or bot) has gained the root access.

Later, I got to know, this has recursed into directory hierarchy through the entire site. And also, I saw that some JavaScript files are also infected. It was shocking! Everything throughout the site can be potentially infected with malicious code and hence unsafe for visitors!! I didn't know how serious the attack was. I have never faced such a situation before, and as the responsible personnel, I have to fix this as soon as possible, with my best efforts.

According to all observations, my conclusion was, this is happened due to the fault of the web hosting provider. I know that CMS' sometimes can contain security holes, but if it was, there should be at least something on the site logs.

All of the above is the summary of my first blog post, mentioned at the top of this post. The next few paragraphs in this post explain how I performed the disinfection.


The only backup we had was bit old, so I forget the idea of restoring from a bacup archive. The challenge was to find out how serious the attack was, and to disinfect everything.

What I suspect so far:
Every JavaScript file and index document is infected -- but not sure about other text-based file formats.

So I have to check each file for malicious code, and then clean them.

First, I thought of writing a PHP script for the purpose. But, PHP is bit insecure with this work. I know, it's not a big deal to fix the security with PHP, but, I was more interested in bash scripting. As a daily Linux-only computer user, I am very familiar with bash, and feel more reliability with that.

Luckily, the web hosting service provider has offered remote access through ssh. Yes, that's great! I was very keen, the rest's gonna be a party!! ;)


Here's the match highlights... :P

Access through ssh, compress the entire site, and then download it. This is necessary because the safe way is to keep a backup + do a testing when doing something serious. One mistake, could ruin everything!!

Here we go, ssh

$ ssh user@mysite.com

Create an archive, (make it tar.bz2 for higher compression ratio -- easy to download). Then exit ssh.

$ tar cvfj mysite.tar.bz2 mysite/
$ exit

Download the backup, through ssh copy.

$ scp user@mysite.com:/home/user/mysite.tar.bz2 /home/shaakunthala/

Unpack on my computer, to be tested with the script.

$ tar xvjf mysite.tar.bz

Now, next step is to write the script. Fired up my favourite vim editor, and then started thinking. ;) Before writing the script it's necessary to exactly identify the nature of the malicious code. Here's what I've identified:

• If a file is infected, the malicious code is at the end of the file.
• The foreign code snippet is different from point to point. But, following text portions can be recognized as a common pattern.
  • GNU GPL
  • window.onload
  • .replace
• Although it seemed like the infection is only with JavaScript and index documents, I refused to accept that. Also, as we didn't have any gigantic files with our website, I decided the script to test all files throughout the site.

Although it was such an easy task to write a script for malware removal, I had to separate the program into two scripts because find -exec does not recognize functions in bash. So, here's what I wrote:

#!/bin/bash
# Author: Sameera Shaakunthala

rm fixlog.txt
rootdir=`pwd`/mysite/
sup=`pwd`"/fixfile.sh"
find $rootdir -exec $sup {} \;
echo "JOB DONE!"

#!/bin/bash
# Author: Sameera Shaakunthala

echo "Processing file: "$1
code=`tail --lines=1 $1 | grep "GNU GPL" | grep window.onload | grep .replace`
l=`echo $code | wc -m | awk '{ print $1 }'`

if [ $l -ne 1 ]
then
 lc=`wc -l $1 | awk '{ print $1 }'`
 lc=`expr $lc - 1`
 head $1 -n $lc > tempfile.tmp
 mv tempfile.tmp $1
 echo "File "$1" has been fixed!" | tee -a fixlog.txt
fi

Now, the next task is the test run on my local machine. If this succeeds, it is safe to run the script on the server.

$ chmod +x sitefix.sh fixfile.sh
$ ./sitefix.sh

After execution, I checked the fixlog.txt, which is the output log of the script. OMG! 602 infected files!! :-O I vigorously checked some randomly selected files, they were clean, and as everything seemed to be clean, I uploaded the script to the server, and then executed. :)

$ scp sitefix.sh fixfile.sh user@mysite.com:/home/user
$ ssh user@mysite.com
$ chmod +x sitefix.sh fixfile.sh
$ ./sitefix.sh

Finally, we have set this as a cron job, till we find the actual security hole.

The final result was, the disinfection of the entire website, within few minutes. As I got to know that virus scanners no longer block our website, it was confirmed that the site is clean. Just see the spirit of Linux bash scripting! :)


Finally, I put a link to a shocking article that must be read... Just click and see! :(