Programming

Django/Python: UnicodeDecodeError error printing Youtube unicoded data

Leave a comment

I was having a problem printing Youtube’s Unicode data using my print method:

print "<p>Video: desc=%s</p>" % (vid.desc)

I’m not well versed with Unicode data, so I was just able to brute force out of this problem, and get rid of the UnicodeDecodeError “ordinal out of range”, by doing the following:

print "<p>Video: desc=%s</p>" % (unicode(vid.desc,'iso-8859-1'))

PS: My database information is in utf-8 format. So, in my understanding, this is converting that utf-8 data into iso-8859-1 to show to the users.

Helpful Tips

Installing VirtualMin – “Error: Missing Dependency: libGeoIP.so.1 is needed by package proftpd”

Leave a comment

On my Red Hat 5.5 box (similar to CentOS), I was trying to install Virtualmin.

I got two errors while installing.

#1.
-This system is not registered with RHN.
RHN support will be disabled.
\Error: Missing Dependency: ruby-rdoc is needed by package rubygems-0.9.2-1.el5.noarch (virtualmin)
Error: Missing Dependency: perl(XML::Parser) is needed by package perl-XML-Simple-2.14-4.fc6.noarch (virtualmin)
Error: Missing Dependency: libdistcache.so.1()(64bit) is needed by package 2:mod_ssl-2.2.3-43.3.vm.x86_64 (virtualmin)
Error: Missing Dependency: libaprutil-1.so.0()(64bit) is needed by package 1:httpd-2.2.3-43.3.vm.x86_64 (virtualmin)
Error: Missing Dependency: libnal.so.1()(64bit) is needed by package 2:mod_ssl-2.2.3-43.3.vm.x86_64 (virtualmin)
This simply meant that I had to run rhn_register to register using the Red Hat login/password that I had received. Because you cannot run yum/update without registering with RHN, unlike CentOS which can update without registering. Simple enough.

#2.
Error: Missing Dependency: libGeoIP.so.1 is needed by package proftpd
This happens because apparently Virtualmin updated to proftpd from 1.3.2 to 1.3.3 without testing properly (source: Virtualmin Forums). To fix, I installed RPMForge and ran install script again.

Programming

Coldfusion solution to Oracle’s “string literal too long” (4k chars limit)

Leave a comment

Working on a Coldfusion app with Oracle database, I wanted to import large amounts of data into “CLOB” fields (capable of handing GBs of data). I tried using SQL Developer (by Oracle) and another user tried SQL Loader, but we were both getting the same error on the INSERT statement:

ORA-01704: string literal too long
Cause: The string literal is longer than 4000 characters.
Action: Use a string literal of at most 4000 characters. Longer values may only be entered using bind variables.

We searched online and noticed that the fix to this problem required bind variables and/or creating a procedure, etc. Really a roadblock if you’re not familiar with PL/SQL and Oracle.

But the problem could be solved using Coldfusion’s JDBC connector to Oracle. I simply wrote up the following cfml code, and noticed that the cfsqltype=”cf_sql_clob” takes care of this 4000 (4k) chars limit problem.

Code:

<cfquery datasource="dsn" username="user" password="pass">
	INSERT INTO logs_table VALUES (
		1,
		<cfqueryparam value="TEST" cfsqltype="cf_sql_varchar">,
		<cfqueryparam value="HUGE AMOUNT OF TEXT HERE (use cfsavecontent and output here)" cfsqltype="cf_sql_clob">
	)
</cfquery>
Programming

Chrome’s javascript sort array function is different, yet proper

1 Comment

Chrome’s javascript sort function behaves more like the ECMA standards than the sort function implemented in other browsers (like Firefox, IE, etc), which try to maintain some backwards compatibility with legacy javascript code.

Generally, when sorting an array, I’ve always found the following (incorrect) code:

result.sort(function(a,b) { return a > b } );

According to the ECMA standards, the right way to do it is not to return true or false, like the above function, but to return -1, 0, or 1, depending on how the two strings compare, like the following:

function sortfn (a,b) {
	var upA = a.toUpperCase();
	var upB = b.toUpperCase();
	return (upA < upB) ? -1 : (upA > upB) ? 1 : 0;
}
Helpful Tips

Django on CentOS Python 2.6 VirtualEnv Using GeekyMedia RPMs

Leave a comment

Django on centos geekymedia

for setuptools (easy_install):

wget http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c11-py2.6.egg#md5=bfa92100bd772d5a213eedd356d64086
easy_install *setuptools*

use it to install pip:
easy_install pip

download MySQLdb and install by:
python26 setup.py build
python26 setup.py install

download virtualenv
mkdir ~/.virtualenvs

add to .bashrc
VIRTUALENVWRAPPER_PYTHON=/usr/bin/python26
source /usr/bin/virtualenvwrapper.sh

initialize virtualenv
mkdir dev
virtualenv dev

Start virtualenv for current session
source dev/bin/activate

now install packages, they will go inside virtualenv (since we are activated)
pip install django
pip install south
pip install pil

Create django project and app
cd dev/
django-admin.py startproject myproj
cd myproj
python manage.py startapp polls

Helpful Tips

Install Python 2.6 on CentOS 5.x

1 Comment

Steps to success:

  1. Download all the Python 2.6 rpm for your CentOS (i386 or x86_64) from geekymedia.
  2. Install tcl, tk, tix (required dependencies): yum -y install tcl tk tix
  3. Install the geekymedia rpms.

    Note that the python26-libs-2.6-geekymedia1.*.rpm and python26-2.6-geekymedia1.*.rpm must be installed together like this rpm -Uvh python26-libs-2.6-geekymedia1.*.rpm python26-2.6-geekymedia1.*.rpm. Similarly, *tools* and *tkinter* rpm must be installed together like this rpm -Uvh *tools*rpm *tkinter*rpm.

Helpful Tips

Yum update stops resolving dependencies for clamav-server

1 Comment

On my CentOS/VirtualMin server configured for RPMForge, I wanted to run yum update, but I was getting the following errors while resolving dependencies for the clam* packages (you can list them by typing: rpm -qa | grep clam.

clamav-server-sysv-0.96-1.vm.el5.i386 from installed has depsolving problems
--> Missing Dependency: clamav-server = 0.96-1.vm.el5 is needed by package clamav-server-sysv-0.96-1.vm.el5.i386 (installed)
Error: Missing Dependency: clamav-server = 0.96-1.vm.el5 is needed by package clamav-server-sysv-0.96-1.vm.el5.i386 (installed)

I’ve had this problem on other servers where I had simply ignored it. But it was getting too much. So I researched and found the following solution working perfectly.

Solution

  1. Remove the clam rpms without removing the dependencies (virtualmin-base) by typing: rpm --nodeps -e clamav-0.96-1.vm.el5 and rpm --nodeps -e clamav-server-sysv-0.96-1.vm.el5.
  2. Run yum update

And voila, I get the following:

Dependencies Resolved

=====================================================================================================================================================================================================
 Package                                                  Arch                                    Version                                            Repository                                 Size
=====================================================================================================================================================================================================
Installing:
 clamav                                                   i386                                    0.96.4-1.el5.rf                                    rpmforge                                  2.2 M
     replacing  clamav-lib.i386 0.96-1.vm.el5

 clamav-db                                                i386                                    0.96.4-1.el5.rf                                    rpmforge                                   25 M
     replacing  clamav-update.i386 0.96-1.vm.el5

 clamd                                                    i386                                    0.96.4-1.el5.rf                                    rpmforge                                  233 k
     replacing  clamav-server.i386 0.96-1.vm.el5

Another solution: I have had servers that had conflicts even on the yum update, because maybe dependent packages came from conflicting repos. Specifically, clamav dependencies of rpmforge conflicted with clamav dependencies of virtualmin. To solve this, I ran: yum --disablerepo=* --enablerepo=virtualmin install clamav clamav-filesystem clamav-data clamav-lib

Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package clamav.x86_64 0:0.96.5-1.vm.el5 set to be updated
---> Package clamav-data.x86_64 0:0.96.5-1.vm.el5 set to be updated
---> Package clamav-filesystem.x86_64 0:0.96.5-1.vm.el5 set to be updated
---> Package clamav-lib.x86_64 0:0.96.5-1.vm.el5 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

===========================================================================================================================================
 Package                               Arch                       Version                             Repository                      Size
===========================================================================================================================================
Installing:
 clamav                                x86_64                     0.96.5-1.vm.el5                     virtualmin                     934 k
 clamav-data                           x86_64                     0.96.5-1.vm.el5                     virtualmin                      25 M
 clamav-filesystem                     x86_64                     0.96.5-1.vm.el5                     virtualmin                      18 k
 clamav-lib                            x86_64                     0.96.5-1.vm.el5                     virtualmin                     4.4 M

Transaction Summary
===========================================================================================================================================
Install       4 Package(s)
Upgrade       0 Package(s)

Total download size: 31 M
Is this ok [y/N]: y
Downloading Packages:
(1/4): clamav-filesystem-0.96.5-1.vm.el5.x86_64.rpm                                                                 |  18 kB     00:00     
(2/4): clamav-0.96.5-1.vm.el5.x86_64.rpm             (0%) 17% [=======                                   ]  0.0 B/s | 160 kB     --:-- ETA 
(3/4): clamav-lib-0.96.5-1.vm.el5.x86_64.rpm         (3%)  3% [=-                                        ]  0.0 B/s | 168 kB     --:-- ETA 
(4/4): clamav-data-0.96.5-1.vm.el5.x86_64.rpm       (17%)  0% [                                          ]  0.0 B/s |  40 kB     --:-- ETA 
-------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                      914 kB/s |  31 MB     00:34     
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing     : clamav-filesystem                                                                                                   1/4 
  Installing     : clamav-data                                                                                                         2/4 
  Installing     : clamav-lib                                                                                                          3/4 
  Installing     : clamav                                                                                                              4/4 

Installed:
  clamav.x86_64 0:0.96.5-1.vm.el5            clamav-data.x86_64 0:0.96.5-1.vm.el5        clamav-filesystem.x86_64 0:0.96.5-1.vm.el5       
  clamav-lib.x86_64 0:0.96.5-1.vm.el5

Along with the above solution, you must remember to disable the conflicting repository. In this case, rpmforge. If rpmforge ever has a newer version of one of the clamav packages, and anyone runs yum update, the dependencies will again stop resolving (because rpmforge and virtualmin repositories do not have a symbiotic relationship). To disable rpmforge, edit /etc/yum.repos.d/rpmforge.repo and add/replace enabled=0 to the [rpmforge] section. Also note, that anytime you want to explicitly use rpmforge, you can do yum --disablerepo=* --enablerepo=rpmforge (or if you want to use it in conjunction with other enabled repositories, remove the –disablerepo flag).

Projects

Punjabi Chat iPhone app version 1

2 Comments

I have been running PunjabiJanta.com for over three years. At times, it has grown as a community and at times, it has shrunk in size. I rarely get time to work on the features of the website, so I felt like I owed the site something special, something the other Punjabi social networking websites lacked.

One of the most widely used features of Punjabi Janta is the PJ Chat. It has been the focus point of Punjabi Janta from the start. In fact I would go so far as to argue that it has been the cornerstone of PJ. So to propel that cornerstone, I have developed the Punjabi Chat iPhone app, which requires a PJ account to work.

Features:

  • Chat with users of PJ and on iPhone
  • Private message users
  • Smileys
  • Status: Away, BRB, Busy, Online
  • Rooms

Screenshots:

PS: You must have a activated (email authenticated) account on PJ for it to work. Please sign up now at PunjabiJanta.com.

Helpful Tips

Nginx with m4a/aac/mp4 seek support on CentOS 5

2 Comments

I was working on Sikh Sangeet website which serves a lot of static audio media files. For the past couple of months, we’ve been paying overage (i.e. bandwidth charges for surpassing limits). I either had the option of paying much more (hundreds) for a better server or keep paying the overage charges (usually less than a hundred). So, I wanted to convert all the listening audio to a smaller format, such that I could provide full quality mp3s for downloading and provide smaller format audio for listening. I ended up choosing H264’s aac-HE, because it is supported in flash, android and iphone.

Previously on our dedicated Sikh Sangeet server, I ran nginx because the server was very limited with hardware, yet needed to serve the huge files that were mp3s. I had the latest nginx rpm (0.6.39-5.el5) from epel setup, but to get mp4 support, I had to install nginx from source with mp4 support.

I will reproduce the steps in the hopes that if I or anybody needs to do this in the future, they can get a quick idea:

Install nero encoders (best encoders for aac).
Convert mp3 to wav: mplayer -vc null -vo null "orig_file.mp3" -ao pcm:fast:file="/tmp/out.wav"
Convert wav to aac/m4a: neroAacEnc -br 56000 -if /tmp/out.wav -of "new_file.m4a"
Tag the m4a file with proper info: neroAacTag "new_file.m4a" -meta:title="title" -meta:artist="artist" -meta:track="track" -meta:album="album" -meta:genre="genre"

Now we have a m4a (aac) file, but this file is still not interleaved like an mp4 should be for nginx to stream parts of it. We must now convert the m4a (aac) file to mp4.

Install MP4Box (best way to convert regular audio into mp4 containers).
Convert m4a to mp4: MP4Box -add new_file.m4a:sbr final_file.mp4

Now we have our audio part ready. Now we just need to setup nginx:
Use the compile/make instructions from the original creators of the mp4 streaming module for nginx.

Notes:

  • If using the latest nginx server, the configure statement will not work. For two reasons:
    1. The nginx source requires pcre, even if you have pcre and pcre-dev(el) packages installed. For some reason nginx ./configure wasn't picking them up, so I downloaded pcre from pcre ftp. And without running make/configure or anything on it, used the following ./configure statement for nginx: ./configure --add-module=$HOME/nginx_mod_h264_streaming-2.2.7 --sbin-path=/usr/local/sbin --with-debug --with-pcre=$HOME/pcre-8.10
    2. The nginx_mod_h264_streaming-2.2.7 module has an incompatibility with the latest nginx (confirmed with 0.8.53):
      nginx_mod_h264_streaming-2.2.7/src/ngx_http_streaming_module.c:158: error: ângx_http_request_tâ has no member named âzero_in_uriâ.
      So, you must patch the ngx_http_streaming_module.c file using this patch.
  • For Red Hat/CentOS/Fedora users: If you want to use service nginx start/stop/reload/etc, download this Nginx Init Script.

Once you have nginx ready serving regular files from your document root (public_html, www, etc). Then simply put this directive inside the server context:

location ~ \.mp4$ {
root /your/document/root/full/path/here;
mp4;
}

The mp4 line will take care of the final_file.mp4?start=25 variable (i.e. start at 25 seconds) that flash players send to the server and cut off the mp4 at the right place.