Monthly Archives: October 2010

Helpful Tips

Python Scraping: Scrapy and BeautifulSoup

3 Comments

When I search for solutions to my problems, I often search the internet for “compare and contrast” or analytical posts on the best tools for the job, which in turn help me make an informed decision.

Recently, my problem was scraping a website for data using python. I searched online and a lot of users recommended Scrapy over BeautifulSoup. Well, that was easy, I naively said. Scrapy probably is the better option for most people (it supports XPath right out the box). Like Scrapy’s docs put it:

comparing BeautifulSoup (or lxml) to Scrapy is like comparing jinja2 to Django.

But Scrapy didn’t settle well with my Cent OS platform (or Google Apps Engine). For one, there were a whole lot of problems trying to install Scrapy in my virtualenv (safe python environment system) because of it’s dependency on libxml2/libxslt and their bindings. Examples:


etree.so "undefined symbol: libiconv"
Version 2.6.26 found. You need at least libxml2 2.6.27 for this version of libxslt
ImportError: /pyenv/test/lib/python2.6/site-packages/libxml2mod.so: undefined symbol: xmlTextReaderSetup
No module named libxml2
Failed to find headers. "update includes_dir"

Note: This may look overly dramatic. And it maybe is a little dramatic, because a lot of these errors/problems do have solutions. Most of them can be searched out of Google results.

I endlessly chased solutions at trying to integrate libxml2, libxml2 python bindings, libxslt and lxml in a virtualenv (with python 2.6; note Cent OS/RHEL only have python2.4 in their repositories). I eventually grew tired of trying to find what is linking to what shared library and what seems to be the missing culprit. And I figured, let me just give BeautifulSoup a try. I thought I’d spend the extra time learning the library that BeautifulSoup is, as opposed to learning the “framework” that Scrapy is.

In the end, BeautifulSoup was not that hard. It may be missing XPath support in its default setup, but I could easily implement the XPaths that I had with ones using BeautifulSoup syntax.

Lesson: Don’t let your ego get into it. Save time by going for fairly-efficient solutions that can be implemented in fairly-optimal time (as my Algorithms professor used to say).

Programming

Convert Django MySQL Database Tables to Unicode

Leave a comment

When I created a Django application, I hadn’t noticed that my MySQL was defaulted to latin character set (probably by Virtualmin or CentOS’s default MySQL values). So I didn’t want to delete my current project and start again. So here are the commands to convert a database to unicode:

for the database

ALTER DATABASE djangodb CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci

on each table do

ALTER TABLE djangotablename CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci
Programming

Django: “Error importing authentication backend”

11 Comments

This is probably a very rare error that one may encounter in Django. But I think I should share it here, as it would save about an hour of anybody else who has this problem.

Problem
Exception Type: ImproperlyConfigured at /
Error importing authentication backend

Probable Cause
I was very desparate to change the name of an app inside my Django project. I renamed the folder name and all possible mentions of the application name anywhere in the code and the database tables (Please note: This is not recommended, there is probably a better solution to do this). Once I faced that problem with no clear indication of where I was going wrong, I looked everywhere in the code and the database. After going into panic mode, I tried desperately changing and removing anything that may break. In the end, I ran out of places to find the application name but the error still existed.

Solution
I had noticed after looking at my cookies that I still had cookies from my session, which meant that everytime I connected to the server, I was trying to pass my “delicious” cookies. But just deleting your own cookies won’t do it. The session object of the user was cached in the database inside the table “django_session”. This especially stores the “AUTHENTICATION_BACKENDS” last used. So, truncate the table: TRUNCATE TABLE django_session to finally get rid of this nasty problem.

Projects

DholCutz Bhangra Radio Android App v1

1 Comment

DholCutz Bhangra Radio Android App is out now! Get it from the Android Market.

The Story: I had planned to do a DholCutz Bhangra Radio app about 4-5 months ago, but I never got the time to do it because my initial plans were to do one big app. Apparently, the demand of the app was too much that I just let it go and programmed a quick version for now. The Android Market approval process (1 day) is the best, light years better than iPhone App Store (3 weeks).

The App: I decided to price this app for $1 USD, which might change when I release a much much better version of this app that allows requesting songs from inside the app. Until now, this will do for the punjabi music fans. The app plays music from the radio and shows the current playing song.

The Screenshots:

Thank you.