Although I am mainly a PHP developer, I have some knowledge of Python too. I really like Python, because it’s a well structured, easy to learn, neat and powerful language. It can be used in many areas, like Web Development (did someone mention Django?), Desktop Application Development as well as Scientific Computations.
What is a web bot?
A web bot (or webot) is a program which is used to read and parse data from the web. Web bots have many usages, like processing large amounts of data from sites (like Twitter) for business purposes, indexing keywords for later use (by search engines like Google) and many more. If you are reading this far, I assume you have a reason to use a web bot, so on we go!
You can write a web bot in many languages out there, but why should you choose Python? First of all, it’s easy to program in Python. Second, Python has many great features which make it a good language for data processing. Third, You have several options available for writing a web bot in Python (see below).
Options for writing a web bot in Python
There are mainly three options (which I know of) for writing a web bot in Python:
- Using the PycURL library: This is a Python interface for the very famous libcurl library. It supports many protocols and is very feature rich. If you are coming from another language which has an interface to cURL (like PHP), then this option will be very easy to use for you.
- Using the classes from PyQt (QtWebKit): PyQt is the Python interface to the Qt framework. Qt (and therefore PyQt) has the WebKit engine embedded at it’s heart(!), so you can create a complete Web Browser using PyQt! This option had a plus and a minus for me. The minus is that, installing QtWebKit (WebKit classes for Qt) on some operating systems (like CentOS 6) might cause you some headache. But the plus is that, you can do virtually anything with QtWebKit. For example you can use QtWebKit to create screenshots of your web pages! So, sky is the limit.
- The last option (that I know of) is to use a library called Mechanize. Mechanize is really great for creating web bots. It is very simple and easy to use. You can do many crazy things with Mechanize with little code.
In the next parts of this tutorial series, I will be going through many examples of using Mechanize and doing interesting things with it. So, stay tuned and don’t forget to leave comments!