Due: November 22, 2022
A web crawler is a program that starts at a web site. It then follows the links on that page recursively. If you think of the links as branches of a tree, and the original web site as the root, the crawler follows the branches to the next “root” (i.e., web site) and then repeats this process some number of times (the depth).
Your job is to write a simple web crawler. We will do this in 2 steps — that makes it less overwhelming. Also, we will focus only on one type of link, that which begins with “http:”.
All links will look like this: href="link’s url" so you have to extract the ‘link’s url” part. You can do this in two ways. First, you can use the string method find() to look for “href="”, and then for the closing “"”, and extract the string between the two. Either returns a list of links, for example
[ ’http://nob.cs.ucdavis.edu/mhi289i/sub1/index.html’, ’http://nob.cs.ucdavis.edu/secure-exer/index.html’, ’http://nob.cs.ucdavis.edu/mhi289i/sub2/index.html’, ’http://nob.cs.ucdavis.edu/mhi289i/next.html’ ]Print the links in the following form (this is a partial listing only):
http://nob.cs.ucdavis.edu/mhi289i/index.html contains: http://nob.cs.ucdavis.edu/mhi289i/sub1/index.html http://nob.cs.ucdavis.edu/secure-exer/index.html http://nob.cs.ucdavis.edu/mhi289i/sub2/index.html http://nob.cs.ucdavis.edu/mhi289i/next.html http://nob.cs.ucdavis.edu/mhi289i/sub1/index.html contains: http://nob.cs.ucdavis.edu/mhi289i/sub2/index.html http://nob.cs.ucdavis.edu/mhi289i/index.html(the indentation is 4 spaces) or, if there are no links, print:
http://nob.cs.ucdavis.edu/secure-exer/index.html contains no linksCall this program “crawler1.py” when you submit it.
A good web site to test your program on is http://nob.cs.ucdavis.edu/mhi289i/index.edu.
Hint: Use a dictionary for this. The key would be the URL of the current web page and the value would be the list of links. Then, when you visit a web page, check that its URL is not in the dictionary. If it is, you already visited it and all its links, so simply return.
Call this program “crawler2.py” when you submit it.
MHI 289I, Programming in Health Informatics
Version of November 15, 2022 at 10:10PM
|You can also obtain a PDF version of this.|