GitHub - praveen97uma/Web-Crawler: Code Arx Project

Branches Tags

Name		Name	Last commit message	Last commit date
Latest commit History 39 Commits
Models		Models
README.txt		README.txt
counter.py		counter.py
crawler.py		crawler.py
downloader.py		downloader.py
indexer.py		indexer.py
linkanalyser.py		linkanalyser.py
mathutils.py		mathutils.py
parser.py		parser.py
porter.py		porter.py
retriever.py		retriever.py
search.py		search.py
stopwords.txt		stopwords.txt
tokenizedocuments.py		tokenizedocuments.py

Repository files navigation

Required Libraries: BeautifulSoup (for parsing)
                      can be installed in Ubuntu by sudo easy_install BeautifulSoup
                      
Invoke 'python crawler.py' in the terminal to initiate the crawler.

A prompt asks for the starting url.

Enter 'www.iitr.ac.in' and the crawler starts crawling url which are in the same domain only.

The crawler also maintains log of all the discarded, crawled and already visited links in log.txt.

*The crawler still needs lot of improvement to be done.*