Share your repls and programming experiences

← Back to all posts
Python Web Crawler
MrSprinkle

I just made a simple little Python web crawler. It starts at the URL provided (START_URL at the top of the program) and it visits all the links on that page, all the links on those pages, etc. etc. Eventually it will build a collection of URLs and their page titles in the results.txt file (which should have been automatically generated when the program was ran)

It will stop searching for URLs either when you press enter or the repl runs out of memory (usually after ~200-300 URLs have been found)
You may see a lot of errors upon stopping the program, this is normal

Voters
sobakarooted
MrSprinkle
Comments
hotnewtop
sobakarooted

👍