AdCharity (1322)

Preface: the only links in there are google, some bing stuff, repl, and stack overflow


Need I say more? The fake search engine is here. It's pretty much missing the most critical parts of an engine (like pagination, mobile support, and indexing) but I think it captures the essence of finding link matches.


You can fork the project and take a look at config.js so you can input your own types of links (the crawler will crawl all links and then crawl those links etc.) It's fairly unstable since I used request and cheerio for images and the crawler crashes but oh well.


  • Featured right now: image and link search
  • Coming: pagination, indexing, mobile support (if I don't abandon the project)
  • Update 1: images that don't work are removed
  • Update 2: Scraping slightly better (still bad)
  • Update 3: Matches partial matches (like spaces in between)
You are viewing a single comment. View All
StringentDev (229)

seems the crawler keeps crashing

AdCharity (1322)

@Lethdev2019 yeah it either means the links you're using have firewalls of some sort or the link isn't valid. Try removing the image crawl part; that's the part that crashes the most

StringentDev (229)

maby another error.catch would work? @AdCharity

AdCharity (1322)

@Lethdev2019 the request module is both unstable and depreciated so I'll probably change it soon (catch does not work on failed requests)