Submit templates for repl.it/templates here.

← Back to all posts
Web scraping with Nodejs
h
HackermonDev (2013)

What is web scraping and how does it work

Web scraping is when you go on a website then you find links on that website and then you can on those links to find more links and it basically keeps doing that and the links are stored in a database. This is how most search engine works. They use a web scraper to find links and then they show them to you when you need it.

Can you show me an example of a web scraper?

Sure! If you scroll down you will see a repl, the repl is an example of a web scraper. The default site is repl.it but you can change it. It uses cheerio to parse the HTML and then find links on the site.

What is cheerio?

Cheerio is a node library to parse HTML. If you ever see a web scraper chances are they are made with cheerio. Cheerio is really easy to use and you can find the docs on this page.

How do I make a search engine?

Well if you want to see me a template on that either upvote this post or tell me in the comments!

Conclusion

Web scraping is very cool