I make webcrawler I think. This is what I think a webcrawler is like. Is it? someone please tell me why it isnt.
youtube web crawler - https://repl.it/@generationXcode/youtube-logger
please suggest more websites to crawl in and if you want, fork this repl and change the url3 variable to be whichever website's URL you want to crawl and let it go.
see what you have crawled at https://regularwordydisc.generationxcode.repl.co/see
this one is safe, only shows like 100 results i think
see all that has ever been crawled at https://regularwordydisc.generationxcode.repl.co/POWER
beware they may be too many results and big boom
@ch1ck3n hi, I'm building it to be better now. Connecting it to another service so that you all can use it as well and run it on your own repls. Rn there is a small issue with the service so I am playing around. Run it again, it works now. Also I removed the part of the program to write a file full of everything, I have a db where i store it all
@ch1ck3n haha ur using it now. I can see hehehehe. I have another service running rn that I just started up and boom so many results. Umm, I don't want to make you do any work, but if you could help out by just forking this repl and editting the url3 variable to be any other website you want this crawler to crawl thru. At this very momenbt I am working on making the results open to everyone, but I need your help here. Thanks
Hi, this is unlisted as this contains less code. If you want me to relist this, write 10 to 15 more lines of code