Share your repls and programming experiences

← Back to all posts
webcrawler lol
h
generationXcode (323)

I make webcrawler I think. This is what I think a webcrawler is like. Is it? someone please tell me why it isnt.

youtube web crawler - https://repl.it/@generationXcode/youtube-logger

please suggest more websites to crawl in and if you want, fork this repl and change the url3 variable to be whichever website's URL you want to crawl and let it go.

see what you have crawled at https://regularwordydisc.generationxcode.repl.co/see

this one is safe, only shows like 100 results i think

see all that has ever been crawled at https://regularwordydisc.generationxcode.repl.co/POWER

beware they may be too many results and big boom

Comments
hotnewtop
generationXcode (323)

@ch1ck3n hi, I'm building it to be better now. Connecting it to another service so that you all can use it as well and run it on your own repls. Rn there is a small issue with the service so I am playing around. Run it again, it works now. Also I removed the part of the program to write a file full of everything, I have a db where i store it all

generationXcode (323)

@ch1ck3n haha ur using it now. I can see hehehehe. I have another service running rn that I just started up and boom so many results. Umm, I don't want to make you do any work, but if you could help out by just forking this repl and editting the url3 variable to be any other website you want this crawler to crawl thru. At this very momenbt I am working on making the results open to everyone, but I need your help here. Thanks

generationXcode (323)

@ch1ck3n I wonder which website you will choose hmmmmmmmm. already have

generationXcode (323)

@generationXcode oh no it is looping over one link

generationXcode (323)

@ch1ck3n maybe some server issue with replit servers???

generationXcode (323)

@ch1ck3n Good point, can you wait about 5 mins. I'll quickly whip up a webpage for you to see all the links

generationXcode (323)

@ch1ck3n https://regularwordydisc.generationxcode.repl.co/see
go here. the results arent properly sorted yet so I'll sort them out nicely and all. have fun!

generationXcode (323)

@ch1ck3n thats cus I have sorted it wrong and I cant have all 1000 results loaded on ur pc ur internet someday will go boom. Currently trying to find a way to sort

generationXcode (323)

i give proof ur stuff there wait a sec

ch1ck3n (1626)

@generationXcode i have a good computer and my internet provider is always bragging about their products, time to show the power of developers

ch1ck3n (1626)

@generationXcode aha

you dont have to order it

generationXcode (323)

@ch1ck3n ok I make special page 4 u. POWER

ch1ck3n (1626)

@generationXcode WOOOOOOO YEEEAAHHHH BOOOIII

generationXcode (323)

@ch1ck3n why did u want 2020 results tho???

generationXcode (323)

@ch1ck3n ok... umm I'm starting the webcrawler again now. Is that alright?

generationXcode (323)

@ch1ck3n ok im running it thru youtube but results arent going thru. Hmm. ig i'll work on that later now. I had a great time. Thanks!

generationXcode (323)

Oh lmao u already ran it thru youtube

hg0428 (192)

Yes, this is a web crawler. It works well. I have a few recommendations for you though. I sent it to you on discord.

LegendaryWolf (701)

Hi, this is unlisted as this contains less code. If you want me to relist this, write 10 to 15 more lines of code

generationXcode (323)

@LegendaryWolf oof. What is the requirement? 60 lines?

LegendaryWolf (701)

lol, you got fooled! I'm not a mod. the minimum is 50 lines BTW

generationXcode (323)

@LegendaryWolf umm ok... odd you would pretend to be a mod but ok lmao

LegendaryWolf (701)

OKIE, i updoot, I think you good at beautiful soup, can ya help me make a search engine? pls

generationXcode (323)

@LegendaryWolf I'll see what I can do. Are you good with requests and P5js? I have a plan. Also, how many years have you programmed. I have a certain search engine I want to make that is different to the rest.

LegendaryWolf (701)

Info -

  • Programing experience = 4 months 1 week
  • good with requests = True / False
  • if good with requests == True / False:
    good with requests = medium
    know JS = True
    know p5.js = False
generationXcode (323)

@LegendaryWolf lmao I was going to come up with an NDA but who rlly cares. Ok when I start making it all, I'll make sure you know about it. Maybe you could help out with the design part and quality of all the code.

generationXcode (323)

@LegendaryWolf idek if that counts but nice to know

CodingCactus (4191)

the minimum is 50 lines BTW

@LegendaryWolf no

IntellectualGuy (710)

@ch1ck3n @generationXcode I don't think there is any minimum, but It's suggested to not post low effort posts.

Whippingdot (573)

koi. Also have you seen that every comment of LegendaryWolf's has one updoot? Can you check into this... @CodingCactus

CodingCactus (4191)

@Whippingdot erm you know when you view the comments on their profiles, it shows the 'hot' ones, so of course they have a few upvotes

Whippingdot (573)

no, I am talking about in this comment thread @CodingCactus

CodingCactus (4191)

@Whippingdot graphegg is probably just being nice and upvoting their replies, same is happening to me

generationXcode (323)

@CodingCactus i don;t like when I see the black coloured upvote signs. The green ones calm me.

Whippingdot (573)

lul but you shouldn't upvote everything for that @generationXcode

Whippingdot (573)

k. i am just cautious cause he reached the amount of cycles I did in less than a month @CodingCactus

generationXcode (323)

@Whippingdot lmao. I like doing it. I cant stop

Whippingdot (573)

u gotta stop. You can be banned for that @generationXcode

generationXcode (323)

@Whippingdot I can upvote whatever I want. How can I be banned for that???

Whippingdot (573)

welp u r spam upvoting. That is giving free cycles to everyone (including me) which is the internet points for replit so... @generationXcode