Share your repls and programming experiences

← Back to all posts
webcrawler lol
generationXcode

I make webcrawler I think. This is what I think a webcrawler is like. Is it? someone please tell me why it isnt.

youtube web crawler - https://repl.it/@generationXcode/youtube-logger

please suggest more websites to crawl in and if you want, fork this repl and change the url3 variable to be whichever website's URL you want to crawl and let it go.

see what you have crawled at https://regularwordydisc.generationxcode.repl.co/see

this one is safe, only shows like 100 results i think

see all that has ever been crawled at https://regularwordydisc.generationxcode.repl.co/POWER

beware they may be too many results and big boom

Voters
RonaldMilton
mkhoi
LegendaryWolf
coding1001
angrydoge
Muhammad_SJC
Brendan23
JWZ6
G0RG3
hg0428
Comments
hotnewtop
RohithSai2

it takes a lot of time

generationXcode

@RohithSai2 isn't functional anymore :(

angrydoge

cool +1 :P

ch1ck3n

i get this error

generationXcode

@ch1ck3n hi, I'm building it to be better now. Connecting it to another service so that you all can use it as well and run it on your own repls. Rn there is a small issue with the service so I am playing around. Run it again, it works now. Also I removed the part of the program to write a file full of everything, I have a db where i store it all

generationXcode

@ch1ck3n haha ur using it now. I can see hehehehe. I have another service running rn that I just started up and boom so many results. Umm, I don't want to make you do any work, but if you could help out by just forking this repl and editting the url3 variable to be any other website you want this crawler to crawl thru. At this very momenbt I am working on making the results open to everyone, but I need your help here. Thanks

ch1ck3n
generationXcode

@ch1ck3n I wonder which website you will choose hmmmmmmmm. already have

generationXcode

@generationXcode oh no it is looping over one link

ch1ck3n

@generationXcode

hmmm

i put in https://ch1ck3n.com

and how did that happen

ch1ck3n

@generationXcode gasp how do i see

generationXcode

@ch1ck3n I have no idea

ch1ck3n
generationXcode

@ch1ck3n maybe some server issue with replit servers???

ch1ck3n

@generationXcode how do i see the links

generationXcode

@ch1ck3n Good point, can you wait about 5 mins. I'll quickly whip up a webpage for you to see all the links

generationXcode

@ch1ck3n https://regularwordydisc.generationxcode.repl.co/see
go here. the results arent properly sorted yet so I'll sort them out nicely and all. have fun!

ch1ck3n

@generationXcode my stuff arnt showing up

generationXcode

@ch1ck3n thats cus I have sorted it wrong and I cant have all 1000 results loaded on ur pc ur internet someday will go boom. Currently trying to find a way to sort

generationXcode

i give proof ur stuff there wait a sec

ch1ck3n

@generationXcode i have a good computer and my internet provider is always bragging about their products, time to show the power of developers

ch1ck3n

@generationXcode aha

you dont have to order it

generationXcode

@ch1ck3n ok I make special page 4 u. POWER

generationXcode
ch1ck3n

@generationXcode WOOOOOOO YEEEAAHHHH BOOOIII

generationXcode
generationXcode
ch1ck3n

@generationXcode

delete 5 results please

generationXcode

@ch1ck3n done did

ch1ck3n
generationXcode
generationXcode

@ch1ck3n yep done

generationXcode

@ch1ck3n why did u want 2020 results tho???

ch1ck3n

@generationXcode the perfect number

generationXcode

@ch1ck3n ok... umm I'm starting the webcrawler again now. Is that alright?

ch1ck3n
generationXcode

@ch1ck3n ok im running it thru youtube but results arent going thru. Hmm. ig i'll work on that later now. I had a great time. Thanks!

generationXcode

Oh lmao u already ran it thru youtube

hg0428

Yes, this is a web crawler. It works well. I have a few recommendations for you though. I sent it to you on discord.

LegendaryWolf

Hi, this is unlisted as this contains less code. If you want me to relist this, write 10 to 15 more lines of code

generationXcode

@LegendaryWolf oof. What is the requirement? 60 lines?

LegendaryWolf

lol, you got fooled! I'm not a mod. the minimum is 50 lines BTW

generationXcode

@LegendaryWolf umm ok... odd you would pretend to be a mod but ok lmao

LegendaryWolf

OKIE, i updoot, I think you good at beautiful soup, can ya help me make a search engine? pls

generationXcode

@LegendaryWolf I'll see what I can do. Are you good with requests and P5js? I have a plan. Also, how many years have you programmed. I have a certain search engine I want to make that is different to the rest.

LegendaryWolf

Info -

  • Programing experience = 4 months 1 week
  • good with requests = True / False
  • if good with requests == True / False:
    good with requests = medium
    know JS = True
    know p5.js = False
generationXcode

@LegendaryWolf lmao I was going to come up with an NDA but who rlly cares. Ok when I start making it all, I'll make sure you know about it. Maybe you could help out with the design part and quality of all the code.

LegendaryWolf
generationXcode

@LegendaryWolf idek if that counts but nice to know

CodingCactus

the minimum is 50 lines BTW

@LegendaryWolf no

generationXcode
ch1ck3n

@CodingCactus then what is the minimum

generationXcode
IntellectualGuy

@ch1ck3n @generationXcode I don't think there is any minimum, but It's suggested to not post low effort posts.

generationXcode

@IntellectualGuy ig ur right

CodingCactus

@ch1ck3n no specific minimum

Whippingdot

koi. Also have you seen that every comment of LegendaryWolf's has one updoot? Can you check into this... @CodingCactus

CodingCactus

@Whippingdot erm you know when you view the comments on their profiles, it shows the 'hot' ones, so of course they have a few upvotes

Whippingdot

no, I am talking about in this comment thread @CodingCactus

CodingCactus

@Whippingdot graphegg is probably just being nice and upvoting their replies, same is happening to me

generationXcode

@CodingCactus i don;t like when I see the black coloured upvote signs. The green ones calm me.

Whippingdot

lul but you shouldn't upvote everything for that @generationXcode

Whippingdot

k. i am just cautious cause he reached the amount of cycles I did in less than a month @CodingCactus

generationXcode

@Whippingdot lmao. I like doing it. I cant stop

Whippingdot

u gotta stop. You can be banned for that @generationXcode

generationXcode

@Whippingdot I can upvote whatever I want. How can I be banned for that???

Whippingdot

welp u r spam upvoting. That is giving free cycles to everyone (including me) which is the internet points for replit so... @generationXcode

generationXcode

@Whippingdot oh dang