Skip to content
← Back to Community
COVID 19 live tracker web scraper
Profile icon
[deleted]

Ok so Im working on a web scraper that will scrape off data from a website, (This one!) and I managed to scrape off some data, but the data I want wont scrape. Heres what I mean: If you look at the website, you will see around 37M cases as of when this post was created. And when I scrape that data off with this code:

const postJobButton = data('a#valueTot.numbers').text();

it only returns 0, why is that? And ignore the goodCode.js, it does nothing. I just save my good code there that file.

Answered by Coder100 [earned 5 cycles]
View Answer
Voters
Profile icon
DynamicSquid
Comments
hotnewtop
Profile icon
Coder100

Here's what happened:

image

the source code says 0! So where does the actual number come from then? JS, how pesky >:(

image
so basically what this means is that you can't scrape the data using your current method :( You are going to have to execute js code too (which isn't possible with your current setup)

hey hey whats this

image

if you could find a way to eval() this piece of code you will be able to get covid data!

psst...

use an API

Profile icon
ALpixel

FUCK

Profile icon
DynamicSquid

I don't do JS, but I would recommend Python for web scraping in the future. Literally takes 2 lines of code.

Profile icon
[deleted]

@DynamicSquid I have not coded in Python in so long I forgot how to do some stuff now lol. And I am also going to make a discord bot that keeps track of the live covid count. Im used to node js for networking and discord bots. But thanks! I will try it some time

Profile icon
Coder100

python and node.js will have no effect on source code smh @DynamicSquid

Profile icon
DynamicSquid

@Coder100 No, Python's way simpler

Profile icon
Coder100

simple != work @DynamicSquid

Profile icon
DynamicSquid

@Coder100 no python's simplicity works

Profile icon
DynamicSquid

@Coder100 my web scraper used python and it worked great!

Profile icon
Coder100

but that's repl.it, they have the data in the source @DynamicSquid

Profile icon
DynamicSquid

@Coder100 wdym?

Profile icon
Coder100

it's hard-coded inside in fact to prove it to you, try scraping the covid19 tracker in question, you'll see there is no difference @DynamicSquid

Profile icon
Coder100

@DynamicSquid
image

seee the comments are hard-coded

Profile icon
DynamicSquid

@Coder100 wait what's hardcoded?

Profile icon
Coder100

the comments smh @DynamicSquid

Profile icon
DynamicSquid

@Coder100 but what do the comments have to do with anything?

Profile icon
Coder100

thats what you are scraping is it not @DynamicSquid

Profile icon
Coder100

what I am trying to say is that all the data is already hard-coded in the HTML:

<div class="comment"> oh wow python can definitely do this better than node.js </div>

@DynamicSquid

Profile icon
DynamicSquid

@Coder100 why would you scrap comments? And yes python is better than js since it's much cleaner

Profile icon
Coder100

@DynamicSquid yuck ruby better

Profile icon
Coder100

oh well I will not argue with idiots
oh well I will not argue when I am losing @DynamicSquid

Profile icon
DynamicSquid
Profile icon
Coder100