Skip to content
Sign upLog in
← Back to Community

How to send a request and capture data in source using Python requests and re

Profile icon
jhash

If you ever wanted to do simple scraping in python, then this is the tutorial for you.

This example will send a request to Google and capture the word "Google" from the title tags.

r = requests.get("https://google.com")

Sends a normal request to google. Typically, your browser would do this to almost any site. We must set this request to a variable to have any use for it.

print(r.text)

Prints response from website.

Now this is the tricky part. This will parse or capture specific data from the website.

print_google = re.search("<title>(.*)</title>", r.text).group(1)

You must be specific when telling Python what to capture. I told Python to capture or parse everything in-between the title tags in the response source.
The syntax for this is

variablename = re.search("leftstring(.*)rightstring", VariableNameThatIwantToParseStuffFrom).group(1)

You can see an example of this in the provided code below.

Voters
Profile icon
wolfieboy09
Profile icon
hazelpy
Profile icon
mysticwill
Profile icon
mohabelarbi1
Profile icon
ina23HAli
Profile icon
DynamicSquid
Profile icon
jhash
Comments
hotnewtop
Profile icon
wolfieboy09

First Comment