Skip to content
Sign upLog in
← Back to Community

How to automate your webdriver

Profile icon
antoon

How to automate your webdriver

In this tutorial you gonna learn how to automate your browser with Selenium.

Selenium is a Python package for communicating with a webdriver.

Importing Selenium

First you need to import Selenium:

from selenium import webdriver

Now we need a topic to look up on Google:

subject = input("client: ")

Connect with your webdriver

Then let's connect to the driver with webdriver.Firefox():

driver = webdriver.Firefox()

When we have a connection with the driver, we need to look up an URL with the help of driver.get(URL):

driver.get("https://www.google.com/search?q=" + subject + "&oq=python&aqs=chrome..69i57j69i59l3j0i7i30l2j69i60l2.1672j0j4&sourceid=chrome&ie=UTF-8")

Now that we've looked up the URL, let's get the elements that Google returned.

How to find elements

There are a few ways to find elements:

  • find_element_by_id()
  • find_element_by_name()
  • find_element_by_xpath()
  • find_element_by_link_text()
  • find_element_by_partial_link_text()
  • find_element_by_tag_name()
  • find_element_by_class_name()
  • find_element_by_css_selector()

These ways only fetch one element. But we want all the elements, you can use these commands for that:

  • find_elements_by_name()
  • find_elements_by_xpath()
  • find_elements_by_link_text()
  • find_elements_by_partial_link_text()
  • find_elements_by_tag_name()
  • find_elements_by_class_name()
  • find_elements_by_css_selector()

To retrieve the elements, use this piece of code:

elems = driver.find_elements_by_xpath("//div[@class = 'yuRUbf']/a")

The code is going to find a-elements (links) in divs with the class: yuRUbf. Google uses the classname yuRUbf for the div where the link in is.

Printing the URLs

Now that we have the links, we're just going to print them out:

print("\n\nserver:") for x in enumerate(elems): print(" " + str(x[0] + 1) + ". " + x[1].get_attribute("href"))

To print the link we use .get_attribute("href")

Then all we have to do is close the driver with driver.close()

Now the program is ready, so test it.

Final Code

from selenium import webdriver subject = input("client: ") driver = webdriver.Firefox() driver.get("https://www.google.com/search?q=" + subject + "&oq=python&aqs=chrome..69i57j69i59l3j0i7i30l2j69i60l2.1672j0j4&sourceid=chrome&ie=UTF-8") elems = driver.find_elements_by_xpath("//div[@class = 'yuRUbf']/a") print("\n\nserver:") for x in enumerate(elems): print(" " + str(x[0] + 1) + ". " + x[1].get_attribute("href")) driver.close()

Bonus: YouTube bot

You can also make this program work on YouTube. You just need to change the URL and the classes.

Here is the YouTube code:

from selenium import webdriver subject = input("client: ") driver = webdriver.Firefox() driver.get("https://www.youtube.com/results?search_query=" + subject) elems = driver.find_elements_by_xpath("//h3/a[@id = 'video-title']") print("\n\nserver:") for x in enumerate(elems): print(" " + str(x[0] + 1) + ". " + x[1].get_attribute("href")) driver.close()

I hope you enjoyed the tutorial with making your own Google and YouTube.

Voters
Profile icon
M33rschaum
Profile icon
AdamTrabant
Profile icon
rahaouismail
Profile icon
DucDang4
Profile icon
DynamicSquid
Profile icon
antoon
Comments
hotnewtop
Profile icon
COROVICD

Could this run on Replit or is this only running from a PC?

Profile icon
antoon

@COROVICD
You can use the code with both.