How to automate your webdriver

In this tutorial you gonna learn how to automate your browser with Selenium.

Selenium is a Python package for communicating with a webdriver.

Importing Selenium

First you need to import Selenium:

from selenium import webdriver

Now we need a topic to look up on Google:

subject = input("client: ")

Connect with your webdriver

Then let's connect to the driver with webdriver.Firefox():

driver = webdriver.Firefox()

When we have a connection with the driver, we need to look up an URL with the help of driver.get(URL):

driver.get("https://www.google.com/search?q=" + subject + "&oq=python&aqs=chrome..69i57j69i59l3j0i7i30l2j69i60l2.1672j0j4&sourceid=chrome&ie=UTF-8")

Now that we've looked up the URL, let's get the elements that Google returned.

How to find elements

There are a few ways to find elements:

• find_element_by_id()

• find_element_by_name()

• find_element_by_xpath()

• find_element_by_link_text()

• find_element_by_partial_link_text()

• find_element_by_tag_name()

• find_element_by_class_name()

• find_element_by_css_selector()

These ways only fetch one element. But we want all the elements, you can use these commands for that:

• find_elements_by_name()

• find_elements_by_xpath()

• find_elements_by_link_text()

• find_elements_by_partial_link_text()

• find_elements_by_tag_name()

• find_elements_by_class_name()

• find_elements_by_css_selector()

To retrieve the elements, use this piece of code:

elems = driver.find_elements_by_xpath("//div[@class = 'yuRUbf']/a")

The code is going to find a-elements (links) in divs with the class: yuRUbf. Google uses the classname yuRUbf for the div where the link in is.

Printing the URLs

Now that we have the links, we're just going to print them out:

print("\n\nserver:")

for x in enumerate(elems):
  print("  " + str(x[0] + 1) + ". " + x[1].get_attribute("href"))

To print the link we use .get_attribute("href")

Then all we have to do is close the driver with driver.close()

Now the program is ready, so test it.

Final Code

from selenium import webdriver

subject = input("client: ")

driver = webdriver.Firefox()
driver.get("https://www.google.com/search?q=" + subject + "&oq=python&aqs=chrome..69i57j69i59l3j0i7i30l2j69i60l2.1672j0j4&sourceid=chrome&ie=UTF-8")

elems = driver.find_elements_by_xpath("//div[@class = 'yuRUbf']/a")

print("\n\nserver:")

for x in enumerate(elems):
  print("  " + str(x[0] + 1) + ". " + x[1].get_attribute("href"))

driver.close()

Bonus: YouTube bot

You can also make this program work on YouTube. You just need to change the URL and the classes.

Here is the YouTube code:

from selenium import webdriver

subject = input("client: ")

driver = webdriver.Firefox()
driver.get("https://www.youtube.com/results?search_query=" + subject)

elems = driver.find_elements_by_xpath("//h3/a[@id = 'video-title']")

print("\n\nserver:")

for x in enumerate(elems):
  print("  " + str(x[0] + 1) + ". " + x[1].get_attribute("href"))

driver.close()

I hope you enjoyed the tutorial with making your own Google and YouTube.