How to Retrieve a Web Page using the Selenium Module in Python



Python


In this article, we show how to retrieve a web page using the Selenium Module in Python.

We will show how to retrieve web pages from multiple browsers.

We will first show how to retrieve a web page using the Firefox web browser. We then show how to retrieve a web page using the Google Chrome web browser.



How to Retrieve a Web Page from the Firefox Web Browser using the Selenium Module in Python

We will now show how to retrieve a web page from the Firefox web browser.

Before you start any of the Python code, you must install the Selenium module and the geckodriver software, which gives us the webdriver function to work with web browsers. The Selenium module can be obtained simply through the line, pip install selenium, in the Command Prompt in Windows. We show in full how to install the geckodriver software for the windows operating system in the article, How to Install Geckodriver for Python in Windows.

So once you have the selenium module and the geckodriver installed, we can now go to the Python code to retrieve web pages.

So this is done in the selenium module through the get() function.

The get() function communicates with the server to return the web page that we request.

In the following code below, we retrieve the home page for wikipedia.com. We then retrieve a page on Python's official programming language website.



The first thing we have to do is import webdriver from the selenium module. This allows us to work with web browers; in this case, Firefox.

We then create a variable, browser, which we set equal to, webdriver.Firefox(). This allows us to make a connection to the Firefox browser.

Next we use the get() function to retrieve the home page of the wikipedia website.

The get() function is how we can retrieve any page using the Selenium module in Python.

This can be a home page or a specific page within a website.

After retrieving the web page, you can then do operations on the page, such as looking for links or searching for specific text, etc.



How to Retrieve a Web Page from the Google Chrome Web Browser using the Selenium Module in Python

We will now show how to retrieve a web page from the Google Chrome web browser.

Before you start any of the Python code, you must install the Selenium module if you haven't already and the geckodriver software, which gives us the webdriver function to work with web browsers. The Selenium module can be obtained simply through the line, pip install selenium, in the Command Prompt in Windows. We show in full how to install the ChromeDriver software for the windows operating system in the article, How to Install ChromeDriver for Python in Windows.

So once you have the selenium module and the ChromeDriver installed, we can now go to the Python code to retrieve web pages.

So this is done in the selenium module through the get() function, which allows us to retrieve web pages from the browser.

In the following code below, we retrieve the home page for Microsoft and Apple.



The first thing we have to do is import webdriver from the selenium module. This allows us to work with web browers; in this case, Chrome.

We then create a variable, browser, which we set equal to, webdriver.Chrome(). This allows us to make a connection to the Chrome browser.

Next we use the get() function to retrieve the home pages for Microsoft and Apple.

And this is how to retrieve a web page with either the Firefox or Chrome web browsers.


Related Resources

How to Create a Zip File in Python

How to Extract All Files and Folders from a Zip File in Python

How to Read the Contents of a Zip File in Python



HTML Comment Box is loading comments...