How to Download a Web Page in Python
In this article, we show how to download a web page in Python.
We can download a web page in Python using the requests module.
In order to use the requests module, the first thing you have to do is install the requests module.
This can be done by opening up your computer's command prompt and specifying the full path to the Scripts folder in the Python package that you downloaded when you installed Python. Once you have this path, you simply have to enter, pip install requests. This downloads the requests module.
Once the requests module has been successfully installed, then you have everything you need to download a web page and the code below should work.
This downloading of a web page works with HTML web pages. It will not work with web pages such as PHP, which are server-side pages, because those are encrypted web pages. It will only work with non-encrypted web pages such as HTML.
So, if you are trying to get this code to work with a PHP page or an ASP page, it will not work, as intended. Python cannot gain access to these pages, or else a web site could have a major security vulnerability.
Thus, the code below works on HTML pages.
The code to download the web page, http://www.learningaboutelectronics.com/Articles/How-to-test-a-capacitor.html, is shown below.
So, first, we must import the requests module.
Then after this, we create a variable called webpage. We set this variable equal to the requests.get() function. Inside of this parameter, we specify the web page that we want to read.
We then print out the contents of this web page through the line, print(webpage.text).
You will then see the entire contents of the web page displayed in the Python shell.
Again, this works very well for plain HTML pages, but does not work with server-side pages like PHP and ASP (due to the encryption that server-side
langugages have).
Related Resources
How to Randomly Select From or Shuffle a List in Python