Chapagain Hands-On Web Scraping with Python

Perform advanced scraping operations using various Python libraries and tools such as Selenium, Regex, and others
1. Auflage 2019
ISBN: 978-1-78953-619-5
Verlag: De Gruyter
Format: EPUB
Kopierschutz: Adobe DRM (»Systemvoraussetzungen)

E-Book, Englisch, 350 Seiten

No detailed description available for "Hands-On Web Scraping with Python".

Autoren/Hrsg.

Chapagain, Anish

Fachgebiete

• Mathematik | Informatik EDV | Informatik Daten / Datenbanken Automatische Datenerfassung, Datenanalyse
• Mathematik | Informatik EDV | Informatik Informatik Künstliche Intelligenz
• Mathematik | Informatik EDV | Informatik Daten / Datenbanken Data Mining

Inhaltsverzeichnis

Table of Contents - Web Scraping Fundamentals
- Python and the Web - Using urllib and Requests
- Using LXML, XPath, and CSS Selectors
- Scraping Using pyquery - a Python Library
- Web Scraping Using Scrapy and Beautiful Soup
- Working with Secure Web
- Data Extraction Using Web-Based APIs
- Using Selenium to Scrape the Web
- Using Regex to Extract Data
- Next Steps

Leseproben

Loading URLs Now that we've confirmed the required libraries and system requirements, we will proceed with loading the URLs. While looking for contents from a URL, it is also necessary to confirm and verify the exact URL that has been chosen for the required content. Contents can be found on single web pages or scattered across multiple pages, and it might not always be the HTML sources we are looking for. We will load some URLs and explore the content using a couple of tasks. Before loading URLs using Python script, it's also advisable to verify the URLs are working properly and contain the detail we are looking for, using web browsers. Developer tools can also be used for similar scenarios, as discussed in Chapter 1, Web Scraping Fundamentals, in the Developer tools section. Task 1: To view data related to the listings of the most popular websites from Wikipedia. We will identify data from the Site, Domain, and Type columns in the page source. We will follow the steps at the following link to achieve our task (a data extraction-related activity will be done in Chapter 3, Using LXML, XPath and CSS Selectors): https://en.wikipedia.org/wiki/List_of_most_popular_websites.  Search Wikipedia for the information we are looking for. The preceding link can be easily viewed in a web browser. The content is in tabular format (as shown in the following screenshot), and so the data can be collected by repeatedly using the select, copy, and paste actions, or by collecting all the text inside the table. However, such actions will not result in the content that we are interested in being in a desirable format, or it will require extra editing and formatting tasks being performed on the text to achieve the desired result. We are also not interested in the page source that's obtained from the browser: Page from Wikipedia, that is, https://en.wikipedia.org/wiki/List_of_most_popular_websites After finalizing the link that contains the content we require, let's load the link using Python. We are making a request to the link and willing to see the response returned by both libraries, that is, urllib and requests: Let's use urllib: >>> import urllib.request as req #import module request from urllib >>> link = "https://en.wikipedia.org/wiki/List_of_most_popular_websites" >>> response = req.urlopen(link) #load the link using method urlopen() >>> print(type(response)) #print type of response object >>> print(response.read()) #read response content b'\n\n\n\n\n