Chapagain Hands-On Web Scraping with Python

Perform advanced scraping operations using various Python libraries and tools such as Selenium, Regex, and others
1. Auflage 2024
ISBN: 978-1-78953-619-5
Verlag: De Gruyter
Format: EPUB
Kopierschutz: Adobe DRM (»Systemvoraussetzungen)

E-Book, Englisch, 350 Seiten

Collect and scrape different complexities of data from the modern Web using the latest tools, best practices, and techniquesKey Features - Learn different scraping techniques using a range of Python libraries such as Scrapy and Beautiful Soup
- Build scrapers and crawlers to extract relevant information from the web
- Automate web scraping operations to bridge the accuracy gap and manage complex business needs
Book DescriptionWeb scraping is an essential technique used in many organizations to gather valuable data from web pages. This book will enable you to delve into web scraping techniques and methodologies. The book will introduce you to the fundamental concepts of web scraping techniques and how they can be applied to multiple sets of web pages. You'll use powerful libraries from the Python ecosystem such as Scrapy, lxml, pyquery, and bs4 to carry out web scraping operations. You will then get up to speed with simple to intermediate scraping operations such as identifying information from web pages and using patterns or attributes to retrieve information. This book adopts a practical approach to web scraping concepts and tools, guiding you through a series of use cases and showing you how to use the best tools and techniques to efficiently scrape web pages. You'll even cover the use of other popular web scraping tools, such as Selenium, Regex, and web-based APIs. By the end of this book, you will have learned how to efficiently scrape the web using different techniques with Python and other popular tools.What you will learn - Analyze data and information from web pages
- Learn how to use browser-based developer tools from the scraping perspective
- Use XPath and CSS selectors to identify and explore markup elements
- Learn to handle and manage cookies
- Explore advanced concepts in handling HTML forms and processing logins
- Optimize web securities, data storage, and API use to scrape data
- Use Regex with Python to extract data
- Deal with complex web entities by using Selenium to find and extract data
Who this book is forThis book is for Python programmers, data analysts, web scraping newbies, and anyone who wants to learn how to perform web scraping from scratch. If you want to begin your journey in applying web scraping techniques to a range of web pages, then this book is what you need! A working knowledge of the Python programming language is expected.

Autoren/Hrsg.

Chapagain, Anish

Fachgebiete

• Mathematik | Informatik EDV | Informatik Daten / Datenbanken Data Mining
• Mathematik | Informatik EDV | Informatik Daten / Datenbanken Automatische Datenerfassung, Datenanalyse
• Mathematik | Informatik EDV | Informatik Informatik Künstliche Intelligenz

Inhaltsverzeichnis

Table of Contents - Web Scraping Fundamentals
- Python and the Web - Using urllib and Requests
- Using LXML, XPath, and CSS Selectors
- Scraping Using pyquery - a Python Library
- Web Scraping Using Scrapy and Beautiful Soup
- Working with Secure Web
- Data Extraction Using Web-Based APIs
- Using Selenium to Scrape the Web
- Using Regex to Extract Data
- Next Steps

Leseproben

Loading URLs Now that we've confirmed the required libraries and system requirements, we will proceed with loading the URLs. While looking for contents from a URL, it is also necessary to confirm and verify the exact URL that has been chosen for the required content. Contents can be found on single web pages or scattered across multiple pages, and it might not always be the HTML sources we are looking for. We will load some URLs and explore the content using a couple of tasks. Before loading URLs using Python script, it's also advisable to verify the URLs are working properly and contain the detail we are looking for, using web browsers. Developer tools can also be used for similar scenarios, as discussed in Chapter 1, Web Scraping Fundamentals, in the Developer tools section. Task 1: To view data related to the listings of the most popular websites from Wikipedia. We will identify data from the Site, Domain, and Type columns in the page source. We will follow the steps at the following link to achieve our task (a data extraction-related activity will be done in Chapter 3, Using LXML, XPath and CSS Selectors): https://en.wikipedia.org/wiki/List_of_most_popular_websites.  Search Wikipedia for the information we are looking for. The preceding link can be easily viewed in a web browser. The content is in tabular format (as shown in the following screenshot), and so the data can be collected by repeatedly using the select, copy, and paste actions, or by collecting all the text inside the table. However, such actions will not result in the content that we are interested in being in a desirable format, or it will require extra editing and formatting tasks being performed on the text to achieve the desired result. We are also not interested in the page source that's obtained from the browser: Page from Wikipedia, that is, https://en.wikipedia.org/wiki/List_of_most_popular_websites After finalizing the link that contains the content we require, let's load the link using Python. We are making a request to the link and willing to see the response returned by both libraries, that is, urllib and requests: Let's use urllib: >>> import urllib.request as req #import module request from urllib >>> link = "https://en.wikipedia.org/wiki/List_of_most_popular_websites" >>> response = req.urlopen(link) #load the link using method urlopen() >>> print(type(response)) #print type of response object >>> print(response.read()) #read response content b'\n\n\n\n\n