By hmimcu 
A web crawler, also known as a web spider or a web robot, is a program or script that automatically browses and retrieves web page information according to certain rules. Web crawlers can automatically request web pages and grab the required data. By processing the captured data, valuable information can be extracted.
Meet reptiles
The series of search engines we are familiar with are large web crawlers, such as Baidu, Sogou, 360 Browser, Google Search and so on. Each search engine has its own crawler program. For example, the crawler of 360 Browser is called 360Spider, and the crawler of Sogou is called Sogouspider.
Baidu search engine, in fact, can be more vividly called Baidu spider (Baiduspider), which crawls and collects high-quality information from massive Internet information every day. When a user retrieves a keyword through Baidu, Baidu will first analyze the keyword entered by the user, then find out the relevant webpages from the included webpages, sort the webpages according to the ranking rules, and finally present the sorted results to user. In this process, Baidu Spider played a very important role.
Baidu engineers have written corresponding crawling algorithms for “Baidu Spider”. By applying these algorithms, “Baidu Spider” can implement corresponding search strategies, such as filtering out duplicate web pages, screening high-quality web pages, and so on. Applying different algorithms, the crawler’s operating efficiency and crawling results will be different.
Reptile classification
Crawlers can be divided into three categories: general-purpose web crawlers, focused web crawlers, and incremental web crawlers.
General web crawler: It is an important part of the search engine, which has been introduced above, and will not be repeated here. General web crawlers need to abide by the robots protocol, through which websites tell search engines which pages can be crawled and which pages are not allowed to be crawled.
Robots agreement: It is a kind of “agreement and common name” agreement, which does not have legal effect. It embodies the “contract spirit” of Internet people. Practitioners in the industry will consciously abide by the agreement, so it is also known as the “gentlemen’s agreement”.
Focused web crawler: It is a web crawler program for specific needs. The difference between it and the general crawler is that the focused crawler will screen and process the webpage content when implementing webpage crawling, and try to ensure that only the webpage information related to the demand is crawled. Focusing on web crawlers greatly saves hardware and network resources. Due to the small number of saved pages, the update speed is very fast, which also satisfies the needs of some specific groups of people for information in specific fields.
Incremental web crawler: refers to the incremental update of downloaded web pages. It is a crawler program that only crawls newly generated or changed web pages. It can guarantee that the crawled pages are up to date to a certain extent. page.
crawler application
With the rapid development of the network, the World Wide Web has become the carrier of a large amount of information. How to effectively extract and use this information has become a huge challenge. Therefore, crawlers have emerged. It can not only be used in the field of search engines, but also in big data analysis, And the commercial field has been applied on a large scale.
1) Data analysis
In the field of data analysis, web crawlers are usually an essential tool for collecting massive amounts of data. For data analysts, to conduct data analysis, they must first have data sources, and learning crawlers can obtain more data sources. During the collection process, data analysts can collect more valuable data according to their own purposes, and filter out invalid data.
2) Business area
For enterprises, it is very important to obtain market dynamics and product information in a timely manner. Enterprises can purchase data through third-party platforms, such as Guiyang Big Data Exchange, Datatang, etc. Of course, if your company has a crawler engineer, you can obtain the desired information through crawlers.
Reptiles are a double-edged sword
Crawlers are a double-edged sword. While it brings us convenience, it also brings hidden dangers to network security. Some lawbreakers use crawlers to illegally collect information of netizens on the Internet, or use crawlers to maliciously attack other people’s websites, resulting in serious consequences of website paralysis.
In order to limit the danger brought by crawlers, most websites have good anti-crawling measures, and further explanations are made through the robots.txt protocol. The following is the content of Taobao robots.txt:
User-agent: Baiduspider
Disallow: /baidu Disallow: /s?
Disallow: /ulink?
Disallow: /link?
Disallow: /home/news/data/
Disallow: /bh
…..
User-agent: *
Disallow: /
It can be seen from the content of the agreement that Taobao has made regulations on pages that cannot be crawled. Therefore, when using crawlers, you must consciously abide by the robots agreement, and do not illegally obtain other people’s information, or do things that endanger other people’s websites.
Why use Python as a crawler
First of all, you should make it clear that not only Python can be used as a crawler, but PHP, Java, and C/C++ can all be used to write crawlers, but in comparison, Python is the easiest crawler. Here is a brief comparison of their advantages and disadvantages:
PHP: It does not support multi-threading and asynchronously very well, and its concurrent processing ability is weak; Java is also often used to write crawler programs, but the Java language itself is very cumbersome and has a large amount of code, so it is a barrier to entry for beginners High; C/C++ has high operating efficiency, but the cost of learning and development is high. Writing a small crawler can take a long time.
The Python language has beautiful syntax, concise code, high development efficiency, and supports multiple crawler modules, such as urllib, requests, Bs4, etc. Python’s request module and parsing module are rich and mature, and it also provides a powerful Scrapy framework, making it easier to write crawlers. So writing a crawler program in Python is a very good choice.
The process of writing a crawler
The crawler program is different from other programs, and its thinking logic is generally similar, so we don’t need to spend a lot of time on logic. The following is a brief description of the process of writing a crawler program in Python:
First, open the URL with the request method of the urllib module to get the HTML object of the web page.
Use a browser to open the source code of the web page to analyze the web page structure and element nodes.
Extract data via Beautiful Soup or regular expressions.
Store data to local disk or database.
Of course, it is not limited to the above-mentioned one process. To write a crawler program, you need to have good Python programming skills, so that you will be handy during the writing process. The crawler program needs to try to pretend to be an adult visiting the website instead of a machine, otherwise it will be restricted by the website’s anti-crawling strategy, or even directly block the IP.
How to learn reptiles?
The latest Python tutorial for dark horse programmers, from entry to proficiency in python in 8 days, learning python and watching this set is enough
Video Tutorial –> Dark Horse Programmer easily obtains epidemic data in 180 minutes, an introductory course on Python reptiles
Know what a reptile is
Master the use of the requests library
Master the BeautifulSoup parsing library to extract data
Master regular expressions to extract data
Mastering is to use the json module to process json format data
Master the realization of the epidemic crawler project
Learn about epidemic data visualization