欧美成人午夜免费全部完,亚洲午夜福利精品久久,а√最新版在线天堂,另类亚洲综合区图片小说区,亚洲欧美日韩精品色xxx

千鋒扣丁學堂Python培訓之代理IP爬蟲的新手使用教程

2019-09-20 13:42:50 5369瀏覽

今天千鋒扣丁學堂Python培訓老師給大家分享一篇Python代理IP爬蟲新手使用的詳細介紹，首先Python爬蟲要經(jīng)歷爬蟲、爬蟲被限制、爬蟲反限制的過程。當然后續(xù)還要網(wǎng)頁爬蟲限制優(yōu)化，爬蟲再反限制的一系列道高一尺魔高一丈的過程。爬蟲的初級階段，添加headers和ip代理可以解決很多問題。

運行環(huán)境

Python3.7,Pycharm

這些需要大家直接去搭建好環(huán)境...

準備工作

爬取IP地址的網(wǎng)站(國內高匿代理)

校驗IP地址的網(wǎng)站

爬取IP的完整代碼

PS:簡單的使用bs4獲取IP和端口號,沒有啥難度,里面增加了一個過濾不可用IP的邏輯

關鍵地方都有注釋了

# -*- coding: utf-8 -*-
# @Time : 2018/11/22 
# @Author : liangk
# @Site :
# @File : auto_archive_ios.py
# @Software: PyCharm
 
 
import requests
from bs4 import BeautifulSoup
import json
 
 
class GetIp(object):
 """抓取代理IP"""
 
 def __init__(self):
 """初始化變量"""
 self.url = 'http://www.xicidaili.com/nn/'
 self.check_url = 'https://www.ip.cn/'
 self.ip_list = []
 
 @staticmethod
 def get_html(url):
 """請求html頁面信息"""
 header = {
  'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36'
 }
 try:
  request = requests.get(url=url, headers=header)
  request.encoding = 'utf-8'
  html = request.text
  return html
 except Exception as e:
  return ''
 
 def get_available_ip(self, ip_address, ip_port):
 """檢測IP地址是否可用"""
 header = {
  'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36'
 }
 ip_url_next = '://' + ip_address + ':' + ip_port
 proxies = {'http': 'http' + ip_url_next, 'https': 'https' + ip_url_next}
 try:
  r = requests.get(self.check_url, headers=header, proxies=proxies, timeout=3)
  html = r.text
 except:
  print('fail-%s' % ip_address)
 else:
  print('success-%s' % ip_address)
  soup = BeautifulSoup(html, 'lxml')
  div = soup.find(class_='well')
  if div:
  print(div.text)
  ip_info = {'address': ip_address, 'port': ip_port}
  self.ip_list.append(ip_info)
 
 def main(self):
 """主方法"""
 web_html = self.get_html(self.url)
 soup = BeautifulSoup(web_html, 'lxml')
 ip_list = soup.find(id='ip_list').find_all('tr')
 for ip_info in ip_list:
  td_list = ip_info.find_all('td')
  if len(td_list) > 0:
  ip_address = td_list[1].text
  ip_port = td_list[2].text
  # 檢測IP地址是否有效
  self.get_available_ip(ip_address, ip_port)
 # 寫入有效文件
 with open('ip.txt', 'w') as file:
  json.dump(self.ip_list, file)
 print(self.ip_list)
 
 
# 程序主入口
if __name__ == '__main__':
 get_ip = GetIp()
 get_ip.main()

使用方法完整代碼

PS:主要是通過使用隨機的IP來爬取,根據(jù)request_status來判斷這個IP是否可以用.

為什么要這樣判斷?主要是雖然上面經(jīng)過了過濾,但是不代表在你爬取的時候是可以用的,所以還是得多做一個判斷.

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# @Time : 2018/11/22 
# @Author : liangk
# @Site :
# @File : get_douban_books.py
# @Software: PyCharm
 
from bs4 import BeautifulSoup
import datetime
import requests
import json
import random
 
ip_random = -1
article_tag_list = []
article_type_list = []
 
 
def get_html(url):
 header = {
  'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36'
 }
 global ip_random
 ip_rand, proxies = get_proxie(ip_random)
 print(proxies)
 try:
  request = requests.get(url=url, headers=header, proxies=proxies, timeout=3)
 except:
  request_status = 500
 else:
  request_status = request.status_code
 print(request_status)
 while request_status != 200:
  ip_random = -1
  ip_rand, proxies = get_proxie(ip_random)
  print(proxies)
  try:
   request = requests.get(url=url, headers=header, proxies=proxies, timeout=3)
  except:
   request_status = 500
  else:
   request_status = request.status_code
  print(request_status)
 ip_random = ip_rand
 request.encoding = 'gbk'
 html = request.content
 print(html)
 return html
 
 
def get_proxie(random_number):
 with open('ip.txt', 'r') as file:
  ip_list = json.load(file)
  if random_number == -1:
   random_number = random.randint(0, len(ip_list) - 1)
  ip_info = ip_list[random_number]
  ip_url_next = '://' + ip_info['address'] + ':' + ip_info['port']
  proxies = {'http': 'http' + ip_url_next, 'https': 'https' + ip_url_next}
  return random_number, proxies
 
 
# 程序主入口
if __name__ == '__main__':
 """只是爬取了書籍的第一頁,按照評價排序"""
 start_time = datetime.datetime.now()
 url = 'https://book.douban.com/tag/?view=type&icn=index-sorttags-all'
 base_url = 'https://book.douban.com/tag/'
 html = get_html(url)
 soup = BeautifulSoup(html, 'lxml')
 article_tag_list = soup.find_all(class_='tag-content-wrapper')
 tagCol_list = soup.find_all(class_='tagCol')
 
 for table in tagCol_list:
  """ 整理分析數(shù)據(jù) """
  sub_type_list = []
  a = table.find_all('a')
  for book_type in a:
   sub_type_list.append(book_type.text)
  article_type_list.append(sub_type_list)
 
 for sub in article_type_list:
  for sub1 in sub:
   title = '==============' + sub1 + '=============='
   print(title)
   print(base_url + sub1 + '?start=0' + '&type=S')
   with open('book.text', 'a', encoding='utf-8') as f:
    f.write('\n' + title + '\n')
    f.write(url + '\n')
   for start in range(0, 2):
    # (start * 20) 分頁是0 20 40 這樣的
    # type=S是按評價排序
    url = base_url + sub1 + '?start=%s' % (start * 20) + '&type=S'
    html = get_html(url)
    soup = BeautifulSoup(html, 'lxml')
    li = soup.find_all(class_='subject-item')
    for div in li:
     info = div.find(class_='info').find('a')
     img = div.find(class_='pic').find('img')
     content = '書名:<%s>' % info['title'] + ' 書本圖片:' + img['src'] + '\n'
     print(content)
     with open('book.text', 'a', encoding='utf-8') as f:
      f.write(content)
 
 end_time = datetime.datetime.now()
 print('耗時: ', (end_time - start_time).seconds)

以上就是關于千鋒扣丁學堂Python培訓之代理IP爬蟲新手使用的全部內容，想要了解更多關于Python和人工智能方面內容的小伙伴，請關注扣丁學堂Python培訓官網(wǎng)、微信等平臺，扣丁學堂IT職業(yè)在線學習教育平臺為您提供權威的Python開發(fā)環(huán)境搭建視頻，Python培訓后的前景無限，行業(yè)薪資和未來的發(fā)展會越來越好的，扣丁學堂老師精心推出的Python視頻教程定能讓你快速掌握Python從入門到精通開發(fā)實戰(zhàn)技能。扣丁學堂Python技術交流群：279521237。

                          

      【關注微信公眾號獲取更多學習資料】         【掃碼進入Python全棧開發(fā)免費公開課】

查看更多關于"Python開發(fā)資訊"的相關文章>