blocks|key|1625796|text|井|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1625797|主要是您必须分离“刮刀器”/爬虫--+python+/program/函数，它将从webserver和解析器中下载文件/数据，解析器将读取这些数据并解释数据。在我的例子中，我不得不放弃并获得一些“开放”的政府信息，但不是对下载/数据友好的。在这个项目中，我使用了scrapy1。|1625798|主要是设置'starter_urls‘，这是我的机器人将爬行/获取的urls，在我使用一个函数’解析器‘检索/解析这个数据之后。|1625799|为了进行解析/检索，您将需要一些html、lxml提取器，因为您90%25的数据都是这样。|1625800|现在集中在你的问题上：|1625801|用于数据爬行|1625802|刮痕|ordered-list-item|1625803|请求2|1625804|Urllib+3|1625805|用于分析数据|1625806|Scrapy/lxml或scrapy%2Bother|1625807|lxml4|1625808|丰盛的汤5|1625809|请记住，“爬行”和“报废”不仅适用于网络，还包括电子邮件。你可以在这里检查另一个问题|1625810|1=+http://scrapy.org/|offset|length|1625811|2-+http://docs.python-requests.org/en/latest/|1625812|3-+http://docs.python.org/library/urllib.html|1625813|4-+http://lxml.de/|1625814|5-+http://www.crummy.com/software/BeautifulSoup/|1625815|6-+Python读取我的outlook电子邮件邮箱并解析消息|1625816|entityMap|0|LINK|mutability|MUTABLE|url|http://scrapy.org/|1|http://docs.python-requests.org/en/latest/|2|http://docs.python.org/library/urllib.html|3|http://lxml.de/|4|http://www.crummy.com/software/BeautifulSoup/|5|https://stackoverflow.com/questions/12291294/python-read-my-outlook-email-mailbox-and-parse-messages/12342273#12342273^0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|3|I|0|0|3|16|1|0|3|16|2|0|3|F|3|0|3|19|4|0|3|S|5|0^^$0|@$1|2|3|4|5|6|7|1Y|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|1Z|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|20|8|@]|9|@]|A|$]]|$1|F|3|G|5|6|7|21|8|@]|9|@]|A|$]]|$1|H|3|I|5|6|7|22|8|@]|9|@]|A|$]]|$1|J|3|K|5|6|7|23|8|@]|9|@]|A|$]]|$1|L|3|M|5|N|7|24|8|@]|9|@]|A|$]]|$1|O|3|P|5|N|7|25|8|@]|9|@]|A|$]]|$1|Q|3|R|5|N|7|26|8|@]|9|@]|A|$]]|$1|S|3|T|5|6|7|27|8|@]|9|@]|A|$]]|$1|U|3|V|5|N|7|28|8|@]|9|@]|A|$]]|$1|W|3|X|5|N|7|29|8|@]|9|@]|A|$]]|$1|Y|3|Z|5|N|7|2A|8|@]|9|@]|A|$]]|$1|10|3|11|5|6|7|2B|8|@]|9|@]|A|$]]|$1|12|3|13|5|6|7|2C|8|@]|9|@$14|2D|15|2E|1|2F]]|A|$]]|$1|16|3|17|5|6|7|2G|8|@]|9|@$14|2H|15|2I|1|2J]]|A|$]]|$1|18|3|19|5|6|7|2K|8|@]|9|@$14|2L|15|2M|1|2N]]|A|$]]|$1|1A|3|1B|5|6|7|2O|8|@]|9|@$14|2P|15|2Q|1|2R]]|A|$]]|$1|1C|3|1D|5|6|7|2S|8|@]|9|@$14|2T|15|2U|1|2V]]|A|$]]|$1|1E|3|1F|5|6|7|2W|8|@]|9|@$14|2X|15|2Y|1|2Z]]|A|$]]|$1|1G|3|-4|5|6|7|30|8|@]|9|@]|A|$]]]|1H|$1I|$5|1J|1K|1L|A|$1M|1N]]|1O|$5|1J|1K|1L|A|$1M|1P]]|1Q|$5|1J|1K|1L|A|$1M|1R]]|1S|$5|1J|1K|1L|A|$1M|1T]]|1U|$5|1J|1K|1L|A|$1M|1V]]|1W|$5|1J|1K|1L|A|$1M|1X]]]]

Well

Mainly you have to separate the 'scraper'/crawler the python lib/program/function that will download the files/data from the webserver and the Parser that will read this data and interpret the data.
In my case I had to scrap and get some govt info that is 'open' but not download/data friendly.
For this project I used scrapy[1]. 

Mainly I set the 'starter_urls' that are the urls my robot will crawl/get and after I use a function 'parser' to retrieve/parse this data.

For parsing/retrieving you are going to need some html,lxml extractor as the 90% of your data will be that.

Now focusing in your question:

For data crawling

<ol>
<li>Scrapy</li>
<li>Requests [2]</li>
<li>Urllib [3] </li>
</ol>

For parsing data

<ol>
<li>Scrapy/lxml or scrapy+other</li>
<li>lxml[4]</li>
<li>beatiful soup [5] </li>
</ol>

And please remember 'crawling' and scrapping is not only for web, emails too. you can check another question about that here [6] 

[1] = <a href="http://scrapy.org/" rel="nofollow noreferrer">http://scrapy.org/</a>

[2] - <a href="http://docs.python-requests.org/en/latest/" rel="nofollow noreferrer">http://docs.python-requests.org/en/latest/</a>

[3] - <a href="http://docs.python.org/library/urllib.html" rel="nofollow noreferrer">http://docs.python.org/library/urllib.html</a>

[4] - <a href="http://lxml.de/" rel="nofollow noreferrer">http://lxml.de/</a>

[5] - <a href="http://www.crummy.com/software/BeautifulSoup/" rel="nofollow noreferrer">http://www.crummy.com/software/BeautifulSoup/</a>

[6] - <a href="https://stackoverflow.com/questions/12291294/python-read-my-outlook-email-mailbox-and-parse-messages/12342273#12342273">Python read my outlook email mailbox and parse messages</a>

blocks|key|1625746|text|在lxml和漂亮的汤之间，lxml更等同于nokogiri，因为它基于libxml2，并且支持xpath/css。|type|ordered-list-item|depth|inlineStyleRanges|entityRanges|data|1625747|网络/http的等价物是urllib2|1625748|unstyled|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|G|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|H|8|@]|9|@]|A|$]]|$1|D|3|-4|5|E|7|I|8|@]|9|@]|A|$]]]|F|$]]

<ol>
<li>Between lxml and beautiful soup, lxml is more equivalent to nokogiri
because it is based on libxml2 and it has xpath/css support.</li>
<li>The equivalent of net/http is urllib2</li>
</ol>

blocks|key|2103139|text|实际上，Python中的真实世界HTML解析器是美汤。目前，Python库很受欢迎(尽管标准库具有类似的功能，但使用的请求相当繁琐)。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|2103140|好斗和收割机项目是真实世界里的爬虫机，只是为了爬行而定制的。|2103141|entityMap|0|LINK|mutability|MUTABLE|url|http://www.crummy.com/software/BeautifulSoup/|1|http://docs.python-requests.org/en/latest/index.html^0|O|2|0|1N|2|1|0|0^^$0|@$1|2|3|4|5|6|7|P|8|@]|9|@$A|Q|B|R|1|S]|$A|T|B|U|1|V]]|C|$]]|$1|D|3|E|5|6|7|W|8|@]|9|@]|C|$]]|$1|F|3|-4|5|6|7|X|8|@]|9|@]|C|$]]]|G|$H|$5|I|J|K|C|$L|M]]|N|$5|I|J|K|C|$L|O]]]]

The de facto real world HTML parser in Python is <a href="http://www.crummy.com/software/BeautifulSoup/" rel="nofollow">beautiful soup</a>. The Python <a href="http://docs.python-requests.org/en/latest/index.html" rel="nofollow">requests</a> library is popular these days for HTTP (although the standard library has similar functionality but with a rather cumbersome API). 

The scrappy and harvestman projects are real world crawlers that have been custom built just for the purpose of crawling.

blocks|key|2103269|text|我还使用了美汤，这是一种非常简单的解析方法。当我爬行一些网页时，我也使用ElementTree+XML。就我个人而言，我非常喜欢ElementTree库(它易于解析)。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|2103270|entityMap|0|LINK|mutability|MUTABLE|url|http://www.crummy.com/software/BeautifulSoup/|1|http://docs.python.org/library/xml.etree.elementtree.html^0|5|2|0|10|F|1|0^^$0|@$1|2|3|4|5|6|7|N|8|@]|9|@$A|O|B|P|1|Q]|$A|R|B|S|1|T]]|C|$]]|$1|D|3|-4|5|6|7|U|8|@]|9|@]|C|$]]]|E|$F|$5|G|H|I|C|$J|K]]|L|$5|G|H|I|C|$J|M]]]]

I also use <a href="http://www.crummy.com/software/BeautifulSoup/" rel="nofollow">Beautiful Soup</a>, its very easy way how to parse HTML. When i was crawling some web pages i also use <a href="http://docs.python.org/library/xml.etree.elementtree.html" rel="nofollow">The ElementTree XML API</a>. Personally, i really like The ElementTree library(its easy to parse XML).

I've started to learn python the past couple of days. I want to know the equivalent way of writing crawlers in python.

so In ruby I use:

<ol>
<li><code>nokogiri</code> for crawling html and getting content through css tags</li>
<li><code>Net::HTTP</code> and <code>Net::HTTP::Get.new(uri.request_uri).body</code> for getting JSON data from a url</li>
</ol>

what are equivalents of these in python?

Going from Ruby to Python : Crawlers

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

教程

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云智能顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

EdgeOne AI 安全实战专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

聚焦“写作效率、视觉美观与运行性能”三方面进行全面升级，为您提供更高效、稳定的创作环境

社区富文本&Markdown编辑器全新改版上线，欢迎大家体验!

诚挚邀请您参与本次调研，分享您的真实使用感受与建议。您的反馈至关重要，感谢您的支持与参与！

社区新版编辑器体验调研

在过去的几天里我开始学习蟒蛇。我想知道用python编写爬虫的等效方法。所以在红宝石里我用：nokogiri，用于爬行html并通过css标记获取内容。用于从url获取JSON数据的Net::HTTP和Net::HTTP::Get.new(uri.request_uri).body在python中，它们的等价物是什么？

问从Ruby到Python :爬虫
EN

回答 4

Stack Overflow用户

Stack Overflow用户

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问从Ruby到Python :爬虫EN

回答 4

Stack Overflow用户

Stack Overflow用户

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问从Ruby到Python :爬虫
EN