urllib2，机械不返回相同的浏览器-还有什么可欺骗的？
我试图创建一个脚本(纯粹的学习目的)来翻译一个给定的词与几个不同的字典。我用我的方式完成了两个，使用urllib2和美丽汤获得和解析翻译，然后转到谷歌翻译。
我很快发现它返回403禁止的错误。添加用户代理将得到翻译，但只有一个单词翻译。为了举例说明，请转到http://translate.google.com/?text=test&sl=en&tl=es，您将得到翻译(在一个名为“hps”的类中)和一个动词、名词和形容词的列表。但是使用下面的脚本，并且html是不同的，主翻译只返回，并且在
span id=result_box动词、名词等是找不到的。
在这个过程中，以及相当多的谷歌搜索过程中，我意识到现在有了一个API --而不是免费的。我不打算发布任何最终的脚本，也不打算使用它来违反任何TOS，但是现在我主要想知道为什么浏览器和urllib之间的区别等等。
为此，我尝试了使用用户代理的纯urllib2，并将其机械化-如下所示。所以，我的问题是--除了用户代理，浏览器和python脚本还有什么区别？我试过使用萤火虫，但没什么能从我身上跳出来(尽管我很讨厌它)。谢谢!
编辑:来自firebug的请求头，我的脚本在下面。
import mechanize
import re
import cookielib
# Browser
br = mechanize.Browser()
# Cookie Jar
cj = cookielib.LWPCookieJar()
br.set_cookiejar(cj)
# Browser options
br.set_handle_equiv(True)
br.set_handle_redirect(True)
br.set_handle_referer(True)
br.set_handle_robots(False)
# Follows refresh 0 but not hangs on refresh > 0
br.set_handle_refresh(mechanize._http.HTTPRefreshProcessor(), max_time=1)
# Want debugging messages?
br.set_debug_http(True)
br.set_debug_redirects(True)
br.set_debug_responses(True)
# User-Agent (this is cheating, ok?)
br.addheaders = [('User-agent', 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1')]
# Open some site, let's pick a random one, the first that pops in mind:
r = br.open('http://translate.google.com/?text=test&sl=en&tl=es')
html = r.read()
match = re.findall(r'verb', html)
print match萤火虫：
GET /?text=test&sl=en&tl=es HTTP/1.1
Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Charset ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept-Encoding gzip, deflate
Accept-Language en-us,en;q=0.5
Connection keep-alive
Cookie PREF=ID=298b435815ef8553:U=e7dad4baf65f083b:FF=0:LD=en:CR=2:TM=1327516863:LM=1339428154:S=maktYFZEHXXpMDFg; NID=60=U229h4lzOnjpHyidbhgYecCx72Myp_-XHgupW-R_mWtpuOveDdIOO1uLBq-6ltn-ER15ppJryR7yYOYEhkCfUCl45qNz5aymBQ1CGDHS4UcHu2oIDYAHut0ctnlL76eDW3n7kjOWoz5wNH6NMw
Host translate.google.com
User-Agent Mozilla/5.0 (Windows NT 6.1; WOW64; rv:9.0) Gecko/20100101 Firefox/9.0剧本：
发送：‘获取/?text=test&sl=en&tl=es HTTP/1.1\r\n接受-编码:标识\r\n主机: translate.google.com\r\nConnection:关闭\r\n用户-代理:/?text=test&sl=en&tl=es/5.0 (X11；U；Linux i686；en-US；rv:1.9.0.1) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1\r\n\n‘回复：'HTTP/1.1 200 OK\r\n’标题:日期: Mon，2012年6月11日16:13:42格林尼治时间
标题:截止日期:星期五，1990年1月1日00:00:00格林尼治时间
头:缓存-控制:无缓存，必须重新验证
标题: Pragma:无缓存
标题:X-框架-选项: SAMEORIGIN
标题:内容-类型: text/html；charset=UTF-8
标题:内容-语言: en
标题: Set-Cookie: PREF=ID=6dd42f2264250d7c:TM=1333431222:LM=1339454222:S=k6JXSoGGaAMNmPEo；expires=Wed，11-6月-2014 16:13:42 GMT；path=/；domain=.google.com
标题: Set-Cookie: NID=60=f8czmR413h3sKUGJUUM4PLKl2O7SUtqfW5hss5O54sRKoErf9wIEU4Wu2WCuHzWTJQ3p1Rj7dQv1B4BBmSMY1tmfus7UZGCYFIKaXoKwklZ9tZsr5vds8vvvFjRdZyevn；expires=Tue，2012年12月11日16:13:42 GMT；path=/；domain=.google.com；HttpOnly
标题: P3P: CP=“这不是P3P策略！更多信息请参见http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657。”
标题:X-内容-类型-选项: nosniff
标题:服务器: HTTP server (未知)
标题:XSS保护: 1；mode=block
标题:连接:关闭
回答 1
Stack Overflow用户
发布于 2012-06-11 16:45:07
之所以找不到动词、形容词，是因为它们是通过AJAX调用加载的。您的机器浏览器没有javascript。因此，它无法执行任何AJAX。但是，如果您可以查看浏览器的检查器或其他东西，您将看到调用的标题、URL和参数。现在所要做的就是模仿这个召唤。
我把它卷了起来，得到了JSON的回复：
thrustmaster@thrustmaster:~/Temp$ curl 'http://translate.google.com/translate_a/t?client=t&text=test&hl=en&sl=en&tl=es&multires=1&ssel=0&tsel=0&sc=1' -H 'User-Agent: blah'
[[["prueba","test","",""]],[["noun",["prueba","ensayo","test","examen","an�lisis","criterio","toque","ejercicio","tanteo"],[["prueba",["test","proof","evidence","trial","event","race"]],["ensayo",["test","trial","essay","assay","testing","rehearsal"]],["test",["test"]],["examen",["examination","review","exam","test","inspection","quiz"]],["an�lisis",["analysis","test","review","assay","breakdown"]],["criterio",["criterion","judgment","standard","test","view","yardstick"]],["toque",["touch","stroke","test","knock","blast","chime"]],["ejercicio",["exercise","practice","drill","practicing","test","prosecution"]],["tanteo",["score","scoring","trial","test","try","calculation"]]]],["adjective",["de prueba"],[["de prueba",["test","testing","trial","probationary","corrective"]]]],["verb",["probar","comprobar","ensayar","examinar","poner a prueba","experimentar","someter a prueba","interrogar","hacer investigaciones","justificar","graduar"],[["probar",["test","try","prove","taste","try out","sample"]],["comprobar",["check","test","prove","ascertain","make sure","substantiate"]],["ensayar",["test","try","rehearse","try out","assay","essay"]],["examinar",["examine","consider","review","look at","explore","test"]],["poner a prueba",["test","try","try out","prove","tempt","put through his paces"]],["experimentar",["experience","experiment","undergo","experiment with","feel","test"]],["someter a prueba",["test","try out","touch"]],["interrogar",["question","interrogate","examine","cross-examine","ask","test"]],["hacer investigaciones",["test"]],["justificar",["justify","warrant","substantiate","prove","make good","test"]],["graduar",["graduate","grade","calibrate","time","test"]]]]],"en",,[["prueba",[5],1,0,1000,0,1,0]],[["test",4,,,""],["test",5,[["prueba",1000,1,0],["prueba de",0,1,0],["ensayo",0,1,0],["de prueba",0,1,0],["test",0,1,0]],[[0,4]],"test"]],,,[["en"]],5]thrustmaster@thrustmaster:~/Temp$ 现在，可能在您的脚本中，您必须从下面的URL获得响应：
http://translate.google.com/translate_a/t?client=t&text=test&hl=en&sl=en&tl=es&multires=1&ssel=0&tsel=0&sc=1
PS：
正如您所说，这可能是一个TOS问题，以防您计划使用此脚本。在API上使用它总是一个更好的选择。您所依赖的HTML可以随时更改。
https://stackoverflow.com/questions/10983772
复制相似问题
腾讯云开发者
Copyright © 2013 - 2026 Tencent Cloud. All Rights Reserved. 腾讯云 版权所有
深圳市腾讯计算机系统有限公司 ICP备案/许可证号：粤B2-20090059 
粤公网安备44030502008569号
腾讯云计算（北京）有限责任公司 京ICP证150476号 | 京ICP备11018762号