github-forks
/
webmagic
mirror of https://github.com/code4craft/webmagic.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
859 Commits
50 Branches
37 Tags
19 MiB
Commit Graph

859 Commits (d6cd92b1a889e0d53b71d7a7b3d81eeff28c3a56)
 

Author SHA1 Message Date
yihua.huang 81ce1ffc5f fix ignore
yihua.huang 93764fa2c9 ignore some test
yihua.huang 5706bb90af update xsoup to 0.3.1
yihua.huang 7586e3d75c add some test for github repo downloader
yihua.huang 800f66c4cc Revert "remove some unkown config" 
This reverts commit 0e245c9896.
yihua.huang 73ae7a1d52 remove ci for jdk6
yihua.huang 0e245c9896 remove some unkown config
yihua.huang 9ed06ccdf0 update surefire version
Yihua Huang 9d0eeb9000 Merge pull request from bingoko/master 
添加PhantomJS无界面浏览器支持
Yihua Huang 84b046e4c9 Merge pull request from hsqlu/master 
update deprecated method
Yihua Huang cfde3b7657 Merge pull request from SpenceZhou/master 
Update RedisScheduler.java
SpenceZhou 165e5a72eb Update RedisScheduler.java 
修改redisscheduler中获取爬取总数bug
Yihua Huang 5f9e1a96f2 Merge pull request from x1ny/master 
修正FileCacheQueueScheduler导致程序不能正常结束和未关闭流
Yihua Huang 7d7eb033d3 Merge pull request from chy996633/master 
知乎爬虫抓取
chy996633 afd1617b58 知乎爬虫抓取
x1ny 90e14b31b0 修正FileCacheQueueScheduler导致程序不能正常结束和未关闭流 
FileCacheQueueScheduler中开启了一个线程周期运行来保存数据但在爬虫结束后没有关闭导致程序无法结束，以及没有关闭io流。

解决方法：
让FileCacheQueueScheduler实现Closable接口，在close方法中关闭线程以及流。
在Spider的close方法中添加对scheduler的关闭操作。
Qiannan Lu 155215290f resolve issue
Qiannan Lu 21f81bb8c1 update deprecated method
bingoko 5d365f7bf4 update and validate pom.xml 
Update selenium and GhostDriver (PhantomJSDriver) to latest version.
bingoko d3bbece202 Add PhantomJS support for selenium 
The configuration file is config.ini
The dependencies are updated in pom.xml.
Update SeleniumDownloader and WebDriverPool to support PhantomJS. 
NOTE: The versions of GhostDriver, Selenium, and PhantomJS are stable
and validated.

A GooglePlay Example is under samples package: GooglePlayProcessor.java
yihua.huang 56e0cd513a compile error fix
yihua.huang c5740b1840 change assert
yihua.huang 67eb632f4d test for issue
Yihua Huang b30ca6ce1e Merge pull request from okuc/master 
修正site.setHttpProxy()不起作用的bug
高军 590561a6e4 修正site.setHttpProxy()不起作用的bug
edwardsbean 19474e4716 add SimpleProxyPool and IProxyPool
Yihua Huang 05a1f39569 Merge pull request from EdwardsBean/fix-mppipeline 
Bug fix:MultiPagePipeline and DoubleKeyMap concurrent bug
edwardsbean 74962d69b9 fix bug:MultiPagePipeline and DoubleKeyMap concurrent bug
Yihua Huang 6b9d21fcf3 Merge pull request from EdwardsBean/retry_time 
add retry sleep time
edwardsbean 4978665633 add retry sleep time
yihua.huang 862ceee674 groovy demo
yihua.huang 8ffc1a7093 add NPE check for POST method
dolphineor 7628dc6b63 move PhantomJSDownloader to webmagic-extension
dolphineor 0d9b148fc2 add PhantomJSDownloader
yihua.huang 8551b668a0 remove commented code
Yihua Huang 20422f1b63 Merge pull request from zhugw/patch-4 
Update FileCacheQueueScheduler.java
zhugw eb3c78b9d8 Update FileCacheQueueScheduler.java 
这样是不是更严谨? 否则的话,中断后再次启动时, (第一个)入口地址仍会被添加到队列及写入到文件中. 
但是现在有另外一个问题存在,如第一遍全部抓取完毕了(通过spider.getStatus==Stopped判断),休眠24小时,再来抓取(通过递归调用抓取方法).
这时不同于中断后再启动,lineReader==cursor, 于是初始化时队列为空,入口地址又在urls集合中了, 故导致抓取线程马上就结束了.这样的话就没有办法去抓取网站上的新增内容了.
解决方案一:
判断抓取完毕后,紧接着覆盖cursor文件,第二次来抓取时,curosr为0, 于是将urls.txt中的所有url均放入队列中了, 可以通过这些url来发现新增url.
方案二:
对方案一进行优化,方案一虽然可以满足业务要求,但会做很多无用功,如仍会对所有旧target url进行下载,抽取,持久化等操作.而新增的内容一般都会在HelpUrl中, 比如某一页多了一个新帖子,或者多了几页内容. 故第二遍及以后来爬取时可以仅将HelpUrl放入队列中. 

希望能给予反馈,我上述理解对不对, 有什么没有考虑到的情况或者有更简单的方案?谢谢!
Yihua Huang 3a9c1d3002 Merge pull request from zhugw/patch-3 
Update Site.java
zhugw bc666e927d Update Site.java 
setCycleRetryTimes的javadoc是这么说的:Set cycleRetryTimes times when download fail, 0 by default. Only work in RedisScheduler.
而通过查看源码发现似乎并没有做限制,即只能用于RedisScheduler. 故想问一下该javadoc是否过时了?
yihua.huang 42a30074c9 update urls.contains to DuplicateRemover in FileCacheQueueScheduler
Yihua Huang 689e89a9b2 Merge pull request from zhugw/patch-1 
Update FileCacheQueueScheduler.java
zhugw 1db940a088 Update FileCacheQueueScheduler.java 
在使用过程中发现urls.txt文件存在重复URL的情况,经跟踪源代码,发现初始化加载文件后,读取所有的url放入一集合中,但是之后添加待抓取URL时并未判断是否已存在该集合中(即文件中)了,故导致文件中重复URL的情况.故据此对源码做了修改,还请作者审阅.
yihua.huang 147401ce5e remove duplicate setPath in ProxyPool
yihua.huang 3734865a6a fix package name =.=
yihua.huang e7668e01b8 fix SourceRegion error and add some tests on it
yihua.huang 4e5ba02020 fix test cont'
yihua.huang 4446669c24 fix test
yihua.huang 9866297ec4 Disable jsoup entity escape by Default. Set Html.DISABLE_HTML_ENTITY_ESCAPE to false to enable it.
yihua.huang 4e6e946dd7 more friendly exception message in PlainText
yihua.huang ebb931e0bf update assertj to test scope