github-forks
/
webmagic
mirror of https://github.com/code4craft/webmagic.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
1,102 Commits
50 Branches
37 Tags
20 MiB
Commit Graph

238 Commits (e2c143b52f7685df4a331d954a56556a11bc98a5)

Author SHA1 Message Date
Jsbd 6d78d51fc0 Merge branch 'master' into master
yihua.huang d69204b919 0.6.0
yihua.huang 97592d6720 Version 0.6.0
yihua.huang 00dfebbceb remove guava dep and add fix docs
yihua.huang a960a39c44 fix compile error for example change
yihua.huang a3ee9e3d08 fix example
Jsbd 1b886d48a2 新增PhantomJSDownloader构造函数，支持crawl.js路径自定义，因为当其他项目依赖此jar包时，runtime.exec()执行phantomjs命令时无使用法jar包中的crawl.js
Jsbd d1f2e65e5d 新增PhantomJSDownloader构造函数，支持crawl.js路径自定义，因为当其他项目依赖此jar包时，runtime.exec()执行phantomjs命令时无使用法jar包中的crawl.js
Jsbd ebc61363c8 为PhantomJSDownloader添加新的构造函数，支持phantomjs自定义命令 
为PhantomJSDownloader添加新的构造函数，支持phantomjs自定义命令
 example: 
   *    phantomjs.exe 支持windows环境
   *    phantomjs --ignore-ssl-errors=yes 忽略抓取地址是https时的一些错误
   *    /usr/local/bin/phantomjs 命令的绝对路径，避免因系统环境变量引起的IOException
yihua.huang b92e6b04f0 修复FileCacheQueueScheduler自己设置DuplicateRemover会导致NPE的问题
Yihua Huang 1491033534 Merge pull request from jerry-sc/monitor-bug 
fix the monitor bug which the spider will terminate when a seed url with port
Jerry e56b8c3efc fix the monitor bug which the spider will terminate when a seed url with port
郭玉昆 700898fe8a fixed 修复使用注解抽取JSON数据的问题
Salon.sai f89a6a6826 add: redis scheduler with priority
Yihua Huang 37cb43b667 Merge pull request from lavenderx/master 
add PhantomJSDownloader
Linker Lin 047cb8ff8f updated versions to 0.5.4-SNAPSHOT
yihua.huang c0b8e8f8ae remove .classpath .project
yihua.huang a8e6de4b90 Merge branch 'master' of git.oschina.net:flashsword20/webmagic
yihua.huang ce5495ecd5 remove useless files
yihua.huang 8265c7dade remove submodules for relase
yihua.huang 7edfa26f90 complete javadoc
yihua.huang 8b90b91e33 complete some javadoc
yihua.huang 2b556cf053 update verison to 0.5.3-SNAPSHOT
yihua.huang 9c5716a543 complete javadoc
yihua.huang db3cbf6ca5 update version to 0.5.3-SNAPSHOT
yihua.huang 7586e3d75c add some test for github repo downloader
Yihua Huang cfde3b7657 Merge pull request from SpenceZhou/master 
Update RedisScheduler.java
SpenceZhou 165e5a72eb Update RedisScheduler.java 
修改redisscheduler中获取爬取总数bug
x1ny 90e14b31b0 修正FileCacheQueueScheduler导致程序不能正常结束和未关闭流 
FileCacheQueueScheduler中开启了一个线程周期运行来保存数据但在爬虫结束后没有关闭导致程序无法结束，以及没有关闭io流。

解决方法：
让FileCacheQueueScheduler实现Closable接口，在close方法中关闭线程以及流。
在Spider的close方法中添加对scheduler的关闭操作。
edwardsbean 74962d69b9 fix bug:MultiPagePipeline and DoubleKeyMap concurrent bug
dolphineor 7628dc6b63 move PhantomJSDownloader to webmagic-extension
yihua.huang 8551b668a0 remove commented code
zhugw eb3c78b9d8 Update FileCacheQueueScheduler.java 
这样是不是更严谨? 否则的话,中断后再次启动时, (第一个)入口地址仍会被添加到队列及写入到文件中. 
但是现在有另外一个问题存在,如第一遍全部抓取完毕了(通过spider.getStatus==Stopped判断),休眠24小时,再来抓取(通过递归调用抓取方法).
这时不同于中断后再启动,lineReader==cursor, 于是初始化时队列为空,入口地址又在urls集合中了, 故导致抓取线程马上就结束了.这样的话就没有办法去抓取网站上的新增内容了.
解决方案一:
判断抓取完毕后,紧接着覆盖cursor文件,第二次来抓取时,curosr为0, 于是将urls.txt中的所有url均放入队列中了, 可以通过这些url来发现新增url.
方案二:
对方案一进行优化,方案一虽然可以满足业务要求,但会做很多无用功,如仍会对所有旧target url进行下载,抽取,持久化等操作.而新增的内容一般都会在HelpUrl中, 比如某一页多了一个新帖子,或者多了几页内容. 故第二遍及以后来爬取时可以仅将HelpUrl放入队列中. 

希望能给予反馈,我上述理解对不对, 有什么没有考虑到的情况或者有更简单的方案?谢谢!
yihua.huang 42a30074c9 update urls.contains to DuplicateRemover in FileCacheQueueScheduler
zhugw 1db940a088 Update FileCacheQueueScheduler.java 
在使用过程中发现urls.txt文件存在重复URL的情况,经跟踪源代码,发现初始化加载文件后,读取所有的url放入一集合中,但是之后添加待抓取URL时并未判断是否已存在该集合中(即文件中)了,故导致文件中重复URL的情况.故据此对源码做了修改,还请作者审阅.
yihua.huang 3734865a6a fix package name =.=
yihua.huang e7668e01b8 fix SourceRegion error and add some tests on it
yihua.huang 4e5ba02020 fix test cont'
yihua.huang 2fd8f05fe2 change path seperator for varient OS
yihua.huang 8c33be48a6 Merge branch 'stable' of github.com:code4craft/webmagic
yihua.huang 5f8c3fd5c5 update version
yihua.huang 928f98dd93 auto create folder in JsonFilePipeline
yihua.huang 7fbe18b8c0 implementation of PageMapper
yihua.huang 5dc9fe95a9 interface of PageMapper
yihua.huang 7668731f08 update version to snapshot
yihua.huang 182dd51689 Merge branch 'stable' of github.com:code4craft/webmagic
yihua.huang 81e6e772ac versions back to 0.5.1
yihua.huang feb604da87 Merge branch 'stable' of github.com:code4craft/webmagic
yihua.huang 358e906379 [maven-release-plugin] prepare for next development iteration
yihua.huang 470750fc0d [maven-release-plugin] prepare release WebMagic-0.5.1
yihua.huang 186b90512e refactor redisscheduler
yihua.huang d1140b9e29 add bloom filter for scheduler
yihua.huang e8d4a9be2b fix remove duplicate error
yihua.huang 04ade75606 Merge branch 'stable' of github.com:code4craft/webmagic 
Conflicts:
	README.md
	pom.xml
	webmagic-avalon/pom.xml
	webmagic-core/pom.xml
	webmagic-extension/pom.xml
	webmagic-lucene/pom.xml
	webmagic-samples/pom.xml
	webmagic-saxon/pom.xml
	webmagic-scripts/pom.xml
	webmagic-selenium/pom.xml
yihua.huang a08d8cb167 update verion
yihua.huang 42a2676e8c update version
yihua.huang c25b32f1ca [maven-release-plugin] prepare for next development iteration
yihua.huang 7ff83bb11a [maven-release-plugin] prepare release WebMagic-0.5.0
yihua.huang 1104122979 more abstraction in scheduler
yihua.huang b0fb1c3e10 remove copy-dependcies plugin for m2e error
yihua.huang 94a67165e1 remove jmx server for simplify
yihua.huang 86a45a6643 change SpiderMonitor to singleton
yihua.huang ab4d36806e clean code
yihua.huang 04fde8203b add control for monitor
yihua.huang 2770811a10 update monitor example
yihua.huang 17e95f2a7f comments
yihua.huang 375e64e845 more monitor status
yihua.huang c6661899fd new thread pool
yihua.huang 179baa7a22 return when page is null
yihua.huang 4738ae2d14 change url find to match
yihua.huang f973889cda refactor subpageprossor etc.
yihua.huang acb63d55d7 some check and example
yihua.huang 11ba5beb42 [refactor]move monitor to webmagic-extension
yihua.huang b06aa489fb [BugFix]Only one url from sourceRegion can be extracted
yihua.huang 023c2ac84e spider config draft
yihua.huang a5db6cf292 some monitor and JMX support
yihua.huang aae1ab2cd6 fix compile error
yihua.huang 1fbfc92de2 Inherit support of Field annotation in Model
yihua.huang a03f6a8431 eclipse project
yihua.huang 3a79b1b64a [Bugfix]formatter property does not work when field is String#100
Yihua Huang cc9d319fd9 Merge pull request from sebastian1118/master 
update:PatternHandler
yihua.huang 03c251237b add Json parse support
Tian 99e12aafaa update:PatternHandler
yihua.huang c1e7207869 add FileCacheQueueScheduler support for cycleRetryTimes
yihua.huang 969ad1766b change logger style to slf4j for cleaner code
yihua.huang 9b2cb43f47 ConfigurablePageProcessor
Bo LIANG 159eeea2f5 Remove unused variable to make the project cleaner.
yihua.huang c143fc662c add SubPageProcessor
Yihua Huang 474f785dab Merge pull request from sebastian1118/master 
new feature: PatternProcessor
Tian 38a12f8641 new feature: PatternProcessor
yihua.huang dafd0b5875 [BugFix]multi model in one pageprocessor will be skipped
yihua.huang a1c7e826f7 fix dep of slf4j-log4j12
yihua.huang f3c2503a29 add warning of slf4j
yihua.huang 8958d774f2 add default values for @Formatter
yihua.huang 6c11718566 Clean project structure
yihua.huang 98e2bba099 Merge branch 'master' of github.com:code4craft/webmagic 
Conflicts:
	README.md
	pom.xml
	webmagic-core/pom.xml
	webmagic-extension/pom.xml
	webmagic-scripts/pom.xml
yihua.huang 757cc9b942 [maven-release-plugin] prepare for next development iteration
yihua.huang 63ffb5c792 [maven-release-plugin] prepare release webmaigc-0.4.3
yihua.huang d5a978e00f update version back to 0.4.3
yihua.huang 0e98183f74 Change log4j to slf4j