287 Commits (develop)

Author	SHA1	Message	Date
yao	2e2a0fdf3e	Downloader 提供刷新组件的api,方便在spider中操作	4 years ago
Sutra Zhou	4b902270b4	Bump version number from 0.7.3 to 0.7.4.	4 years ago
Sutra Zhou	5d14efc50f	Serialize request URL only in FileCacheQueueScheduler.	5 years ago
Sutra Zhou	71aa04c89f	Upgrade dependencies.	5 years ago
Sutra Zhou	ba1b4017a7	Mark slf4j-log4j12 as optional.	5 years ago
Sutra Zhou	b98a87e45a	Serialize requests in FileCacheQueueScheduler, so that the extra info of request could be restored.	5 years ago
yihua.huang	c701fe8d38	#702 Refactor: rename CheckForAdditionalInfo to checkForAdditionalInfo	7 years ago
yihy	266083fa07	[Fix] #698  Repair using redis,Request additional information is lost	7 years ago
yihua.huang	e5db538c19	#647 remove ThreadSafe annotation	7 years ago
yihua.huang	e276b11ea9	vesion 0.7.3	8 years ago
yihua.huang	1b6394bef9	version 0.7.2	8 years ago
yihua.huang	76766a7c77	RequestUtils for range #222	8 years ago
yihua.huang	13cdf82695	update version to 0.7.2-SNAPSHOT	8 years ago
yihua.huang	1e9187f24e	version 0.7.1	8 years ago
yihua.huang	5de6205563	use RawText as source for JsonPath #589	8 years ago
yihua.huang	211ac74389	use group 0 of targetUrl instead of group 1 #588	8 years ago
yihua.huang	9b77306098	test of ExtractByUrl #586	8 years ago
yihua.huang	d8bd0637a1	add custom formatter test #586	8 years ago
yihua.huang	6cc647577a	fix test for ExtractLinks #586	8 years ago
yihua.huang	a6f8ed5476	complete formatter refactor by ObjectFormatterBuilder #586	8 years ago
yihua.huang	b1ef61b278	add tests before refactor #586	8 years ago
yihua.huang	b363ee6a9d	move constant to class	8 years ago
yihua.huang	df682857a7	fix test and some refactor	8 years ago
yihua.huang	0e6eb46eba	add test for jsonpath in pagemodel #462	8 years ago
yihua.huang	f5018d569e	remove test_issue409 #409	8 years ago
yihua.huang	17ae500c77	test_issue409 #409	8 years ago
yihua.huang	4cd5b4f93e	test_issue409	8 years ago
yihua.huang	fb0acd710c	complete SimpleHttpClientTest	8 years ago
yihua.huang	d07941d900	SimpleHttpClientTest	8 years ago
yihua.huang	f02f469c69	add test #570	8 years ago
yihua.huang	2d693580fc	add test	8 years ago
yihua.huang	b879b0eed0	fix redisscheduler #583	8 years ago
Yihua Huang	9903f0367d	Merge pull request #570 from SoulZhong/master ... 修复formatter初始化未传参bug	8 years ago
yihua.huang	2e35e149be	for 0.7.1	8 years ago
yihua.huang	17478fcfc4	0.7.0 release	8 years ago
yihua.huang	49de9374cd	new SimpleHttpClient #576	8 years ago
yihua.huang	7ffc6998ef	add isExtractLinks to OOSpider #575	8 years ago
soul	bc828e1384	修复formatter初始化未传参bug	8 years ago
yihua.huang	a8c2e6c729	alpha release	8 years ago
yihua.huang	3c1338193b	for 0.7.0.alpha	8 years ago
yihua.huang	d38d51dfcb	fix javadoc	8 years ago
yihua.huang	1b04a7f2b3	#527 move logic check from downloaderto spider	8 years ago
yihua.huang	74110e6ec5	remove useless file	8 years ago
yihua.huang	b100dfe273	update version	8 years ago
yihua.huang	c13110c4cb	fix samples	8 years ago
yihua.huang	d87c73b472	change check-and-set to atomic sadd for redis DuplicateRemover #368	8 years ago
yihua.huang	aaccc93215	new version	8 years ago
yihua.huang	3e633c6871	version	8 years ago
yihua.huang	f45e2f118b	for release	8 years ago
Jsbd	6d78d51fc0	Merge branch 'master' into master	8 years ago
yihua.huang	d69204b919	0.6.0	8 years ago
yihua.huang	97592d6720	Version 0.6.0	8 years ago
yihua.huang	00dfebbceb	#424 remove guava dep and add fix docs	8 years ago
yihua.huang	a960a39c44	fix compile error for example change	8 years ago
yihua.huang	a3ee9e3d08	fix example	8 years ago
Jsbd	1b886d48a2	新增PhantomJSDownloader构造函数，支持crawl.js路径自定义，因为当其他项目依赖此jar包时，runtime.exec()执行phantomjs命令时无使用法jar包中的crawl.js	8 years ago
Jsbd	d1f2e65e5d	新增PhantomJSDownloader构造函数，支持crawl.js路径自定义，因为当其他项目依赖此jar包时，runtime.exec()执行phantomjs命令时无使用法jar包中的crawl.js	8 years ago
Jsbd	ebc61363c8	为PhantomJSDownloader添加新的构造函数，支持phantomjs自定义命令 ... 为PhantomJSDownloader添加新的构造函数，支持phantomjs自定义命令 example: * phantomjs.exe 支持windows环境 * phantomjs --ignore-ssl-errors=yes 忽略抓取地址是https时的一些错误 * /usr/local/bin/phantomjs 命令的绝对路径，避免因系统环境变量引起的IOException	8 years ago
yihua.huang	b92e6b04f0	#400 修复FileCacheQueueScheduler自己设置DuplicateRemover会导致NPE的问题	8 years ago
Yihua Huang	1491033534	Merge pull request #377 from jerry-sc/monitor-bug ... fix the monitor bug which the spider will terminate when a seed url with port	8 years ago
Jerry	e56b8c3efc	fix the monitor bug which the spider will terminate when a seed url with port	8 years ago
郭玉昆	700898fe8a	fixed #301 修复使用注解抽取JSON数据的问题	9 years ago
Salon.sai	f89a6a6826	add: redis scheduler with priority	9 years ago
Yihua Huang	37cb43b667	Merge pull request #176 from lavenderx/master ... add PhantomJSDownloader	9 years ago
Linker Lin	047cb8ff8f	updated versions to 0.5.4-SNAPSHOT	9 years ago
yihua.huang	c0b8e8f8ae	remove .classpath .project	9 years ago
yihua.huang	a8e6de4b90	Merge branch 'master' of git.oschina.net:flashsword20/webmagic	9 years ago
yihua.huang	ce5495ecd5	remove useless files	9 years ago
yihua.huang	8265c7dade	remove submodules for relase	9 years ago
yihua.huang	7edfa26f90	complete javadoc	9 years ago
yihua.huang	8b90b91e33	complete some javadoc	9 years ago
yihua.huang	2b556cf053	update verison to 0.5.3-SNAPSHOT	9 years ago
yihua.huang	9c5716a543	complete javadoc	9 years ago
yihua.huang	db3cbf6ca5	update version to 0.5.3-SNAPSHOT	9 years ago
yihua.huang	7586e3d75c	add some test for github repo downloader	9 years ago
Yihua Huang	cfde3b7657	Merge pull request #237 from SpenceZhou/master ... Update RedisScheduler.java	9 years ago
SpenceZhou	165e5a72eb	Update RedisScheduler.java ... 修改redisscheduler中获取爬取总数bug	9 years ago
x1ny	90e14b31b0	修正FileCacheQueueScheduler导致程序不能正常结束和未关闭流 ... FileCacheQueueScheduler中开启了一个线程周期运行来保存数据但在爬虫结束后没有关闭导致程序无法结束，以及没有关闭io流。 解决方法： 让FileCacheQueueScheduler实现Closable接口，在close方法中关闭线程以及流。 在Spider的close方法中添加对scheduler的关闭操作。	9 years ago
edwardsbean	74962d69b9	fix bug:MultiPagePipeline and DoubleKeyMap concurrent bug	10 years ago
dolphineor	7628dc6b63	move PhantomJSDownloader to webmagic-extension	10 years ago
yihua.huang	8551b668a0	remove commented code	10 years ago
zhugw	eb3c78b9d8	Update FileCacheQueueScheduler.java ... 这样是不是更严谨? 否则的话,中断后再次启动时, (第一个)入口地址仍会被添加到队列及写入到文件中. 但是现在有另外一个问题存在,如第一遍全部抓取完毕了(通过spider.getStatus==Stopped判断),休眠24小时,再来抓取(通过递归调用抓取方法). 这时不同于中断后再启动,lineReader==cursor, 于是初始化时队列为空,入口地址又在urls集合中了, 故导致抓取线程马上就结束了.这样的话就没有办法去抓取网站上的新增内容了. 解决方案一: 判断抓取完毕后,紧接着覆盖cursor文件,第二次来抓取时,curosr为0, 于是将urls.txt中的所有url均放入队列中了, 可以通过这些url来发现新增url. 方案二: 对方案一进行优化,方案一虽然可以满足业务要求,但会做很多无用功,如仍会对所有旧target url进行下载,抽取,持久化等操作.而新增的内容一般都会在HelpUrl中, 比如某一页多了一个新帖子,或者多了几页内容. 故第二遍及以后来爬取时可以仅将HelpUrl放入队列中. 希望能给予反馈,我上述理解对不对, 有什么没有考虑到的情况或者有更简单的方案?谢谢!	11 years ago
yihua.huang	42a30074c9	update urls.contains to DuplicateRemover in FileCacheQueueScheduler #157	11 years ago
zhugw	1db940a088	Update FileCacheQueueScheduler.java ... 在使用过程中发现urls.txt文件存在重复URL的情况,经跟踪源代码,发现初始化加载文件后,读取所有的url放入一集合中,但是之后添加待抓取URL时并未判断是否已存在该集合中(即文件中)了,故导致文件中重复URL的情况.故据此对源码做了修改,还请作者审阅.	11 years ago
yihua.huang	3734865a6a	fix package name =.=	11 years ago
yihua.huang	e7668e01b8	fix SourceRegion error and add some tests on it #144	11 years ago
yihua.huang	4e5ba02020	fix test cont'	11 years ago
yihua.huang	2fd8f05fe2	change path seperator for varient OS #139	11 years ago
yihua.huang	8c33be48a6	Merge branch 'stable' of github.com:code4craft/webmagic	11 years ago
yihua.huang	5f8c3fd5c5	update version	11 years ago
yihua.huang	928f98dd93	auto create folder in JsonFilePipeline #122	11 years ago
yihua.huang	7fbe18b8c0	implementation of PageMapper #120	11 years ago
yihua.huang	5dc9fe95a9	interface of PageMapper #120	11 years ago
yihua.huang	7668731f08	update version to snapshot	11 years ago
yihua.huang	182dd51689	Merge branch 'stable' of github.com:code4craft/webmagic	11 years ago
yihua.huang	81e6e772ac	versions back to 0.5.1	11 years ago
yihua.huang	feb604da87	Merge branch 'stable' of github.com:code4craft/webmagic	11 years ago
yihua.huang	358e906379	[maven-release-plugin] prepare for next development iteration	11 years ago
yihua.huang	470750fc0d	[maven-release-plugin] prepare release WebMagic-0.5.1	11 years ago
yihua.huang	186b90512e	refactor redisscheduler #118	11 years ago