Commit Graph

1140 Commits (revert-1082-common-downloader-status-process)
 

Author SHA1 Message Date
Sutra Zhou acfbd7b883
Revert "Common the downloader status process and pass error information when …"
Sutra Zhou ee5a0585d7
Merge pull request from vioao/common-downloader-status-process
Common the downloader status process and pass error information when …
vio.ao d01f26333b Common the downloader status process and pass error information when onError
Sutra Zhou 16221e391d Fix xhtml namespace.
Sutra Zhou 6747046b6c
Merge pull request from yokelsoft/develop
change dependency versions into properties
David Hsing 54da7af17e change dependency versions into properties
change dependency versions into properties
update commons-collections from 3.x to 4.4
Sutra Zhou 692605bd75 Merge branch 'develop' of github.com:code4craft/webmagic into develop
Sutra Zhou 34da2fb3a0 Make PageProcessor#getSite be default method. Closes .
Sutra Zhou f110147b62
Merge pull request from carl-don-it/perfect_spider_run
perfect Spider.run to avoid some rare concurrent issue, change the Sp…
carl.don:tjr fcdb9074d6 格式化 Spider.run 代码
carl.don:tjr ab5d81a6b6 perfect Spider.run to avoid some rare concurrent issue, change the Spider.emptySleepTime to long type
Sutra Zhou c5a037a807 Update for next development version
Sutra Zhou 4024230e17 Merge branch 'release/0.7.5' into develop
Sutra Zhou 113eaa4bae Bump version number to 0.7.5.
Sutra Zhou 4e51a4f68b Update versions for release
Sutra Zhou e3e66fb270 Upgrade webmagic-samples dependencies.
Sutra Zhou 14b09a3385 Update maven plugin versions.
Sutra Zhou 31a00f5f8e Set gitflow-maven-plugin versionTagPrefix.
Sutra Zhou db70b6e095 Add maven reports.
Sutra Zhou c7828f44ac
Merge pull request from gkorland/patch-1
Update to Jedis 3.6.0
Guy Korland 189c5962e6
Update to Jedis 3.6.0
Sutra Zhou dba0ddb92c Remove unknown tag from javadoc.
Sutra Zhou 133106a15c
Merge pull request from FreaxLin/Restore_the_already_crawled_example
提交可恢复爬取内容例子
linweisen 76f625c02e 提交可恢复爬取内容例子
Sutra Zhou be6f5ff771 Add missing @Deprecated annotations.
Sutra Zhou 4e8a086dae Pass exception to onError. Fixes .
Sutra Zhou dcfd238413 Polish java version setting.
Sutra Zhou 59fc16101b
Merge pull request from thebirdandfish/develop
增加了List<SpiderStatusMXBean>属性的get方法,供SpiderMonitor的子类获取.
wecandoitjustthink 528a8908af 增加了List<SpiderStatusMXBean>属性的get方法,供SpiderMonitor的子类获取.
Sutra Zhou 71b7dfbf9a
Merge pull request from yqia182/master
SpiderStatus中getPagePerSecond()方法,增加验证逻辑,避免空指针,避免除数为零。
JustThink 54127318a4 SpiderStatus中getPagePerSecond()方法,增加验证逻辑,避免空指针,避免除数为零。
Sutra Zhou d92dc8397f Upgrade htmlcleaner from 2.5 to 2.9, this is the highest version to let Xpath2Selector pass the test cases.
Sutra Zhou 124c52b988 Downgrade htmlcleaner from 2.24 back to 2.5, to make Xpath2Selector pass the test cases.
Sutra Zhou 683db09133 Complete testXPath2 assertion.
Sutra Zhou 2f71f7912c Fix scm tag.
Sutra Zhou d0e2776991 Upgrade xsoup from 0.3.1 to 0.3.2.
Sutra Zhou 0e01550a79 Upgrade dependencies, including the jedis from 2.9.3 to 3.4.1.
Sutra Zhou 0d73f08ef6 Upgrade maven plugins.
Sutra Zhou e14a762632 Add gitflow-maven-plugin.
Sutra Zhou ab6ff7f809 Revert "pageCount修改"
This reverts commit 9a71f0ac92.
Sutra Zhou 30daec4803 Revert "提供出现某种异常刷新代理,异常可配置"
This reverts commit 4a6441e7c5.
Sutra Zhou d0843bee0d Revert "简化代码"
This reverts commit 9cc5287743.
Sutra Zhou 5ceccc62e0 Revert "提供异常刷新httpClient,异常可配置,重写getHttpClient代码"
This reverts commit 19465089c3.
Sutra Zhou 33e3fcdf22 Revert "代理接口的修改,提供刷星代理API。downloader 下载错误时,提供request,exception,proxyProvider三个参数,"
This reverts commit ba69eba669.
Sutra Zhou c489647c4b Revert " Downloader 提供刷新组件的api,方便在spider中操作"
This reverts commit 2e2a0fdf3e.
Sutra Zhou 4bedd97267 Revert " 刷新代理api重构,需要提供旧代理,如果依然是旧代理,才进行刷新,防止应延迟响应造成的过度刷新"
This reverts commit 0aa2c3949d.
Sutra Zhou 3f756c9325 Revert " 代理功能扩展,对原代理提供商进行拆分,加入lombok"
This reverts commit 33906e36f4.
Sutra Zhou aabc5584b8 Revert " bug修改,对结果提供缓存能力"
This reverts commit f68795d7dd.
Sutra Zhou 57dfc7cfb3
Merge pull request from sutra/build
Remove useless imports to fix build.
Sutra Zhou 328c3e0d7d Remove useless imports to fix build.