github-forks
/
webmagic
mirror of https://github.com/code4craft/webmagic.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
1,188 Commits
50 Branches
37 Tags
19 MiB
Commit Graph

1188 Commits (WebMagic-0.7.6)
 

Author SHA1 Message Date
yao 0aa2c3949d 刷新代理api重构，需要提供旧代理，如果依然是旧代理，才进行刷新，防止应延迟响应造成的过度刷新
yao 2e2a0fdf3e Downloader 提供刷新组件的api,方便在spider中操作
yao 19465089c3 提供异常刷新httpClient，异常可配置，重写getHttpClient代码
yao 9cc5287743 简化代码
yao 4a6441e7c5 提供出现某种异常刷新代理，异常可配置
yao ba69eba669 代理接口的修改，提供刷星代理API。downloader 下载错误时，提供request,exception,proxyProvider三个参数，
Sutra Zhou b2aa5e2677
Merge pull request from itranlin/master 
子任务可以使用不同的下载器
itranlin fc7ae9ce28 子任务可以使用不同的下载器。。。
yao 9a71f0ac92 pageCount修改
Sutra Zhou 4b902270b4 Bump version number from 0.7.3 to 0.7.4.
Sutra Zhou 1895dba696
Merge pull request from code4craft/snyk-fix-ad3d8147b4d9640ed91bfc34fbfb9e38 
[Snyk] Security upgrade com.google.guava:guava from 29.0-jre to 30.0-android
snyk-bot c87af365d6
fix: pom.xml to reduce vulnerabilities 
The following vulnerabilities are fixed with an upgrade:
- https://snyk.io/vuln/SNYK-JAVA-COMGOOGLEGUAVA-1015415
Sutra Zhou 541f421011
Merge pull request from code4craft/snyk-fix-9aefd7a47e1f776c68e7fad3e85033a4 
[Snyk] Security upgrade junit:junit from 4.13 to 4.13.1
snyk-bot 2223552aeb
fix: pom.xml to reduce vulnerabilities 
The following vulnerabilities are fixed with an upgrade:
- https://snyk.io/vuln/SNYK-JAVA-JUNIT-1017047
Sutra Zhou 50c4e8ccfe
Merge pull request from code4craft/snyk-fix-5cfb952010fd88492e4e2d5516c6237d 
[Snyk] Security upgrade org.apache.httpcomponents:httpclient from 4.5.12 to 4.5.13
snyk-bot 7f737626b1
fix: pom.xml to reduce vulnerabilities 
The following vulnerabilities are fixed with an upgrade:
- https://snyk.io/vuln/SNYK-JAVA-ORGAPACHEHTTPCOMPONENTS-1016906
Sutra Zhou b4b1df85a0 Fix TLSv1.3. Maybe we should expose a API to allow user to use org.apache.http.ssl.SSLContextBuilder. Fixes .
Sutra Zhou 94ac7ca3b6
Merge pull request from code4craft/snyk-fix-84ae966502c97386fb9c384e5874967b 
[Snyk] Security upgrade com.alibaba:fastjson from 1.2.68 to 1.2.69
snyk-bot e3b3b9afdd
fix: pom.xml to reduce vulnerabilities 
The following vulnerabilities are fixed with an upgrade:
- https://snyk.io/vuln/SNYK-JAVA-COMALIBABA-570967
Sutra Zhou 96ebe608cc
Merge pull request from bytesgo/manage-version 
build: manage plugin version & remove build WARNING
leeyazhou 9aab25f339 build: manage plugin version & remove build WARNING 
## use the new dependency of commons-io

[WARNING] The artifact org.apache.commons:commons-io:jar:1.3.2 has been
relocated to commons-io:commons-io:jar:1.3.2

## manage plugin version of maven-jar-plugin and maven-deploy-plugin

[WARNING]
[WARNING] Some problems were encountered while building the effective
model for us.codecraft:webmagic-core:jar:0.7.3
[WARNING] 'build.plugins.plugin.version' for
org.apache.maven.plugins:maven-jar-plugin is missing. @
us.codecraft:webmagic-parent:0.7.3, /opt/code/git/webmagic/pom.xml, line
263, column 21
[WARNING]
[WARNING] Some problems were encountered while building the effective
model for us.codecraft:webmagic-extension:jar:0.7.3
[WARNING] 'build.plugins.plugin.version' for
org.apache.maven.plugins:maven-jar-plugin is missing. @
us.codecraft:webmagic-parent:0.7.3, /opt/code/git/webmagic/pom.xml, line
263, column 21
[WARNING]
[WARNING] Some problems were encountered while building the effective
model for us.codecraft:webmagic-scripts:jar:0.7.3
[WARNING] 'build.plugins.plugin.version' for
org.apache.maven.plugins:maven-jar-plugin is missing. @ line 61, column
21
[WARNING]
[WARNING] Some problems were encountered while building the effective
model for us.codecraft:webmagic-selenium:jar:0.7.3
[WARNING] 'build.plugins.plugin.version' for
org.apache.maven.plugins:maven-jar-plugin is missing. @
us.codecraft:webmagic-parent:0.7.3, /opt/code/git/webmagic/pom.xml, line
263, column 21
[WARNING] 'build.plugins.plugin.version' for
org.apache.maven.plugins:maven-deploy-plugin is missing. @ line 34,
column 12
[WARNING]
[WARNING] Some problems were encountered while building the effective
model for us.codecraft:webmagic-saxon:jar:0.7.3
[WARNING] 'build.plugins.plugin.version' for
org.apache.maven.plugins:maven-jar-plugin is missing. @
us.codecraft:webmagic-parent:0.7.3, /opt/code/git/webmagic/pom.xml, line
263, column 21
[WARNING] 'build.plugins.plugin.version' for
org.apache.maven.plugins:maven-deploy-plugin is missing. @ line 34,
column 21
[WARNING]
[WARNING] Some problems were encountered while building the effective
model for us.codecraft:webmagic-samples:jar:0.7.3
[WARNING] 'build.plugins.plugin.version' for
org.apache.maven.plugins:maven-jar-plugin is missing. @
us.codecraft:webmagic-parent:0.7.3, /opt/code/git/webmagic/pom.xml, line
263, column 21
[WARNING]
[WARNING] Some problems were encountered while building the effective
model for us.codecraft:webmagic-parent:pom:0.7.3
[WARNING] 'build.plugins.plugin.version' for
org.apache.maven.plugins:maven-jar-plugin is missing. @ line 263, column
21
[WARNING]
[WARNING] It is highly recommended to fix these problems because they
threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support
building such malformed projects.
Sutra Zhou 48bc73fbff New method Proxy#create.
Sutra Zhou 6d3f2d9b64 Wrap URISyntaxException as IllegalArgumentException for Proxy#toURI.
Sutra Zhou 236e5ade44 Update Proxy#toString().
Sutra Zhou 791323a5b0 Add Proxy#scheme.
Sutra Zhou 2413366adb Format code, no actual code changed.
Sutra Zhou 5d14efc50f Serialize request URL only in FileCacheQueueScheduler.
Sutra Zhou 7945c0612d Merge branch 'master' of github.com:code4craft/webmagic
Sutra Zhou 71aa04c89f Upgrade dependencies.
Sutra Zhou 6c05dd8b72 Upgrade maven plugins.
Sutra Zhou a7c4e701e4 Specify the required minimum maven version.
Sutra Zhou f24410d7bf
Merge pull request from Diffblue-benchmarks/add-NumberUtils-tests 
Add unit tests for us.codecraft.webmagic.utils.NumberUtils
Sutra Zhou c966a2015c
Merge pull request from code4craft/snyk-fix-cec34e294bd875db1e52fde5b4b853f1 
[Snyk] Fix for 1 vulnerable dependencies
Sutra Zhou 98281ab26e Upgrade httpclient from 4.5.2 to 4.5.12.
Sutra Zhou 0af0badd25
Merge pull request from code4craft/snyk-fix-e3a42154e495a09f41dc3b2a6e6c9327 
[Snyk] Fix for 15 vulnerable dependencies
Sutra Zhou c97dcfd93b
Merge pull request from code4craft/snyk-fix-f95684b6c6d9efe03af61ca1db60f547 
[Snyk] Security upgrade com.github.dreamhead:moco-core from 0.11.0 to 1.0.0
Sutra Zhou e0d98df20c
Merge pull request from code4craft/snyk-fix-a77ec7619ec5042ae3cc039eeb4b728b 
[Snyk] Security upgrade com.github.dreamhead:moco-core from 0.11.0 to 1.0.0
Sutra Zhou 750d656ae9
Merge pull request from code4craft/snyk-fix-c555226a9da1135c3953aafa295a803d 
[Snyk] Security upgrade com.github.dreamhead:moco-core from 0.11.0 to 1.0.0
Sutra Zhou 06386a72ce
Merge pull request from code4craft/snyk-fix-7d7c3e630152d251e6b5661ebcb7903f 
[Snyk] Security upgrade com.github.dreamhead:moco-core from 0.11.0 to 1.0.0
Sutra Zhou 97738aee46
Merge pull request from code4craft/snyk-fix-991184deb41d150aace33c2c66628d25 
[Snyk] Security upgrade com.github.dreamhead:moco-core from 0.11.0 to 1.0.0
Sutra Zhou d1cb985b57
Merge pull request from code4craft/snyk-fix-3eaeb3b8db2efeeb547c4607ef121a7c 
[Snyk] Fix for 1 vulnerabilities
Sutra Zhou 155825088c
Merge pull request from code4craft/snyk-fix-136a1b205198304fbf279cba094ed737 
[Snyk] Fix for 1 vulnerabilities
Sutra Zhou 410daa94e1
Merge pull request from code4craft/snyk-fix-858549a4b6d8c4cb61600052a08ab66d 
[Snyk] Fix for 1 vulnerabilities
Sutra Zhou 86cc3c8659
Merge pull request from code4craft/snyk-fix-e41912d25ca79fe77c7dc4748b211c9a 
[Snyk] Fix for 1 vulnerabilities
Sutra Zhou fe9dca1247 Upgrade guava from 15.0 to 29.0.
Sutra Zhou d7f8c1d862
Merge pull request from code4craft/snyk-fix-ac48823ab9abac16eb16ea3ff73b9be2 
[Snyk] Fix for 2 vulnerabilities
Sutra Zhou ab5ac9d796 Fix test failure and javadoc failure.
Sutra Zhou 4078766d0e Change log level of supportedProtocols.
Sutra Zhou 3510e74d3f Travis supports openjdk relase number is 9 to 15 now.
Sutra Zhou 3e42523141 TLSv1.3 requires Java 11 at least. Fixes .