Commit Graph

1026 Commits (3510e74d3f024a30aaf7355be7eb3b035c53fc3e)
 

Author SHA1 Message Date
Yihua Huang 0cd2f6031a Merge pull request from GZhY/master
对 WebMagic-0.7.0 的小修补
GZhY 5f34adf938 完善 LinksSelector.selectList 的测试用例
GZhY ce3f0ac239 删除 fixAllRelativeHrefs 并修复 SeleniumDownloader 对 fixAllRelativeHrefs 的依赖
GZhY bc6e81e00f 修复checkElementAndConvert方法注释中注释错误
yihua.huang 4a2c0f4f97 add returnProxy for proxyProvider
yihua.huang 1b04a7f2b3 move logic check from downloaderto spider
Yihua Huang 6ead04a758 Merge pull request from code4craft/proxyRefactor
HttpClient部分重构
yihua.huang 0f4d6e8b12 remove port in UrlUtils.getDomain()
yihua.huang a1ae632b62 test for request cookies and headers
yihua.huang db67db8103 remove fixAllRelativeHrefs by default, get absolute urls for links()
yihua.huang abd020b45b some comments
yihua.huang 2622b448b8 fix test
yihua.huang b06a248c00 fix test
yihua.huang 1cfbd13aae refacor in httpclientdownloader
yihua.huang 83ada9749e fix test
yihua.huang fe95a6842f Request再次重构:去掉params,仅保留HttpRequestBody
yihua.huang 395396c68e 增加HttpRequestBody
Yihua Huang 25df6650d9 Merge pull request from xbynet/master
Request支持设置header与cookie、新增POST请求时,XML、JSON参数支持、Page支持获取响应header
xbynet c93a8a2722 修复字符编码检测BUG
yihua.huang 74110e6ec5 remove useless file
yihua.huang b100dfe273 update version
xbynet@outlook.com 1c24baa8d1 Request支持设置header与cookie
新增POST请求时,XML、JSON参数支持
Page支持获取响应header
xbynet 5824e9515b Merge pull request from code4craft/master
合并最新代码
yihua.huang 6bd197859b fix test
yihua.huang f23e138c72 add response headers to Page
yihua.huang ba000b364c remove site.addStartUrl in samples
yihua.huang 3824232171 remove site.addStartUrl in samples
yihua.huang 8564d51e56 remove site.addStartUrl in samples
yihua.huang c13110c4cb fix samples
yihua.huang c51ac6017c remove Site.addStartRequest() etc.
yihua.huang 68050fc88e test pass
yihua.huang 474b7c9d57 refactor
yihua.huang 25c81013ca new proxy pool api
yihua.huang 46297deaa1 HttpUriRequestConverter
yihua.huang 1d86f7c048 compile passed in httpclientDownloader
yihua.huang b71f379512 fix
yihua.huang a7f9e7cad5 重构一部分httpclient
yihua.huang 221c155060 move release connection before return proxy
yihua.huang 68beff42c5 add test
Yihua Huang d6b4e98a66 Merge pull request from cheiftain/my_change
Bug, add null check to site in HttpClientDownloader & HttpClientGener…
xbynet 76729c9302 Merge pull request from code4craft/master
合并官方最新代码
wuyifan 79522f941e Bug, add null check to site in HttpClientDownloader & HttpClientGenerator
yihua.huang e9341d0291 complete test
yihua.huang e7d35c4846 add params to all method of request
yihua.huang 75bad591d7 rewrite hashCode and equals for params
Yihua Huang 11c32669b2 Merge pull request from xbynet/master
简化POST参数设置.
yihua.huang aa01e27779 change constructor for Proxy to public
yihua.huang 0fbf657d86 update fastjson to 1.2.28
Yihua Huang 676758349f Merge pull request from eyougo/master
fix a bug of RegexSelector when regex has zero-width assertions.
mei 791520e6a0 fix a bug of RegexSelector when regex has zero-width assertions.