Commit Graph

309 Commits (5e8ca02ec670e18f52361296072929fc0a93efc3)

Author SHA1 Message Date
yihua.huang 5e8ca02ec6 contributor 11 years ago
yihua.huang 5f8c3fd5c5 update version 11 years ago
yihua.huang 7a64847a3c Bugfix: selector does not works well in element #113 11 years ago
yihua.huang 8d67fd0357 change back return proxy from spider to httpclientdownloader #128 11 years ago
yihua.huang 40bf8ca58f change return proxy from spider to httpclientdownloader #128 11 years ago
yihua.huang 1f21d9cc14 spell mistake fix #128 11 years ago
Yihua Huang e310139d00 Merge pull request #128 from yxssfxwzy/proxy
多个代理的管理
11 years ago
yihua.huang b165090434 Bugfix:Type convert error in JsonPathSelector #129 11 years ago
yihua.huang a5d1b56e44 fix ut #113 11 years ago
yihua.huang 3939074a23 Bugfix: nodes() only return the first element #113 11 years ago
yihua.huang 41c2ea9498 refactor of selectable cont' #113
1. remove lazy init of Html
2. rename strings to sourceTexts for better meaning
3. make getSourceTexts abstract and DO NOT always store strings
4. instead store parsed elements of document in HtmlNode
11 years ago
yihua.huang f9825c214a refactor selectable for html fragment #113 11 years ago
yihua.huang 03d26c169b Enhance auto charset detect #126
1. Only read from content once to fix stream closed exception
2. invite moco as server test
11 years ago
zwf c146e2c7b4 add proxy pool 11 years ago
yihua.huang 21982d3460 remove cpdetector temporary #126 11 years ago
fengwuze fcbfb75608 修改自动从网页中获取字符的代码块,抽取出来成为单独的方法。 11 years ago
fengwuze 95494d3c4d 增加处理meta的逻辑。
遗留:
3、网页没有指定编码的情况下,需要采用cpdetector,但目前cpdetector这个在Maven的中央库里面没有,不清楚如何解决。
11 years ago
yihua.huang dde2d89bbe Ignore content in json when bracket when remove padding #124 11 years ago
ywooer 259f0a16c5 Update FilePipeline.java 11 years ago
ywooer 26d38851b5 add charset to Writer 11 years ago
yihua.huang 7668731f08 update version to snapshot 11 years ago
yihua.huang 81e6e772ac versions back to 0.5.1 11 years ago
yihua.huang 358e906379 [maven-release-plugin] prepare for next development iteration 11 years ago
yihua.huang 470750fc0d [maven-release-plugin] prepare release WebMagic-0.5.1 11 years ago
yihua.huang 01aec7e1ab extension point of geturl #118 11 years ago
yihua.huang ec1c2e8cbc test and so on 11 years ago
yihua.huang 4f22f1210e some bug fix #118 11 years ago
yihua.huang 56f033ce8d set setDuplicateRemover for chain api #118 11 years ago
yihua.huang d1140b9e29 add bloom filter for scheduler #118 11 years ago
yihua.huang 8e4814bdc5 fix path seperator 11 years ago
yihua.huang e8d4a9be2b fix remove duplicate error #117 11 years ago
yihua.huang a08d8cb167 update verion 11 years ago
yihua.huang 42a2676e8c update version 11 years ago
yihua.huang c25b32f1ca [maven-release-plugin] prepare for next development iteration 11 years ago
yihua.huang 7ff83bb11a [maven-release-plugin] prepare release WebMagic-0.5.0 11 years ago
yihua.huang 1104122979 more abstraction in scheduler 11 years ago
yihua.huang 2770811a10 update monitor example 11 years ago
yihua.huang 5ecd909ef2 add timeout for wait/notify #111 11 years ago
yihua.huang c7afdb516e remove thread utils #110 11 years ago
yihua.huang 17e95f2a7f comments 11 years ago
yihua.huang 05eb7831b6 refactor and comments #110 11 years ago
yihua.huang 375e64e845 more monitor status 11 years ago
yihua.huang 018061d2cd fix error in thread pool 11 years ago
yihua.huang cdc423f2bf log 11 years ago
yihua.huang c6661899fd new thread pool #110 11 years ago
yihua.huang 179baa7a22 return when page is null 11 years ago
yihua.huang 0336f4cdb4 remove IllegalStateException when download error for less error log 11 years ago
yihua.huang 11ba5beb42 [refactor]move monitor to webmagic-extension #98 11 years ago
yihua.huang d61f65cef8 update mbean to mxbean #98 11 years ago
yihua.huang ad6a273b12 update test url 11 years ago