github-forks
/
webmagic
mirror of https://github.com/code4craft/webmagic.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

Commit Graph

  • 97738aee46
         Merge pull request #914 from code4craft/snyk-fix-991184deb41d150aace33c2c66628d25 Sutra Zhou 2020-05-29 14:38:08 +0800
  • d1cb985b57
         Merge pull request #911 from code4craft/snyk-fix-3eaeb3b8db2efeeb547c4607ef121a7c Sutra Zhou 2020-05-29 14:37:45 +0800
  • 155825088c
         Merge pull request #908 from code4craft/snyk-fix-136a1b205198304fbf279cba094ed737 Sutra Zhou 2020-05-29 14:37:24 +0800
  • 410daa94e1
         Merge pull request #899 from code4craft/snyk-fix-858549a4b6d8c4cb61600052a08ab66d Sutra Zhou 2020-05-29 14:37:00 +0800
  • 86cc3c8659
         Merge pull request #897 from code4craft/snyk-fix-e41912d25ca79fe77c7dc4748b211c9a Sutra Zhou 2020-05-29 14:36:21 +0800
  • fe9dca1247 Upgrade guava from 15.0 to 29.0. Sutra Zhou 2020-05-29 14:28:34 +0800
  • d7f8c1d862
         Merge pull request #895 from code4craft/snyk-fix-ac48823ab9abac16eb16ea3ff73b9be2 Sutra Zhou 2020-05-29 14:01:08 +0800
  • ab5ac9d796 Fix test failure and javadoc failure. Sutra Zhou 2020-05-28 19:05:17 +0800
  • 4078766d0e Change log level of supportedProtocols. Sutra Zhou 2020-05-28 14:30:14 +0800
  • 3510e74d3f Travis supports openjdk relase number is 9 to 15 now. Sutra Zhou 2020-05-28 14:18:53 +0800
  • 3e42523141 TLSv1.3 requires Java 11 at least. Fixes #927. Sutra Zhou 2020-05-28 14:14:16 +0800
  • 436af97346 Use spaces as indent. Sutra Zhou 2020-05-28 13:59:05 +0800
  • 423d7558c3
         Merge 99eecb02cf into ba1b4017a7 Xenios 2020-05-27 18:36:05 +0000
  • ba1b4017a7 Mark slf4j-log4j12 as optional. Sutra Zhou 2020-05-21 19:59:29 +0800
  • 30667f4687 Remove oss-parent setting as it is no longer active. Sutra Zhou 2020-05-21 18:51:37 +0800
  • 605c120ebf fix: webmagic-selenium/pom.xml to reduce vulnerabilities snyk-fix-f96b01419b9ec0f23ce96beb068b24af snyk-bot 2020-05-14 09:31:14 +0800
  • c77914886d
         Merge 84f1914900 into e7476cb8dc LeoAugust19 2020-04-30 11:36:56 -0400
  • d53315cb26
         Merge 9d6dd7e170 into e7476cb8dc darknt 2020-04-30 12:29:24 -0300
  • e7476cb8dc Make Request#getExtra be generic. Sutra Zhou 2020-04-19 22:44:06 +0800
  • c46400d126 Fix javadoc of sleep time. Fixes #918. Sutra Zhou 2020-04-12 01:30:57 +0800
  • b98a87e45a Serialize requests in FileCacheQueueScheduler, so that the extra info of request could be restored. Sutra Zhou 2020-04-11 20:21:20 +0800
  • fe3d52e2a4 Add TLSv1.3 support. Sutra Zhou 2020-04-11 18:00:04 +0800
  • d78a94f7f7
         Merge pull request #658 from aristotll/patch-1 Sutra Zhou 2020-04-11 17:51:30 +0800
  • 8327e482c9 fix: pom.xml to reduce vulnerabilities snyk-fix-f95684b6c6d9efe03af61ca1db60f547 snyk-bot 2020-04-09 02:58:45 +0000
  • 38eedabef9 fix: pom.xml to reduce vulnerabilities snyk-fix-a77ec7619ec5042ae3cc039eeb4b728b snyk-bot 2020-04-01 02:58:50 +0000
  • 95da41f08b fix: pom.xml to reduce vulnerabilities snyk-fix-c555226a9da1135c3953aafa295a803d snyk-bot 2020-03-28 02:58:44 +0000
  • c79cd1dfa0 fix: pom.xml to reduce vulnerabilities snyk-fix-7d7c3e630152d251e6b5661ebcb7903f snyk-bot 2020-03-22 02:58:48 +0000
  • c1cc949459 第三次通过git提交到仓库 HaiMing123456 2020-03-19 18:56:27 +0800
  • f3a577710e
         创建php3.php HaiMing123456 2020-03-19 15:28:57 +0800
  • cc2d7af70a fix: pom.xml to reduce vulnerabilities snyk-fix-991184deb41d150aace33c2c66628d25 snyk-bot 2020-03-03 02:59:56 +0000
  • 8335d18c79 fix: pom.xml to reduce vulnerabilities snyk-fix-3eaeb3b8db2efeeb547c4607ef121a7c snyk-bot 2020-02-12 02:58:51 +0000
  • 8815934f88 fix: pom.xml to reduce vulnerabilities snyk-fix-136a1b205198304fbf279cba094ed737 snyk-bot 2020-01-04 02:58:56 +0000
  • 09fb39f431 fix: pom.xml to reduce vulnerabilities snyk-fix-858549a4b6d8c4cb61600052a08ab66d snyk-bot 2019-10-15 02:58:57 +0000
  • fd6037af26 fix: pom.xml to reduce vulnerabilities snyk-fix-e41912d25ca79fe77c7dc4748b211c9a snyk-bot 2019-10-08 02:58:45 +0000
  • 7fd58a94a8 fix: pom.xml to reduce vulnerabilities snyk-fix-ac48823ab9abac16eb16ea3ff73b9be2 snyk-bot 2019-10-03 02:59:35 +0000
  • 77997ca14b
         Merge pull request #1 from code4craft/master qingmo 2019-09-25 15:58:34 +0800
  • 36d70901be 增加支持批量保存为JSON文件的Pipeline。 圣手书生 2019-09-11 12:05:42 +0800
  • 0b8fab1bfa fix: pom.xml to reduce vulnerabilities snyk-fix-e3a42154e495a09f41dc3b2a6e6c9327 snyk-test 2019-09-02 02:58:48 +0000
  • 4bedabcec2
         Update Page.java weilunhui 2019-08-21 16:47:07 +0800
  • 78c914512a
         Update Spider.java weilunhui 2019-08-19 18:02:40 +0800
  • 2fd0e192fd Add unit tests for us.codecraft.webmagic.utils.NumberUtils Thomas Perkins 2019-07-29 15:58:52 +0100
  • a44fa24c91 fix: pom.xml to reduce vulnerabilities snyk-fix-2caf795052fb80730da1ec15fccea6a8 snyk-test 2019-06-28 02:58:49 +0000
  • f1b3a29d6f fix: webmagic-selenium/pom.xml to reduce vulnerabilities snyk-fix-cec34e294bd875db1e52fde5b4b853f1 snyk-test 2019-06-28 01:31:36 +0000
  • 9247eda745 Merge remote-tracking branch 'upstream/master' VATALU 2019-03-27 10:42:52 +0800
  • a142c465d5 Merge remote-tracking branch 'origin/master' Denis Basargin 2018-11-07 11:41:05 +0300
  • c22e579cb0 vesion 0.7.5 Denis Basargin 2018-11-07 11:34:14 +0300
  • 99349dbd4c vesion 0.7.3 Denis Basargin 2018-11-07 11:34:14 +0300
  • 524835045c Fix Spider main thread any problem with redis connection (when working with redis-based schedulers) leads to unhandled exception and release the main (control) thread Denis Basargin 2018-11-07 11:33:54 +0300
  • 99eecb02cf Small change Corey 2018-09-29 21:13:07 -0500
  • ac03b903eb 修改Json#node方法，改为返回List<Json>，方便链式调用 xukeek 2018-09-30 09:36:58 +0800
  • 828398ac83 Made some bug and vulnerability fixes. Xenios91 2018-09-29 03:24:26 -0500
  • 139ed2c504 Made some bug fixes and corrected 2 XML vulnerabilities. Xenios91 2018-09-29 03:02:56 -0500
  • 01ba4c31d8
         Merge 60083bcd75 into be892b80bf Liang Tan 2018-09-27 03:09:25 +0000
  • 0b3fa6cf1e 修正Cookie在子域名无效的问题 suym845 2018-08-25 15:44:59 +0800
  • 8b54094514 支持直接用编译后的RegexSelector进行选择 suym845 2018-08-25 15:35:21 +0800
  • 5e863d28d4 fix getProxy to make poll and offer atomic 延之 2018-08-09 18:11:12 +0800
  • f272796de6 fix constract method 延之 2018-08-09 18:08:35 +0800
  • 8d3dd1ac2f
         Merge 319efc4409 into be892b80bf suym845 2018-08-07 08:43:20 +0000
  • 319efc4409
         支持直接使用RegexSelector进行正则匹配 suym845 2018-08-07 16:43:16 +0800
  • 6c46cbf81c
         支持直接使用RegexSelector进行正则匹配 suym845 2018-08-07 16:41:37 +0800
  • 6d1ff8798b add a new proxy provider named effectiveProxyProvider 延之 2018-08-07 11:17:54 +0800
  • 84f1914900 fix issue #816 LeoAugust19 2018-08-02 09:25:58 +0800
  • 78a1621831 图片下载 apaqi 2018-07-19 01:00:59 +0800
  • 7ef204135d 解决网络协议问题 apaqi 2018-07-17 01:25:08 +0800
  • e6b0ef0ca8 extend the four-component to every request mouzhanpeng 2018-07-09 18:04:43 +0800
  • e5bcf7376e
         Merge 5d94ea1922 into be892b80bf FlyingHJL 2018-07-03 13:00:59 +0000
  • 5d94ea1922 add Steam crawler FlyingHJL 2018-07-03 20:57:42 +0800
  • 60083bcd75
         Merge pull request #1 from code4craft/master Liang Tan 2018-05-24 17:24:51 -0400
  • 810fc3c2f0
         修复因组件更新产生的指定cookie在子域名页面无效的问题 suym845 2018-05-07 01:37:08 +0800
  • 264a9c55a9 upgrade dependency versions, formatted code style waincent 2018-04-18 16:08:01 +0800
  • 80b93fb4cc 修复部分网站重定向出现异常。新增基于反射的解析方法调用 13522234273@139.com 2018-04-18 11:13:26 +0800
  • f013a4b3c1 修复部分网站重定向出现异常。新增基于反射的解析方法调用 13522234273@139.com 2018-04-18 10:52:15 +0800
  • e52a0343b5
         Update HttpClientGenerator.java hupeng 2018-04-08 17:55:31 +0800
  • b74264be69
         : 是非法字符，无法作为文件名 singinginwind 2018-03-28 17:08:26 +0800
  • 1cdb7a9602
         Merge pull request #1 from code4craft/master yetianhua 2018-01-09 23:25:44 +0800
  • 81880d3dc6 Html新增text()方法，便于获取全部文本 VATALU 2017-12-20 16:42:29 +0800
  • d9e6866af9 move httpclient download 臧天才 2017-12-15 18:46:06 +0800
  • 58ea794401 Merge 984fac38ec into be892b80bf 睐笃大大 2017-12-15 08:21:51 +0000
  • 984fac38ec add customization config jitpack 臧天才 2017-12-15 16:17:57 +0800
  • 1381aa9a4a fixed package error test 臧天才 2017-12-15 15:40:03 +0800
  • ad22e6c65f Merge 4238a9b73c into f405e642c0 wanglong9 2017-12-05 11:52:26 +0000
  • 4238a9b73c 1、增加ComopositeDownloader混合Downloader下载器 以及普通http下载、phatomjs下载器、chrome headless下载器 并提供实现参数化配置 2、增加adsl自动拨号、ip代理接口 3、增加错误页面处理PageTrigger 详情看DownloaderTest.java 王龙 2017-12-05 19:44:26 +0800
  • df7cd59990 1、增加ComopositeDownloader混合Downloader下载器 以及普通http下载、phatomjs下载器、chrome headless下载器 并提供实现参数化配置 2、增加adsl自动拨号、ip代理接口 3、增加错误页面处理PageTrigger 详情看DownloaderTest.java 王龙 2017-12-05 19:43:15 +0800
  • 56bf87825f delete test xuefeng.cai 2017-12-04 10:35:56 +0800
  • 36b25fac7c ISSUE#688 增加了ChromeDriverDownloader 取代SeleniumDownloader xuefeng.cai 2017-12-03 19:29:04 +0800
  • be892b80bf update travis ci to openjdk yihua.huang 2017-12-02 10:57:06 +0800
  • fe80950ee9 Merge branch 'master' of github.com:code4craft/webmagic yihua.huang 2017-12-02 10:51:07 +0800
  • c7d1ed7d20 #fix httpcore version: change to 4.4.4 yihua.huang 2017-12-02 10:50:49 +0800
  • 0037529c87
         Merge pull request #703 from Yihy/master Yihua Huang 2017-12-02 10:43:50 +0800
  • c701fe8d38 #702 Refactor: rename CheckForAdditionalInfo to checkForAdditionalInfo yihua.huang 2017-11-30 11:50:52 +0800
  • adf5454837 [Fix] 修正错误方法名 yihy 2017-11-30 11:35:12 +0800
  • b0bf1a9ef2
         Merge pull request #702 from Yihy/master Yihua Huang 2017-11-30 11:30:39 +0800
  • 266083fa07 [Fix] #698  Repair using redis,Request additional information is lost yihy 2017-11-29 20:19:00 +0800
  • 0e3e1328ca Merge e3aa66360c into e5db538c19 念去去云 2017-11-29 05:52:32 +0000
  • e5db538c19 #647 remove ThreadSafe annotation yihua.huang 2017-11-29 13:49:40 +0800
  • b539522ca8 #701 support to tls1.2 yihua.huang 2017-11-29 13:36:19 +0800
  • e3aa66360c [Fix] #698 Repair using redis,Request additional information is lost yihy 2017-11-29 10:19:04 +0800
  • 0daef0b768 [Fix] #698 Repair using redis,Request additional information is lost yihy 2017-11-29 10:08:52 +0800
  • d7230124cc
         Merge pull request #1 from code4craft/master Windsmild 2017-11-12 16:11:46 +0800
  • e277ba05db Merge 843c3069b2 into e276b11ea9 Sherlme 2017-10-18 03:55:09 +0000