github-forks
/
webmagic
mirror of https://github.com/code4craft/webmagic.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

Commit Graph

  • 9fa071232a
         Merge e4ff496615 into 1cd199b160 apaqi 2025-01-17 18:38:23 +0000
  • 96f955edbe
         Merge 8b54094514 into 1cd199b160 suym845 2025-01-17 18:38:13 +0000
  • 35e314a3a8
         Merge 0b3fa6cf1e into 1cd199b160 suym845 2025-01-17 18:38:08 +0000
  • a332bfc3ba
         Merge bba7d7184d into 1cd199b160 qingmo 2025-01-17 18:38:05 +0000
  • 5ed967a54e
         Merge 5e863d28d4 into 1cd199b160 Yanzhi 2025-01-17 18:38:03 +0000
  • eadd168612
         Merge 4c8296a3b9 into 7efc9872df Taher Ghaleb 2025-01-17 18:37:54 +0000
  • 71a8ce07ab
         Merge a694e02cb9 into 7efc9872df Yang 2025-01-17 18:37:52 +0000
  • 8d663205d2
         Merge 4bedabcec2 into 1cd199b160 weilunhui 2025-01-17 18:37:51 +0000
  • 1e85d23951
         Merge 78c914512a into 1cd199b160 weilunhui 2025-01-17 18:37:43 +0000
  • f2b61f68d9
         Merge 77997ca14b into 1cd199b160 qingmo 2025-01-17 18:37:40 +0000
  • cb101373d5
         Merge 80b93fb4cc into 1cd199b160 406965097 2025-01-07 09:59:23 +0800
  • fe772f4f56
         Merge 9247eda745 into 1cd199b160 Sean Gu 2025-01-07 09:59:12 +0800
  • 2a9b638a0e
         Merge b02a9efcf4 into 7efc9872df Niu_XZ 2025-01-07 09:58:12 +0800
  • f4314f5640
         Merge 8531438bc9 into 7efc9872df GG22G2 2025-01-03 14:51:46 +0100
  • 7efc9872df
         Update LICENSE, fix license year (#1186) develop Jason N. White 2024-12-31 12:10:32 -0600
  • b4f5f76528
         Update LICENSE, fix license year JasonnnW3000 2024-12-31 13:01:35 -0500
  • 78ffe7ee7e
         Merge 36d70901be into 1cd199b160 张露兵 2024-12-27 14:51:28 +0800
  • 42a172729e Remove useless modifier. Joe Zhou 2024-12-22 16:48:59 +0800
  • b1cf7ae455 Set page status code for SeleniumDownloader. Fixes #1185 Joe Zhou 2024-12-18 21:05:25 +0800
  • 9bb2417f58
         修改SmartContentSelector threshold可定制化 (#1183) zyw61483 2024-12-11 16:36:20 +0800
  • 76f418ab6d 修改SmartContentSelector threshold可定制化 zhaoyiwei 2024-12-11 16:18:59 +0800
  • 697d9d2b1b 修改SmartContentSelector threshold可定制化 zhaoyiwei 2024-12-11 16:16:06 +0800
  • 1cd199b160
         fix: webmagic-scripts/pom.xml to reduce vulnerabilities (#1181) master Yihua Huang 2024-12-07 13:13:37 +0800
  • 5739c1d977
         fix: webmagic-scripts/pom.xml to reduce vulnerabilities snyk-fix-1ecbed5d36d45ae4bf2e417c53344d18 snyk-bot 2024-12-07 01:59:34 +0000
  • 8dc417452a Update for next development version Joe Zhou 2024-11-27 04:28:50 +0800
  • 851063bce5 Merge tag 'WebMagic-1.0.2' into develop Joe Zhou 2024-11-27 04:28:46 +0800
  • 837253cfc9 Merge branch 'release/1.0.2' WebMagic-1.0.2 Joe Zhou 2024-11-27 04:28:46 +0800
  • bf1088bd67 Update versions for release Joe Zhou 2024-11-27 04:16:05 +0800
  • c20edb8246 Polish code. Joe Zhou 2024-11-27 03:31:30 +0800
  • 0a9fe8d3e0 Add static methods to construct Page. Joe Zhou 2024-11-27 00:49:03 +0800
  • 7e500d7b95
         Updated README.md, fix typo(s) (#1180) Bob Conan 2024-11-22 20:24:58 -0600
  • ddf5553a49 Updated README.md, fix typo(s) BobConanDev 2024-11-22 14:38:38 -0500
  • 6ed83769e0 Update for next development version Joe Zhou 2024-10-26 01:37:09 +0800
  • 63e675066f Merge tag 'WebMagic-1.0.1' into develop Joe Zhou 2024-10-26 01:37:07 +0800
  • e98f7d722a Merge branch 'release/1.0.1' WebMagic-1.0.1 Joe Zhou 2024-10-26 01:37:06 +0800
  • 6eab7a4155 Update versions for release Joe Zhou 2024-10-26 01:02:00 +0800
  • 50026ff937
         Bump commons-io:commons-io from 2.11.0 to 2.14.0 (#1179) dependabot[bot] 2024-10-09 23:38:56 +0800
  • 598f7efb13
         Bump commons-io:commons-io from 2.11.0 to 2.14.0 dependabot[bot] 2024-10-09 15:37:19 +0000
  • 541ced9eea Change the default status code from 200 to 0 & downloadSuccess from true to false, for Page. Joe Zhou 2024-10-09 23:36:02 +0800
  • 19f60bf340
         fix: pom.xml to reduce vulnerabilities (#1178) Yihua Huang 2024-10-05 10:42:57 +0800
  • 8687889e5c
         fix: pom.xml to reduce vulnerabilities snyk-fix-0532fcb31120c3d1c843be536233a913 snyk-bot 2024-10-04 11:58:21 +0000
  • b722b2675e
         Merge 9b46b7cf0f into 15ec80fcf1 VivianDelannoyEtu 2024-08-21 13:25:28 +0800
  • e03de561ae
         Merge 35f5384a60 into 9d75cce16d l1fan 2024-08-21 13:25:27 +0800
  • 15ec80fcf1
         FileCacheQueueScheduler使用BloomFilter进行去重 (#1176) xiezcGitHub 2024-08-19 13:05:28 +0800
  • 74094fa115 FileCacheQueueScheduler使用BloomFilter进行去重 xiezc 2024-08-16 15:34:03 +0800
  • 8ac88d95f3 FileCacheQueueScheduler使用BloomFilter进行去重 xiezhicheng 2024-08-16 11:31:42 +0800
  • 2c135dadce
         #1172 问题的解决 (#1173) xiezcGitHub 2024-08-06 19:29:41 +0800
  • 582ea6e098 #1172 问题的解决 xiezhicheng 2024-08-06 18:45:10 +0800
  • 25c2d95e96 Update for next development version Joe Zhou 2024-07-05 00:27:54 +0800
  • c88330f25d Merge tag 'WebMagic-1.0.0' into develop Joe Zhou 2024-07-05 00:27:50 +0800
  • 9d75cce16d Merge branch 'release/1.0.0' WebMagic-1.0.0 Joe Zhou 2024-07-05 00:27:50 +0800
  • 3e9cd9b5c3 Update versions for release Joe Zhou 2024-07-05 00:20:28 +0800
  • 4d0cdb011f
         stopWhenComplete，增加动态修改完成时停止方法。 (#1169) Niu_XZ 2024-06-17 17:27:28 +0800
  • b02a9efcf4 修改download成功后对状态码的处理，如果状态码不被site.acceptStatCode接收的话就算失败，进行doCycleRetry重试逻辑。并且加入队列前先sleep。 niuxiaozu 2024-06-17 15:34:03 +0800
  • 278403c188 stopWhenComplete，增加动态修改完成时停止方法。 niuxiaozu 2024-06-17 15:24:47 +0800
  • 29506fe81f
         Update README.md Niu_XZ 2024-06-17 14:16:38 +0800
  • 3ace9cae8b
         Update README.md Niu_XZ 2024-06-17 14:16:04 +0800
  • 2474f91ea5 1、修改site.sleepTime的意义，sleep的模式为当前spider所有请求都间隔site.sleepTime。 2、修改download成功后对状态码的处理，如果状态码不被site.acceptStatCode接收的话就算失败，进行doCycleRetry重试逻辑。 3、stopWhenComplete，增加动态修改完成时停止方法。 niuxiaozu 2024-06-17 11:08:36 +0800
  • 973642363b
         Merge 56bf87825f into f7fdcd025f WindAI 2024-06-13 12:34:02 +0000
  • 49a5efff46 Add a private constructor to hide the implicit public one. Joe Zhou 2024-06-04 01:02:45 +0800
  • 5c43e36118 Make sure the contentType of detectCharset could be null. Joe Zhou 2024-06-04 00:59:30 +0800
  • d2aebc60a7 Make getCharset to support null parameter. Joe Zhou 2024-06-04 00:57:28 +0800
  • 7d2d2244b3 Upgrade oxerr-parent from 2.1.0 to 2.2.1. Sutra Zhou 2024-05-21 12:55:05 +0800
  • 16a4fe3e28 Use oxerr-parent instead. Sutra Zhou 2024-05-17 13:17:13 +0800
  • 35a02066ff
         Merge f64400c072 into f7fdcd025f Yihua Huang 2024-05-14 18:14:36 +0800
  • 4ee30c0592 Merge tag 'WebMagic-0.10.3' into develop Sutra Zhou 2024-04-23 23:41:18 +0800
  • 38f240c42e Update to hotfix version Sutra Zhou 2024-04-23 23:39:23 +0800
  • f7fdcd025f Merge branch 'hotfix/0.10.3' WebMagic-0.10.3 Sutra Zhou 2024-04-23 23:39:16 +0800
  • 462c60fef2 Fix for entity is null. Sutra Zhou 2024-04-23 23:37:46 +0800
  • 54aef0f0e0 Update versions for hotfix Sutra Zhou 2024-04-23 23:34:15 +0800
  • 6be02f3a0f Merge tag 'WebMagic-0.10.2' into develop Sutra Zhou 2024-04-23 21:47:57 +0800
  • a81c4e7627 Update to hotfix version Sutra Zhou 2024-04-23 21:46:35 +0800
  • 0dc852e219 Merge branch 'hotfix/0.10.2' WebMagic-0.10.2 Sutra Zhou 2024-04-23 21:46:28 +0800
  • 0dbfaf7284 Remove useless log. Sutra Zhou 2024-04-23 21:34:59 +0800
  • bda51537b6 Update versions for hotfix Sutra Zhou 2024-04-23 21:32:04 +0800
  • 2217b1c8f2 Merge tag 'WebMagic-0.10.1' into develop Sutra Zhou 2024-04-23 20:41:02 +0800
  • 884f51ba3b Update to hotfix version Sutra Zhou 2024-04-23 20:38:03 +0800
  • 2994716ba5 Merge branch 'hotfix/0.10.1' WebMagic-0.10.1 Sutra Zhou 2024-04-23 20:37:56 +0800
  • 78740a4e28 Fix NPE. Sutra Zhou 2024-04-23 20:26:38 +0800
  • 5afe3e7b81 Revert "fix: pom.xml to reduce vulnerabilities (#1134)" Sutra Zhou 2024-04-23 20:21:45 +0800
  • a0ff4a2d2e Fix log message. Sutra Zhou 2024-04-23 20:09:15 +0800
  • 76ef0332f1 Update versions for hotfix Sutra Zhou 2024-04-23 20:06:47 +0800
  • a5144350bd Upgrade maven plugins to latest versions. Joe Zhou 2024-04-22 00:45:54 +0800
  • e34b495625 Upgrade maven-suirefire-plugin from 3.0.0-M7 to 3.2.5. Joe Zhou 2024-04-22 00:17:58 +0800
  • 5344db0106 Upgrade jacoco-maven-plugin from 0.8.8 to 0.8.12. Joe Zhou 2024-04-21 23:35:26 +0800
  • 9ab342c3a7 Remove public modifiers from junit5 test methods. Joe Zhou 2024-04-21 23:31:21 +0800
  • 5196a56ccf Format code. Joe Zhou 2024-04-21 23:30:43 +0800
  • dba1668306 Add tests to test the equals & hashCode of Proxy. Joe Zhou 2024-04-21 23:23:32 +0800
  • b7e0d360ec Upgrade junit from 4.13.2 to 5.10.2. Joe Zhou 2024-04-21 22:04:48 +0800
  • d8321baf56
         Refactored and implement of a template method pattern for logger config in webmagic-scripts (#1158) François Gibier 2024-04-06 01:55:46 +0200
  • aff8097b02
         Merge branch 'code4craft:develop' into develop François Gibier 2024-04-05 21:53:46 +0200
  • 16a9dc5479 Refactored and implement of a template method pattern for logger config François Gibier 2024-04-05 21:47:39 +0200
  • d5c7406bd9 add lombok for getters and setters François Gibier 2024-04-05 21:28:55 +0200
  • 2df7dca871
         Changed refactor of processSingle again, this one is a better version (#1157) François Gibier 2024-04-05 16:50:21 +0200
  • d66abe8904 Changed my refactor of processSingle, this one is a lot better François Gibier 2024-04-05 16:42:59 +0200
  • 1face99eca
         Merge branch 'develop' into develop François Gibier 2024-04-05 16:39:22 +0200
  • 4964d5ae33 Changed my refactor of processSingle, this one is a lot better François Gibier 2024-04-05 16:34:45 +0200
  • 05e5eefc7d
         Refactor of processSingle in PageModelExtractor (#1155) François Gibier 2024-04-05 15:51:08 +0200
  • 5f8fb5be51 Refactor of processSingle in PageModelExtractor François Gibier 2024-04-05 15:25:15 +0200
  • f10fabcb58 Update .gitignore, with merging Maven.gitignore & Global/Eclipse.gitignore in github/gitignore. Joe Zhou 2024-04-03 20:21:02 +0800