github-forks
/
gitea
mirror of https://github.com/go-gitea/gitea.git
1
0
Fork 0
Code Projects Releases Activity
gitea/routers/web/repo
History
Jason Song 375fd15fbf
Refactor indexer () 
Refactor `modules/indexer` to make it more maintainable. And it can be
easier to support more features. I'm trying to solve some of issue
searching, this is a precursor to making functional changes.

Current supported engines and the index versions:

| engines | issues | code |
| - | - | - |
| db | Just a wrapper for database queries, doesn't need version | - |
| bleve | The version of index is **2** | The version of index is **6**
|
| elasticsearch | The old index has no version, will be treated as
version **0** in this PR | The version of index is **1** |
| meilisearch | The old index has no version, will be treated as version
**0** in this PR | - |


## Changes

### Split

Splited it into mutiple packages

```text
indexer
├── internal
│   ├── bleve
│   ├── db
│   ├── elasticsearch
│   └── meilisearch
├── code
│   ├── bleve
│   ├── elasticsearch
│   └── internal
└── issues
    ├── bleve
    ├── db
    ├── elasticsearch
    ├── internal
    └── meilisearch
```

- `indexer/interanal`: Internal shared package for indexer.
- `indexer/interanal/[engine]`: Internal shared package for each engine
(bleve/db/elasticsearch/meilisearch).
- `indexer/code`: Implementations for code indexer.
- `indexer/code/internal`: Internal shared package for code indexer.
- `indexer/code/[engine]`: Implementation via each engine for code
indexer.
- `indexer/issues`: Implementations for issues indexer.

### Deduplication

- Combine `Init/Ping/Close` for code indexer and issues indexer.
- ~Combine `issues.indexerHolder` and `code.wrappedIndexer` to
`internal.IndexHolder`.~ Remove it, use dummy indexer instead when the
indexer is not ready.
- Duplicate two copies of creating ES clients.
- Duplicate two copies of `indexerID()`.


### Enhancement

- [x] Support index version for elasticsearch issues indexer, the old
index without version will be treated as version 0.
- [x] Fix spell of `elastic_search/ElasticSearch`, it should be
`Elasticsearch`.
- [x] Improve versioning of ES index. We don't need `Aliases`:
- Gitea does't need aliases for "Zero Downtime" because it never delete
old indexes.
- The old code of issues indexer uses the orignal name to create issue
index, so it's tricky to convert it to an alias.
- [x] Support index version for meilisearch issues indexer, the old
index without version will be treated as version 0.
- [x] Do "ping" only when `Ping` has been called, don't ping
periodically and cache the status.
- [x] Support the context parameter whenever possible.
- [x] Fix outdated example config.
- [x] Give up the requeue logic of issues indexer: When indexing fails,
call Ping to check if it was caused by the engine being unavailable, and
only requeue the task if the engine is unavailable.
- It is fragile and tricky, could cause data losing (It did happen when
I was doing some tests for this PR). And it works for ES only.
- Just always requeue the failed task, if it caused by bad data, it's a
bug of Gitea which should be fixed.

---------

Co-authored-by: Giteabot <teabot@gitea.io>
..
actions Add actor and status dropdowns to run list ()
setting Support configuration variables on Gitea Actions ()
activity.go Implement FSFE REUSE for golang files ()
attachment.go Fix all possible setting error related storages and added some tests ()
blame.go Sort repo topic labels by name ()
branch.go Use the type RefName for all the needed places and fix pull mirror sync bugs ()
cherry_pick.go Rename almost all Ctx functions ()
commit.go Pass 'not' to commit count ()
compare.go Don't list root repository on compare page if pulls not allowed ()
download.go Fix all possible setting error related storages and added some tests ()
editor.go Fix blank dir message when uploading files from web editor ()
editor_test.go Refactor web package and context package ()
find.go Implement FSFE REUSE for golang files ()
helper.go Add actor and status dropdowns to run list ()
helper_test.go Add actor and status dropdowns to run list ()
http.go Redesign Scoped Access Tokens ()
http_test.go Implement FSFE REUSE for golang files ()
issue.go Refactor indexer ()
issue_content_history.go Move helpers to be prefixed with `gt-` ()
issue_dependency.go Add API to manage issue dependencies ()
issue_label.go Make label templates have consistent behavior and priority ()
issue_label_test.go Refactor web package and context package ()
issue_lock.go Fix improper HTMLURL usages in Go code ()
issue_pin.go Followup to pinned Issues ()
issue_stopwatch.go Speed up HasUserStopwatch & GetActiveStopwatch ()
issue_test.go Implement FSFE REUSE for golang files ()
issue_timetrack.go Fix improper HTMLURL usages in Go code ()
issue_watch.go Rewrite logger system ()
lfs.go Introduce path Clean/Join helper functions ()
main_test.go Implement FSFE REUSE for golang files ()
middlewares.go Show outdated comments in files changed tab ()
migrate.go Make repo migration cancelable and fix various bugs ()
milestone.go Improve Gitea's web context, decouple "issue template" code into service package ()
packages.go Use dynamic package type list ()
patch.go Redirect to the commit page after applying patch ()
projects.go Merge `new project` templates into one ()
projects_test.go Refactor web package and context package ()
pull.go Show outdated comments in files changed tab ()
pull_review.go Show outdated comments in files changed tab ()
release.go Add actor and status dropdowns to run list ()
release_test.go Refactor web package and context package ()
render.go Implement FSFE REUSE for golang files ()
repo.go Fix all possible setting error related storages and added some tests ()
search.go Refactor indexer ()
setting.go Hide 'Mirror Settings' when unneeded, improve hints ()
setting_protected_branch.go Support for status check pattern ()
settings_test.go Refactor web package and context package ()
tag.go Add main landmark to templates and adjust titles ()
topic.go Implement FSFE REUSE for golang files ()
treelist.go Implement FSFE REUSE for golang files ()
view.go Show if File is Executable ()
view_test.go Implement FSFE REUSE for golang files ()
webhook.go Fix `Permission` in API returned repository struct ()
wiki.go Decouple the different contexts from each other ()
wiki_test.go Refactor web package and context package ()