You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
 
 
 
 
Go to file
wxiaoguang 6f9c278559
Rewrite queue (#24505)
# ⚠️ Breaking

Many deprecated queue config options are removed (actually, they should
have been removed in 1.18/1.19).

If you see the fatal message when starting Gitea: "Please update your
app.ini to remove deprecated config options", please follow the error
messages to remove these options from your app.ini.

Example:

```
2023/05/06 19:39:22 [E] Removed queue option: `[indexer].ISSUE_INDEXER_QUEUE_TYPE`. Use new options in `[queue.issue_indexer]`
2023/05/06 19:39:22 [E] Removed queue option: `[indexer].UPDATE_BUFFER_LEN`. Use new options in `[queue.issue_indexer]`
2023/05/06 19:39:22 [F] Please update your app.ini to remove deprecated config options
```

Many options in `[queue]` are are dropped, including:
`WRAP_IF_NECESSARY`, `MAX_ATTEMPTS`, `TIMEOUT`, `WORKERS`,
`BLOCK_TIMEOUT`, `BOOST_TIMEOUT`, `BOOST_WORKERS`, they can be removed
from app.ini.

# The problem

The old queue package has some legacy problems:

* complexity: I doubt few people could tell how it works.
* maintainability: Too many channels and mutex/cond are mixed together,
too many different structs/interfaces depends each other.
* stability: due to the complexity & maintainability, sometimes there
are strange bugs and difficult to debug, and some code doesn't have test
(indeed some code is difficult to test because a lot of things are mixed
together).
* general applicability: although it is called "queue", its behavior is
not a well-known queue.
* scalability: it doesn't seem easy to make it work with a cluster
without breaking its behaviors.

It came from some very old code to "avoid breaking", however, its
technical debt is too heavy now. It's a good time to introduce a better
"queue" package.

# The new queue package

It keeps using old config and concept as much as possible.

* It only contains two major kinds of concepts:
    * The "base queue": channel, levelqueue, redis
* They have the same abstraction, the same interface, and they are
tested by the same testing code.
* The "WokerPoolQueue", it uses the "base queue" to provide "worker
pool" function, calls the "handler" to process the data in the base
queue.
* The new code doesn't do "PushBack"
* Think about a queue with many workers, the "PushBack" can't guarantee
the order for re-queued unhandled items, so in new code it just does
"normal push"
* The new code doesn't do "pause/resume"
* The "pause/resume" was designed to handle some handler's failure: eg:
document indexer (elasticsearch) is down
* If a queue is paused for long time, either the producers blocks or the
new items are dropped.
* The new code doesn't do such "pause/resume" trick, it's not a common
queue's behavior and it doesn't help much.
* If there are unhandled items, the "push" function just blocks for a
few seconds and then re-queue them and retry.
* The new code doesn't do "worker booster"
* Gitea's queue's handlers are light functions, the cost is only the
go-routine, so it doesn't make sense to "boost" them.
* The new code only use "max worker number" to limit the concurrent
workers.
* The new "Push" never blocks forever
* Instead of creating more and more blocking goroutines, return an error
is more friendly to the server and to the end user.

There are more details in code comments: eg: the "Flush" problem, the
strange "code.index" hanging problem, the "immediate" queue problem.

Almost ready for review.

TODO:

* [x] add some necessary comments during review
* [x] add some more tests if necessary
* [x] update documents and config options
* [x] test max worker / active worker
* [x] re-run the CI tasks to see whether any test is flaky
* [x] improve the `handleOldLengthConfiguration` to provide more
friendly messages
* [x] fine tune default config values (eg: length?)

## Code coverage:

![image](https://user-images.githubusercontent.com/2114189/236620635-55576955-f95d-4810-b12f-879026a3afdf.png)
2 years ago
.gitea Issue template form (#16349) 3 years ago
.github Check latest version on CI (#24556) 2 years ago
assets Add RPM registry (#23380) 2 years ago
build Replace placeholders in licenses (#24354) 2 years ago
cmd Rewrite queue (#24505) 2 years ago
contrib Updated upgrade script that is informing user that Gitea service has to be running in order to upgrade it (#24260) 2 years ago
custom/conf Rewrite queue (#24505) 2 years ago
docker Unify nightly naming across binaries and docker images (#24116) 2 years ago
docs Rewrite queue (#24505) 2 years ago
models Rewrite queue (#24505) 2 years ago
modules Rewrite queue (#24505) 2 years ago
options [skip ci] Updated licenses and gitignores 2 years ago
public Add RPM registry (#23380) 2 years ago
routers Rewrite queue (#24505) 2 years ago
services Rewrite queue (#24505) 2 years ago
snap Remove unnecessary whitespace in snapcraft.yaml (#22090) 2 years ago
templates Rewrite queue (#24505) 2 years ago
tests Rewrite queue (#24505) 2 years ago
web_src Add goto issue id function (#24479) 2 years ago
.air.toml Set stop_on_error (#24453) 2 years ago
.changelog.yml Changelog for v1.15.0-rc1 (#16422) 4 years ago
.dockerignore Add .dockerignore (#21753) 2 years ago
.drone.yml Upgrade to Node 20 on CI, enable actions cancellation (#24524) 2 years ago
.editorconfig Add markdownlint (#20512) 3 years ago
.eslintrc.yaml Add eslint-plugin-regexp (#24361) 2 years ago
.gitattributes Mark `/templates/swagger/v1_json.tmpl` as generated file (#24306) 2 years ago
.gitignore Remove git sample files and ignore them (#24271) 2 years ago
.gitpod.yml Remove docs from automatically building in Gitpod (#23978) 2 years ago
.golangci.yml Enable forbidigo linter (#24278) 2 years ago
.ignore Add some .ignore entries (#18296) 3 years ago
.markdownlint.yaml Add markdownlint (#20512) 3 years ago
.npmrc Upgrade to npm lockfile v3 and explicitely set it (#23561) 2 years ago
.spectral.yaml Add spectral linter for Swagger (#20321) 3 years ago
.stylelintrc.yaml Enhance stylelint rule config, remove dead CSS (#24472) 2 years ago
BSDmakefile update BSDmakefile to latest version from upstream (#24063) 2 years ago
CHANGELOG.md Changelog for 1.19.3 (#24495) (#24506) 2 years ago
CODE_OF_CONDUCT.md Add Gitea Community Code of Conduct (#23188) 2 years ago
CONTRIBUTING.md Remove .lgtm and reference to old bot (#24464) 2 years ago
DCO Remove address from DCO (#22595) 2 years ago
Dockerfile Hardcode path to docker images (#23955) 2 years ago
Dockerfile.rootless Hardcode path to docker images (#23955) 2 years ago
LICENSE Fix typo 8 years ago
MAINTAINERS Apply to become maintainer (#24254) 2 years ago
Makefile Makefile: Use portable !, not GNUish -not, with find(1). (#24565) 2 years ago
README.md Fix broken `README` link (#24546) 2 years ago
README_ZH.md link update in README files (#22582) 2 years ago
SECURITY.md Add markdownlint (#20512) 3 years ago
build.go User/Org Feed render description as per web (#23887) 2 years ago
go.mod Add RPM registry (#23380) 2 years ago
go.sum Add RPM registry (#23380) 2 years ago
main.go add CLI command to register runner tokens (#23762) 2 years ago
package-lock.json Remove `font-awesome` and fomantic `icon` module (#24471) 2 years ago
package.json Remove `font-awesome` and fomantic `icon` module (#24471) 2 years ago
playwright.config.js Update JS dependencies and eslint config (#21388) 2 years ago
vitest.config.js Update JS dependencies (#24218) 2 years ago
webpack.config.js Remove `font-awesome` and fomantic `icon` module (#24471) 2 years ago

README.md

Gitea

Gitea - Git with a cup of tea

Contribute with Gitpod

View this document in Chinese

Purpose

The goal of this project is to make the easiest, fastest, and most painless way of setting up a self-hosted Git service.

As Gitea is written in Go, it works across all the platforms and architectures that are supported by Go, including Linux, macOS, and Windows on x86, amd64, ARM and PowerPC architectures. You can try it out using the online demo. This project has been forked from Gogs since November of 2016, but a lot has changed.

Building

From the root of the source tree, run:

TAGS="bindata" make build

or if SQLite support is required:

TAGS="bindata sqlite sqlite_unlock_notify" make build

The build target is split into two sub-targets:

  • make backend which requires Go Stable, required version is defined in go.mod.
  • make frontend which requires Node.js LTS or greater and Internet connectivity to download npm dependencies.

When building from the official source tarballs which include pre-built frontend files, the frontend target will not be triggered, making it possible to build without Node.js and Internet connectivity.

Parallelism (make -j <num>) is not supported.

More info: https://docs.gitea.io/en-us/install-from-source/

Using

./gitea web

NOTE: If you're interested in using our APIs, we have experimental support with documentation.

Contributing

Expected workflow is: Fork -> Patch -> Push -> Pull Request

NOTES:

  1. YOU MUST READ THE CONTRIBUTORS GUIDE BEFORE STARTING TO WORK ON A PULL REQUEST.
  2. If you have found a vulnerability in the project, please write privately to security@gitea.io. Thanks!

Translating

Translations are done through Crowdin. If you want to translate to a new language ask one of the managers in the Crowdin project to add a new language there.

You can also just create an issue for adding a language or ask on discord on the #translation channel. If you need context or find some translation issues, you can leave a comment on the string or ask on Discord. For general translation questions there is a section in the docs. Currently a bit empty but we hope to fill it as questions pop up.

https://docs.gitea.io/en-us/contributing/translation-guidelines/

Crowdin

Further information

For more information and instructions about how to install Gitea, please look at our documentation. If you have questions that are not covered by the documentation, you can get in contact with us on our Discord server or create a post in the discourse forum.

We maintain a list of Gitea-related projects at gitea/awesome-gitea.

The Hugo-based documentation theme is hosted at gitea/theme.

The official Gitea CLI is developed at gitea/tea.

Authors

Backers

Thank you to all our backers! 🙏 [Become a backer]

Sponsors

Support this project by becoming a sponsor. Your logo will show up here with a link to your website. [Become a sponsor]

FAQ

How do you pronounce Gitea?

Gitea is pronounced /ɡɪti:/ as in "gi-tea" with a hard g.

Why is this not hosted on a Gitea instance?

We're working on it.

License

This project is licensed under the MIT License. See the LICENSE file for the full license text.

Screenshots

Looking for an overview of the interface? Check it out!

Dashboard User Profile Global Issues
Branches Web Editor Activity
New Migration Migrating Pull Request View
Pull Request Dark Diff Review Dark Diff Dark