You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
gitea/tests/integration
Jason Song 33cc5837a6
Support compression for Actions logs (#31761)
Support compression for Actions logs to save storage space and
bandwidth. Inspired by
https://github.com/go-gitea/gitea/issues/24256#issuecomment-1521153015

The biggest challenge is that the compression format should support
[seekable](https://github.com/facebook/zstd/blob/dev/contrib/seekable_format/zstd_seekable_compression_format.md).
So when users are viewing a part of the log lines, Gitea doesn't need to
download the whole compressed file and decompress it.

That means gzip cannot help here. And I did research, there aren't too
many choices, like bgzip and xz, but I think zstd is the most popular
one. It has an implementation in Golang with
[zstd](https://github.com/klauspost/compress/tree/master/zstd) and
[zstd-seekable-format-go](https://github.com/SaveTheRbtz/zstd-seekable-format-go),
and what is better is that it has good compatibility: a seekable format
zstd file can be read by a regular zstd reader.

This PR introduces a new package `zstd` to combine and wrap the two
packages, to provide a unified and easy-to-use API.

And a new setting `LOG_COMPRESSION` is added to the config, although I
don't see any reason why not to use compression, I think's it's a good
idea to keep the default with `none` to be consistent with old versions.

`LOG_COMPRESSION` takes effect for only new log files, it adds `.zst` as
an extension to the file name, so Gitea can determine if it needs
decompression according to the file name when reading. Old files will
keep the format since it's not worth converting them, as they will be
cleared after #31735.

<img width="541" alt="image"
src="https://github.com/user-attachments/assets/e9598764-a4e0-4b68-8c2b-f769265183c9">
6 months ago
..
migration-test Preserve BOM in web editor (#28935) 1 year ago
schemas
README.md Revert "Support SAML authentication (#25165)" (#29358) 11 months ago
README_ZH.md Require MySQL 8.0, PostgreSQL 12, MSSQL 2012 (#27337) 1 year ago
actions_trigger_test.go Add permission check when creating PR (#31033) 6 months ago
admin_config_test.go Fix admin config page error, use tests to cover the admin config and 500 error page (#24965) 2 years ago
admin_user_test.go User details page (#26713) 1 year ago
api_actions_artifact_test.go Fix uploaded artifacts should be overwritten (#28726) 1 year ago
api_actions_artifact_v4_test.go Actions Artifacts v4 backend (#28965) 11 months ago
api_activitypub_person_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
api_admin_org_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
api_admin_test.go Add result check in TestAPIEditUser (#29674) 9 months ago
api_branch_test.go Recommend/convert to use case-sensitive collation for MySQL/MSSQL (#28662) 1 year ago
api_comment_attachment_test.go Catch and handle unallowed file type errors in issue attachment API (#30791) 9 months ago
api_comment_test.go Add user blocking (#29028) 11 months ago
api_feed_user_test.go
api_fork_test.go
api_gitignore_templates_test.go Add API for gitignore templates (#22783) 2 years ago
api_gpg_keys_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
api_helper_for_declarative_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
api_httpsig_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
api_issue_attachment_test.go Catch and handle unallowed file type errors in issue attachment API (#30791) 9 months ago
api_issue_config_test.go Update misspell to 0.5.1 and add `misspellings.csv` (#30573) 9 months ago
api_issue_label_test.go Support using label names when changing issue labels (#30943) 9 months ago
api_issue_milestone_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
api_issue_pin_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
api_issue_reaction_test.go Add user blocking (#29028) 11 months ago
api_issue_stopwatch_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
api_issue_subscription_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
api_issue_templates_test.go Refactor issue template parsing and fix API endpoint (#29069) 12 months ago
api_issue_test.go Fix no edit history after editing issue's title and content (#30814) 9 months ago
api_issue_tracked_time_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
api_keys_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
api_label_templates_test.go Add API for Label templates (#24602) 2 years ago
api_license_templates_test.go Add API for License templates (#23009) 2 years ago
api_nodeinfo_test.go Allow non-admin users to delete review requests (#29057) 11 months ago
api_notification_test.go Enable more `revive` linter rules (#30608) 9 months ago
api_oauth2_apps_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
api_org_avatar_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
api_org_test.go Allow non-admin users to delete review requests (#29057) 11 months ago
api_packages_alpine_test.go Integrate alpine `noarch` packages into other architectures index (#29137) 11 months ago
api_packages_cargo_test.go Run `go generate` and `go vet` on all packages (#30529) 10 months ago
api_packages_chef_test.go Resolve lint for unused parameter and unnecessary type arguments (#30750) 9 months ago
api_packages_composer_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
api_packages_conan_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
api_packages_conda_test.go Support compression for Actions logs (#31761) 6 months ago
api_packages_container_test.go Remove sub-path from container registry realm (#31293) 8 months ago
api_packages_cran_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
api_packages_debian_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
api_packages_generic_test.go Azure blob storage support (#30995) 8 months ago
api_packages_goproxy_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
api_packages_helm_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
api_packages_maven_test.go Avoid 500 panic error when uploading invalid maven package file (#31014) 9 months ago
api_packages_npm_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
api_packages_nuget_test.go Fix NuGet Package API for $filter with Id equality (#31188) 8 months ago
api_packages_pub_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
api_packages_pypi_test.go Fix links in PyPI Simple Repository API page (#30594) 10 months ago
api_packages_rpm_test.go Add signature support for the RPM module (#27069) 6 months ago
api_packages_rubygems_test.go Improve rubygems package registry (#31357) 8 months ago
api_packages_swift_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
api_packages_test.go Use `crypto/sha256` (#29386) 11 months ago
api_packages_vagrant_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
api_private_serv_test.go Final round of `db.DefaultContext` refactor (#27587) 1 year ago
api_pull_commits_test.go api: GetPullRequestCommits: return file list (#27483) 1 year ago
api_pull_review_test.go Allow non-admin users to delete review requests (#29057) 11 months ago
api_pull_test.go Add permission check when creating PR (#31033) 6 months ago
api_releases_test.go Protected tag is no internal server error (#30962) 9 months ago
api_repo_archive_test.go Add an immutable tarball link to archive download headers for Nix (#31139) 8 months ago
api_repo_avatar_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
api_repo_branch_test.go Disable query token param in integration tests (#28592) 1 year ago
api_repo_collaborator_test.go Add user blocking (#29028) 11 months ago
api_repo_compare_test.go feat(api): implement branch/commit comparison API (#30349) 10 months ago
api_repo_edit_test.go Add merge style `fast-forward-only` (#28954) 12 months ago
api_repo_file_create_test.go Move context from modules to services (#29440) 11 months ago
api_repo_file_delete_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
api_repo_file_get_test.go Redesign Scoped Access Tokens (#24767) 2 years ago
api_repo_file_helpers.go Fix Bug in Issue Config when only contact links are set (#26521) 1 year ago
api_repo_file_update_test.go Move context from modules to services (#29440) 11 months ago
api_repo_files_change_test.go Move context from modules to services (#29440) 11 months ago
api_repo_get_contents_list_test.go Simplify how git repositories are opened (#28937) 1 year ago
api_repo_get_contents_test.go Simplify how git repositories are opened (#28937) 1 year ago
api_repo_git_blobs_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
api_repo_git_commits_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
api_repo_git_hook_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
api_repo_git_notes_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
api_repo_git_ref_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
api_repo_git_tags_test.go Resolve lint for unused parameter and unnecessary type arguments (#30750) 9 months ago
api_repo_git_trees_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
api_repo_hook_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
api_repo_languages_test.go
api_repo_lfs_locks_test.go Fix #31185 try fix lfs download from bitbucket failed (#31201) 8 months ago
api_repo_lfs_migrate_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
api_repo_lfs_test.go Fix #31185 try fix lfs download from bitbucket failed (#31201) 8 months ago
api_repo_raw_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
api_repo_secrets_test.go feat(api): enhance Actions Secrets Management API for repository (#30656) 9 months ago
api_repo_tags_test.go Add `lint-go-gopls` (#30729) 8 months ago
api_repo_teams_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
api_repo_test.go Don't only list code-enabled repositories when using repository API (#30817) 9 months ago
api_repo_topic_test.go Refactor topic Find functions and add more tests for pagination (#30127) 10 months ago
api_repo_variables_test.go Add API for `Variables` (#29520) 10 months ago
api_settings_test.go
api_team_test.go Allow everyone to read or write a wiki by a repo unit setting (#30495) 10 months ago
api_team_user_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
api_token_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
api_twofa_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
api_user_avatar_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
api_user_block_test.go Add user blocking (#29028) 11 months ago
api_user_email_test.go Unify user update methods (#28733) 1 year ago
api_user_follow_test.go Add user blocking (#29028) 11 months ago
api_user_heatmap_test.go Refactor timeutil package (#28623) 1 year ago
api_user_info_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
api_user_org_perm_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
api_user_orgs_test.go Fix wrong display of recently pushed notification (#25812) 9 months ago
api_user_search_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
api_user_secrets_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
api_user_star_test.go Add user blocking (#29028) 11 months ago
api_user_update_test.go Add an api test for updating user (#30539) 10 months ago
api_user_variables_test.go Clarify Actions resources ownership (#31724) 6 months ago
api_user_watch_test.go Add user blocking (#29028) 11 months ago
api_wiki_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
attachment_test.go Use fetch to send requests to create issues/comments (#25258) 2 years ago
auth_ldap_test.go allow synchronizing user status from OAuth2 login providers (#31572) 7 months ago
avatar.png Add API for changing Avatars (#25369) 2 years ago
benchmarks_test.go Replace deprecated `math/rand` functions (#30733) 9 months ago
branches_test.go Refactor locale&string&template related code (#29165) 12 months ago
change_default_branch_test.go
cmd_keys_test.go Refactor to use urfave/cli/v2 (#25959) 2 years ago
compare_test.go Simplify 404/500 page (#31409) 7 months ago
cors_test.go Refactor CORS handler (#28587) 1 year ago
create_no_session_test.go Refactor tests (#26464) 1 year ago
csrf_test.go
db_collation_test.go Partially enable MSSQL case-sensitive collation support (#29238) 11 months ago
delete_user_test.go
download_test.go
dump_restore_test.go Add `lint-go-gopls` (#30729) 8 months ago
editor_test.go Fix automerge will not work because of some events haven't been triggered (#30780) 9 months ago
empty_repo_test.go Fix wrong display of recently pushed notification (#25812) 9 months ago
eventsource_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
explore_repos_test.go Improve repo search UI (#29767) 11 months ago
explore_user_test.go Remove unnecessary ".Link" usages (#29909) 11 months ago
git_clone_wiki_test.go Use raw Wiki links for non-renderable Wiki files (#30273) 10 months ago
git_helper_for_declarative_test.go Fix push multiple branches error with tests (#31151) 8 months ago
git_push_test.go Fix push multiple branches error with tests (#31151) 8 months ago
git_smart_http_test.go
git_test.go Allow force push to protected branches (#28086) 7 months ago
goget_test.go Support SSH for go get (#24664) 2 years ago
gpg_git_test.go Add `lint-go-gopls` (#30729) 8 months ago
html_helper.go
incoming_email_test.go Next round of `db.DefaultContext` refactor (#27089) 1 year ago
integration_test.go Refactor names (#31405) 8 months ago
issue_test.go Prevent simultaneous editing of comments and issues (#31053) 8 months ago
lfs_getobject_test.go Remove GetByBean method because sometimes it's danger when query condition parameter is zero and also introduce new generic methods (#28220) 1 year ago
lfs_local_endpoint_test.go
lfs_view_test.go
linguist_test.go Add support for `linguist-detectable` and `linguist-documentation` (#29267) 11 months ago
links_test.go Simplify 404/500 page (#31409) 7 months ago
markup_external_test.go Run `go generate` and `go vet` on all packages (#30529) 10 months ago
migrate_test.go Upgrade xorm to new version which supported update join for all supported databases (#28590) 1 year ago
mirror_pull_test.go Move migration functions to services layer (#29497) 11 months ago
mirror_push_test.go Move context from modules to services (#29440) 11 months ago
nonascii_branches_test.go Refactor route path normalization (#31381) 8 months ago
oauth_test.go add `username` to OIDC introspection response (#31688) 6 months ago
org_count_test.go Use db.Find instead of writing methods for every object (#28084) 1 year ago
org_project_test.go Fix data-race during testing (#30999) 9 months ago
org_team_invite_test.go Update team invitation email link (#26550) 1 year ago
org_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
private-testing.key
privateactivity_test.go Convert to url auth to header auth in tests (#28484) 1 year ago
project_test.go Rename project board -> column to make the UI less confusing (#30170) 8 months ago
pull_compare_test.go Fix wrong display of recently pushed notification (#25812) 9 months ago
pull_create_test.go Fix wrong display of recently pushed notification (#25812) 9 months ago
pull_diff_test.go Add commits dropdown in PR files view and allow commit by commit review (#25528) 2 years ago
pull_merge_test.go Fix wrong display of recently pushed notification (#25812) 9 months ago
pull_review_test.go Fix wrong display of recently pushed notification (#25812) 9 months ago
pull_status_test.go Fix wrong display of recently pushed notification (#25812) 9 months ago
pull_update_test.go Performance improvements for pull request list page (#29900) 11 months ago
release_test.go Restyle release list, fix branch dropdown (#30837) 9 months ago
rename_branch_test.go Fix rename branch 500 when the target branch is deleted but exist in database (#30430) 10 months ago
repo_activity_test.go Fix wrong display of recently pushed notification (#25812) 9 months ago
repo_archive_test.go Skip gzip for some well-known compressed file types (#30796) 9 months ago
repo_branch_test.go Fix wrong display of recently pushed notification (#25812) 9 months ago
repo_commits_search_test.go Integration Test for Commit Search containing Square Brackets (#28751) 1 year ago
repo_commits_test.go Restore warning commit status (#27504) 1 year ago
repo_fork_test.go Fix wrong display of recently pushed notification (#25812) 9 months ago
repo_generate_test.go Substitute variables in path names of template repos too (#25294) 2 years ago
repo_mergecommit_revert_test.go Run `go generate` and `go vet` on all packages (#30529) 10 months ago
repo_migrate_test.go
repo_search_test.go Resolve lint for unused parameter and unnecessary type arguments (#30750) 9 months ago
repo_tag_test.go Protected tag is no internal server error (#30962) 9 months ago
repo_test.go Fix repo home UI when there is no repo description (#30552) 9 months ago
repo_topic_test.go Refactor topic Find functions and add more tests for pagination (#30127) 10 months ago
repo_watch_test.go Replace 'userxx' with 'orgxx' in all test files when the user type is org (#27052) 1 year ago
repo_webhook_test.go Fix incorrect menu/link on webhook edit page (#29709) 11 months ago
repofiles_change_test.go Refactor names (#31405) 8 months ago
session_test.go Fix session key conflict with database keyword (#28613) 1 year ago
setting_test.go Introduce `flex-list` & `flex-item` elements for Gitea UI (#25790) 2 years ago
signin_test.go Refactor locale&string&template related code (#29165) 12 months ago
signout_test.go Use JSON response for "user/logout" (#25522) 2 years ago
signup_test.go Allow to change primary email before account activation (#29412) 11 months ago
ssh_key_test.go Redesign Scoped Access Tokens (#24767) 2 years ago
timetracking_test.go Replace 'userxx' with 'orgxx' in all test files when the user type is org (#27052) 1 year ago
user_avatar_test.go Reserve ".png" suffix for user/org names (#23992) 2 years ago
user_settings_test.go Allow disabling authentication related user features (#31535) 7 months ago
user_test.go Fix user router possbile panic (#29751) 11 months ago
version_test.go
view_test.go
webfinger_test.go Replace `interface{}` with `any` (#25686) 2 years ago
xss_test.go

README.md

Integration tests

Integration tests can be run with make commands for the appropriate backends, namely:

make test-sqlite
make test-pgsql
make test-mysql
make test-mssql

Make sure to perform a clean build before running tests:

make clean build

Run tests via local act_runner

Run all jobs

act_runner exec -W ./.github/workflows/pull-db-tests.yml --event=pull_request --default-actions-url="https://github.com" -i catthehacker/ubuntu:runner-latest

Warning: This file defines many jobs, so it will be resource-intensive and therefor not recommended.

Run single job

act_runner exec -W ./.github/workflows/pull-db-tests.yml --event=pull_request --default-actions-url="https://github.com" -i catthehacker/ubuntu:runner-latest -j <job_name>

You can list all job names via:

act_runner exec -W ./.github/workflows/pull-db-tests.yml --event=pull_request --default-actions-url="https://github.com" -i catthehacker/ubuntu:runner-latest -l

Run sqlite integration tests

Start tests

make test-sqlite

Run MySQL integration tests

Setup a MySQL database inside docker

docker run -e "MYSQL_DATABASE=test" -e "MYSQL_ALLOW_EMPTY_PASSWORD=yes" -p 3306:3306 --rm --name mysql mysql:latest #(just ctrl-c to stop db and clean the container)
docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" --rm --name elasticsearch elasticsearch:7.6.0 #(in a second terminal, just ctrl-c to stop db and clean the container)

Start tests based on the database container

TEST_MYSQL_HOST=localhost:3306 TEST_MYSQL_DBNAME=test TEST_MYSQL_USERNAME=root TEST_MYSQL_PASSWORD='' make test-mysql

Run pgsql integration tests

Setup a pgsql database inside docker

docker run -e "POSTGRES_DB=test" -p 5432:5432 --rm --name pgsql postgres:latest #(just ctrl-c to stop db and clean the container)

Start tests based on the database container

TEST_PGSQL_HOST=localhost:5432 TEST_PGSQL_DBNAME=test TEST_PGSQL_USERNAME=postgres TEST_PGSQL_PASSWORD=postgres make test-pgsql

Run mssql integration tests

Setup a mssql database inside docker

docker run -e "ACCEPT_EULA=Y" -e "MSSQL_PID=Standard" -e "SA_PASSWORD=MwantsaSecurePassword1" -p 1433:1433 --rm --name mssql microsoft/mssql-server-linux:latest #(just ctrl-c to stop db and clean the container)

Start tests based on the database container

TEST_MSSQL_HOST=localhost:1433 TEST_MSSQL_DBNAME=gitea_test TEST_MSSQL_USERNAME=sa TEST_MSSQL_PASSWORD=MwantsaSecurePassword1 make test-mssql

Running individual tests

Example command to run GPG test:

For SQLite:

make test-sqlite#GPG

For other databases(replace mssql to mysql, or pgsql):

TEST_MSSQL_HOST=localhost:1433 TEST_MSSQL_DBNAME=test TEST_MSSQL_USERNAME=sa TEST_MSSQL_PASSWORD=MwantsaSecurePassword1 make test-mssql#GPG

Setting timeouts for declaring long-tests and long-flushes

We appreciate that some testing machines may not be very powerful and the default timeouts for declaring a slow test or a slow clean-up flush may not be appropriate.

You can either:

  • Within the test ini file set the following section:
[integration-tests]
SLOW_TEST = 10s ; 10s is the default value
SLOW_FLUSH = 5S ; 5s is the default value
  • Set the following environment variables:
GITEA_SLOW_TEST_TIME="10s" GITEA_SLOW_FLUSH_TIME="5s" make test-sqlite