[PATCH 0/4] Prepare webpage for hosting via gitlab pages
As the progress on the gitlab feature we need to do this seems good: https://gitlab.com/gitlab-org/gitlab-pages/-/issues/601 this series prepares the libvirt project to do the hosting and will allow us to validate it before switching over once the feature is enabled. Patches 1/4 and 2/4 can be pushed right away: - Patch 1/4 prepares the job to publish the page (currently via the libvirt.gitlab.io/libvirt URL each project gets automatically - Patch 2/4 adds the '_redirects' file which is populated from adapted 'mod_rewrite' rules from the current server. This file serves for validation of the rules and will allow us to validate them before switching over The last 2 patches are for the subsequent steps: - 3/4 removes the 'legacy' webpage job, this can be pushed either when we switch over to gitlab pages or when the old server will fetch the archive from the new artifacts file - 4/4 will be needed right before swithching over to gitlab pages as the redirects will need to be adapted for custom domain hosting The new 'pages' artifact from a CI run of this series can be viewed at: https://gitlab.com/pipo.sk/libvirt/-/jobs/5900957150 The 'redirect' validation (from an older version of this series) can be viewed at: https://pipo.sk.gitlab.io/libvirt/_redirects (beware that this will cause a certificate failure, as gitlab pages doesn't work well if you have a dot in your username ...) Peter Krempa (4): gitlab-ci: Configure website build to be published via gitlab pages docs: Add redirects configuration for gitlab pages gitlab-ci: Drop the legacy web page job docs: gitlab redirects: Drop '/libvirt' prefix for hosting the web through gitlab pages .gitlab-ci.yml | 80 +++++++++++++++++++++++++++++++++--------- docs/.gitlab_redirects | 54 ++++++++++++++++++++++++++++ 2 files changed, 117 insertions(+), 17 deletions(-) create mode 100644 docs/.gitlab_redirects -- 2.43.0
Modify the build process of the web page to publish the contents of the web page also via gitlab pages. The current webpage job is renamed to 'website_prebuilt_env_legacy' but preserves the naming of artifacts. This is needed to preserve the current approach where libvirt.org is hosted from the old server by pulling the artifacts. New jobs are added, which use the new artifact naming scheme needed for gitlab pages to host them. Artifacts are named 'pages' and contain a 'public' directory now. The new jobs are: - pages: - triggered on push to master - website_prebuilt_env: - triggered on any remaining trigger of 'gitlab_native_build_job_prebuilt_env' except for the push to master - artifact naming uses the new scheme - website_local_env: - triggered same way as gitlab_native_build_job_local_env - uses new naming scheme Doing the above allows us to have gitlab pages coexist with the old setup until etiher switching to gitlab pages completely (once the remote redirects feature gets enabled for the main instance) or changing the old server to pull data from the new job artifact location: https://gitlab.com/libvirt/libvirt/-/jobs/artifacts/master/download?job=page... Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- .gitlab-ci.yml | 86 +++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 81 insertions(+), 5 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 8bad761b32..6c7d54c311 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -78,7 +78,8 @@ include: - .cross_build_job - .gitlab_cross_build_job_local_env - +# LEGACY web page build +# # This artifact published by this job is downloaded by libvirt.org to # be deployed to the web root: # https://gitlab.com/libvirt/libvirt/-/jobs/artifacts/master/download?job=webs... @@ -97,7 +98,7 @@ include: paths: - website -website_prebuilt_env: +website_prebuilt_env_legacy: extends: - .website_job - .gitlab_native_build_job_prebuilt_env @@ -107,13 +108,88 @@ website_prebuilt_env: variables: NAME: almalinux-8 + +# This job builds the website and publishes it in the following ways: +# - 'pages' +# - publishing for gitlab pages +# - done only on push to the default branch +# - 'website*' +# - for CI purposes +# - job name based on how it was invoked: +# - 'website_prebuilt_env' +# - when the CI runs on the container images build by the upstream libvirt project +# - 'website_local_env' +# - when run in locally-rebuilt environment +# +# The job exposes artifacts containing a directory named 'public'. +# +.pages_job: + script: + - source ci/jobs.sh + - run_website_build + after_script: + - test "$CI_JOB_STATUS" != "success" && exit 1; + - mv install/usr/share/doc/libvirt/html/ public + artifacts: + expose_as: 'pages' + name: 'pages' + paths: + - public + +pages: + extends: + - .pages_job + needs: + - job: x86_64-fedora-39-container + optional: true + variables: + NAME: fedora-39 + stage: builds + image: $CI_REGISTRY/$RUN_UPSTREAM_NAMESPACE/libvirt/ci-$NAME:latest + before_script: + - cat /packages.txt + rules: + - if: '$CI_PROJECT_NAMESPACE == $RUN_UPSTREAM_NAMESPACE && $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH' + when: on_success + - when: never + +website_prebuilt_env: + extends: + - .pages_job + - .gitlab_native_build_job_prebuilt_env + needs: + - job: x86_64-fedora-39-container + optional: true + variables: + NAME: fedora-39 + rules: + # forks: pushes to a branch when a pipeline run in upstream env is explicitly requested + - if: '$CI_PROJECT_NAMESPACE != $RUN_UPSTREAM_NAMESPACE && $CI_PIPELINE_SOURCE == "push" && $RUN_PIPELINE_UPSTREAM_ENV' + when: on_success + + # upstream: other web/api/scheduled pipelines targeting the default branch + - if: '$CI_PROJECT_NAMESPACE == $RUN_UPSTREAM_NAMESPACE && $CI_PIPELINE_SOURCE =~ /(web|api|schedule)/ && $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH' + when: on_success + + # upstream+forks: merge requests targeting the default branch, without CI changes + - if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == $CI_DEFAULT_BRANCH' + changes: + - ci/gitlab/container-templates.yml + - ci/containers/$NAME.Dockerfile + when: never + - if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == $CI_DEFAULT_BRANCH' + when: on_success + + # upstream+forks: that's all folks + - when: never + website_local_env: extends: - - .website_job + - .pages_job - .gitlab_native_build_job_local_env variables: - IMAGE: docker.io/library/almalinux:8 - NAME: almalinux-8 + IMAGE: registry.fedoraproject.org/fedora:39 + NAME: fedora-39 .codestyle_job: -- 2.43.0
Adapt the configuration of redirects from the server hosting libvirt.org The redirect to the 'libvirt-console-proxy' Golang module is not adapted as it doesn't exist on the current server. NOTES: - The redirects are currently configured for hosting via the 'gitlab.io/libvirt' URI. For hosting via custom domain it will need to be rewritten to drop the '/libvirt' prefixes. - gitlab pages doesn't currently support redirects to outside content, thus most of the redirects don't actually work: https://gitlab.com/gitlab-org/gitlab-pages/-/issues/601 - The redirects file is only installed in the webpage job but is not actually distributed. - The validity of the redirects can be checked by visiting: https://libvirt.gitlab.io/_redirects Having them installed allows us to validate them before we'll be switching to use gitlab pages completely. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- .gitlab-ci.yml | 1 + docs/.gitlab_redirects | 54 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 55 insertions(+) create mode 100644 docs/.gitlab_redirects diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 6c7d54c311..1ee945aa0a 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -130,6 +130,7 @@ website_prebuilt_env_legacy: after_script: - test "$CI_JOB_STATUS" != "success" && exit 1; - mv install/usr/share/doc/libvirt/html/ public + - cp docs/.gitlab_redirects public/_redirects artifacts: expose_as: 'pages' name: 'pages' diff --git a/docs/.gitlab_redirects b/docs/.gitlab_redirects new file mode 100644 index 0000000000..26e64563ee --- /dev/null +++ b/docs/.gitlab_redirects @@ -0,0 +1,54 @@ +# Redirects to golang module pages +/libvirt/libvirt-go /libvirt-go.html 200 +/libvirt/libvirt-go-xml /libvirt-go-xml.html 200 +/libvirt/go/libvirt /go/libvirt.html 200 +/libvirt/go/libvirtxml /go/libvirtxml.html 200 + +# Redirects to the download server +/libvirt/sources/* https://download.libvirt.org/:splat 301 +/libvirt/ruby/download/* https://download.libvirt.org/ruby/:splat 301 +/libvirt/maven2/* https://download.libvirt.org/maven2/:splat 301 + +# Redirects to subproject pages +/libvirt/ruby/* https://ruby.libvirt.org/:splat 301 +/libvirt/ocaml/* https://ocaml.libvirt.org/:splat 301 +/libvirt/php/* https://php.libvirt.org/:splat 301 +/libvirt/libvirt-appdev-guide-python/en-US/html/* https://libvirt.gitlab.io/libvirt-appdev-guide-python/:splat: 301 +/libvirt/java.html https://java.libvirt.org 301 +# Redirect to the proper javadoc directory on the subproject page +/libvirt/sources/java/javadoc/* https://java.libvirt.org/javadoc/:splat 301 + +# Redirects from old gitweb location (see below) +/libvirt/git https://gitlab.com/libvirt/ 301 + +# The above rules are adapted from the following set of 'mod_rewrite' rules used +# originally on libvirt.org: +# +# RewriteRule ^/libvirt-go$ /libvirt-go.html [L] +# RewriteRule ^/libvirt-go-xml$ /libvirt-go-xml.html [L] +# RewriteRule ^/go/libvirt$ /go/libvirt.html [L] +# RewriteRule ^/go/libvirtxml$ /go/libvirtxml.html [L] +# RewriteRule ^/sources/(.*) https://download.libvirt.org/$1 [L] +# RewriteRule ^/ruby/download/(.*) https://download.libvirt.org/ruby/$1 [L] +# RewriteRule ^/(maven2/.*) https://download.libvirt.org/$1 [L] +# RewriteRule ^/ocaml/(.*) https://ocaml.libvirt.org/$1 [L] +# RewriteRule ^/ruby/(.*) https://ruby.libvirt.org/$1 [L] +# RewriteRule ^/php/(.*) https://php.libvirt.org/$1 [L] +# RewriteRule ^/java.html https://java.libvirt.org [L] +# RewriteRule ^/docs/libvirt-appdev-guide-python/en-US/html/(.*) https://libvirt.gitlab.io/libvirt-appdev-guide-python/$1 [L] +# RewriteRule ^/git https://gitlab.com/libvirt/ [L] +# +# Redirect replacing 'gitweb'. The 'gitweb' interface was originally replaced +# by the following redirect condition: +# +# RewriteCond %{QUERY_STRING} p=([-a-zA-Z0-9]+).git +# RewriteRule ^/git/$ https://gitlab.com/libvirt/%1 [L] +# +# That unfortunately can't be represented in gitlab redirects as it doesn't +# support redirects based on query strings. Given that the above redirect broke +# most gitweb links anyways, due to handling only the 'p=' argument, git gitlab +# redirect will break the rest of them. +# +# The following rule was dropped as the page never existed: +# +# RewriteRule ^/libvirt-console-proxy$ /libvirt-console-proxy.html [L] -- 2.43.0
On Wed, Jan 10, 2024 at 02:30:23PM +0100, Peter Krempa wrote:
Adapt the configuration of redirects from the server hosting libvirt.org
The redirect to the 'libvirt-console-proxy' Golang module is not adapted as it doesn't exist on the current server.
NOTES: - The redirects are currently configured for hosting via the 'gitlab.io/libvirt' URI. For hosting via custom domain it will need to be rewritten to drop the '/libvirt' prefixes.
- gitlab pages doesn't currently support redirects to outside content, thus most of the redirects don't actually work:
https://gitlab.com/gitlab-org/gitlab-pages/-/issues/601
- The redirects file is only installed in the webpage job but is not actually distributed.
- The validity of the redirects can be checked by visiting:
https://libvirt.gitlab.io/_redirects
Having them installed allows us to validate them before we'll be switching to use gitlab pages completely.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- .gitlab-ci.yml | 1 + docs/.gitlab_redirects | 54 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 55 insertions(+) create mode 100644 docs/.gitlab_redirects
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 6c7d54c311..1ee945aa0a 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -130,6 +130,7 @@ website_prebuilt_env_legacy: after_script: - test "$CI_JOB_STATUS" != "success" && exit 1; - mv install/usr/share/doc/libvirt/html/ public + - cp docs/.gitlab_redirects public/_redirects artifacts: expose_as: 'pages' name: 'pages' diff --git a/docs/.gitlab_redirects b/docs/.gitlab_redirects new file mode 100644 index 0000000000..26e64563ee --- /dev/null +++ b/docs/.gitlab_redirects @@ -0,0 +1,54 @@
+/libvirt/maven2/* https://download.libvirt.org/maven2/:splat 301
Maven has this unhelpful behaviour where by when you add libvirt.org as a maven repository host, it'll now consult libvirt.org for *all* Java JARs, not solely the libvirt JARs. For all of last week we were serving about 1 million hits a day for /maven, of which the libvirt JAR requests were about 0.5%, the rest were 404s ! This is why libvirt.org was painfully slow for most of last week. This was all made worse by us redirecting from libvirt.org to download.libvirt.org, needlessly doubling the number of requests had to serve. I changed the /maven2/* redirect to /maven2/org/libvirt/* which lets libvirt.org service an immediate 404, avoiding hitting download.libvirt.org, cutting our requests by 50%. Fortunately traffic has died down to more normal levels after the "back to work mass code recompile" spike has gone. Anyway lets do the same for this gitlab redirect to keep the bogus maven traffic for non-existent files off download.libvirt.org With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Wed, Jan 10, 2024 at 15:02:25 +0000, Daniel P. Berrangé wrote:
On Wed, Jan 10, 2024 at 02:30:23PM +0100, Peter Krempa wrote:
Adapt the configuration of redirects from the server hosting libvirt.org
The redirect to the 'libvirt-console-proxy' Golang module is not adapted as it doesn't exist on the current server.
NOTES: - The redirects are currently configured for hosting via the 'gitlab.io/libvirt' URI. For hosting via custom domain it will need to be rewritten to drop the '/libvirt' prefixes.
- gitlab pages doesn't currently support redirects to outside content, thus most of the redirects don't actually work:
https://gitlab.com/gitlab-org/gitlab-pages/-/issues/601
- The redirects file is only installed in the webpage job but is not actually distributed.
- The validity of the redirects can be checked by visiting:
https://libvirt.gitlab.io/_redirects
Having them installed allows us to validate them before we'll be switching to use gitlab pages completely.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- .gitlab-ci.yml | 1 + docs/.gitlab_redirects | 54 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 55 insertions(+) create mode 100644 docs/.gitlab_redirects
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 6c7d54c311..1ee945aa0a 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -130,6 +130,7 @@ website_prebuilt_env_legacy: after_script: - test "$CI_JOB_STATUS" != "success" && exit 1; - mv install/usr/share/doc/libvirt/html/ public + - cp docs/.gitlab_redirects public/_redirects artifacts: expose_as: 'pages' name: 'pages' diff --git a/docs/.gitlab_redirects b/docs/.gitlab_redirects new file mode 100644 index 0000000000..26e64563ee --- /dev/null +++ b/docs/.gitlab_redirects @@ -0,0 +1,54 @@
+/libvirt/maven2/* https://download.libvirt.org/maven2/:splat 301
Maven has this unhelpful behaviour where by when you add libvirt.org as a maven repository host, it'll now consult libvirt.org for *all* Java JARs, not solely the libvirt JARs.
For all of last week we were serving about 1 million hits a day for /maven, of which the libvirt JAR requests were about 0.5%, the rest were 404s ! This is why libvirt.org was painfully slow for most of last week.
This was all made worse by us redirecting from libvirt.org to download.libvirt.org, needlessly doubling the number of requests had to serve.
I changed the /maven2/* redirect to /maven2/org/libvirt/* which lets libvirt.org service an immediate 404, avoiding hitting download.libvirt.org, cutting our requests by 50%. Fortunately traffic has died down to more normal levels after the "back to work mass code recompile" spike has gone.
Anyway lets do the same for this gitlab redirect to keep the bogus maven traffic for non-existent files off download.libvirt.org
Ah, right. I've captured the redirect list before the holliday break (actually the original version is from April) so it didn't have this one and didn't bother to check once more. The final check needs to be done before switching over anyways. I've changed the rule in my branch to: /libvirt/maven2/org/libvirt* https://download.libvirt.org/maven2/org/libvirt/:splat 301
The web page is now consumed via the 'pages' job and thus we don't need to keep the old one around. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- .gitlab-ci.yml | 31 ------------------------------- 1 file changed, 31 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 1ee945aa0a..90b1cfc3e4 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -78,37 +78,6 @@ include: - .cross_build_job - .gitlab_cross_build_job_local_env -# LEGACY web page build -# -# This artifact published by this job is downloaded by libvirt.org to -# be deployed to the web root: -# https://gitlab.com/libvirt/libvirt/-/jobs/artifacts/master/download?job=webs... -.website_job: - script: - - source ci/jobs.sh - - run_website_build - after_script: - - test "$CI_JOB_STATUS" != "success" && exit 1; - - mv install/usr/share/doc/libvirt/html/ website - artifacts: - expose_as: 'Website' - name: 'website' - when: on_success - expire_in: 30 days - paths: - - website - -website_prebuilt_env_legacy: - extends: - - .website_job - - .gitlab_native_build_job_prebuilt_env - needs: - - job: x86_64-almalinux-8-container - optional: true - variables: - NAME: almalinux-8 - - # This job builds the website and publishes it in the following ways: # - 'pages' # - publishing for gitlab pages -- 2.43.0
The prefix needs to be dropped for the redirects to work properly once hosting 'libvirt.org' via gitlab pages. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- docs/.gitlab_redirects | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/docs/.gitlab_redirects b/docs/.gitlab_redirects index 26e64563ee..64b4843821 100644 --- a/docs/.gitlab_redirects +++ b/docs/.gitlab_redirects @@ -1,25 +1,25 @@ # Redirects to golang module pages -/libvirt/libvirt-go /libvirt-go.html 200 -/libvirt/libvirt-go-xml /libvirt-go-xml.html 200 -/libvirt/go/libvirt /go/libvirt.html 200 -/libvirt/go/libvirtxml /go/libvirtxml.html 200 +/libvirt-go /libvirt-go.html 200 +/libvirt-go-xml /libvirt-go-xml.html 200 +/go/libvirt /go/libvirt.html 200 +/go/libvirtxml /go/libvirtxml.html 200 # Redirects to the download server -/libvirt/sources/* https://download.libvirt.org/:splat 301 -/libvirt/ruby/download/* https://download.libvirt.org/ruby/:splat 301 -/libvirt/maven2/* https://download.libvirt.org/maven2/:splat 301 +/sources/* https://download.libvirt.org/:splat 301 +/ruby/download/* https://download.libvirt.org/ruby/:splat 301 +/maven2/* https://download.libvirt.org/maven2/:splat 301 # Redirects to subproject pages -/libvirt/ruby/* https://ruby.libvirt.org/:splat 301 -/libvirt/ocaml/* https://ocaml.libvirt.org/:splat 301 -/libvirt/php/* https://php.libvirt.org/:splat 301 -/libvirt/libvirt-appdev-guide-python/en-US/html/* https://libvirt.gitlab.io/libvirt-appdev-guide-python/:splat: 301 -/libvirt/java.html https://java.libvirt.org 301 +/ruby/* https://ruby.libvirt.org/:splat 301 +/ocaml/* https://ocaml.libvirt.org/:splat 301 +/php/* https://php.libvirt.org/:splat 301 +/libvirt-appdev-guide-python/en-US/html/* https://libvirt.gitlab.io/libvirt-appdev-guide-python/:splat: 301 +/java.html https://java.libvirt.org 301 # Redirect to the proper javadoc directory on the subproject page -/libvirt/sources/java/javadoc/* https://java.libvirt.org/javadoc/:splat 301 +/sources/java/javadoc/* https://java.libvirt.org/javadoc/:splat 301 # Redirects from old gitweb location (see below) -/libvirt/git https://gitlab.com/libvirt/ 301 +/git https://gitlab.com/libvirt/ 301 # The above rules are adapted from the following set of 'mod_rewrite' rules used # originally on libvirt.org: -- 2.43.0
participants (2)
-
Daniel P. Berrangé -
Peter Krempa