9:30 p.m.
As the progress on the gitlab feature we need to do this seems good:
https://gitlab.com/gitlab-org/gitlab-pages/-/issues/601
this series prepares the libvirt project to do the hosting and will
allow us to validate it before switching over once the feature is
enabled.
Patches 1/4 and 2/4 can be pushed right away:
- Patch 1/4 prepares the job to publish the page (currently via the
libvirt.gitlab.io/libvirt URL each project gets automatically
- Patch 2/4 adds the '_redirects' file which is populated from adapted
'mod_rewrite' rules from the current server. This file serves for
validation of the rules and will allow us to validate them before
switching over
The last 2 patches are for the subsequent steps:
- 3/4 removes the 'legacy' webpage job, this can be pushed either when
we switch over to gitlab pages or when the old server will fetch the
archive from the new artifacts file
- 4/4 will be needed right before swithching over to gitlab pages as
the redirects will need to be adapted for custom domain hosting
The new 'pages' artifact from a CI run of this series can be viewed at:
https://gitlab.com/pipo.sk/libvirt/-/jobs/5900957150
The 'redirect' validation (from an older version of this series) can be
viewed at:
https://pipo.sk.gitlab.io/libvirt/_redirects
(beware that this will cause a certificate failure, as gitlab pages
doesn't work well if you have a dot in your username ...)
Peter Krempa (4):
gitlab-ci: Configure website build to be published via gitlab pages
docs: Add redirects configuration for gitlab pages
gitlab-ci: Drop the legacy web page job
docs: gitlab redirects: Drop '/libvirt' prefix for hosting the web
through gitlab pages
.gitlab-ci.yml | 80 +++++++++++++++++++++++++++++++++---------
docs/.gitlab_redirects | 54 ++++++++++++++++++++++++++++
2 files changed, 117 insertions(+), 17 deletions(-)
create mode 100644 docs/.gitlab_redirects
--
2.43.0
9:30 p.m.
New subject: [PATCH 1/4] gitlab-ci: Configure website build to be published via gitlab pages
Modify the build process of the web page to publish the contents of the
web page also via gitlab pages.
The current webpage job is renamed to 'website_prebuilt_env_legacy' but
preserves the naming of artifacts. This is needed to preserve the
current approach where libvirt.org is hosted from the old server by
pulling the artifacts.
New jobs are added, which use the new artifact naming scheme needed for
gitlab pages to host them.
Artifacts are named 'pages' and contain a 'public' directory now.
The new jobs are:
- pages:
- triggered on push to master
- website_prebuilt_env:
- triggered on any remaining trigger of
'gitlab_native_build_job_prebuilt_env'
except for the push to master
- artifact naming uses the new scheme
- website_local_env:
- triggered same way as gitlab_native_build_job_local_env
- uses new naming scheme
Doing the above allows us to have gitlab pages coexist with the old
setup until etiher switching to gitlab pages completely (once the remote
redirects feature gets enabled for the main instance) or changing the
old server to pull data from the new job artifact location:
https://gitlab.com/libvirt/libvirt/-/jobs/artifacts/master/download?job=p...
Signed-off-by: Peter Krempa <pkrempa(a)redhat.com>
---
.gitlab-ci.yml | 86 +++++++++++++++++++++++++++++++++++++++++++++++---
1 file changed, 81 insertions(+), 5 deletions(-)
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 8bad761b32..6c7d54c311 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -78,7 +78,8 @@ include:
- .cross_build_job
- .gitlab_cross_build_job_local_env
-
+# LEGACY web page build
+#
# This artifact published by this job is downloaded by libvirt.org to
# be deployed to the web root:
# https://gitlab.com/libvirt/libvirt/-/jobs/artifacts/master/download?job=w...
@@ -97,7 +98,7 @@ include:
paths:
- website
-website_prebuilt_env:
+website_prebuilt_env_legacy:
extends:
- .website_job
- .gitlab_native_build_job_prebuilt_env
@@ -107,13 +108,88 @@ website_prebuilt_env:
variables:
NAME: almalinux-8
+
+# This job builds the website and publishes it in the following ways:
+# - 'pages'
+# - publishing for gitlab pages
+# - done only on push to the default branch
+# - 'website*'
+# - for CI purposes
+# - job name based on how it was invoked:
+# - 'website_prebuilt_env'
+# - when the CI runs on the container images build by the upstream libvirt project
+# - 'website_local_env'
+# - when run in locally-rebuilt environment
+#
+# The job exposes artifacts containing a directory named 'public'.
+#
+.pages_job:
+ script:
+ - source ci/jobs.sh
+ - run_website_build
+ after_script:
+ - test "$CI_JOB_STATUS" != "success" && exit 1;
+ - mv install/usr/share/doc/libvirt/html/ public
+ artifacts:
+ expose_as: 'pages'
+ name: 'pages'
+ paths:
+ - public
+
+pages:
+ extends:
+ - .pages_job
+ needs:
+ - job: x86_64-fedora-39-container
+ optional: true
+ variables:
+ NAME: fedora-39
+ stage: builds
+ image: $CI_REGISTRY/$RUN_UPSTREAM_NAMESPACE/libvirt/ci-$NAME:latest
+ before_script:
+ - cat /packages.txt
+ rules:
+ - if: '$CI_PROJECT_NAMESPACE == $RUN_UPSTREAM_NAMESPACE &&
$CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH ==
$CI_DEFAULT_BRANCH'
+ when: on_success
+ - when: never
+
+website_prebuilt_env:
+ extends:
+ - .pages_job
+ - .gitlab_native_build_job_prebuilt_env
+ needs:
+ - job: x86_64-fedora-39-container
+ optional: true
+ variables:
+ NAME: fedora-39
+ rules:
+ # forks: pushes to a branch when a pipeline run in upstream env is explicitly
requested
+ - if: '$CI_PROJECT_NAMESPACE != $RUN_UPSTREAM_NAMESPACE &&
$CI_PIPELINE_SOURCE == "push" && $RUN_PIPELINE_UPSTREAM_ENV'
+ when: on_success
+
+ # upstream: other web/api/scheduled pipelines targeting the default branch
+ - if: '$CI_PROJECT_NAMESPACE == $RUN_UPSTREAM_NAMESPACE &&
$CI_PIPELINE_SOURCE =~ /(web|api|schedule)/ && $CI_COMMIT_REF_NAME ==
$CI_DEFAULT_BRANCH'
+ when: on_success
+
+ # upstream+forks: merge requests targeting the default branch, without CI changes
+ - if: '$CI_PIPELINE_SOURCE == "merge_request_event" &&
$CI_MERGE_REQUEST_TARGET_BRANCH_NAME == $CI_DEFAULT_BRANCH'
+ changes:
+ - ci/gitlab/container-templates.yml
+ - ci/containers/$NAME.Dockerfile
+ when: never
+ - if: '$CI_PIPELINE_SOURCE == "merge_request_event" &&
$CI_MERGE_REQUEST_TARGET_BRANCH_NAME == $CI_DEFAULT_BRANCH'
+ when: on_success
+
+ # upstream+forks: that's all folks
+ - when: never
+
website_local_env:
extends:
- - .website_job
+ - .pages_job
- .gitlab_native_build_job_local_env
variables:
- IMAGE: docker.io/library/almalinux:8
- NAME: almalinux-8
+ IMAGE: registry.fedoraproject.org/fedora:39
+ NAME: fedora-39
.codestyle_job:
--
2.43.0
9:30 p.m.
New subject: [PATCH 2/4] docs: Add redirects configuration for gitlab pages
Adapt the configuration of redirects from the server hosting libvirt.org
The redirect to the 'libvirt-console-proxy' Golang module is not adapted
as it doesn't exist on the current server.
NOTES:
- The redirects are currently configured for hosting via the
'gitlab.io/libvirt' URI. For hosting via custom domain it will need
to be rewritten to drop the '/libvirt' prefixes.
- gitlab pages doesn't currently support redirects to outside content,
thus most of the redirects don't actually work:
https://gitlab.com/gitlab-org/gitlab-pages/-/issues/601
- The redirects file is only installed in the webpage job but is not
actually distributed.
- The validity of the redirects can be checked by visiting:
https://libvirt.gitlab.io/_redirects
Having them installed allows us to validate them before we'll be
switching to use gitlab pages completely.
Signed-off-by: Peter Krempa <pkrempa(a)redhat.com>
---
.gitlab-ci.yml | 1 +
docs/.gitlab_redirects | 54 ++++++++++++++++++++++++++++++++++++++++++
2 files changed, 55 insertions(+)
create mode 100644 docs/.gitlab_redirects
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 6c7d54c311..1ee945aa0a 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -130,6 +130,7 @@ website_prebuilt_env_legacy:
after_script:
- test "$CI_JOB_STATUS" != "success" && exit 1;
- mv install/usr/share/doc/libvirt/html/ public
+ - cp docs/.gitlab_redirects public/_redirects
artifacts:
expose_as: 'pages'
name: 'pages'
diff --git a/docs/.gitlab_redirects b/docs/.gitlab_redirects
new file mode 100644
index 0000000000..26e64563ee
--- /dev/null
+++ b/docs/.gitlab_redirects
@@ -0,0 +1,54 @@
+# Redirects to golang module pages
+/libvirt/libvirt-go /libvirt-go.html 200
+/libvirt/libvirt-go-xml /libvirt-go-xml.html 200
+/libvirt/go/libvirt /go/libvirt.html 200
+/libvirt/go/libvirtxml /go/libvirtxml.html 200
+
+# Redirects to the download server
+/libvirt/sources/* https://download.libvirt.org/:splat 301
+/libvirt/ruby/download/* https://download.libvirt.org/ruby/:splat 301
+/libvirt/maven2/* https://download.libvirt.org/maven2/:splat 301
+
+# Redirects to subproject pages
+/libvirt/ruby/* https://ruby.libvirt.org/:splat 301
+/libvirt/ocaml/* https://ocaml.libvirt.org/:splat 301
+/libvirt/php/* https://php.libvirt.org/:splat 301
+/libvirt/libvirt-appdev-guide-python/en-US/html/*
https://libvirt.gitlab.io/libvirt-appdev-guide-python/:splat: 301
+/libvirt/java.html https://java.libvirt.org 301
+# Redirect to the proper javadoc directory on the subproject page
+/libvirt/sources/java/javadoc/* https://java.libvirt.org/javadoc/:splat 301
+
+# Redirects from old gitweb location (see below)
+/libvirt/git https://gitlab.com/libvirt/ 301
+
+# The above rules are adapted from the following set of 'mod_rewrite' rules used
+# originally on libvirt.org:
+#
+# RewriteRule ^/libvirt-go$ /libvirt-go.html [L]
+# RewriteRule ^/libvirt-go-xml$ /libvirt-go-xml.html [L]
+# RewriteRule ^/go/libvirt$ /go/libvirt.html [L]
+# RewriteRule ^/go/libvirtxml$ /go/libvirtxml.html [L]
+# RewriteRule ^/sources/(.*) https://download.libvirt.org/$1 [L]
+# RewriteRule ^/ruby/download/(.*) https://download.libvirt.org/ruby/$1 [L]
+# RewriteRule ^/(maven2/.*) https://download.libvirt.org/$1 [L]
+# RewriteRule ^/ocaml/(.*) https://ocaml.libvirt.org/$1 [L]
+# RewriteRule ^/ruby/(.*) https://ruby.libvirt.org/$1 [L]
+# RewriteRule ^/php/(.*) https://php.libvirt.org/$1 [L]
+# RewriteRule ^/java.html https://java.libvirt.org [L]
+# RewriteRule ^/docs/libvirt-appdev-guide-python/en-US/html/(.*)
https://libvirt.gitlab.io/libvirt-appdev-guide-python/$1 [L]
+# RewriteRule ^/git https://gitlab.com/libvirt/ [L]
+#
+# Redirect replacing 'gitweb'. The 'gitweb' interface was originally
replaced
+# by the following redirect condition:
+#
+# RewriteCond %{QUERY_STRING} p=([-a-zA-Z0-9]+).git
+# RewriteRule ^/git/$ https://gitlab.com/libvirt/%1 [L]
+#
+# That unfortunately can't be represented in gitlab redirects as it doesn't
+# support redirects based on query strings. Given that the above redirect broke
+# most gitweb links anyways, due to handling only the 'p=' argument, git gitlab
+# redirect will break the rest of them.
+#
+# The following rule was dropped as the page never existed:
+#
+# RewriteRule ^/libvirt-console-proxy$ /libvirt-console-proxy.html [L]
--
2.43.0
11:02 p.m.
New subject: [PATCH 2/4] docs: Add redirects configuration for gitlab pages
On Wed, Jan 10, 2024 at 02:30:23PM +0100, Peter Krempa wrote:
Adapt the configuration of redirects from the server hosting
libvirt.org
The redirect to the 'libvirt-console-proxy' Golang module is not adapted
as it doesn't exist on the current server.
NOTES:
- The redirects are currently configured for hosting via the
'gitlab.io/libvirt' URI. For hosting via custom domain it will need
to be rewritten to drop the '/libvirt' prefixes.
- gitlab pages doesn't currently support redirects to outside content,
thus most of the redirects don't actually work:
https://gitlab.com/gitlab-org/gitlab-pages/-/issues/601
- The redirects file is only installed in the webpage job but is not
actually distributed.
- The validity of the redirects can be checked by visiting:
https://libvirt.gitlab.io/_redirects
Having them installed allows us to validate them before we'll be
switching to use gitlab pages completely.
Signed-off-by: Peter Krempa <pkrempa(a)redhat.com>
---
.gitlab-ci.yml | 1 +
docs/.gitlab_redirects | 54 ++++++++++++++++++++++++++++++++++++++++++
2 files changed, 55 insertions(+)
create mode 100644 docs/.gitlab_redirects
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 6c7d54c311..1ee945aa0a 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -130,6 +130,7 @@ website_prebuilt_env_legacy:
after_script:
- test "$CI_JOB_STATUS" != "success" && exit 1;
- mv install/usr/share/doc/libvirt/html/ public
+ - cp docs/.gitlab_redirects public/_redirects
artifacts:
expose_as: 'pages'
name: 'pages'
diff --git a/docs/.gitlab_redirects b/docs/.gitlab_redirects
new file mode 100644
index 0000000000..26e64563ee
--- /dev/null
+++ b/docs/.gitlab_redirects
@@ -0,0 +1,54 @@
+/libvirt/maven2/* https://download.libvirt.org/maven2/:splat 301
Maven has this unhelpful behaviour where by when you add libvirt.org
as a maven repository host, it'll now consult libvirt.org for *all*
Java JARs, not solely the libvirt JARs.
For all of last week we were serving about 1 million hits a day
for /maven, of which the libvirt JAR requests were about 0.5%,
the rest were 404s ! This is why libvirt.org was painfully slow
for most of last week.
This was all made worse by us redirecting from libvirt.org to
download.libvirt.org, needlessly doubling the number of requests
had to serve.
I changed the /maven2/* redirect to /maven2/org/libvirt/* which
lets libvirt.org service an immediate 404, avoiding hitting
download.libvirt.org, cutting our requests by 50%. Fortunately
traffic has died down to more normal levels after the "back
to work mass code recompile" spike has gone.
Anyway lets do the same for this gitlab redirect to keep the
bogus maven traffic for non-existent files off download.libvirt.org
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
5:45 a.m.
New subject: [PATCH 2/4] docs: Add redirects configuration for gitlab pages
On Wed, Jan 10, 2024 at 15:02:25 +0000, Daniel P. Berrangé wrote:
On Wed, Jan 10, 2024 at 02:30:23PM +0100, Peter Krempa wrote:
> Adapt the configuration of redirects from the server hosting libvirt.org
>
> The redirect to the 'libvirt-console-proxy' Golang module is not adapted
> as it doesn't exist on the current server.
>
> NOTES:
> - The redirects are currently configured for hosting via the
> 'gitlab.io/libvirt' URI. For hosting via custom domain it will need
> to be rewritten to drop the '/libvirt' prefixes.
>
> - gitlab pages doesn't currently support redirects to outside content,
> thus most of the redirects don't actually work:
>
> https://gitlab.com/gitlab-org/gitlab-pages/-/issues/601
>
> - The redirects file is only installed in the webpage job but is not
> actually distributed.
>
> - The validity of the redirects can be checked by visiting:
>
> https://libvirt.gitlab.io/_redirects
>
> Having them installed allows us to validate them before we'll be
> switching to use gitlab pages completely.
>
> Signed-off-by: Peter Krempa <pkrempa(a)redhat.com>
> ---
> .gitlab-ci.yml | 1 +
> docs/.gitlab_redirects | 54 ++++++++++++++++++++++++++++++++++++++++++
> 2 files changed, 55 insertions(+)
> create mode 100644 docs/.gitlab_redirects
>
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 6c7d54c311..1ee945aa0a 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -130,6 +130,7 @@ website_prebuilt_env_legacy:
> after_script:
> - test "$CI_JOB_STATUS" != "success" && exit 1;
> - mv install/usr/share/doc/libvirt/html/ public
> + - cp docs/.gitlab_redirects public/_redirects
> artifacts:
> expose_as: 'pages'
> name: 'pages'
> diff --git a/docs/.gitlab_redirects b/docs/.gitlab_redirects
> new file mode 100644
> index 0000000000..26e64563ee
> --- /dev/null
> +++ b/docs/.gitlab_redirects
> @@ -0,0 +1,54 @@
> +/libvirt/maven2/* https://download.libvirt.org/maven2/:splat 301
Maven has this unhelpful behaviour where by when you add libvirt.org
as a maven repository host, it'll now consult libvirt.org for *all*
Java JARs, not solely the libvirt JARs.
For all of last week we were serving about 1 million hits a day
for /maven, of which the libvirt JAR requests were about 0.5%,
the rest were 404s ! This is why libvirt.org was painfully slow
for most of last week.
This was all made worse by us redirecting from libvirt.org to
download.libvirt.org, needlessly doubling the number of requests
had to serve.
I changed the /maven2/* redirect to /maven2/org/libvirt/* which
lets libvirt.org service an immediate 404, avoiding hitting
download.libvirt.org, cutting our requests by 50%. Fortunately
traffic has died down to more normal levels after the "back
to work mass code recompile" spike has gone.
Anyway lets do the same for this gitlab redirect to keep the
bogus maven traffic for non-existent files off download.libvirt.org
Ah, right. I've captured the redirect list before the holliday break
(actually the original version is from April) so it didn't have this
one and didn't bother to check once more.
The final check needs to be done before switching over anyways.
I've changed the rule in my branch to:
/libvirt/maven2/org/libvirt* https://download.libvirt.org/maven2/org/libvirt/:splat 301
9:30 p.m.
New subject: [PATCH 3/4] gitlab-ci: Drop the legacy web page job
The web page is now consumed via the 'pages' job and thus we don't need
to keep the old one around.
Signed-off-by: Peter Krempa <pkrempa(a)redhat.com>
---
.gitlab-ci.yml | 31 -------------------------------
1 file changed, 31 deletions(-)
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 1ee945aa0a..90b1cfc3e4 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -78,37 +78,6 @@ include:
- .cross_build_job
- .gitlab_cross_build_job_local_env
-# LEGACY web page build
-#
-# This artifact published by this job is downloaded by libvirt.org to
-# be deployed to the web root:
-# https://gitlab.com/libvirt/libvirt/-/jobs/artifacts/master/download?job=w...
-.website_job:
- script:
- - source ci/jobs.sh
- - run_website_build
- after_script:
- - test "$CI_JOB_STATUS" != "success" && exit 1;
- - mv install/usr/share/doc/libvirt/html/ website
- artifacts:
- expose_as: 'Website'
- name: 'website'
- when: on_success
- expire_in: 30 days
- paths:
- - website
-
-website_prebuilt_env_legacy:
- extends:
- - .website_job
- - .gitlab_native_build_job_prebuilt_env
- needs:
- - job: x86_64-almalinux-8-container
- optional: true
- variables:
- NAME: almalinux-8
-
-
# This job builds the website and publishes it in the following ways:
# - 'pages'
# - publishing for gitlab pages
--
2.43.0
9:30 p.m.
New subject: [PATCH 4/4] docs: gitlab redirects: Drop '/libvirt' prefix for hosting the web through gitlab pages
The prefix needs to be dropped for the redirects to work properly once
hosting 'libvirt.org' via gitlab pages.
Signed-off-by: Peter Krempa <pkrempa(a)redhat.com>
---
docs/.gitlab_redirects | 28 ++++++++++++++--------------
1 file changed, 14 insertions(+), 14 deletions(-)
diff --git a/docs/.gitlab_redirects b/docs/.gitlab_redirects
index 26e64563ee..64b4843821 100644
--- a/docs/.gitlab_redirects
+++ b/docs/.gitlab_redirects
@@ -1,25 +1,25 @@
# Redirects to golang module pages
-/libvirt/libvirt-go /libvirt-go.html 200
-/libvirt/libvirt-go-xml /libvirt-go-xml.html 200
-/libvirt/go/libvirt /go/libvirt.html 200
-/libvirt/go/libvirtxml /go/libvirtxml.html 200
+/libvirt-go /libvirt-go.html 200
+/libvirt-go-xml /libvirt-go-xml.html 200
+/go/libvirt /go/libvirt.html 200
+/go/libvirtxml /go/libvirtxml.html 200
# Redirects to the download server
-/libvirt/sources/* https://download.libvirt.org/:splat 301
-/libvirt/ruby/download/* https://download.libvirt.org/ruby/:splat 301
-/libvirt/maven2/* https://download.libvirt.org/maven2/:splat 301
+/sources/* https://download.libvirt.org/:splat 301
+/ruby/download/* https://download.libvirt.org/ruby/:splat 301
+/maven2/* https://download.libvirt.org/maven2/:splat 301
# Redirects to subproject pages
-/libvirt/ruby/* https://ruby.libvirt.org/:splat 301
-/libvirt/ocaml/* https://ocaml.libvirt.org/:splat 301
-/libvirt/php/* https://php.libvirt.org/:splat 301
-/libvirt/libvirt-appdev-guide-python/en-US/html/*
https://libvirt.gitlab.io/libvirt-appdev-guide-python/:splat: 301
-/libvirt/java.html https://java.libvirt.org 301
+/ruby/* https://ruby.libvirt.org/:splat 301
+/ocaml/* https://ocaml.libvirt.org/:splat 301
+/php/* https://php.libvirt.org/:splat 301
+/libvirt-appdev-guide-python/en-US/html/*
https://libvirt.gitlab.io/libvirt-appdev-guide-python/:splat: 301
+/java.html https://java.libvirt.org 301
# Redirect to the proper javadoc directory on the subproject page
-/libvirt/sources/java/javadoc/* https://java.libvirt.org/javadoc/:splat 301
+/sources/java/javadoc/* https://java.libvirt.org/javadoc/:splat 301
# Redirects from old gitweb location (see below)
-/libvirt/git https://gitlab.com/libvirt/ 301
+/git https://gitlab.com/libvirt/ 301
# The above rules are adapted from the following set of 'mod_rewrite' rules used
# originally on libvirt.org:
--
2.43.0
241
days inactive
241
days old
6 comments
2 participants
participants (2)
-
Daniel P. Berrangé -
Peter Krempa