Automatically generate and commit site HTML

.github/actions/build-html/action.yml

@@ -0,0 +1,22 @@
+name: Build HTML
+
+description: Set up Ruby and run the Jekyll build.
+
+inputs:
+  ruby-version:
+    description: Ruby version to use.
+    required: false
+    default: "3.4"
+
+runs:
+  using: composite
+  steps:
+  - name: Set up Ruby and Bundler
+    uses: ruby/setup-ruby@v1
+    with:
+      ruby-version: ${{ inputs.ruby-version }}
+      # This will use the version of Bundler specified in `Gemfile.lock`.
+      bundler-cache: true
+  - name: Run documentation build
+    shell: bash
+    run: bundle exec jekyll build

.github/workflows/html-build.yml

@@ -0,0 +1,16 @@
+name: Build HTML
+
+on:
+  pull_request:
+    branches:
+    - asf-site
+
+jobs:
+  build:
+    name: Build HTML
+    runs-on: ubuntu-24.04
+    steps:
+    - name: Checkout Spark Website repository
+      uses: actions/checkout@v7
+    - name: Build HTML
+      uses: ./.github/actions/build-html

.github/workflows/html-push.yml

@@ -0,0 +1,43 @@
+name: Build and Push HTML
+
+on:
+  push:
+    branches:
+    - asf-site
+
+jobs:
+  commit:
+    name: Build and commit HTML to `asf-site`
+    # This condition is important. We don't want to trigger this job if the last
+    # commit was created _by_ this job!
+    if: >-
+      !(
+        contains(github.event.head_commit.message, '[html]') &&
+        github.event.head_commit.author.name == 'github-actions[bot]'
+      )
+    # Not technically necessary, but helps avoid spurious failures if multiple
+    # commits are pushed in rapid succession.
+    concurrency:
+      group: html-push-${{ github.ref }}
+      cancel-in-progress: true
+    runs-on: ubuntu-24.04
+    permissions:
+      contents: write
+    steps:
+    - name: Checkout Spark Website repository
+      uses: actions/checkout@v7
+    - name: Build HTML
+      uses: ./.github/actions/build-html
+    - name: Commit and push generated HTML
+      run: |
+        git config user.name "github-actions[bot]"
+        git config user.email "github-actions[bot]@users.noreply.github.com"
+        # `-f` because we told git to otherwise ignore `site/`
+        git add -f site/
+        if git diff --cached --quiet; then
+          echo "No changes to commit."
+        else
+          COMMIT_TITLE=$(git log -1 --pretty=%s)
+          git commit -m "[html] $COMMIT_TITLE"
+          git push
+        fi

.gitignore

@@ -4,4 +4,4 @@ target/
 .jekyll-cache/
 .jekyll-metadata
 .local_ruby_bundle
-site/python
+site/

Gemfile.lock

@@ -79,4 +79,4 @@ RUBY VERSION
    ruby 3.2.3p157
 
 BUNDLED WITH
-   2.4.19
+   2.4.22

README.md

@@ -1,10 +1,34 @@
-## Generating the website HTML
+# Apache Spark Main Website
 
-In this directory you will find text files formatted using Markdown, with an `.md` suffix.
+This repository captures the main Apache Spark website located at https://spark.apache.org. The programming docs under https://spark.apache.org/docs/ are [in the main Spark repo][main], not here. They are built separately for each release of Spark and then copied to the website under the `docs/` directory.
 
-Building the site requires [Ruby 3](https://www.ruby-lang.org), [Jekyll](http://jekyllrb.com/docs), and
-[Rouge](https://github.com/rouge-ruby/rouge). The most reliable way to ensure a compatible environment
-is to use the official Docker build image from the Apache Spark repository.
+[main]: https://github.com/apache/spark/tree/master/docs#readme
+
+## Contributing
+
+To contribute changes, build and test the site locally, then submit a pull request with your changes. You only need to commit changes to the Markdown source. A [GitHub Actions workflow](.github/workflows/html-push.yml) will generate the corresponding HTML under `site/` and push it for you.
+
+The `site/` directory is tracked in git as a deployment artifact. To hide local changes to it from `git status` and `git diff`, run:
+
+```sh
+git ls-files -z site/ | xargs -0 git update-index --skip-worktree
+```
+
+To undo this, run the same command with `--no-skip-worktree` instead.
+
+## Building the site locally
+
+Building the site requires [Ruby 3](https://www.ruby-lang.org), [Jekyll](http://jekyllrb.com/docs), and [Rouge](https://github.com/rouge-ruby/rouge).
+
+```
+gem install bundler -v 2.4.22
+bundle install
+bundle exec jekyll serve
+```
+
+### Building the site with Docker
+
+The most reliable way to ensure a compatible environment is to use the official Docker build image from the Apache Spark repository.
 
 If you haven't already, clone the [Apache Spark](https://github.com/apache/spark) repository. Navigate to
 the Spark root directory and run the following command to create the builder image:
@@ -21,29 +45,16 @@ the Markdown files in the Docker container.
 .dev/build-docs.sh
 ```
 
-## Docs sub-dir
-
-The docs are not generated as part of the website. They are built separately for each release 
-of Spark from the Spark source repository and then copied to the website under the docs 
-directory. See the instructions for building those in the readme in the Spark 
-project's `/docs` directory.
-
-## Rouge and Pygments
-
-We also use [Rouge](https://github.com/rouge-ruby/rouge) for syntax highlighting in documentation Markdown pages.
-Its HTML output is compatible with CSS files designed for [Pygments](https://pygments.org/).
+## Deploying to production
 
-To mark a block of code in your Markdown to be syntax highlighted by `jekyll` during the 
-compile phase, use the following syntax:
+The website is deployed automatically by [ASF Infra][infra]. The deployment configuration is tracked by [.asf.yaml](./.asf.yaml) and is [documented here][asf-docs].
 
-    {% highlight scala %}
-    // Your Scala code goes here, you can replace Scala with many other
-    // supported languages too.
-    {% endhighlight %}
+One deployment detail that appears to be critical is the presence of the [`content`](./content/) symlink to `site/`. Even though ASF Infra is [aware of Jekyll][jek], we perhaps do not have the exact setup required for them to automatically use our [Jekyll config](./_config.yml) to understand where the site content lives. Without the `content` symlink, the website will just show a plain directory listing of the files in this repo.
 
-You probably don't need to install that unless you want to regenerate the Pygments CSS file.
-It requires Python, and can be installed by running `sudo easy_install Pygments`.
+[infra]: https://infra.apache.org
+[asf-docs]: https://github.com/apache/infrastructure-asfyaml/tree/main#readme
+[jek]: https://github.com/apache/infrastructure-asfyaml/tree/76d241ccef02e5397e10c173ebf04c07525311ea#jekyll_cms
 
 ## Merge PR
 
-To merge pull request, use the `merge_pr.py` script which also squashes the commits.
+To merge a pull request, use the `merge_pr.py` script. This script also squashes the commits.

committers.md

@@ -265,9 +265,8 @@ an `apache` remote (the default `PUSH_REMOTE_NAME`, used for pushing the squashe
 ```
 
 The script is interactive and walks you through selecting the PR and pushing the merge to the
-`asf-site` branch. Before merging, make sure the pull request includes the regenerated HTML under
-`site/` in addition to the Markdown/HTML source changes; see the repository's `README.md` for how
-the site is built.
+`asf-site` branch. See the repository's [README](https://github.com/apache/spark-website#readme)
+for more details on how the site is built.
 
 <h3>Policy on backporting bug fixes</h3>
 

sitemap.xml

@@ -151,9 +151,18 @@ sitemap: false
   <changefreq>weekly</changefreq>
 </url>
 {% endfor %}
-{% for page in site.pages %}{% if page.sitemap != false %}<url>
+{%- comment -%}
+Explicitly sort `site.pages` so that the order is consistent and we don't get spurious git diffs.
+`site.posts` doesn't have this issue because it's already sorted.
+See: https://jekyllrb.com/docs/variables/#site-variables
+{%- endcomment -%}
+{%- assign sorted_pages = site.pages | sort: "url" -%}
+{%- for page in sorted_pages -%}
+{%- if page.sitemap != false -%}
+<url>
   <loc>{{ site.url }}{{ page.url }}</loc>
   <changefreq>weekly</changefreq>
-</url>{% endif %}
-{% endfor %}
+</url>
+{% endif %}
+{%- endfor -%}
 </urlset>