PROJECTS NOTES HOME

Building a development notes blog with denote and org-publish Notes

Building a blog with Emacs, customizing it, etc, can be quite confusing for the first timer.

It can also be little bit confusing for me as well, after I come back to it after a year or so.

That is why I have built this step by step documentation of how this blog was built. If others find benefit in this - even better.

Why You Should Always Teach What You Learn - https://medium.com/accelerated-intelligence/explanation-effect-why-you-should-always-teach-what-you-learn-9800983a0ea1

1 Why

1.1 Why build a website for development notes anyway

I have a lot of notes. Over a few systems(at home, at work, under descriptions of various projects).

So I wanted to gather everything in one place and somehow link them together. It is also nice if I can share that with others. Also, as of [2023-12-16 Sat], having everything about you/your way of working, etc written in one place, is a perfect for AI systems to grab. All in one place, no need to scrape the whole Internet. So the AI will for sure know how I like to work and my weak spots(what I usually write here is what I just learned or don't want to forget).

Also, all of my notes are in .org files written in emacs, so of course my website will also be written in Emacs.

There are also many inspirational websites that look nice, are clean and simple, but not necessarily do the purpose of "development notes storage". Unless maybe this one - http://tips.defun.work/, which is also my initial inspiration.

Have your devnotes reachable anywhere + be able to share them with others(in a readable html format in a form of a blog). I am in.

2 How

2.1 Choosing an Emacs tool to build such website

Emacs has both built in and external packages that help you to generate .html files from .org note files.

Some of them are:

2.2 ox-publish VS org-static-blog

At first([2023-12-15 Fri]) I was thinking to write the website with org-static-blog, almost had the blog setup and ready, converted around 50 pages to html already, displayed them under my domain. But then kept seeing some features that ox-publish has and org-static-blog does not. Thought okay, let's rethink that. Asked ChatGPT for help:

The choice between using the default ox-publish functionality in Org Mode or the org-static-blog package to create your Emacs website depends on your specific requirements and preferences. Let's explore both options:

ox-publish in Org Mode:

Pros:

Flexibility: ox-publish is part of Org Mode and provides considerable flexibility in customizing your website. You can define various publishing options, including different output formats and configurations.

Integration: It seamlessly integrates with Org Mode, making it easy to manage content directly within Emacs.

Customization: Allows extensive customization using Emacs Lisp, enabling you to tailor your website to your preferences.

Cons:

Steep Learning Curve: Setting up and configuring ox-publish might have a learning curve, especially if you're new to Emacs Lisp and detailed configuration.

org-static-blog Package:

Pros:

Simplicity: org-static-blog is designed specifically for creating simple blogs or websites using Org Mode. It's relatively straightforward to set up and use, making it ideal if you want a quick solution.

Minimal Configuration: It requires minimal configuration, making it accessible for beginners.

Focused Purpose: Primarily built for blogs, it might suit your needs if you're looking for a straightforward blogging platform within Emacs.

Cons:

Limited Features: It might lack certain advanced features or customization options compared to ox-publish, especially if you need more complex functionality.

Less Flexible: As it's tailored for static blogs, if your website needs evolve beyond blogging, you might face limitations.

Decision Factors:

Complexity vs. Simplicity: Consider how complex or simple you want your website setup to be.

Customization Needs: Evaluate how much flexibility and customization options you require for your website.

Future Growth: Consider whether your website's needs might expand beyond a blog and if you need features beyond what a blogging-centric package provides.

Familiarity with Emacs Lisp: Assess your comfort level with Emacs Lisp, as ox-publish might require more familiarity.

Recommendation:

If you prioritize simplicity, quick setup, and your website's focus is primarily a blog, org-static-blog could be a good starting point.

If you foresee needing extensive customization, flexibility, or plan for features beyond a blog, investing time in learning and using ox-publish might be more beneficial in the long run.

Experiment with both to understand their functionalities and how they align with your specific website goals. You can start with org-static-blog for a quick setup and later explore ox-publish for more intricate customization and flexibility if needed.

These were my thoughts AFTER I have read the ChatGPT response:

Of course after reading this I leaned toward ox-publish instead of org-static-blog. True, I have noticed that it lacks configuration options and is kind of restraining. I read other people's tutorials on how they use ox-publish and all the settings they have and I can not try to reuse them with org-static-blog. Not nice..

Also there is this issue that I am facing today(re renders each post). - https://github.com/bastibe/org-static-blog/issues/126

I just hope that tag feature works in ox-publish(it should) - then I am all good. Let's dive in.

[2023-12-16 Sat] Unfortunately, the tags don't work in ox-publish way of making a website, so I will have to stick with org-static-blog. Not complaining. It will force me to read thought the source code more often and and see how I can use pieces of ox-publish in org-static-blog configuration.

[2023-12-19 Tue] The issue I have mentioned above still bugs me. I can use tags in my website, BUT I can not compile it from the comfortable place of a custom script(instead of having the config in my main config), it is not native Emacs, stuff is built on top of it.. I once again decided to switch my mind and build my website with built in tools - Build a website with emacs and ox-publish. I will have ALL the features of org mode and standard publishing tools, exports will be clean, I will also be able to have multiple configs for each blog/website and. I will simply not have the tags.. which I will build on top by myself with the inspiration of =org-static-blog=(someday in the future). The notes will not be rendered into html that often, they won't be categorized by tags, but that's not a big deal. Primary audience for these notes is still - me. I need to be able to search through them and find the information in them. If it's not as easy to do that on the web - so be it. I will work on that later.

2.3 What is "ox-publish"

Best description in a video format is here.

First things first of course:

3 Build

3.1 Installation of "ox-publish"

This package is already installed with Emacs by default.

3.1.1 Decide where to place the configuration

Ox-publish configuration might get large if you start modifying the head, preamble, postamble.. Also if you have multiple blogs/websites that you want to publish this way, you need to have two separate configurations.

Eventually you realize that you can't store multiple configurations in your emacs config file.

So the solution to this is to create a "build" folder and in it have such files:

devnotes/build/
├── build.sh
└── build-site.el

build.sh should be made to be executable chmo o+x build.sh and it's contents are:

#!/bin/sh
emacs -Q --script build-site.el

and build-site.el content would contain your ox-publish configuration(we will remove the configuration from our emacs config and place it in this file with a couple of additions).

That's it. Now when we will run the bash script (./build.sh), it will fetch the dependencies into the /build directory. It will then do the html conversion from org files as you have specified and at the end it will print out "Build complete!"

3.2 Creating .org files to be rendered

3.2.1 Separate repositories

We have one dir/repo for source files, another for exported html files. Having them in separate repos allows us to use ripgrep search easier, allows us to use project.el simpler, not seeing duplicates of files(one .org, one .html). We can version control the changes of the .org files as well as html files separately.

/home/nixos/GIT:
drwxr-xr-x 6 nixos users   4096 May 27 15:42 .
drwx------ 6 nixos users   4096 May 27 15:44 ..
drwxr-xr-x 5 nixos users  20480 May 28 15:28 arvydasg.github.io # html files lie here
drwxr-xr-x 6 nixos users  20480 May 28 15:30 devnotes # source .org (could be other type) files lie here

3.2.2 Folder structure of source dir

When we have at least one .org file in the source directory, we can think about exporting it to .html.

Structure of devnotes folder, an example:

/home/nixos/GIT/devnotes:
drwxr-xr-x 6 nixos users 20480 May 28 15:35 .
drwxr-xr-x 6 nixos users  4096 May 27 15:42 ..
drwxr-xr-x 2 nixos users  4096 May 28 13:59 build # folder in which the build script lives
drwxr-xr-x 8 nixos users  4096 May 28 15:36 .git  # git folder
drwxr-xr-x 3 nixos users  4096 May 27 15:00 media # images folder
drwxr-xr-x 4 nixos users  4096 May 27 15:00 static # css/js/ico folder
-rw-r--r-- 1 nixos users  1404 May 27 16:30 20231125T235251--api-tools__api.org # denote file
-rw-r--r-- 1 nixos users   646 May 27 15:00 20231126T001334--raspberrypi-first-setup__raspberrypi.org # denote file
-rw-r--r-- 1 nixos users   790 May 27 15:00 20231126T001417--magic-mirror-setup__raspberrypi.org # denote file
-rw-r--r-- 1 nixos users   324 May 27 15:00 20231126T001642--raspberrypi-pins__raspberrypi.org # denote file
  -rw-r--r-- 1 nixos users   225 May 27 15:00 contact.org # simple page that
-rw-r--r-- 1 nixos users   164 May 27 15:00 .dir-locals.el # read "denote features" page for explanation to this, has no influence to blog
-rw-r--r-- 1 nixos users   800 May 27 15:00 freelancing.org # random page
-rw-r--r-- 1 nixos users   814 May 27 15:00 index.org # index(front) page of the website
-rw-r--r-- 1 nixos users  1505 May 27 15:00 projects.org # custom list of projects, created the links myself
-rw-r--r-- 1 nixos users    36 May 27 15:00 readme.md
-rw-r--r-- 1 nixos users 23584 May 28 15:28 sitemap.org # automatically generated sitemap
-rw-r--r-- 1 nixos users   817 May 28 15:06 statusupdates.org # random page
-rw-r--r-- 1 nixos users  2728 May 27 15:00 uses.org # random page

3.2.3 Creating new notes with Denote

I am using Denote package, so I can use it's functionality to create a new entry. Here are some denote features.

I use Denote for my personal note taking, now I will also use it for development note taking. Win win.

3.3 The build process

3.3.1 "ox-publish" config file itself

Now since we have this list of files and folders, we are ready to export them to .html.

For that we will need the build script. It's is a single file in which we describe HOW we want our build configuration to look like.

There are many options available. I have written ALL of them (don't get intimidated by that) so I know what is available. It's better to know what is available than not to know :)

So here is the build script (latest version of it):

(require 'ox-publish)

;; run this with 'emacs -Q --script build-site.el' command

(setq org-publish-project-alist
      '(
        ("notes"
         :base-directory "../"
         :base-extension "org"
         :publishing-directory "../../arvydasg.github.io"
         :recursive t
         ;; use :noexport: tag if you want headings that have it not to be exported
         ;; :exclude
         ;; :include
         :publishing-function org-html-publish-to-html ;converts org files to HTML
         ;; :preparation-function
         ;; :completion-function
         ;; -----------------------------------------------------------------------------------
         ;; the DETAILS: options set within the individual .org file can
         ;; override all of the settings below:
         ;; you can specify majority the options that you see below in each
         ;; individual file. Find out syntax for it in =C-h v
         ;; org-publish-project-alist= and choose the configuration you want to
         ;; see the syntax for.
         :author "Arvydas Gasparavicius"
         ;; :creator
         :email "arvydas.gaspa@gmail.com"
         ;; :exclude-tags "noexport"
         :headline-levels 3
         :language "en"
         :preserve-breaks nil
         :section-numbers t
         ;; :select-tags
         :time-stamp-file t
         ;; :with-archived-trees
         :with-author t
         :with-creator t
         :with-date t                   ;date from the filename?
         ;; :with-drawers
         :with-email t
         :with-emphasize t
         :with-entities t
         :with-fixed-width t
         :with-footnotes t
         :with-inlinetasks t
         :with-latex t
         :with-planning t
         :with-priority t
         :with-properties t
         ;; :with-smart-quotes
         :with-special-strings t
         :with-statistics-cookies' t
         :with-sub-superscript nil
         :with-toc t
         :with-tables t
         :with-tags t
         :with-tasks t
         :with-timestamps t
         :with-title t
         :with-todo-keywords t
         ;; kita dalis
         :auto-sitemap t
         :sitemap-filename "sitemap.org"
         :sitemap-title "Notes"
         :sitemap-style tree
         :sitemap-format-entry ag/org-sitemap-date-entry-format
         ;; :sitemap-function
         ;; :sitemap-sort-folders
         :sitemap-sort-files anti-chronologically
         ;; :sitemap-ignore-case
         ;; :makeindex
         ;; :body-only t
         ;; -----------------------------------------------------------------------------------
         ;; ox-html.el.gz

         :html-doctype "html5"
         :html-container "div"
         :html-content-class "content"
         :description "welcome to my site"
         :keywords "emacs antanas geles"
         :html-html5-fancy t
         ;; :html-link-use-abs-url
         ;; :html-link-home "sitemap.html"
         :html-link-home "index.html"
         :html-link-up "sitemap.html"
         ;; :html-mathjax
         ;; :html-equation-reference-format
         ;; :html-postamble nil
         ;; :html-postamble auto                ;with-author/email/creator/date
         ;; :html-postamble t               ;from :html-postamble-format
         :html-postamble "<hr/>
         <footer>
         <div class=\"copyright-container generated\">
         <div class=\"copyright\">
         Copyright &copy; 2023-2024 Arvydas Gasparavicius
         </div>
         </div>
         <p class=\"date\">This org file is <u>exported</u> to HTML: %T</p>
         <p class=\"date\">This org file is last <u>modified</u>: %C</p>
         <p class=\"date\">This org file is <u>created</u>: %d</p>
         <div class=\"generated\">
        Created with %c on <a href=\"https://www.gnu.org\">GNU</a>/<a href=\"https://www.kernel.org/\">Linux</a>
        </div>
        </footer>
         <button onclick=\"topFunction()\" id=\"myBtn\" title=\"Go to top\">Top</button>
         <script src=\"./static/js/generic.js\"></script>
         <script src=\"./static/js/scroll-to-top.js\"></script>
         <script src=\"./static/js/lightbox.js\"></script>
         <script src=\"./static/js/test.js\"></script>"
         ;; :html-preamble t                 ;insert a default one, which in none
         :html-preamble  "<div id=\"updated\">Updated: %C</div>"
         ;; :html-head
         :html-head "
<link rel=\"stylesheet\" href=\"./static/css/org-html-style-default.css\" type=\"text/css\"/>
<link rel=\"stylesheet\" href=\"./static/css/generic.css\" type=\"text/css\"/>
<link rel=\"stylesheet\" href=\"./static/css/taingram.css\" type=\"text/css\"/>
<link rel=\"stylesheet\" href=\"./static/css/lightbox.css\" type=\"text/css\"/>
<link rel=\"stylesheet\" href=\"./static/css/scroll-to-top.css\" type=\"text/css\"/>
<link rel=\"icon\" href=\"./static/ag.ico\">
"
         ;; :html-head-extra "<link rel=\"stylesheet\" href=\"https://arvydas.dev/static/style.css\" type=\"text/css\"/>"
         ;; :html-head-extra "<link rel=\"stylesheet\" href=\"./static/style.css\" type=\"text/css\"/>"
         ;; :subtitle "this is a subtitle"
         :html-head-include-default-style nil         ;turning them off, but adding in a separate file called org-html-style-default.css
         :html-head-include-scripts nil
         :html-allow-name-attribute-in-anchors nil
         :html-divs ((preamble "div" "preamble")
                     (content "div" "content")
                     (postamble "div" "postamble"))
         ;; :html-checkbox-type
         :html-extension "html"
         :html-footnote-format "<sup>%s</sup>"
         :html-footnote-separator "<sup>, </sup>"
         :html-footnotes-section "<div id=\"footnotes\">\n<h2 class=\"footnotes\">%s: </h2>\n<div id=\"text-footnotes\">\n%s\n</div>\n</div>"
         ;; :html-format-drawer-function
         ;; :html-format-headline-function
         ;; :html-format-inlinetask-function
         :html-home/up-format "<div id=\"org-div-home-and-up\">\n <a accesskey=\"h\" href=\"%s\">NOTES </a>\n \n <a accesskey=\"H\" href=\"%s\"> HOME </a>\n</div>"
         :html-indent nil
         ;; :html-infojs-options
         ;; :html-infojs-template
         ;; :html-inline-image-rules
         :html-link-org-files-as-html t
         ;; :html-mathjax-options
         ;; :html-mathjax-template
         :html-metadata-timestamp-format "%Y-%m-%d %a %H:%M"
         ;; :html-postamble-format
         ;; :html-preamble-format
         ;; :html-prefer-user-labels nil
         :html-self-link-headlines nil
         :html-table-align-individual-fields t
         ;; :html-table-caption-above t
         :html-table-data-tags ("<td%s>" . "</td>")
         :html-table-header-tags ("<th scope=\"%s\"%s>" . "</th>")
         :html-table-use-header-tags-for-first-column nil
         :html-tag-class-prefix "gaid"
         :html-text-markup-alist ((bold . "<b>%s</b>")
                                  (code . "<code>%s</code>")
                                  (italic . "<i>%s</i>")
                                  (strike-through . "<del>%s</del>")
                                  (underline . "<span class=\"underline\">%s</span>")
                                  (verbatim .  "<code>%s</code>"))
         :html-todo-kwd-class-prefix ""
         :html-toplevel-hlevel 2
         ;; :html-use-infojs
         :html-validation-link nil
         ;; :html-validation-link "<a href=\"https://validator.w3.org/check?uri=referer\">Validate</a>"
         :html-viewport  ((width "device-width")
                          (initial-scale "1")
                          (minimum-scale "")
                          (maximum-scale "")
                          (user-scalable ""))
         :html-inline-images t
         :html-table-attributes (:border "2" :cellspacing "0" :cellpadding "6" :rules "groups" :frame "hsides")
         :html-table-row-open-tag "<tr>"
         :html-table-row-close-tag "</tr>"
         :html-xml-declaration (("html" . "<?xml version=\"1.0\" encoding=\"%s\"?>")
                                ("php" . "<?php echo \"<?xml version=\\\"1.0\\\" encoding=\\\"%s\\\" ?>\"; ?>"))
         :html-wrap-src-lines nil
         :html-klipsify-src nil         ;making it to t - jupyterhub like code blocks. Insane!
         ;; :html-klipse-css
         ;; :html-klipse-js
         ;; :html-klipse-selection-script
         ;; :infojs-opt
         ;; :creator
         :with-latex t
         ;; :latex-header
         )

        ("static"
         :base-directory "../static"
         :publishing-directory "../../arvydasg.github.io/static"
         :base-extension "css\\|js\\|ico\\|svg"
         :recursive t
         :publishing-function org-publish-attachment ;copies the files verbatim
         )

        ("media"
         :base-directory "../media"
         :publishing-directory "../../arvydasg.github.io/media"
         :base-extension "png\\|jpg\\|jpeg\\|pdf"
         :recursive t
         :publishing-function org-publish-attachment ;copies the files verbatim
         )

        ("build-site.el" :components ("notes" "static" "media"))))

(defun ag/org-sitemap-date-entry-format (entry style project)
  "Format ENTRY in org-publish PROJECT Sitemap format ENTRY ENTRY
 STYLE format that includes date."
  (let ((filename (org-publish-find-title entry project)))
    (if (= (length filename) 0)
        (format "*%s*" entry)
      (format "{{{timestamp(%s)}}} [[file:%s][%s]]"
              (format-time-string "%Y-%m-%d"
                                  (org-publish-find-date entry project))
              entry
              filename))))

(setq org-export-global-macros
      '(("timestamp" . "@@html:<span class=\"timestamp\">[$1]</span>@@")))

;; publish all the projects described in this file :components part
(org-publish-all t)

;; publish only specific project
;; The t parameter tells it to regenerate all ;; files regardless of when they
;; were last generated)so no need to add the ;; force prefix with C-c C-e C-f
(org-publish "my-org-site" t)

(message "Build complete!")

You can see everything in it… How I specify directories, how I name the author, what is my nav(preamble), what is my footer (postamble), how I added css files, how I added js files, how I added a theme, how I created folders for images and css/js files, etc….

3.3.2 Running the build (generating the .html files from .org files)

After you have studied the config file and you understand what's happening, we can now launch this script while in /home/nixos/GIT/devnotes/build. Write ./build.sh in shell and it will run the exportation.

./build.sh is basically creates a "naked" emacs instance which runs our configuration file:

#!/bin/sh
emacs -Q --script build-site.el

After the export - check the html file folder, you should see the html files generated as you like. Open them in the browser to check. Commit if happy, modify the build script if not happy.

3.3.3 Previewing the generated site

Now that we have a few .org file notes in our directory and we have exported them into html, we can preview them. Open the .html file and it will open in the browser.

  1. Preview the html file in the browser

    If the file on my wsl lies here:

    file://wsl.localhost/NixOS/home/nixos/GIT/devnotes/html/index.html, I would open the whole html directory instead of a single file (file://wsl.localhost/NixOS/home/nixos/GIT/devnotes/html/) to preview the contents of it in the browser. This allows me to quickly open the files that I want and see what has changed.

    how to implement - here https://systemcrafters.net/publishing-websites-with-org-mode/building-the-site/

  2. Preview with simple-httpd

    Or you can use simple-httpd. It's a package that can host your files as a website on your local machine so that you can pull it up in your browser.

    You can install simple-httpd from MELPA using M-x package-install or by putting the following snippet in your Emacs configuration if you have use-package installed and MELPA configured as a package source(install it globally in your emacs config, not in build-site.el file):

    (use-package simple-httpd
      :ensure t)
    

    Now you can run M-x httpd-serve-directory. It will prompt you for a directory to serve from within Emacs.

    Select the path of your public/ directory and then open your browser to http://localhost:8080 to see the preview of your site! Set httpd-port to change the default port if necessary.

    Any time you regenerate the site files, you can just reload the page to see the result of changes you made! (if you need to do a refresh, then it's the same as opening in the browser…). That's why I don't bother with simple-httpd.

4 Publish

I am using github pages. It is possible to host static html pages from it for free. Here is a tutorial how it is done - deploy a static website with github pages.

To later add a domain name to your site, follow this tut - setup custom domain for github pages

4.1 Manual workflow

Good if you want to preview the changes locally first, only then commit.

  • I first make changes in my devnote directory
  • Run ./build.sh locally
  • Check the output(html files) in arvydasg.github.io local repository
  • If all is good, commit the .org files to devnote directory
  • Commit the .html files to arvydasg.github.io directory

This was the primary way of working before setting up github actions before [2024-06-03 Mon].

At first I was trying to create this workflow myself, manually, dealing with two repos, figuring out the paths, realizing that I need to use some secret keys, etc… but it was a struggle and then I found this - https://cpina.github.io/push-to-another-repository-docs/, but it did not work for me as expected, so I went back to manual way of creating a github action that I want. Each mistake led me to the direction I needed to go. Did not give up and made it :)

Here is how I made it:

4.2 Automated workflow with github actions

Took me the whole day today ([2024-06-03 Mon]) to figure this one out…

I learned about stepping into the github action containe midrun, a very cool way to debug your github action.

I learned that I must create a personal access token here - https://github.com/settings/tokens (mark repository tick) NOTE: [2024-07-11 Thu] regenerated this token for 60 more days. and then add the token to the devnotes repo(from which you are running the action) here - https://github.com/azegas/test/settings/secrets/actions. Name the secret variable GH_PAT.

And here is the first working config (current version):

name: Publish to arvydasg.github.io

on:
  push:
    branches:
      - master

jobs:
  publish:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout devnotes repository
	uses: actions/checkout@v3

      - name: Install Emacs
	run: |
	  # install terminal version of emacs
	  sudo apt install emacs-nox --yes

      - name: Run build.sh Script
	run: |
	  cd build/
	  ./build.sh

      # directory after build.sh run script - /home/runner/work/devnotes/devnotes
      # source directory - /home/runner/work/devnotes/devnotes
      # destination directory - /home/runner/work/devnotes/arvydasg.github.io
      # tmp directory - /tmp
      # test repo to which to push stuff - https://github.com/azegas/test.git

      - name: Clone arvydasg.github.io repo
	run: |
	  cd /tmp
	  git clone https://github.com/azegas/arvydasg.github.io.git
	  ls -la

      - name: Copy content from source to arvydasg.github.io repo
	run: |
	  cd /tmp/arvydasg.github.io
	  cp -r /home/runner/work/devnotes/arvydasg.github.io/* .
	  ls -la

      - name: Authenticate myself to git
	run: |
	  cd /tmp/arvydasg.github.io
	  pwd
	  git ls-remote --get-url origin
	  git config --global user.email "azegaspa@gmail.com"
	  git config --global user.name "BOT"
	  git config --global -l

      - name: Commit the changes
	run: |
	  cd /tmp/arvydasg.github.io
	  git add .
	  git commit -m "yay"
	  git status

      - name: Push changes to arvydasg.github.io repo (using GitHub personal access token)
	env:
	  GH_PAT: ${{ secrets.GH_PAT }}
	run: |
	  # so it's only necessary to set up the github secret for devnotes repo, not for arvydasg.github.io repo
	  cd /tmp/arvydasg.github.io
	  git remote set-url origin https://$GH_PAT@github.com/azegas/arvydasg.github.io.git
	  git push origin master

      # - name: Setup tmate session
      #   uses: mxschmitt/action-tmate@v3

Now by having this in .github/workflows/deploy.yml file, whenever you make a commit to devnotes repo, arvydasg.github.io repo will get updated with the html content, effectively updating the arvydas.dev site.

So now I will never have to manually do a push again. Its good that I dont have to push to see the site, can develop locally, test locally, push only when the changes ready.

4.3 Fix commit message

I want to make the devnotes repo commit link to show in arvydasg.github.io repo commit that was automatically made with github action.

Tried checking out this documentation - https://docs.github.com/en/actions/learn-github-actions/variables#default-environment-variables

And then from it, tried using some of the "context" variables like so:

# env:
#   ORIGINAL_COMMIT_MESSAGE: ${{ github.event.head_commit.message }}
#   GITHUB_RUN_ID: ${{ github.run_id }}
# git commit -m "Automated commit. Original commit: $ORIGINAL_COMMIT_MESSAGE - GitHub Actions Run ID: $GITHUB_RUN_ID"
# Automated commit. Original commit: mod - GitHub Actions Run ID: 93562……34702

# git commit -m "$GITHUB_SHA, $GITHUB_WORKFLOW_SHA, $GITHUB_TRIGGERING_ACTOR"
# bed1324733be364f7e7f61417c77f57e3873da08, bed1324733be364f7e7f61417c7……7f57e3873da08, azegas

But I did not like that there were no hyperlinks, only some jibberish. But from the example above, you can find t he commit with it - bed1324733be364f7e7f61417c77f57e3873da08. But no hyperlink…

I finally found a way to make a commit message with a hyperlink:

- name: Commit the changes with dynamic message
  env:
    GITHUB_SHA: ${{ github.sha }}
    REPO: ${{ github.repository }}
  run: |
    cd /tmp/arvydasg.github.io
    COMMIT_URL="https://github.com/${REPO}/commit/${GITHUB_SHA}"
    COMMIT_MESSAGE="Automated commit for - ${COMMIT_URL}"
    git add .
    git commit -m "${COMMIT_MESSAGE}"
    git status

Added this to my main config.

4.4 Problem with "date modified"

Commit that did this change - https://github.com/azegas/devnotes/commit/f5471491056b4e055fbc8a9556d18fc141ffd2f5

At the top of each page, I would have this rendered for example in the preamble - "Updated: 2024-06-04 Tue 04:46". Basically an identification when the particular post was last updated.

Now this will be removed, and I will explain why below.

But first, a few issues with this "updated date" box in the first place:

  • It shows wrong time, not the current Lithuanian time
  • IF a small change is made in the pramble or postamble of the whole site, the pages get REGENERATED and therefore, their date when they are updated get changed. Even though the content of the page did not change. I don't like that. The idea was to track only the date modified of the post, not of the html that surrounds it.
  • IF you delete some files in the arvydasg.github.io repo and then regenerate them again - "updated" date will be inaccurate again . Not good.

Basically the last point is the main reason why I have to remove this feature.

4.4.1 Workflow before github actions

Before github actions, only the pages that have changed would get regenerated.

Now - since it's happening in the pre-commit and is temporary, all the files are being freshly generated by each github action run. Therefore rendering this "updated" field useless.

So I will simply leave the "created" field at the bottom of the page for those who are interested and remove everything else.

5 Bits and pieces

5.1 Add a "google translate" button to the site

5.2 Individual files customization options

We are specifying all the configuration in the build-site.el file, however, you can still make some changes to the individual files.

Check like it's done in index.html

Here are the things you can modify:

#+title: Freelancing
#+date: <2020-01-01 Sun 01:01>
#+description:
#+filetags:
#+OPTIONS: toc:nil
#+OPTIONS: num:nil

5.3 Generating pages with code blocks

So far we've been looking at a very simple example page that doesn't really have much on it. What happens when we try to generate a more elaborate Org file containing code blocks?

Let's try it out with another file, a version of my literate Emacs configuration called Emacs.org!

If you're generating a site that features code blocks like a coding blog or a literate Emacs configuration, you'll probably see an error like this when you generate the site:

Cannot fontify source block (htmlize.el >= 1.34 required)

To resolve this issue, you will need to install the htmlize package from MELPA. We can automate the process of installing this package by adding the following snippet to our build-site.el file:

;; Set the package installation directory so that packages aren't stored in the
;; ~/.emacs.d/elpa path.
(require 'package)
(setq package-user-dir (expand-file-name "./.packages"))
(setq package-archives '(("melpa" . "https://melpa.org/packages/")
                         ("elpa" . "https://elpa.gnu.org/packages/")))

;; Initialize the package system
(package-initialize)
(unless package-archive-contents
  (package-refresh-contents))

;; Install dependencies
(package-install 'htmlize)

In this snippet, we load Emacs' package manager and configure the package-user-dir to be a sub-directory of our project folder. This allows you to install packages for the script without mixing them up with the packages of your personal Emacs configuration!

The next thing we do is add MELPA to the package archive list and refresh the package archive so that htmlize can be found. Finally, we call package-install to install it!

In a future video I'll show you how to convert your Emacs color theme to a CSS file that you can use to colorize source blocks with exactly the same colors used in the theme!

5.4 Important information about linking between .org files

Github link to issue - https://github.com/protesilaos/denote/issues/369

As of [2023-12-17 Sun] denote links do not work with exporting to html. Talked with Prot about it, there is no fix for it now. So what I do now is I create file links(C-c C-l file: <and your filename>) instead of denote links, these do work. Also, if you get a link wrong and it is not possible to render it, the compilation of html pages will break. That's good, because by checking the logs you will see the broken link and can fix it. If you don't want the compilation to stop, you can add this in your config:

(setq org-export-with-broken-links 'mark)

You will simply see this in your html page(I prefer to know that there are broken links during the html compilation, so I don't put the code above in my config).

[BROKEN LINK: denote:20231125T212326]

Naming the links:

Example taken from here - https://orgmode.org/manual/Links-in-HTML-export.html

#+ATTR_HTML: :title The Org mode website :style color:red;
https://orgmode.org

5.5 CSS support in HTML export

Official docs for css support in HTML export - https://orgmode.org/manual/CSS-support.html

There are a few options when it comes to custom CSS on your website generated with Emacs.

5.5.1 Use the default styles from ox-html

You have to do nothing, ox-html - org-html-style-default got you covered. And it looks pretty good already by default.

But you can do like I did, went to org-html-style-default and copied the CSS from there, placed in my own stylesheet. I can customize the look myself now a little. Don't have to rewrite everything - just the parts that I want.

Also it's good to have the default styles as a reference close to me rather than somewhere deep in the emacs docs.

5.5.2 Add custom classes to blocks of text/code

5.5.3 import custom css

Default html export engine already has us covered with basic css, you can enable/disable it like so:

:html-head-include-default-style t/nil

But you might want to change the default css. Best bet is to do something like this:

:html-head "<link rel=\"stylesheet\" href=\"../static/style.css\" type=\"text/css\"/>"

Basically whatever you put in style.css now will end up in every html page of yours. We can add any information you like there. But for now I am concerned about css and favicon. This will link to my own style.css file, but you can also use CDN's or bootstrap or whatever else.

Use your own config from another website/blog for example:

:html-head-extra "<link rel=\"stylesheet\" href=\"https://arvydas.dev/static/style.css\" type=\"text/css\"/>"

5.5.4 org-html-themes

ORRR last thing that you can try when it comes to styling your website, try using one of these, org-html-themes.

You can point to one of these themes like such, by placing this code at the top of your.org file:

Or if you have downloaded the code(more future proof solution), add such line at the top of your .org file.

#+SETUPFILE: PATH/TO/GIT/REPO/org/theme-NAME-local.setup

Of course we can also modify our setq org-static-blog-page-header to contain the org-html-themes, but I won't do that this time since I know I will be using my own custom stylesheet.

5.5.5 Note to self

Some css examples here(built with ox-publish though) - https://github.com/arvydasg/emacs_blog/blob/master/css/style.css

5.6 Javascript support in HTML export

Official docs - https://orgmode.org/manual/JavaScript-support.html

Add the .js files the same way as you would add .css file.

5.7 Images in HTML export

5.7.1 Display images

NOTE: gif's are not rendered in github pages for some reason (julijaconsulting and facebook project gif's)

Official docs - https://orgmode.org/manual/Images-in-HTML-export.html My docs - images in org files

5.7.2 One option is to add images as plain html

But then you can not change the size of it in lightbox.

1.png

5.7.3 Another option is to include a link to a file(the usual way)

To toggle the images(show/hide), use org-toggle-inline-images.

water.jpg

Okay, so no now we know that we can create a yasnippet for this(I have mine bound to img) and then I can drop images wherever.

5.7.4 Org download package for inserting images in org files

Optional, does not really work if in WSL - using org download package in WSL

5.7.5 Add a lightbox effect on the images

5.8 Preamble(nav) and postamble(footer)

Docs about them - https://orgmode.org/manual/HTML-preamble-and-postamble.html

Postamble - https://taingram.org/blog/org-mode-blog.html Postamble - nice footer and more? - https://ict4g.net/adolfo/notes/emacs/linking-my-html-pages-to-source-code.html

Check other inspirational websites that are build with Emacs for some inspiration.

Then simply modify - :html-preamble and :html-postamble in your config.

5.9 Add a search to the sitemap.org

So my thought was that… if I create lots of small files and all of those small files are listed in one list in sitemap.org, then it will be difficult to locate the files.. The tags are currently not visible in this list, you only have to search by keywords that you know. For example you search for Django, but then you get a search bar with results of 20 entries. Have to go through each one of them… Pain.

What if I had a simple search bar in sitemap.org file which would allow me to filter the list results by the words I write in the search bar?

5.9.1 Print "hello world" first in the sitemap.org

Sounds good, but the problem is that I do not control the behavior of the sitemap.org generation. At least not originally. But it's emacs, so I can.

I see that my build configuration has :sitemap-function option, which is currently commented out. Perhaps this is the place that is responsible for creating the sitemap.org file?

I go to /nix/store/jm91p103vzqyvjcgk44ga6r5p5fhj45i-emacs-29.3/share/emacs/29.3/lisp/org and find ox-publish.el.gz package. In it I see org-publish-sitemap-default function that looks like this:

(defun org-publish-sitemap-default (title list)
  "Default site map, as a string.
TITLE is the title of the site map.  LIST is an internal
representation for the files to include, as returned by
`org-list-to-lisp'.  PROJECT is the current project."
  (concat "#+TITLE: " title "\n\n"
          (org-list-to-org list)))

First I want to simply print "hello world". Let's modify this function like so:

(defun ag-org-publish-sitemap-default (title list)
  "Default site map, as a string.
TITLE is the title of the site map.  LIST is an internal
representation for the files to include, as returned by
`org-list-to-lisp'."
  (concat "#+TITLE: " title "\n\n"
          "* Hello World\n\n"
          (org-list-to-org list)))

Then in the build config, let's use this function:

:sitemap-function ag-org-publish-sitemap-default

Build the site.

You should see the hello world in the sitemap.org. Great, we have found a way to control the sitemap!

ag-org-publish-sitemap-default

5.9.2 Add the search bar in plain html

To include an HTML search bar in the sitemap.org file generated by our custom function, we need to embed the HTML directly into the Org file. Org-mode supports including raw HTML, which will be preserved when the Org file is exported to HTML.

Let's modify our custom function to include the search bar:

(defun ag-org-publish-sitemap-default (title list)
  "Default site map, as a string.
TITLE is the title of the site map.  LIST is an internal
representation for the files to include, as returned by
`org-list-to-lisp'."
  (concat "#+TITLE: " title "\n\n"
          "#+OPTIONS: toc:nil\n\n"   ;; Disable the table of contents for better formatting
          "#+BEGIN_EXPORT html\n"
          "<div class=\"search-wrapper\">\n"
          "<label for=\"search\">Search</label>\n"
          "<input type=\"search\" id=\"search\" data-search>\n"
          "</div>\n"
          "#+END_EXPORT\n\n"
          (org-list-to-org list)))

Now create a search.js file in static/js folder. Add it in the html-postamble.

Content of the search.js:

  // This code waits for the entire HTML document to load before running
document.addEventListener("DOMContentLoaded", function() {
    // Select all list items (notes) within the unordered list with class "org-ul"
    const allNotes = document.querySelectorAll(".org-ul li");

    // Select the search input field with the data attribute "data-search"
    const searchInput = document.querySelector("[data-search]");

    // Add an event listener to the search input field that listens for "input" events
    searchInput.addEventListener("input", function(e) {
        // Get the current value of the search input field and convert it to lowercase
        const value = e.target.value.toLowerCase();

        // Loop through each note in the list of notes
        allNotes.forEach(function(note) {
            // Get the text content of the note and convert it to lowercase
            const noteText = note.textContent.toLowerCase();

            // Check if the note's text includes the search input value
            if (noteText.includes(value)) {
                // If it does, display the note
                note.style.display = "";
            } else {
                // If it doesn't, hide the note
                note.style.display = "none";
            }
        });
    });
});

Rebuild the site. Search should work.

5.10 Extra's

Extra tips:

5.10.1 You can also include other content from other files!

Like such - https://orgmode.org/manual/Include-Files.html

I have done this to include recent posts into index.org page.

5.10.2 Creating a HTML page with my .emacs configuration

Simply made a symlink to my .org configuration file and it renders it together with the rest of the .org files.

[nixos@nixos:~/GIT/devnotes/build]$ ln -s ../../dotfiles/.emacs.d/init.org init.org

UNFORTUNATELY - if you will chose to deploy to github pages, it will not understand the symlink, the build will fail with such message:

Error: No such file or directory @ rb_check_realpath_internal - /github/workspace/org/init.org

So I ditched this idea of having a symlink to my init.org. IF I will ever host this site on another platform - I might try this workaround again.

5.10.3 Adding a date to post title in sitemap

Inspiration here - https://taingram.org/blog/org-mode-blog.html

We can customize the entry format with a sitemap-format-entry function. In my case I wanted to show the date inline with the blog post listings:

(defun ag/org-sitemap-date-entry-format (entry style project)
  "Format ENTRY in org-publish PROJECT Sitemap format ENTRY ENTRY STYLE format that includes date."
  (let ((filename (org-publish-find-title entry project)))
    (if (= (length filename) 0)
        (format "*%s*" entry)
      (format "{{{timestamp(%s)}}} [[file:%s][%s]]"
              (format-time-string "%Y-%m-%d"
                                  (org-publish-find-date entry project))
              entry
              filename))))

Notice above we insert an Org macro called timestamp, it is defined as follows:

(setq org-export-global-macros
      '(("timestamp" . "@@html:<span class=\"timestamp\">[$1]</span>@@")))

We then also must inform org-static-project-alist about this change by using our custom function in:

:sitemap-format-entry ag/org-sitemap-date-entry-format

Re-render your .org files.

This macro adds some HTML around the timestamp for CSS styling, it has to be done as a macro as otherwise Org escapes the HTML tags. The results are:

#+BEGIN_EXPORT html

6 Customize the automated build

I want to remember the ways it's possible to modify the export process.

  • Official docs - https://orgmode.org/worg/org-tutorials/org-publish-html-tutorial.html
  • A single line does this - #+INCLUDE: "sitemap.org" :lines "12-20"

    This will include 12-20th lines of the sitemap.org page. Can simply open sitemap.org page to check what lines you want to include.

    Useful when you want to add a list of sitemap in index.html page for example.

    #+INCLUDE: "sitemap.org" :lines "-6"
    
  • See a default suggested template C-c C-e # (org-export-dispatch)

    # html template:
    #+options: html-link-use-abs-url:nil html-postamble:auto html-preamble:t
    #+options: html-scripts:nil html-style:t html5-fancy:nil tex:t
    #+html_doctype: xhtml-strict
    #+html_container: div
    #+html_content_class: content
    #+description:
    #+keywords:
    #+html_link_home:
    #+html_link_up:
    #+html_mathjax:
    #+html_equation_reference_format: \eqref{%s}
    #+html_head:
    #+html_head_extra:
    #+subtitle:
    #+infojs_opt:
    #+creator: <a href="https://www.gnu.org/software/emacs/">Emacs</a> 29.3 (<a href="https://orgmode.org">Org</a> mode 9.6.15)
    #+latex_header:
    
     # Default template
    #+options: ':nil *:t -:t ::t <:t H:3 \n:nil ^:t arch:headline author:t
    #+options: broken-links:nil c:nil creator:nil d:(not "LOGBOOK") date:t e:t
    #+options: email:nil f:t inline:t num:t p:nil pri:nil prop:nil stat:t tags:t
    #+options: tasks:t tex:t timestamp:t title:t toc:t todo:t |:t
    #+title: *Org Src 20231215T182523--building-a-development-notes-blog-with-denote-and-org-publish__denote_emacs_websites.org[ org ]*
    #+date: <2024-06-06 Thu>
    #+author: nixos
    #+email: nixos@nixos
    #+language: en
    #+select_tags: export
    #+exclude_tags: noexport
    #+creator: Emacs 29.3 (Org mode 9.6.15)
    #+cite_export:
    
  • Export settings - https://orgmode.org/manual/Export-Settings.html
  • HTML specific export settings - https://orgmode.org/manual/HTML-specific-export-settings.html
  • Source code:

    /org/ox-html.el.gz
    /org/ox-publish.el.gz
    
  • org-html-preamble-format some useful %a %c %T functions
  • IF you want to add html into an org file (not code block), can do it like such:

    #+HTML: <div">Updated:
    #+HTML: hello
    #+HTML: </div>
    
  • Create editable text areas Docs - https://orgmode.org/manual/Text-areas-in-HTML-export.html

    If you ever want to try using this feature, here is an example: