Posting to a Static Site by Email with GitHub Actions

The basic requirement is to fetch email from a dedicated mailbox and save each new message to a post file in git. I'm already using GitHub Actions for deployment, so I chose that as a handy serverless runtime environment.

Ideally, new posts would be processed immediately. Rather than tuning a cron schedule (on the order of minutes?), I deployed a Cloudflare Email Worker that pings GitHub and triggers email ingestion via repository_dispatch as soon as an email lands. I'm already using Cloudflare for domain registration and have email routes there. Email Workers, currently in beta, are perfectly suited to a glue handler like this (for a hobby project–I can't speak to production readiness).

To make GitHub API calls, I created a GitHub App and authenticate as an app installation with a private key. The one-time App/App installation/private key setup process is well documented; setup and coding to migrate from using a personal access token to app installation² took about an hour. One trick is required to make the GitHub private key loadable in Cloudflare³.

The result is a nice event-driven workflow. I can post from my phone–including formatted text and images–with an easy, quick feedback loop. Within a few seconds of hitting send in the email app, the message is processed to orgmode and pushed to a PR. From the PR, I can preview the post in the GitHub code browser (.org renders just like markdown) or jump over to the staging site. If there are any issues, I can edit in place using the GitHub app (web also works, but the web editor is harder to use). Merging the PR deploys the post to the main site.

Here's the high level workflow:

The Email Worker's essential task is to make a single GitHub API call. It also does sender validation.

Here's how it calls GitHub using octokit:

const app = new App({
        appId,
        privateKey,
});

const {
        data: { id },
} = await app.octokit.request('GET /repos/{owner}/{repo}/installation', {
        owner,
        repo,
        headers: {
                'X-GitHub-Api-Version': '2022-11-28'
        }
})

const octokit = await app.getInstallationOctokit(id);

await octokit.rest.repos.createDispatchEvent({
        owner: OWNER,
        repo: REPO,
        event_type: "email_received",
        client_payload: {},
});

The bulk of the GitHub workflow functionality is in ingest-email.py. Here's how it fetches email via IMAP:

import dkim
import email
import imaplib

imap = imaplib.IMAP4_SSL(imap_server)
imap.login(imap_username, imap_password)
imap.select(imap_folder)

status, email_ids = imap.search(None, "UNSEEN")
email_id_list = email_ids[0].split()
for email_id in email_id_list:
    # Fetch with PEEK to keep the message unread pending successful processing.
    status, email_data = imap.fetch(email_id, "(BODY.PEEK[])")
    if status != "OK":
        raise Exception(f"Fetch {email_id} failed: {status}.")

    msg_bytes = email_data[0][1]

    # Scream if a message shows up failing ARC authentication.
    if not arc_authenticated(msg_bytes):
        raise Exception(f": {sender} ({subject})")

    post = extract_post(email.message_from_bytes(msg_bytes), org_dest=notes_dest, img_dest=img_dest)

    check_in(post['subject'], notes_dest, img_dest)

    imap.store(email_id, "+FLAGS", "\Seen")

imap.logout()

The script extracts the message body from a multipart email (plain text email is also handled), does minor HTML cleanup, and converts to orgmode using the venerable, amazing pandoc.

for part in msg.walk():
    # ... validate headers and content type

    if content_type == "text/html":
        html = BeautifulSoup(payload.decode(), 'html.parser')
        for br in html.find_all('br'):
            br.extract()
        content = subprocess.run(
            ["pandoc", "--from=html", "--to=org"],
            input=str(html),
            stdout=subprocess.PIPE,
            text=True,
        ).stdout

    # ... handle plain text

A separate multipart walk (not shown) extracts image parts to disk in the GitHub runner workspace.

Next, image src attributes are rewritten from email cid references to file system relative paths. Post content is written to disk in a basic orgmode template.

image_link_base = pathlib.Path('..').joinpath(image_path.relative_to(image_path.parts[0]))
for img in images:
    org_link = f"file:{image_link_base}/{img['filename']}"
    content = content.replace(f"cid:{img['content_id']}", org_link)

# Create post file (orgmode template)
note_file_path = os.path.join(org_dest, f"{slug(subject)}.org")
with open(note_file_path, "w") as f:
    post_timestamp = pendulum.now().format('YYYY-MM-DD ddd HH:mm')
    f.write(f"""#+title: {subject}
#+date: <{post_timestamp}>
#+setupfile: org.txt

{content}""")

Everyday git commands commit new files to git and create a PR. Lastly, the deploy workflow is explicitly triggered to build the branch⁴.

After a bit of setup, processing email to text in GitHub Actions is easy to use. With the flexibility of HTML converters and different attachment types, the same concept could be extended to any static content format.