David Eisinger

Elsewhere > Pandoc: A Tool I Use and Like

Posted 2022-05-25 on viget.com

Today I want to talk to you about one of my favorite command-line tools, Pandoc. From the project website:

If you need to convert files from one markup format into another, pandoc is your swiss-army knife.

I spend a lot of time writing, and I love Vim, Markdown, and the command line (and avoid browser-based WYSIWYG editors when I can), so that’s where a lot of my Pandoc use comes in, but it has a ton of utility outside of that – really, anywhere you need to move between different text-based formats, Pandoc can probably help. A few examples from recent memory:

Markdown ➞ Craft Blog Post

This website you’re reading presently uses Craft CMS, a flexible and powerful content management system that doesn’t perfectly match my writing process1. Rather than composing directly in Craft, I prefer to write locally, pipe the output through Pandoc, and put the resulting HTML into a text block in the CMS. This gets me a few things I really like:

By default, Pandoc uses Pandoc Markdown when converting Markdown docs to other formats, an “extended and slightly revised version” of the original syntax, which is how footnotes and a bunch of other things work.

Markdown ➞ Rich Text (Basecamp)

I also sometimes find myself writing decently long Basecamp posts. Basecamp 3 has a fine WYSIWYG editor (🪦 Textile), but again, I’d rather be in Vim. Pasting HTML into Basecamp doesn’t work (just shows the code verbatim), but I’ve found that if I convert my Markdown notes to HTML and open the HTML in a browser, I can copy and paste that directly into Basecamp with good results. Leveraging MacOS’ open command, this one-liner does the trick2:

cat [filename.md] \
  | pandoc -t html \
  > /tmp/output.html \
  && open /tmp/output.html \
  && read -n 1 \
  && rm /tmp/output.html

This will convert the contents to HTML, save that to a file, open the file in a browser, wait for the user to hit enter, and the remove the file. Without that read -n 1, it’ll remove the file before the browser has a chance to open it.

HTML ➞ Text

We built an app for one of our clients that takes in news articles (in HTML) via an API and sends them as emails to their clients (think big brands) if certain criteria are met. Recently, we were making improvements to the plain text version of the emails, and we noticed that some of the articles were coming in without any linebreaks in the content. When we removed the HTML (via Rails’ strip_tags helper), the resulting content was all on one line, which wasn’t very readable. So imagine an article like this:

<h1>Headline</h1> <p>A paragraph.</p> <ul><li>List item #1</li> <li>List item #2</li></ul>

Our initial approach (with strip_tags) gives us this:

Headline A paragraph. List item #1 List item #2

Not great! But fortunately, some bright fellow had the idea to pull in Pandoc, and some even brighter person packaged up some Ruby bindings for it. Taking that same content and running it through PandocRuby.html(content).to_plain gives us:


A paragraph.

-   List item #1
-   List item #2

Much better, and though you can’t tell from this basic example, Pandoc does a great job with spacing and wrapping to generate really nice-looking plain text from HTML.

HTML Element ➞ Text

A few months ago, we were doing Pointless Weekend and needed a domain for our Thrillr app. A few of us were looking through lists of fun top-level domains, but we realized that AWS Route 53 only supports a limited set of them. In order to get everyone the actual list, I needed a way to get all the content out of an HTML <select> element, and you’ll never guess what I did (unless you guessed “use Pandoc”). In Firefox:

The result is something like this:

.ac - $76.00
.academy - $12.00
.accountants - $94.00
.agency - $19.00
.apartments - $47.00
.associates - $29.00
.au - $15.00
.auction - $29.00

Preview Mermaid/Markdown (--standalone)

A different client recently asked for an architecture diagram of a complex system that Andrew and I were working on, and we opted to use Mermaid (which is rad BTW) to create sequence diagrams to illustrate all of the interactions. Both GitHub and GitLab support Mermaid natively, which is really neat, but we wanted a way to quickly iterate on our diagrams without having to push changes to the remote repo.

We devised a simple build chain (demo version available here) that watches for changes to a Markdown file, converts the Mermaid blocks to SVG, and then uses Pandoc to take the resulting document and convert it to a styled HTML page using the --standalone option (here’s the key line). Then we could simply make our changes and refresh the page to see our progress.

Generate a PDF

Finally, and this is not something I need to do very often, but Pandoc also includes several ways to create PDF documents. The simplest (IMO) is to install wkhtmltopdf, then instruct Pandoc to convert its input to HTML but use .pdf in the output filename, so something like:

echo "# Hello\n\nIs it me you're looking for?" \
  | pandoc -t html -o hello.pdf

The result is quite nice.

I think that’s about all I have to say about Pandoc for today. A couple final thoughts:

So install Pandoc with your package manager of choice and give it a shot. I think you’ll find it unexpectedly useful.

Swiss army knife icons created by smalllikeart - Flaticon

  1. My writing process is (generally):

    1. Write down an idea in my notebook
    2. Gradually add a series of bullet points (this can sometimes take awhile)
    3. Once I feel like I have a solid outline, copy that into a Markdown file
    4. Start collecting links (in the [1]: footnote style)
    5. Write a intro
    6. Convert the bullet points to headers, edit + rearrange
    7. Fill in all the sections, write jokes, etc.
    8. Write a conclusion
    9. Create a Gist, get feedback from the team
    10. Convert Markdown to HTML, copy to clipboard (cat [file] | pandoc -t html | pbcopy)
    11. Create a new post in Craft, add a text section, flip to code view, paste clipboard contents
    12. Fill in the rest of the post metadata
    13. 🚢
  2. I’ve actually got this wired up as a Vim command in .vimrc:

    command Mdpreview ! cat %
      \ | pandoc -t html
      \ > /tmp/output.html
      \ && open /tmp/output.html
      \ && read -n 1
      \ && rm /tmp/output.html