Breaking distributed systems for fun and profit | Kyle Kingsbury (Jepsen)
Kyle (Aphyr) is the creator of Jepsen
Well-known for his insightful and meticulous write-ups on testing distributed systems, Kyle (aka Aphyr) joins the show to chat about the origins of Jepsen, how he built a business around testing distributed systems, his writing process, favorite databases, and more.
Ronak & Guang’s Picks
#1 Key to being a better writer is to reduce filter
Like many, I want to write better and, more importantly, write more often, but have a hard time starting. Kyle's pro-tips: "This is obviously not great advice, but some people say get drunk; maybe have a gummy – the idea is to reduce your filter. You want to get yourself in a state, whether sober and just doing it emotionally or with your favorite whiskey in hand, of being able to just spew thoughts onto the page and then refine. Uh, but if you're struggling to get over the hump, I find that often the challenge is you're trying to get everything perfect upfront. And I like to think of writing as a tool for thinking, and nobody else is going to see your bad draft; you can always throw it away."
I'm gonna give this a try with a PBR next time 😏
#2 Finding a pricing model that works
Pricing is hard, especially for a consulting business where hourly-based pricing is too open-ended for the clients and project-based pricing is too risky for Overruns. Kyle puts it well - “In the software world, numbers are insane. Nobody knows what anything costs. Any figure you quote is going to be both 10X less and 100X more than two different clients are willing to pay”.
The issue Kyle ran into earlier on was that “I didn't anticipate that I would spend so much time doing lead acquisition and bookkeeping. And also that when you charge less and you do sort of piece work, such as 30 minutes here, two hours there, clients will want you to work for an hour here and then three hours next week and two hours next week. And it should all be on their schedule. And what you end up with is a work week that contains like 10 hours of actual work.” This led to Kyle significantly undervaluing his labor.
The solution? “Rather than offer an hour here an hour there, you engage me for a number of weeks and we can decide how many that is. You could say, 'Okay, next week is our last week.' That's great. Or you could say, 'I want to extend.' That's great. But you're buying a whole week of time. And then I can really sink my teeth into the problem, have all the context I need, and, ideally give you a good result.”
Segments:
(00:03:29) From Physics to Software Engineering
(00:07:47) The origins of Jepsen
(00:09:41) Turning Jepsen into a full-time venture
(00:13:14) Jepsen's testing philosophy
(00:16:30) The consulting journey
(00:19:16) Structuring a consultancy
(00:22:32) Setting boundaries
(00:24:32) Pricing misadventures
(00:29:17) Pros and cons of being an independent consultant
(00:32:08) Managing your time when working for yourself
(00:38:23) Best part of the job
(00:41:13) Early writing influences
(00:45:25) LLMs and AI-generated content
(00:48:17) “The period where you can trust what you read is actually very recent”
(00:51:33) How to become a better writer
(00:54:25) Developing a formal understanding of distributed systems
(00:59:30) Common faults in distributed systems
(01:01:17) The complexity of testing distributed systems
(01:07:32) Communicating criticism effectively
(01:10:26) Advice for distributed systems engineers
(01:13:46) “Anybody trying to sell you a distributed lock is selling you sawdust and lies”
(01:16:31) Failure mode documentation
(01:18:52) The pitfalls of containerization
(01:20:17) Lightning round - favorite databases
Show Notes:
“Anybody who is trying to sell you a distributed lock is trying to sell you sawdust and Lies”: https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html
Kyle’s excellent write-ups on testing distributed systems: https://jepsen.io/analyses
Kyle’s blog: https://aphyr.com/posts
Training courses that Kyle runs: https://jepsen.io/services/training
Stay in touch:
👋 Make Ronak’s day by leaving us a review and let us know who we should talk to next! hello@softwaremisadventures.com
Music: Vlad Gluschenko — Forest License: Creative Commons Attribution 3.0 Unported: https://creativecommons.org/licenses/by/3.0/deed.en