Lessons from the early days building Kafka and Confluent | Jay Kreps

Jay is the CEO of Confluent and co-creator of Apache Kafka

---

From writing the first lines of Kafka over a Christmas break as a LinkedIn engineer to running a public company as the CEO of Confluent, Jay joins the show to chat about how he and his co-founders convinced investors to take a chance on their vision, what many engineers get wrong about communication, and why engineers can make great CEOs - even when coding is not in the job description. And much more.

Highlights

#1 Do engineers make good CEOs?

Engineering is probably the least useful skill for becoming a CEO since you don’t need - nor want - the CEO to write a lot of code. But there are 2 attributes of an engineer that translate quite well into the CEO role:

The ability to learn new things, quickly

As an engineer, you're constantly learning new things, which builds the confidence in tackling unfamiliar subjects. This ability to learn and understand various aspects of the business is crucial for a CEO, who needs to oversee multiple functions and make informed decisions.

Systems thinking

As companies scale, they become complex systems that require a unique kind of thinking. Engineers, particularly those with experience in distributed systems, possess this skillset. They understand how to design and optimize systems, identifying areas of high coupling and optimizing feedback loops.

This system thinking is crucial in software engineering, and it's equally valuable in running a larger company. Engineers know how to peer inside a "module" (or department), get it in good shape, and measure its outputs. They can reason about inputs and investments, and make data-driven decisions.

#2 Many engineers communicate in a way that provokes the opposite of what they want to happen

As engineers, we often focus on conveying facts, but that's only half the battle. To truly communicate effectively, we need to consider our audience's perspective and tailor our message to drive the desired outcome.

Unfortunately, many engineers fall into the trap of communicating in a way that actually provokes the opposite reaction. The solution isn't to become overly cautious or vague, but to develop a deeper understanding of our audience and what will drive change. By focusing on the impact we want to have and honing our communication skills over time, we can become more effective leaders and change-makers.

Segments:

(00:01:16) The Shaved Head Bet

(00:04:07) Fundraising

(00:12:16) The Role of Technical Background in VCs

(00:15:48) The power of believing in the possibility of important changes

(00:18:29) The Journey to starting Confluent

(00:27:11) Kafka's Controversial Beginnings

(00:34:30) Effective Communication in Engineering

(00:44:20) The Early Days of Kafka

(00:48:31) The Power of Storytelling

(00:57:19) Early days of Confluent

(01:03:06) Do Engineers Make Good CEOs?

(01:07:59) A Typical Day in the Life of a CEO

(01:12:24) The Evolution of Data Streaming

Show Notes:

• “The log” blog post that solidified Jay and his co-founders’ conviction to found Confluent: https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying

• Jay on twitter: https://x.com/jaykreps

Stay in touch:

👋 Make Ronak’s day by leaving us a review and let us know who we should talk to next! hello@softwaremisadventures.com

Music: Vlad Gluschenko — Forest License: Creative Commons Attribution 3.0 Unported: https://creativecommons.org/licenses/by/3.0/deed.en