Become a LLM-ready Engineer | Maxime Beauchemin (Airflow, Preset)

Max is the creator of Apache Airflow, Apache Superset and the founder of Preset.

---

If you’ve worked on data problems, you probably have heard of Airflow and Superset, two powerful tools that have cemented their place in the data ecosystem. Building successful open source softwares is no easy feat, and even fewer engineers have done this back to back. In Part 1 of this conversation, we chat about how to adapt to the LLM-age as engineers.

Highlights:

#1: Building a reflex for trying with AI first.

The AI revolution is fundamentally changing the way we live and work, comparable only to the impact of the internet. To thrive in this new era, it's essential to develop a "first reflex" to leverage AI in our daily workflows. This means trying to solve problems with AI assistance before attempting to do them on our own.

In practice, this means integrating AI tools, such as language models like Claude, into our daily tasks. Before starting a project, ask your AI assistant for help, and see if it can provide useful code snippets, input documentation, or even assist with writing emails or designing data models. By working with AI, you can create a feedback loop that streamlines your work and increases productivity. Don't try to do it alone – let AI be your first reflex.

#2: being a SQL monkey is probably not going to cut it anymore when AI is a better SQL monkey than we are

The data engineering landscape is on the cusp of a significant shift. With the advent of AI-powered tools, being a skilled SQL expert may no longer be enough. According to Max, founder of Apache Superset, AI is rapidly becoming a better "SQL monkey" than humans. While AI excels at writing SQL, it lacks essential skills like executive judgment, long-term memory, and business context.

As AI takes over routine SQL tasks, data engineers will need to adapt and focus on higher-level skills like providing context, understanding data models, and making strategic decisions.

#3: Promptimize: bringing Test Driven Development to Prompt Engineering

In the world of AI-generated data models, prompt engineering is crucial for achieving accurate results. However, measuring the quality of prompts and comparing their performance can be a daunting task. This is where Promptimize comes in - a toolkit designed to bring scientific rigor to prompt engineering.

With Promptimize, you can write test cases for your prompts, similar to unit testing frameworks, and measure their success rates against different AI models like GPT-3.5 Turbo or GPT-4. This allows you to identify strengths and weaknesses, compare performance, and optimize your prompts for better outcomes.

Segments:

[00:01:59] The Rise and Fall of the Data Engineer

[00:11:13] The Importance of Executive Skill in the Era of AI

[00:13:53] Developing the first reflex to use AI.

[00:17:47] What are LLMs good at?

[00:25:33] Text to SQL

[00:28:19] Promptimize

[00:32:16] Using tools LangChain

[00:35:02] Writing better prompts

Show Notes:

• Max on Linkedin: https://www.linkedin.com/in/maximebeauchemin/

• Rise of the Data Engineer: https://medium.com/free-code-camp/the-rise-of-the-data-engineer-91be18f1e603

• Downfall of the Data Engineer: https://maximebeauchemin.medium.com/the-downfall-of-the-data-engineer-5bfb701e5d6b

• Promptimize: https://github.com/preset-io/promptimize

Stay in touch:

👋 Make Ronak’s day by leaving us a review and let us know who we should talk to next! hello@softwaremisadventures.com