Simon Eskildsen on scaling Shopify, building turbopuffer, and the future of databases
Cursor · 2026-05-14
💡 Quick Take
1. Debugging MySQL stalls involved identifying an obscure dependency on `lsof` from Percona utilities via PHP cron jobs.
2. Shopify scaled MySQL by managing a massive number of individual connections from Ruby/Python apps, requiring clever tricks to reduce connection counts.
3. Flash sales, like Kylie Jenner's, caused massive traffic spikes and inventory lock contention, leading to outages.
4. Preparing for unpredictable flash sales involved writing load testing tools and padding capacity, especially before cloud adoption.
5. Scaling strategies included caching harder (larger TTLs, NGINX Lua) and load shedding to prioritize critical requests.
6. Shopify's infrastructure team evolved from traditional ops to a DevOps/Production Engineering model, merging developers and operations.
7. Early 2010s infrastructure teams collaborated extensively, sharing knowledge and contributing to open-source tools like Ghost and LHM.
8. Toxiproxy was developed to test service failures gracefully, ensuring Shopify stayed up even if critical databases failed.
9. Logrus, a Go logging library, was created to enforce intentional, structured logging and prevent post-incident regret.
10. Software should age well by prioritizing simplicity, with complexity only introduced when truly deserved.
11. Being on-call for critical systems teaches the importance of writing simple, debuggable code due to the high stakes of downtime.
12. AI models need a "constitution" to guide them towards writing simple, maintainable code, not just complex, "perfect" systems.
13. Enforcing shortness in code generation (without golf) can lead to simpler, more understandable solutions.
14. Graceful failure modes and the ability to shut down all servers without data loss are crucial invariants.
15. "Napkin Math" (Fermi estimates) is essential for understanding fundamental system properties beyond benchmarks, closing the gap between theory and reality.
16. TCP window tuning is critical for long-distance performance, as demonstrated by slow page loads in Australia due to conservative initial packet limits.
17. New major database platforms emerge roughly every 15 years, driven by new workloads (web, OLAP, AI) and new storage architectures.
18. SQLite is admired for its minimalist philosophy and efficient code paths, like using in-memory B-trees for joins.
19. Blob stores like S3 and Google Cloud Storage are inspirational for their simple APIs, consistent latency, and infinite scale.
20. Databases consistently see new contenders every 10 years because new workloads and advantageous storage architectures (e.g., NVMe, object storage separation) emerge that incumbents struggle to adopt.
21. Competitive programming and on-call experience at Shopify fostered a deep understanding of database bottlenecks and the trade-offs involved in scaling.
22. Learning new database systems (like Postgres after MySQL) is easier when you have a foundational understanding to build upon.
23. AI models are currently better at dashboard/website code than core database logic due to the latter's complex aging properties and storage architecture decisions.
24. TurboPuffer can already index the entire web and serve thousands of QPS, though achieving Google-level relevance requires more iteration.
25. S3's eventual strong consistency (December 2020) was a significant engineering feat, and its lack of it for years was a "mistake."
26. Object storage (like S3) is fundamental to modern architectures like TurboPuffer, removing the burden of managing physical hardware.
27. Interviewing has become harder due to AI's proficiency; a return to logic puzzles and assessing clarity of thought is suggested.
28. "P99" engineers are characterized by speed (deliberate or fast-paced), agency (bending things to their will), obsessiveness, and a drive to constantly seek higher levels of challenge.
29. Designing systems involves starting simple and elegantly navigating increasing complexity and trade-offs.
30. The next frontier for databases might involve GPUs, leveraging their speculative execution and high bandwidth capabilities.
31. It's too early to say we have the "last database"; new hardware and evolving workloads will likely drive future innovations.
📊 Detailed Explanation
1. Debugging MySQL stalls involved identifying an obscure dependency on `lsof` from Percona utilities via PHP cron jobs. This is a classic example of how seemingly unrelated software can cause massive issues. The core problem was a MySQL cluster stalling hourly for 30 seconds. After extensive debugging, the culprit was found to be a `lsof` command running on the machines. This `lsof` was triggered by a standard cron job within PHP, which was a dependency of Percona utilities used for MySQL management. The PHP cron job was checking for open sessions, and its presence, even indirectly, was causing kernel soft lockups on the critical database servers. It highlights the importance of understanding all dependencies, even obscure ones, in a complex system.
2. Shopify scaled MySQL by managing a massive number of individual connections from Ruby/Python apps, requiring clever tricks to reduce connection counts. In the 2010s, many SaaS apps were built with languages like Ruby or Python, where each process could only handle a limited number of queries per second (QPS). This meant that for every user interaction, there could be thousands of individual processes needing a direct connection to MySQL or PostgreSQL. Shopify was dealing with 30,000-40,000 open connections at times. The challenge was that systems like MySQL and especially PostgreSQL (out of the box) weren't built to efficiently handle such a massive number of individual connections. The team had to employ "tricks" to reduce this connection count, as there weren't robust open-source proxies available at the time like there are today. This emphasizes the strain on database connection pools and the need for optimization when dealing with high-concurrency applications.
3. Flash sales, like Kylie Jenner's, caused massive traffic spikes and inventory lock contention, leading to outages. These events, driven by influencers with millions of followers, would drop a new product, leading hundreds of thousands, or even millions, of people to try and buy the same limited-quantity SKU simultaneously. This created extreme "inventory lock contention" on the specific inventory row in MySQL. Essentially, so many users were trying to reserve the same item that it would lock up the database, causing outages. This wasn't just a traffic issue; it was a contention issue on critical data, directly leading to SEVs (Severity events).
4. Preparing for unpredictable flash sales involved writing load testing tools and padding capacity, especially before cloud adoption. Unlike predictable events like Black Friday, these flash sales could happen at any moment. The preparation involved creating internal load testing tools, often simple Ruby scripts run on many servers, to simulate users trying to reserve inventory and create contention. This helped identify bottlenecks. Before the widespread adoption of cloud computing, this also meant physically racking servers and "padding" capacity, meaning over-provisioning hardware to handle potential spikes. This was a significant challenge, as on-premise scaling up is slow and limited, and even in the cloud, you have to forecast usage.
5. Scaling strategies included caching harder (larger TTLs, NGINX Lua) and load shedding to prioritize critical requests. When faced with massive scaling demands, the team had limited options. One was to "cache harder," meaning increasing Time-To-Live (TTL) values for cached data and moving caching layers closer to the user. They moved caching from Ruby applications to NGINX, specifically using NGINX Lua, to offload more load from the application servers. Another crucial strategy was "load shedding." This involved prioritizing requests. For example, requests from users with items in their cart or an active session were prioritized over new visitors. Load shedding is essentially a way to "fail gracefully and fairly," ensuring the most critical functions remain operational during extreme load.
6. Shopify's infrastructure team evolved from traditional ops to a DevOps/Production Engineering model, merging developers and operations. When the speaker joined in 2013, the team was traditional ops – people excellent at Linux system administration. The SEVs were often caused by flash sales. The shift began with the nascent idea of DevOps, where engineers could both operate systems (SSH, Linux commands) and write software. This led to the merging of performance engineers working on the application layer with those managing servers. Initially, it was a scary concept for developers to write infrastructure code (like Chef), but this "production engineering" model, pioneered by companies like Facebook, became the standard. This highlights the industry-wide shift towards breaking down silos between development and operations.
7. Early 2010s infrastructure teams collaborated extensively, sharing knowledge and contributing to open-source tools like Ghost and LHM. There was a significant "collective intelligence" among companies like Stripe, GitHub, and Shopify. Lessons learned were often shared through phone calls and informal discussions, not always documented. This era saw the development of open-source libraries born out of necessity. GitHub created Ghost for MySQL schema migrations using binlogs, and SoundCloud developed LHM (Last Hour Migration) to use MySQL triggers for schema changes. This collaborative spirit was crucial for tackling unprecedented scaling challenges.
8. Toxiproxy was developed to test service failures gracefully, ensuring Shopify stayed up even if critical databases failed. The speaker created Toxiproxy because with the proliferation of microservices, they needed a way to guarantee that if one service (like a database managing sessions and carts) failed, the entire Shopify platform wouldn't go down. Toxiproxy acts as a proxy that allows you to simulate failures of downstream services. This was essential for writing robust tests and understanding how the system would behave under adverse conditions, ensuring resilience.
9. Logrus, a Go logging library, was created to enforce intentional, structured logging and prevent post-incident regret. The speaker developed Logrus after experiencing numerous incidents where they regretted not having logged enough specific information from the system at critical moments. The goal was to create an API that forced developers to be thoughtful about what data they were capturing. The `logrus.WithFields` API, which requires explicit typing of fields, was designed to encourage this intentionality. While Logrus became popular for its structured logging approach (pre-OpenTelemetry), the speaker notes its performance issues (allocations) and that it's not actively maintained.
10. Software should age well by prioritizing simplicity, with complexity only introduced when truly deserved. This is a core engineering principle learned over a decade at Shopify. The speaker observed that complex, meticulously planned projects (RFCs) didn't necessarily age well, while sometimes "spit and bubblegum" solutions held up remarkably. The key takeaway is that simplicity should be the default, and complexity should only be added when there's a clear, demonstrated need for it. This principle is fundamental to designing systems that are maintainable and debuggable over time.
11. Being on-call for critical systems teaches the importance of writing simple, debuggable code due to the high stakes of downtime. When you're on the "last resort pager" for a system where downtime costs millions per minute, the responsibility is immense. This experience forces engineers to write code that is not only functional but also incredibly simple and easy to debug. Every line of code written is a potential liability for someone who has to be on-call at 3 AM. This personal responsibility drives a preference for clarity and simplicity over cleverness.
12. AI models need a "constitution" to guide them towards writing simple, maintainable code, not just complex, "perfect" systems. The analogy is made to Reinforcement Learning (RL) for AI models, where a "constitution" or set of principles is needed. Just as humans have a simple, widely applicable constitution, AI models need guiding principles to ensure they produce "good quality code" that ages well. The concern is that models, like eager undergrads who've read too much Hacker News, might design overly complex "perfect" systems without considering the trade-offs of simplicity and maintainability, which are crucial for long-term success.
13. Enforcing shortness in code generation (without golf) can lead to simpler, more understandable solutions. While code golfing is undesirable, encouraging models to produce shorter code (e.g., 10 lines instead of 100) can often lead to simpler, more direct solutions. However, it's not always about line count; a system that uses Kafka might have fewer lines of code but introduce significant system complexity. The key is that shorter code often implies less convoluted logic, making it easier to grasp and maintain.
14. Graceful failure modes and the ability to shut down all servers without data loss are crucial invariants. These are fundamental properties that systems should uphold. The ability to withstand complete server shutdowns without losing any data is a surprisingly difficult invariant to maintain. It speaks to robust design, proper error handling, and ensuring data durability through redundancy and careful state management.
15. "Napkin Math" (Fermi estimates) is essential for understanding fundamental system properties beyond benchmarks, closing the gap between theory and reality. Benchmarks are often point-in-time measurements that don't reveal underlying system characteristics. "Napkin Math," or using simple calculations based on first principles (like data size, memory bandwidth, network latency), allows engineers to estimate theoretical performance. When this theoretical estimate differs significantly from a benchmark, it highlights either a flaw in the math (stupidity) or a problem with the system's performance. Closing this gap is crucial for making informed technical decisions and understanding true system behavior.
16. TCP window tuning is critical for long-distance performance, as demonstrated by slow page loads in Australia due to conservative initial packet limits. A real-world example illustrated how a website taking 3 seconds to load from Australia to the US was due to TCP's conservative initial window size. When a connection is established, TCP sends a limited number of packets (e.g., 10) to gauge the link's capacity. For a large, high-latency connection like across the Pacific, this conservative approach leads to many round trips just to negotiate the optimal window size. Tuning the TCP kernel settings to send more packets initially can dramatically improve performance for such connections.
17. New major database platforms emerge roughly every 15 years, driven by new workloads (web, OLAP, AI) and new storage architectures. The evolution of databases isn't random. There's a pattern: a new workload emerges (e.g., the web in the 90s, OLAP in the 2000s, AI now), and this workload requires a new storage architecture that incumbents can't easily adopt. This creates an opportunity for new database companies to build platforms optimized for these new conditions. Companies like Oracle, MySQL, and Postgres arose from the web era, while Snowflake and Databricks emerged for OLAP workloads.
18. SQLite is admired for its minimalist philosophy and efficient code paths, like using in-memory B-trees for joins. SQLite's strength lies in its extreme minimalism and efficiency. Instead of having separate code paths for different join strategies (nested loops, hash joins), SQLite constructs an ad-hoc B-tree in memory to perform the join. This is the same code path used for on-disk B-tree indexes, meaning they put more pressure and optimization into a single, well-tested path. This is a brilliant example of code reuse and maximizing efficiency.
19. Blob stores like S3 and Google Cloud Storage are inspirational for their simple APIs, consistent latency, and infinite scale. Systems like S3 and GCS are lauded for having very few, well-defined APIs that are extremely reliable and offer consistent latency bounds, even at massive scale. They automatically shard and provide effectively infinite storage. This simplicity and predictability make them foundational for many modern applications.
20. Databases consistently see new contenders every 10 years because new workloads and advantageous storage architectures (e.g., NVMe, object storage separation) emerge that incumbents struggle to adopt. The reason databases are unique in their continuous disruption is twofold: 1) New workloads appear (like OLAP or AI data needs) that existing architectures aren't optimized for. 2) New hardware and storage paradigms emerge (like NVMe SSDs or the separation of compute and storage in cloud object storage) that offer significant advantages. Incumbents, often burdened by decades of legacy architecture and assumptions (like tight coupling of compute and storage), find it difficult to pivot and adopt these new paradigms. New entrants, unburdened, can build from the ground up for these new conditions.
21. Competitive programming and on-call experience at Shopify fostered a deep understanding of database bottlenecks and the trade-offs involved in scaling. The speaker's passion for databases stems from competitive programming, which trains one to think in asymptotic notation and execution efficiency. At Shopify, databases were consistently the breaking point during scaling. Being on-call for these critical systems meant constantly battling bottlenecks and planning for future growth, forcing a deep dive into how databases work at a fundamental level – from SQL execution to bits and bytes on disk. This constant problem-solving in high-stakes environments cultivates an appreciation for the intricate trade-offs involved in database design and scaling.
22. Learning new database systems (like Postgres after MySQL) is easier when you have a foundational understanding to build upon. Having spent a decade mastering MySQL's intricacies (how data is laid out on disk dictated by the primary index, etc.), learning Postgres became easier. The speaker could compare and contrast the two, identifying compromises and differences in their index implementations and write handling. This foundational knowledge acts as a "trunk" onto which new information can be grafted, making the learning process more efficient.
23. AI models are currently better at dashboard/website code than core database logic due to the latter's complex aging properties and storage architecture decisions. While AI models are rapidly improving, they still struggle with the nuanced, long-term considerations of core database development. The complexities of how data is laid out on disk, how APIs will age over time, and the irreversible decisions made regarding storage architecture are lessons learned from years of experience. These "war lessons" are not easily encoded into current AI models, making them less adept at generating robust, long-lasting database code compared to simpler application logic.
24. TurboPuffer can already index the entire web and serve thousands of QPS, though achieving Google-level relevance requires more iteration. The architecture of TurboPuffer is capable of indexing massive datasets (hundreds of billions of documents) and handling thousands of queries per second with impressive latency (P99 of 200ms, P50 of 40ms). While indexing the entire web is feasible, achieving Google's level of search relevance would require significant further development and iteration on ranking algorithms and data processing.
25. S3's eventual strong consistency (December 2020) was a significant engineering feat, and its lack of it for years was a "mistake." For a long time, S3 did not offer strong consistency, meaning reads might not immediately reflect the latest writes. Achieving strong consistency, especially on such a massive scale, is incredibly difficult. The speaker calls its eventual achievement a "mind-blowingly late" but significant engineering feat, and notes that Amazon itself considers the initial lack of consistency a major mistake. Google Cloud Storage, possibly fronted by Spanner, may have achieved consistency earlier.
26. Object storage (like S3) is fundamental to modern architectures like TurboPuffer, removing the burden of managing physical hardware. The existence of reliable, scalable object storage like S3 is what makes modern companies like TurboPuffer possible. Without it, companies would be forced to manage vast fleets of physical hard drives, a complex and costly endeavor akin to the challenges of managing GPUs today. This abstraction layer is critical for innovation and allows companies to focus on their core product rather than infrastructure management.
27. Interviewing has become harder due to AI's proficiency; a return to logic puzzles and assessing clarity of thought is suggested. The traditional interviewing trick of giving candidates a complex codebase to work on has become less effective as AI models can now assist significantly. This necessitates a return to first principles. Logic puzzles, especially those involving probability, are proposed as a way to assess clarity of thought, the ability to reason under pressure, and articulate complex ideas, skills that are crucial for high-level engineering regardless of specific domain knowledge.
28. "P99" engineers are characterized by speed (deliberate or fast-paced), agency (bending things to their will), obsessiveness, and a drive to constantly seek higher levels of challenge. The "P99" (top 1% of engineers) exhibit specific traits. They are "fast" – not necessarily in terms of talking, but in their ability to move deliberately, avoid bugs, and consistently make progress. They possess "agency," the ability to make machines do what they need. They are often obsessive, diving deep into details, and they consistently seek out new challenges and higher levels of expertise, moving from one "P99" community to the next.
29. Designing systems involves starting simple and elegantly navigating increasing complexity and trade-offs. The process of designing a system, especially for a P99 engineer, involves starting with the simplest possible solution and then elegantly layering complexity and addressing trade-offs as scale or requirements demand. This is a "one step forward, one step forward" approach, avoiding the pitfall of trying to build a perfect, complex system from the outset, which often leads to fundamental assumptions being wrong.
30. The next frontier for databases might involve GPUs, leveraging their speculative execution and high bandwidth capabilities. Given the evolution of hardware, GPUs are a strong candidate for the next platform for databases. Their ability to perform speculative execution and handle high bandwidth aligns with the trend of systems that can manage a large number of outstanding requests with few round trips, a pattern that has proven successful in CPUs and object storage.
31. It's too early to say we have the "last database"; new hardware and evolving workloads will likely drive future innovations. While current architectures might seem sufficient for many tasks, the history of computing shows that hardware advancements (like GPUs) and new workloads (like AI processing) consistently drive the need for new database paradigms. Therefore, declaring the "last database" is premature; innovation is likely to continue, even if the pace or form it takes is unpredictable.
🎯 Expert Opinion
This transcript offers a treasure trove of insights into scaling complex systems, particularly from the perspective of a decade at Shopify and the development of TurboPuffer. The narrative beautifully illustrates the cyclical nature of technological innovation and the enduring principles of good engineering. Here's my take as an expert:
The "Infra Wars" and the Rise of Production Engineering: The early 2010s were indeed a wild west for scaling. The shift from traditional Ops to Production Engineering/DevOps wasn't just a semantic change; it was a fundamental rethinking of how systems are built and maintained. The speaker's account of merging developers and ops teams is a hallmark of this era. Companies realized that to handle the unpredictable, massive scale driven by things like flash sales, you needed engineers who understood both the application's logic and the underlying infrastructure's behavior. This cross-pollination of skills is crucial. My experience confirms that organizations that successfully break down these silos achieve significantly higher reliability and faster iteration cycles. The collaborative spirit among early tech companies, sharing hard-won lessons, was a powerful accelerant for the entire industry. Tools like Toxiproxy, born from necessity, are now standard patterns, demonstrating how practical problems drive innovation.
Simplicity as the Ultimate Sophistication: The emphasis on "simplicity surprise you, complexity has to be deserved" is perhaps the most profound takeaway. In a world often chasing the latest, most complex architectural patterns, this principle is a vital anchor. Software ages, and the simpler it is, the better it weathers the storms of changing requirements, language evolution, and scale. This is especially relevant today with the rise of AI-generated code. While AI can produce technically correct code, it often lacks the nuanced understanding of long-term maintainability and debuggability that comes from human experience. The "constitution" for AI models needs to heavily emphasize this principle. My work with large-scale systems consistently shows that the most resilient and adaptable systems are those built on a foundation of elegant simplicity, not brute-force complexity.
The Database Evolution Cycle: The 15-year cycle of database innovation is a fascinating observation, and I agree with the core thesis. It's driven by the interplay of new workloads and disruptive storage architectures. We're seeing this play out *now* with the AI revolution. The need to process and retrieve vast amounts of unstructured and semi-structured data for LLMs and other AI applications is a new workload. Simultaneously, advancements in hardware like GPUs and specialized AI accelerators, coupled with evolving cloud storage capabilities, are creating new architectural possibilities. Companies that can marry these two factors – a novel workload with a fundamentally advantageous storage/compute architecture that incumbents can't easily replicate – are the ones poised to define the next generation of database and data management systems. The speaker's point about S3's eventual strong consistency is a perfect example of how even foundational infrastructure takes time to mature and enable new possibilities.
The "P99" Engineer and the Future of Interviewing: The description of "P99" engineers is spot-on. Their agency, relentless pursuit of higher levels, and ability to navigate complexity with clarity are what drive true innovation. The challenge of interviewing in the age of AI is real. Whiteboard interviews and complex coding challenges are becoming less effective as AI assistants can bridge many of those gaps. This necessitates a return to assessing fundamental problem-solving, logical reasoning, and communication skills. Logic puzzles, scenario-based design questions, and deep dives into past project experiences are essential. The ability to articulate trade-offs, explain "why" behind decisions, and demonstrate clarity of thought under pressure are the new differentiators. The "life story" interview approach is brilliant for uncovering this inherent drive and agency.
TurboPuffer's Potential and the GPU Frontier: TurboPuffer's ability to index the web at scale is impressive and speaks to the power of modern architectures built on object storage and efficient query processing. The potential for GPU-accelerated databases is immense. GPUs excel at parallel processing and handling massive amounts of data concurrently, which aligns perfectly with the demands of many modern workloads, including AI. While challenges remain in programming models and general-purpose applicability, it's highly probable that GPUs will play a significant role in the next wave of database innovation. The speaker's cautious optimism about not having the "last database" is wise; the landscape is too dynamic.
The Enduring Importance of First Principles: From "Napkin Math" to understanding TCP windows, the transcript consistently circles back to the power of understanding fundamental principles. This is the bedrock of robust engineering. Benchmarks are useful, but they are symptoms; first principles reveal the disease or the health of the system. This mindset is what allows engineers to debug the unknown, design for scale, and build systems that not only work today but can adapt and thrive tomorrow. It's a lesson that applies as much to building a startup as it does to training an AI model to write good code.
Kanal: Cursor