<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom" xmlns:media="http://search.yahoo.com/mrss/">
  <title>The Fly Blog</title>
  <subtitle>News, tips, and tricks from the team at Fly</subtitle>
  <id>https://fly.io/blog/</id>
  <link href="https://fly.io/blog/"/>
  <link href="https://fly.io/blog/" rel="self"/>
  <updated>2026-11-13T00:00:00+00:00</updated>
  <author>
    <name>Fly</name>
  </author>
  <entry>
    <title>Throwing Out The Phonebook</title>
    <link rel="alternate" href="https://fly.io/blog/throwing-out-the-phonebook/"/>
    <id>https://fly.io/blog/throwing-out-the-phonebook/</id>
    <published>2026-11-13T00:00:00+00:00</published>
    <updated>2026-11-17T14:21:47+00:00</updated>
    <media:thumbnail url="https://fly.io/blog/throwing-out-the-phonebook/assets/frankie-on-the-phone-with-servers-c20b.webp"/>
    <content type="html">&lt;div class="lead"&gt;&lt;p&gt;There’s really only one hard problem in software engineering: synchronization. OK, maybe two: cache invalidation and synchronization. Synchronization is the harder one though, especially when the thing on the other side is eventually consistent. This year we rebuilt 6PN, the private network between your Fly Machines, and we did it by avoiding the hard thing.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;To start, here&amp;rsquo;s the status quo: every Fly Machine gets a private IPv6 address, and every other Machine in your organization can reach it. We call that network 6PN. It&amp;rsquo;s what &lt;code&gt;my-db.internal&lt;/code&gt; points at, and it&amp;rsquo;s how your app server talks to your database without either of them touching the public internet.&lt;/p&gt;

&lt;p&gt;Until this year, every host in our fleet carried a phonebook: a map of every Fly Machine that had ever moved, and where it lives now. Hundreds of thousands of entries, on every host, kept up to date in real time. Phonebooks are supposed to fit inside your house, and this was getting, uh…unwieldy. So we changed it! The new design keeps, in total, somewhere between 100 and 200 records. There&amp;rsquo;s no clever memorization mechanics and when a Machine calls a number it doesn&amp;rsquo;t know, the host looks it up, writes it down, and moves on.&lt;/p&gt;

&lt;p&gt;Here&amp;rsquo;s the story of how we got into the phonebook business, and how we got out by the skin of our teeth. (It wasn&amp;rsquo;t that dramatic, but this design does work better.)&lt;/p&gt;
&lt;h2 id='the-address-is-the-route' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#the-address-is-the-route' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;The address is the route&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;A 6PN address isn&amp;rsquo;t random. It carries its own routing information:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative "&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-gju0za4"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-gju0za4"&gt;fdaa:netid:netid:hostid:hostid:mach:mach:2
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;&lt;code&gt;netid&lt;/code&gt; is your organization&amp;rsquo;s private network. &lt;code&gt;hostid&lt;/code&gt; is the physical server the Machine was born on. &lt;code&gt;mach&lt;/code&gt; is the Machine. Our servers talk to each other over a &lt;a href='/blog/corrosion/' title=''&gt;fully connected WireGuard mesh&lt;/a&gt;, and each host owns a &lt;code&gt;/48&lt;/code&gt; on it: &lt;code&gt;fdaa:hostid:hostid::/48&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;You&amp;rsquo;ll notice the host ID sits in the wrong place for that. So &lt;code&gt;udp-xdp&lt;/code&gt;, an eBPF program hooked onto every Machine&amp;rsquo;s network interface, shuffled the bytes on the way out:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative "&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-upx74mwa"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-upx74mwa"&gt;fdaa:netid:netid:hostid:hostid:mach:mach:2   # what the Machine sees
fdaa:hostid:hostid:netid:netid:mach:mach:2   # what goes over the wire
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;Now the packet matches a kernel route for the destination host, WireGuard carries it there, and the far side swaps it back. There aren&amp;rsquo;t any lookups (or state, or databases). The address is the route.&lt;/p&gt;

&lt;p&gt;A truly elegant design! Until a Machine moves.&lt;/p&gt;
&lt;h2 id='machines-learn-to-move' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#machines-learn-to-move' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Machines learn to move&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;For a long list of reasons, sometimes we need to migrate Machines between hosts. A Machine that moves to a new host still has the old host&amp;rsquo;s ID baked into its address, which meant that it was pointing in the wrong direction.&lt;/p&gt;

&lt;p&gt;Our first fix was &lt;code&gt;dnat6pn&lt;/code&gt;: remember every address a Machine has ever had, and rewrite traffic aimed at an old one to the new one. That table grew faster than a 12th century rat population, and it was equally tricky to manage. The address a Machine saw on its own interface wasn&amp;rsquo;t the address other Machines used to reach it. Move an app to a different organization and it fell apart entirely. It even ended up breaking our own apps. It was strapped on top of the swapping design rather than built into it. Mistakes had, indeed, been made.&lt;/p&gt;

&lt;p&gt;So we flipped it around. A Machine&amp;rsquo;s canonical address became its &lt;em&gt;original&lt;/em&gt; address, forever, no matter where it lives. &lt;code&gt;udp-xdp&lt;/code&gt; took on translating every 6PN address into a &amp;ldquo;routable&amp;rdquo; one, with the host ID replaced by whichever host the Machine is on &lt;em&gt;right now&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;Which means &lt;code&gt;udp-xdp&lt;/code&gt;, on every host, needed to know where every migrated Machine in the fleet currently lives.&lt;/p&gt;

&lt;p&gt;That&amp;rsquo;s the phonebook. A BPF map on every host, fed by &lt;a href='/blog/corrosion/' title=''&gt;Corrosion&lt;/a&gt;, our gossip-replicated SQLite, through a subscription to every Machine update across the whole fleet.&lt;/p&gt;
&lt;h2 id='a-phonebook-in-elvish' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#a-phonebook-in-elvish' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;A phonebook in Elvish&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;All this worked…mostly.&lt;/p&gt;

&lt;p&gt;&lt;strong class='font-semibold text-navy-950'&gt;Life in a perpetual growth spurt.&lt;/strong&gt; eBPF can&amp;rsquo;t call out to userspace to fetch an entry when it needs one, so every entry has to be loaded ahead of time. The map grew steadily with no sign of slowing down. When we last looked, we were within the same order of magnitude as the cap we&amp;rsquo;d set on it.&lt;/p&gt;

&lt;p&gt;&lt;strong class='font-semibold text-navy-950'&gt;Every host had a different phonebook.&lt;/strong&gt; Every host was supposed to hold the same map. They held vastly different numbers of entries. There&amp;rsquo;s only one way to read that: synchronization wasn&amp;rsquo;t working, and we couldn&amp;rsquo;t tell you where.&lt;/p&gt;

&lt;p&gt;&lt;strong class='font-semibold text-navy-950'&gt;We couldn&amp;rsquo;t see it.&lt;/strong&gt; We had incidents where syncing Corrosion into the map broke. But we had no good way to monitor the map itself, so if it caused problems we didn&amp;rsquo;t hear about directly, we wouldn&amp;rsquo;t have known. And when we did go looking, the rewriting made debugging miserable: the address on the sending end and the address on the receiving end were different, so nothing lined up.&lt;/p&gt;

&lt;p&gt;&lt;strong class='font-semibold text-navy-950'&gt;It couldn&amp;rsquo;t route around anything.&lt;/strong&gt; Traffic over the WireGuard mesh went to the host in the address, full stop. Teaching it to take a detour around a bad link meant more logic in &lt;code&gt;udp-xdp&lt;/code&gt;, the component we already couldn&amp;rsquo;t debug.&lt;/p&gt;

&lt;p&gt;Sadly, you can&amp;rsquo;t fix a fundamental design flaw with a patch. Holding a real-time copy of an eventually consistent database, on every host in the fleet, inside the kernel; that&amp;rsquo;s the hardest kind of problem, and we were kind of sick of holding it.&lt;/p&gt;
&lt;h2 id='the-network-already-asks' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#the-network-already-asks' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;The network already asks&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;For unrelated reasons (Egress IPs), we&amp;rsquo;d ended up running a Layer 2 network, VXLAN, on top of the WireGuard mesh.&lt;/p&gt;

&lt;p&gt;VXLAN gives you an Ethernet segment over any IP network; all it needs is a UDP port. On a real Ethernet segment, a machine that doesn&amp;rsquo;t know where an address lives shouts for it with ARP or Neighbor Discovery, and someone answers. VXLAN can&amp;rsquo;t shout across an IP network. You either tell it about every peer in advance, or you fake the answer.&lt;/p&gt;

&lt;p&gt;If we have to tell it about every peer in advance, we&amp;rsquo;re back in the phonebook industry. So we fake it. When VXLAN sees an address it can&amp;rsquo;t map to a peer, it emits a netlink event, and &lt;code&gt;drift-agent&lt;/code&gt; listens for those and configures the peer on the spot.&lt;/p&gt;
&lt;div class="callout"&gt;&lt;p&gt;Drift is another clever internal tool. Look out for an upcoming post that will tell you all about it.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;Here&amp;rsquo;s the part that matters. That &amp;ldquo;I don&amp;rsquo;t know this address&amp;rdquo; event fires for &lt;em&gt;any&lt;/em&gt; unknown address VXLAN sees. Not just VXLAN peers. Anything. We&amp;rsquo;d built a mechanism that tells us, lazily and precisely, the moment a host needs to know something it doesn&amp;rsquo;t.&lt;/p&gt;

&lt;p&gt;That hefty phonebook is now smart directory assistance.&lt;/p&gt;
&lt;h2 id='ask-write-it-down-move-on' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#ask-write-it-down-move-on' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Ask, write it down, move on&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;&lt;code&gt;s6pnforwarderd&lt;/code&gt; is the daemon that used to load every migrated Machine into &lt;code&gt;udp-xdp&lt;/code&gt;&amp;lsquo;s map. Now it does almost nothing until someone asks.&lt;/p&gt;

&lt;p&gt;All machine 6PN traffic goes to its own routing table, &lt;code&gt;0xfdaa&lt;/code&gt;, which starts out with one rule:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative "&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-375gh4v9"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-375gh4v9"&gt;fdaa::/16 dev vxlan0
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;When a Machine sends a packet to another Machine&amp;rsquo;s 6PN address, &lt;code&gt;vxlan0&lt;/code&gt; has no idea where that is. It only knows about hosts. So it raises the &amp;ldquo;who is this?&amp;rdquo; event. &lt;code&gt;drift-agent&lt;/code&gt; handles host peers itself and passes everything else on over an HTTP SSE stream. &lt;code&gt;s6pnforwarderd&lt;/code&gt; is subscribed to that stream. It asks Corrosion&amp;rsquo;s local SQLite two questions: does this 6PN address exist, and which host is that Machine on right now? Then it writes one route:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative "&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-v6r3hujj"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-v6r3hujj"&gt;fdaa:netid:netid:hostid:hostid:mach:mach:2 via fdaf:newhostid:newhostid:: dev vxlan0
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;Every packet after that goes straight through. Every so often, &lt;code&gt;s6pnforwarderd&lt;/code&gt; re-checks the routes it has loaded in case a Machine has moved, and it uses Corrosion&amp;rsquo;s lightweight per-row update notifications to notice sooner. That part&amp;rsquo;s optional; the periodic check covers it.&lt;/p&gt;

&lt;p&gt;No translation needs to happen either. The address the Machine sees on its interface is the address on the wire is the address at the other end. When something goes wrong, both sides finally agree on what they&amp;rsquo;re looking at.&lt;/p&gt;
&lt;h2 id='what-changed' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#what-changed' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;What changed&lt;/span&gt;&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong class='font-semibold text-navy-950'&gt;The table is tiny.&lt;/strong&gt; On most hosts, &lt;code&gt;0xfdaa&lt;/code&gt; holds somewhere between 100 and 200 routes. The old map held hundreds of thousands.
&lt;/li&gt;&lt;li&gt;&lt;strong class='font-semibold text-navy-950'&gt;There&amp;rsquo;s no fleet-wide subscription.&lt;/strong&gt; A host learns about the Machines its Machines actually talk to, and nothing else.
&lt;/li&gt;&lt;li&gt;&lt;strong class='font-semibold text-navy-950'&gt;Nothing is permanently fatal.&lt;/strong&gt; Corrosion propagation is slow? A SQL query fails? The route gets re-checked on the next loop. And if a lookup fails outright, the sending Machine will try again anyway (TCP retransmits, that&amp;rsquo;s its whole personality), which fires another event, which triggers another lookup.
&lt;/li&gt;&lt;li&gt;&lt;strong class='font-semibold text-navy-950'&gt;We can finally see it.&lt;/strong&gt; We&amp;rsquo;ve put real work into monitoring for when this goes wrong, because we&amp;rsquo;ve essentially scoped down what each host cares about. Much more of this logic now uses standard kernel components and an observable userspace control plane. We had nothing equivalent for the old map.
&lt;/li&gt;&lt;/ul&gt;
&lt;div class="callout"&gt;&lt;p&gt;The &lt;code&gt;via&lt;/code&gt; host doesn’t have to be the host running the Machine. With forwarding on, a host that gets traffic for a Machine it no longer has can pass it to the host that does. Instead of breaking the connection, a stale route just needs an extra hop and voila. It also means we can send traffic around a bad link on purpose.&lt;/p&gt;
&lt;/div&gt;&lt;h2 id='safer-too' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#safer-too' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Safer too&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;We didn&amp;rsquo;t delete &lt;code&gt;udp-xdp&lt;/code&gt;. We let it keep doing its one job, since it&amp;rsquo;s fairly good at it: checking that the source and destination 6PN addresses belong to the same network. Your Machines can reach your Machines and nobody else&amp;rsquo;s.&lt;/p&gt;

&lt;p&gt;When &lt;code&gt;udp-xdp&lt;/code&gt; validates a packet, it stamps it with a &lt;code&gt;0xfdaa&lt;/code&gt; fwmark, and only stamped packets get routed through the &lt;code&gt;0xfdaa&lt;/code&gt; table to &lt;code&gt;vxlan0&lt;/code&gt;. That order is the important bit. If &lt;code&gt;udp-xdp&lt;/code&gt; fails to load, nothing gets stamped, and nothing crosses. The network fails closed. If you do it the other way around (route everything, then have a filter drop the bad stuff), then a filter that fails to load means that everything crosses.&lt;/p&gt;

&lt;p&gt;This check also doesn&amp;rsquo;t fit well in &lt;code&gt;nftables&lt;/code&gt;, because it compares fields from the packet against each other. In &lt;code&gt;udp-xdp&lt;/code&gt; it&amp;rsquo;s a few lines of code that depend on no external data at all. That&amp;rsquo;s the kind of thing that belongs in eBPF. Complex logic and a synchronized copy of the fleet don&amp;rsquo;t, and longer term we&amp;rsquo;d like to move UDP encapsulation out of there too.&lt;/p&gt;
&lt;h2 id='what-it-costs' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#what-it-costs' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;What it costs&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Lazy lookups have a price. In this case, it&amp;rsquo;s the first packet that&amp;rsquo;s forking over the toll.&lt;/p&gt;

&lt;p&gt;The lookup itself is fast, usually under 100ms on our side, mostly database I/O. But the first packet to a Machine that your Machine has never talked to arrives at a host that doesn&amp;rsquo;t have a route yet. With TCP, the retry is up to your Machine&amp;rsquo;s kernel, and Linux&amp;rsquo;s default retransmission timeout is one second. So the first connection between two Machines that have never spoken can take about a second. Every connection after that goes straight through.&lt;/p&gt;

&lt;p&gt;We&amp;rsquo;re going to fix that, most likely by holding the first few packets (up to a cap) while the lookup runs, instead of letting them fall on the floor.&lt;/p&gt;

&lt;p&gt;The other cost: if &lt;code&gt;drift-agent&lt;/code&gt; or &lt;code&gt;s6pnforwarderd&lt;/code&gt; crashes, routes that already exist keep working. What breaks is new conversations: Machines that have never talked to each other, and Machines that just migrated. The second one isn&amp;rsquo;t new; a freshly moved Machine was a failure mode under the old design too. The first one is new, but compared to loading every route in the fleet onto every host, all the time, well…this is the lesser evil we&amp;rsquo;ve chosen.&lt;/p&gt;
&lt;h2 id='changing-the-engine-at-cruising-altitude' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#changing-the-engine-at-cruising-altitude' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Changing the engine at cruising altitude&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;You don&amp;rsquo;t get to schedule a maintenance window for every private network on the platform. So we didn&amp;rsquo;t swap one system for the other. We ran both, and let each host opt in.&lt;/p&gt;

&lt;p&gt;The new VXLAN path only touches packets carrying the &lt;code&gt;0xfdaa&lt;/code&gt; fwmark, and &lt;code&gt;udp-xdp&lt;/code&gt; only sets that mark when all of these are true:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The new &lt;code&gt;s6pnforwarderd&lt;/code&gt; and &lt;code&gt;drift-agent&lt;/code&gt; are running on the host.
&lt;/li&gt;&lt;li&gt;&lt;code&gt;s6pnforwarderd&lt;/code&gt; has flipped a feature flag in &lt;code&gt;udp-xdp&lt;/code&gt;.
&lt;/li&gt;&lt;li&gt;The packet is outgoing 6PN traffic headed for another host, or it&amp;rsquo;s incoming traffic that arrived on &lt;code&gt;vxlan0&lt;/code&gt;, and specifically &lt;em&gt;not&lt;/em&gt; on &lt;code&gt;wg0&lt;/code&gt;.
&lt;/li&gt;&lt;/ol&gt;

&lt;p&gt;That last condition is what made it safe. A host on the new system sends over VXLAN. If the destination host hasn&amp;rsquo;t been upgraded yet, it sends its replies the old way over &lt;code&gt;wg0&lt;/code&gt;, where the swapping code still applies and nothing gives the packet the new mark. We kept the old map up to date the whole time, so old-style hosts kept working.&lt;/p&gt;

&lt;p&gt;The route is asymmetric for a while: out over VXLAN, back over WireGuard. That&amp;rsquo;s fine. We told the kernel to accept it.&lt;/p&gt;

&lt;p&gt;Then we turned it on, host by host, across the fleet. The rollout wasn&amp;rsquo;t noticeable in most regions; during the trial we had some temporary Flycast trouble in &lt;code&gt;yyz&lt;/code&gt;. Once every host was on the new path, we deleted the old one. As of last month, the swapping-based 6PN code is gone.&lt;/p&gt;
&lt;h2 id='rerouting-on-purpose' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#rerouting-on-purpose' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Rerouting, on purpose&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Because a route is just &amp;ldquo;send this to that host over VXLAN&amp;rdquo;, the next host doesn&amp;rsquo;t have to be the final one. That means we can send traffic around a degraded link, and build highly available gateways that don&amp;rsquo;t each need a full copy of the fleet.&lt;/p&gt;

&lt;p&gt;If you&amp;rsquo;ve been on Fly.io a while, rerouting might sound familiar. &lt;code&gt;fly-proxy&lt;/code&gt;, our load balancer, has done it for years. But it only ever covered traffic coming in through Anycast and Flycast, and it took what felt like a thousand state machines in the load balancer to pull off.&lt;/p&gt;

&lt;p&gt;Rerouting in the network itself works for basically everything, and it&amp;rsquo;s a lot simpler. So the &lt;code&gt;fly-proxy&lt;/code&gt; version is going to be put out to pasture soon. It was rerouting at the wrong layer.&lt;/p&gt;

&lt;p&gt;The old design knew where everything was, all the time, but it still couldn&amp;rsquo;t tell us when it was wrong. The new one knows almost nothing, and it&amp;rsquo;s the first version of 6PN we can trust to tell us when it breaks.&lt;/p&gt;

&lt;p&gt;Good riddance to the phonebook! Turns out you can just call the operator and ask.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Well, That Wasn't Fun: October 16 '26</title>
    <link rel="alternate" href="https://fly.io/blog/well-that-wasnt-fun/"/>
    <id>https://fly.io/blog/well-that-wasnt-fun/</id>
    <published>2026-11-06T00:00:00+00:00</published>
    <updated>2026-11-17T14:21:47+00:00</updated>
    <media:thumbnail url="https://fly.io/blog/well-that-wasnt-fun/assets/robot-bugs-firing-lasers-at-a-disk-eb13.webp"/>
    <content type="html">&lt;div class="lead"&gt;&lt;p&gt;October 16, 2026 began like any other day at Fly.io (meaning, we were anticipating only a &lt;em&gt;medium&lt;/em&gt; amount of chaos). By that evening, we had patched and rebooted our entire fleet in response to &lt;a href="https://www.cve.org/CVERecord/?id=CVE-2026-99999"&gt;a Linux kernel bug&lt;/a&gt;. The full rollout took 10 hours, but the emotional damage persists.&lt;/p&gt;
&lt;/div&gt;&lt;h2 id='cve-2026-99999' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#cve-2026-99999' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;CVE-2026-99999&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;The rest of this won&amp;rsquo;t make much sense without a clear read on the bug itself, so: the &lt;a href='https://lore.kernel.org/linux-cve-announce/2026101533-CVE-2026-99999-beee@gregkh/' title=''&gt;Linux kernel CVE team&lt;/a&gt; and &lt;a href='https://pwn.ai/blog/pwnescape' title=''&gt;pwn.ai&lt;/a&gt; announced CVE-2026-99999 on the morning of the 15th. We heard about it that night, and started working on mitigations.&lt;/p&gt;
&lt;h2 id='ip-sleight-of-hand' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#ip-sleight-of-hand' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;IP sleight of hand&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Protocols like TCP and UDP include &lt;a href='https://en.wikipedia.org/wiki/Checksum' title=''&gt;checksums&lt;/a&gt; to detect corruption on the way across the internet. Calculating one means reading the whole message, which adds up at high speeds. Linux avoids that with &lt;a href='https://docs.kernel.org/networking/checksum-offloads.html' title=''&gt;checksum offload&lt;/a&gt;: instead of calculating the checksum immediately, the kernel stores metadata saying where the checksum should read from and write to. If the checksum turns out not to be needed, the kernel never spends the CPU on it.&lt;/p&gt;
&lt;div class="right-sidenote"&gt;&lt;p&gt;Not needed, for example, when sending to localhost, or when the network card can add its own checksums.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;Checksum offload works the same way for virtual machines. The virtual network card accepts &lt;a href='https://docs.oasis-open.org/virtio/virtio/v1.3/virtio-v1.3.html#:~:text=5.1.6.2%20Packet%20Transmission' title=''&gt;checksum metadata&lt;/a&gt; and passes it to the host. On the host side, the packet is treated like any other offloaded packet, using the metadata the guest provided.&lt;/p&gt;

&lt;p&gt;The offload code has checks meant to reject bad metadata. For IPv4, they were wrong. To exploit that, an attacker needs to do three things: point the checksum at a different header field, pick a message that produces the checksum they want, and force the host to calculate the checksum in software. The &lt;a href='https://pwn.ai/blog/pwnescape' title=''&gt;pwn.ai post&lt;/a&gt; goes into detail on pulling that off. The end result: the host receives the packet, parses it, decides where to send it, &lt;em&gt;then&lt;/em&gt; applies the checksum, in the wrong place.&lt;/p&gt;

&lt;p&gt;The attacker now controls two bytes of an IPv4 header the host has already trusted.&lt;/p&gt;

&lt;p&gt;Based on network activity during the rollout, we believe it&amp;rsquo;s unlikely that any of our customers&amp;rsquo; data was actually exposed. It is theoretically possible that between disclosure and our final reboot, a malicious app could have read small chunks of network traffic from other apps on the same physical host, but we have no evidence this occurred.&lt;/p&gt;
&lt;div class="callout"&gt;&lt;p&gt;&lt;strong&gt;If you ran on Fly.io between October 15 and 17:&lt;/strong&gt; the exploit is hard to detect, so out of caution, rotate any credentials you sent over HTTP or another unencrypted protocol in that window. That includes unencrypted traffic over your 6PN.&lt;/p&gt;
&lt;/div&gt;&lt;h2 id='two-bytes-is-plenty' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#two-bytes-is-plenty' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Two bytes is plenty&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Our job is to help you run your amazing things, which is why we take &lt;a href='https://docs.fly.io/reference/architecture' title=''&gt;strong hardware virtualization&lt;/a&gt; very seriously. It&amp;rsquo;s the least flaky way to keep workloads apart from each other. A guest should not be able to influence the host outside of a few narrowly defined paths. A bug like this one punches holes in that. Two holes, in this case.&lt;/p&gt;

&lt;p&gt;The first is a firewall bypass. A rewrite that targets the source or destination IP lands after the host has already applied routing and firewall rules. The upstream fix is now in place.&lt;/p&gt;

&lt;p&gt;The second is an out-of-bounds read. A rewrite that targets the header length tricks the host into treating adjacent memory as part of the packet. Rewriting the smallest IPv4 header to the largest captures 32 bytes of memory as &lt;a href='https://en.wikipedia.org/wiki/Internet_Protocol_Options' title=''&gt;options&lt;/a&gt; on the outgoing packet. That memory can contain host packets, kernel data structures, and packets belonging to the VM next door. A leak from a neighboring VM is, all by itself, a reason to drop everything.&lt;/p&gt;

&lt;p&gt;Detecting this kind of activity is hard. Firewall-bypass packets can be perfectly well-formed. Out-of-bounds-read packets probably carry a bad header checksum once the options are included, but by the time you can see that, it&amp;rsquo;s already too late. We looked at adapting our existing &lt;a href='https://fly.io/blog/bpf-xdp-packet-filters-and-udp/' title=''&gt;XDP&lt;/a&gt; rules, but the rewrite happens after they run. Sure, we could disable checksum offload on the host, but that needs a reboot. And if we were going to reboot, we were going to reboot into a patched kernel.&lt;/p&gt;
&lt;h2 id='ten-hours-of-reboots' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#ten-hours-of-reboots' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Ten hours of reboots&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Each of our hosts takes around 15 to 20 minutes to reboot (server hardware is cool like that), and every Fly Machine running on it is down for that period. Our placement also scatters Machines randomly across hosts, so we couldn&amp;rsquo;t reboot more than one host at a time without taking down every Machine some unlucky subset of apps had.&lt;/p&gt;
&lt;div class="right-sidenote"&gt;&lt;p&gt;Single-Machine apps were going down no matter what we did.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;So, we planned around Managed Postgres. We split each region into waves so that no MPG cluster had both nodes in the same wave. Even with a limited number of MPG clusters, our largest region, &lt;code&gt;iad&lt;/code&gt;, needed 17 waves.&lt;/p&gt;

&lt;p&gt;Of course, computers behaved like computers. Most hosts came back within the estimate we&amp;rsquo;d announced, but some didn&amp;rsquo;t. A few rebooted without essential software and needed reinstalls. Others got stuck in BIOS, while a four-node chassis in &lt;code&gt;gru&lt;/code&gt; lost its power supply mid-incident, with no spare on site. Machines on those hosts stayed down until we restored them on the 20th. But at the end of the day, no Fly Machines or customer data were lost.&lt;/p&gt;
&lt;h2 id='the-communication-catch-22' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#the-communication-catch-22' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;The communication Catch-22&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;At 16:22 UTC we posted an emergency maintenance notice on the status page and started telling our largest customers. The notice said &amp;ldquo;maintenance,&amp;rdquo; not security, and didn&amp;rsquo;t name the CVE. Folks were not shy about telling us this felt unacceptable, which makes perfect sense. A same-day &amp;ldquo;emergency maintenance&amp;rdquo; that reboots your whole app looks like we&amp;rsquo;ve done something stupid and are patching wildly without telling anyone why.&lt;/p&gt;

&lt;p&gt;It didn&amp;rsquo;t feel great to us either, but telling the internet we were vulnerable (and for how long) felt like a significantly worse idea. This situation was particularly problematic because the bug was disclosed along with a detailed writeup of how to exploit it.&lt;/p&gt;

&lt;p&gt;In the moment, we decided it was most important to avoid confirming that our hosts were vulnerable &lt;em&gt;right now&lt;/em&gt;. Could we have run our communication channels better anyway? &lt;em&gt;Always.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;As part of the post-mortem, our Head of Security revamped our incident communications plan into an Incident Communications Plan. Previously, the folks in an incident channel would decide on a comms plan, since they were closest to the metal. Now, we&amp;rsquo;re taking that responsibility off their backs (since they&amp;#39;re…actively solving an incident) and putting it on our exec team. Might as well make them useful! They&amp;rsquo;re all now armed with &amp;ldquo;Do not pass go&amp;rdquo; level instructions on exactly who says what, posts where, and when. This should make it easier to get the right level of communication out at the right time when (not if, because something is always going to go sideways) the next thing happens.&lt;/p&gt;
&lt;h2 id='meanwhile' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#meanwhile' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Meanwhile…&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;While we dealt with…all of that, the world seemed to be catching fire around us. October was a wild ride (not sure we needed an entire Cybersecurity Awareness &lt;em&gt;Month&lt;/em&gt;, but okay). We have nothing but #HugOps for our industry peers who have had a rougher time. It&amp;rsquo;s not easy out here in the security trenches right now.&lt;/p&gt;

&lt;p&gt;But this is the world in which you (and your agent) are building software. It&amp;rsquo;s kinda gross. More bugs exist, and they&amp;rsquo;re getting found faster and exploited more easily. We&amp;rsquo;re doing what we can on our end, but we wanted to offer up a few things that really do reduce the blast zone if your project is compromised (even if they feel tedious to do in the moment).&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;strong class='font-semibold text-navy-950'&gt;Encrypt traffic between your own services, even on a private network.&lt;/strong&gt; The one piece of advice we had to give after CVE-2026-99999 was &amp;ldquo;rotate whatever you sent in plaintext.&amp;rdquo;
&lt;/li&gt;&lt;li&gt;&lt;strong class='font-semibold text-navy-950'&gt;Give agents credentials scoped to the job, and make them expire.&lt;/strong&gt; An agent&amp;rsquo;s environment is the first thing a successful exploit reads, so a token that can deploy one app for one hour makes your life easier than one that&amp;rsquo;s org-wide (or, God forbid, that lives forever). If you can &lt;a href='https://fly.io/connectors/' title=''&gt;keep credentials away from agents&lt;/a&gt;, even better.
&lt;/li&gt;&lt;li&gt;&lt;strong class='font-semibold text-navy-950'&gt;Assume every boundary leaks a little.&lt;/strong&gt; Hardware virtualization is the strongest wall we have, but for a day and a half a guest could still pull 32 bytes at a time through it. Try not to put an agent, its secrets &lt;em&gt;and&lt;/em&gt; your production data in the same place on the theory that the wall will hold.
&lt;/li&gt;&lt;li&gt;&lt;strong class='font-semibold text-navy-950'&gt;Treat what the agent reads as hostile.&lt;/strong&gt; Obviously &lt;em&gt;your&lt;/em&gt; agent is an angel, but you never really know who it&amp;rsquo;s hanging out with. Be a suspicious parent, and only give it the permissions you&amp;rsquo;d hand over to the author of the worst web pages, issues or emails it might read.
&lt;/li&gt;&lt;li&gt;&lt;strong class='font-semibold text-navy-950'&gt;Rehearse rotation.&lt;/strong&gt; &amp;ldquo;Rotate your credentials&amp;rdquo; is easy to say, but like…you have to know where they are, in five seconds. Where are they? Go look.
&lt;/li&gt;&lt;/ul&gt;

&lt;p&gt;Nothing here makes anyone invincible, but these can at least be some hard constraints when bad things happen.&lt;/p&gt;

&lt;p&gt;So! None of this was fun, and we&amp;rsquo;ll probably do it all again soon. See you then.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Feng Shui for Robots: The Fly MCP Server</title>
    <link rel="alternate" href="https://fly.io/blog/feng-shui-for-robots/"/>
    <id>https://fly.io/blog/feng-shui-for-robots/</id>
    <published>2026-10-23T00:00:00+00:00</published>
    <updated>2026-10-26T15:35:04+00:00</updated>
    <media:thumbnail url="https://fly.io/blog/feng-shui-for-robots/assets/mcp-server-4f95.webp"/>
    <content type="html">&lt;div class="lead"&gt;&lt;p&gt;In Databricks’ &lt;a href="https://www.databricks.com/sites/default/files/2026-01/State-of-AI-Agents-2026-Final.pdf"&gt;State of AI Agents 2026&lt;/a&gt; report, there’s a chart from Neon’s telemetry that’s somehow both mind-blowing and unsurprising. In October 2023, AI agents created 0.1% of the databases on Neon. By October 2025 they created 80% of them.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;We&amp;rsquo;re seeing this play out all across the infra landscape, including here at Fly.io, where agents have overtaken humans as our most common customer species. Thing is, we think it&amp;rsquo;s important to remember that there&amp;rsquo;s a human with aspirations behind those agents. Often one trying to build something that actually makes it to prod.&lt;/p&gt;

&lt;p&gt;If we want to make that easy, &amp;ldquo;computers for agents&amp;rdquo; can&amp;rsquo;t just be a sandbox where an agent can vibe. Agents need a whole cloud now. Somewhere safe to try dumb stuff, sure, but also a dead simple migration path to, and all the requisite parts of, a scalable production environment.&lt;/p&gt;

&lt;p&gt;This is kinda our whole thing right now. So we built &lt;code&gt;mcp.fly.io&lt;/code&gt;: one MCP server that connects your agent to Sprites and to your Fly.io apps.&lt;/p&gt;
&lt;h2 id='start-with-a-sprite' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#start-with-a-sprite' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Start with a Sprite&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Building something real with an agent should start in a sandbox. The speed and scale at which you can test different approaches and solutions is easily one of the most significant benefits of building with agents. So you want them to be able to kick all the tires to find out which of them explode.&lt;/p&gt;

&lt;p&gt;This is why the sandbox you choose in this step is so important. There are plenty of places to safely execute agent code. But what you want is to actually install packages, write files, start a server on a port, hit the server, and see what restarting the server does in real life. This is how you get from vibes to a robust product. It&amp;rsquo;s why you should tell your agent to build in a Sprite.&lt;/p&gt;
&lt;h2 id='just-linux-all-the-way-down' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#just-linux-all-the-way-down' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Just Linux, all the way down&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;When your agent builds something on a Sprite, it builds it the way it would on any Linux box. Packages go where packages go. Files sit on a disk that survives sleeps and restarts. The server runs as a &lt;strong class='font-semibold text-navy-950'&gt;service&lt;/strong&gt;, a long-running process that survives the Sprite rebooting. It gets an HTTPS URL, so the webhook the agent wired up can actually receive webhooks. And before the agent tries something that might break everything, it can &lt;strong class='font-semibold text-navy-950'&gt;checkpoint&lt;/strong&gt; the whole filesystem and roll back when it does, in fact, break everything.&lt;/p&gt;
&lt;h2 id='the-not-so-great-migration' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#the-not-so-great-migration' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;The not-so-great migration&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;In what feels like a raised middle finger to their skeptics, agents are actually building things that deserve to graduate to production infra now. Like, actual products that people rely on.&lt;/p&gt;

&lt;p&gt;The most common reason we&amp;rsquo;re seeing for migrating something out of a Sprite is the need for it to be always on. Sprites are really good at sleeping when nobody&amp;rsquo;s using them and waking on the next request. And with scheduled jobs landing soon with Sprites 2.0, serving your product from a Sprite is totally viable. But when you start to care more about availability to users than you do about the environment being perfect for your dev agent, it&amp;rsquo;s often a sign to migrate it to good ol&amp;rsquo; Fly Machines.&lt;/p&gt;

&lt;p&gt;Did the word &amp;ldquo;migrate&amp;rdquo; spike your blood pressure right there? Let&amp;rsquo;s reframe what migrate means now, because it really doesn&amp;rsquo;t have to be that stressful anymore. A Sprite isn&amp;rsquo;t a sandbox pretending to be a computer. It&amp;rsquo;s a real Linux machine with its own kernel, a disk that stays put and services that keep running, and so is a Fly Machine. So when it&amp;rsquo;s time to re-home your project, there&amp;rsquo;s no foreign runtime to translate out of and no function-shaped thing to reassemble into a server. Your agent built a Linux service on a Linux box, and all it&amp;rsquo;s doing is moving it to another Linux box.&lt;/p&gt;

&lt;p&gt;The Fly MCP server talks to both Sprites and Machines. Once the app you&amp;rsquo;re moving into exists and you&amp;rsquo;ve granted the agent access to it, the agent can create the Machine, its volume and its secrets from the same conversation it built the thing in.&lt;/p&gt;

&lt;p&gt;It&amp;rsquo;s impossible to build a catchall &amp;ldquo;promote this Sprite&amp;rdquo; flow, because who knows what you&amp;rsquo;ll build in your Sprites. But giving your agent (which built the project, and can read everything it installed and configured on the Sprite) an MCP server that connects it to everything it needs for the migration is about as close as you can get to one. Reverse-engineering a disposable environment into an always-on production one is a tough ask for any agent, but moving a Linux service between two Linux boxes is a robot cakewalk.&lt;/p&gt;
&lt;h2 id='eyes-on-mpg' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#eyes-on-mpg' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Eyes on MPG&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Before you ask, we are deep into investigating how to bring MPG skills to the Fly MCP server. It will be real fun to watch the agent that built your app on a Sprite and moved it to a Machine create the Postgres cluster behind it, with automatic backups and failover, and attach it to the app in the same conversation. Watch this space.&lt;/p&gt;
&lt;h2 id='you-beta-try-it' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#you-beta-try-it' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;You beta try it&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Obviously you want to be able to tell people you&amp;rsquo;ve been using this since it was in beta, so &lt;a href='https://mcp.fly.io/' title=''&gt;get on it&lt;/a&gt;. When you install the Fly MCP server, you&amp;rsquo;ll authorize it in a browser. You&amp;rsquo;ll be able to select the organization, the apps your agent can work on and what it can do with Sprites. There is no API token to paste. &lt;code&gt;mcp.fly.io&lt;/code&gt; only works on the apps you select: their Machines, volumes and secrets, with IP addresses and certificates readable but not writable. Discovery only returns the permissions and apps you approved, so your agent won&amp;rsquo;t even know the apps you didn&amp;rsquo;t select exist.&lt;/p&gt;
&lt;h2 id='computers-for-agents-from-prototype-to-production' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#computers-for-agents-from-prototype-to-production' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Computers for agents, from prototype to production&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;We feel a little like overbearing parents with big expectations of the Fly MCP server. We expect it to develop into something that helps your agent do everything from building a quick, shareable demo to a stable, robust production application. And it takes a village to raise an MCP server, so we appreciate you co-parenting with us.&lt;/p&gt;

&lt;p&gt;&lt;a href='https://mcp.fly.io/' title=''&gt;Connect your agent&lt;/a&gt; and take it for a spin.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>SBD: We Have a Filesystem at Home</title>
    <link rel="alternate" href="https://fly.io/blog/sprites-block-device/"/>
    <id>https://fly.io/blog/sprites-block-device/</id>
    <published>2026-10-22T00:00:00+00:00</published>
    <updated>2026-10-22T11:27:19+00:00</updated>
    <media:thumbnail url="https://fly.io/blog/sprites-block-device/assets/pasta-robot.webp"/>
    <content type="html">&lt;div class="lead"&gt;&lt;p&gt;I have unkind things to say about people who make their own pasta. Here, I’ll only use them as an example of how not to think when you’re building infrastructure. It’s unfairly easy to believe that the harder you work on something, the better it gets. But infra is about making the $1 bag of spaghetti: a simple thing that works well (and doesn’t require you to spend a week scraping flour out of the corners of your kitchen).&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;When work on Sprites started, it was a new kind of infra for a new kind of product. So when we couldn’t find the right filesystem in the pasta section, we made our own. It was clever, and that’s exactly why we didn’t like it. This post introduces its replacement, the Sprites Block Device (SBD). It&amp;rsquo;s the bag of spaghetti you grab off the shelf, and the only difference you’ll notice is better reliability.&lt;/p&gt;
&lt;h2 id='it-was-always-object-storage' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#it-was-always-object-storage' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;It was always object storage&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Our stance on one core aspect of Sprites&amp;rsquo; storage hasn&amp;rsquo;t changed from v1: the root of a Sprite&amp;rsquo;s disk is object storage, not the NVMe in whatever server the Sprite happens to be running on. That&amp;rsquo;s what lets a Sprite sleep, wake on a different server, and still find its files where it left them. The durable state of a Sprite is, more or less, a location in a bucket. &lt;a href='https://fly.io/blog/design-and-implementation/' title=''&gt;We wrote about why&lt;/a&gt;, at length, with feeling.&lt;/p&gt;

&lt;p&gt;The thing about object stores is they can&amp;rsquo;t change part of an object. To change one byte, you have to replace the whole thing. That&amp;rsquo;s a bad cultural fit for Linux programs which rewrite a few bytes in the middle of a file all the time. Something has to handle that translation.&lt;/p&gt;

&lt;p&gt;At launch, that something was a heavily modded version of JuiceFS. In stock form, JuiceFS splits storage into the data (as chunks in the bucket) and the metadata (a map of which chunks make up which files) in a database. That&amp;rsquo;s not how we used it, though. Our hot rodded version ran a rewritten SQLite metadata backend, kept durable with Litestream. It stored exactly one file: a 100 GB disk image, mounted on a loop device with ext4 inside. Linux did all the ordinary ext4 things. JuiceFS just held the image.&lt;/p&gt;

&lt;p&gt;It worked fine. Sprites shipped on it, people made a ton of them, and checkpoints were fast because checkpointing was mostly just shuffling metadata around. But as their usage grew, cracks began to emerge.&lt;/p&gt;
&lt;h2 id='manifests-that-lie' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#manifests-that-lie' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Manifests that lie&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;A filesystem with a metadata database has two inseparable components: the chunks in the bucket, and the database that explains how those chunks fit together. When they agree, you have files. When they don&amp;rsquo;t, you&amp;rsquo;re screwed.&lt;/p&gt;

&lt;p&gt;Every layer we built on top had to inherit that arrangement and add to it. A checkpoint under JuiceFS was a copy-on-write clone of the disk image, so the metadata database tracked the chunks for every checkpoint as well as the live disk. A second SQLite database, &lt;code&gt;checkpoints.db&lt;/code&gt;, listed the checkpoints themselves and which image file to mount on the loop device for each one.&lt;/p&gt;

&lt;p&gt;So a Sprite&amp;rsquo;s disk was chunks, plus a metadata database saying which chunks made up each image, plus a checkpoint database saying which image was which. Each one was made durable separately, so each one could be right while the others were wrong.&lt;/p&gt;
&lt;h2 id='what-could-go-wrong' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#what-could-go-wrong' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;What could go wrong?&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;A story which may or may not be based on lived experience: one day a Sprite stops booting. JuiceFS is asking object storage for a chunk and getting a 404. We wanted to go fast, so we ran JuiceFS in writeback mode. It records a chunk in its metadata first and uploads the bytes after. If the machine is killed at any time between those two steps, the metadata points at a chunk that doesn&amp;rsquo;t exist.&lt;/p&gt;

&lt;p&gt;So the ext4 filesystem living on top finds a directory block that failed its checksum and goes read-only. Moving the Sprite to a new machine won&amp;rsquo;t help, nor will restoring a checkpoint, because JuiceFS checkpoints were clones. They shared the same chunks, so every checkpoint would have the same gap.&lt;/p&gt;
&lt;h2 id='my-agent-ate-my-dogfood' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#my-agent-ate-my-dogfood' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;My agent ate my dogfood&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;It was less the technical details of issues like this and more their timid nature that brought about the single most intense period of dogfooding in Fly.io history. And if you&amp;rsquo;re going to be dogfooding computers for agents, it better be the agents doing the eating.&lt;/p&gt;

&lt;p&gt;So we built a harness that drives fleets of Sprites through write, checkpoint, sleep, wake and restore, over and over, verifying every byte that comes back. We pointed it at 400 JuiceFS Sprites for about 5,000 cycles. A nonzero number of them permanently lost the ability to checkpoint. Well … fuck.&lt;/p&gt;

&lt;p&gt;For a while, we continued to interrogate our recipe. There was more reconciliation, more repair tooling, and increasingly clever ways to keep three ledgers in agreement. But spaghetti that comes with a repair kit is gonna leave a weird taste in your mouth, however you cook it.&lt;/p&gt;
&lt;h2 id='just-a-disk' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#just-a-disk' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Just a disk&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Enter: the Sprites Block Device (SBD). It takes a bucket and presents it to the Linux kernel as a block device, over NBD. The Sprite formats that device as ext4 and mounts it, the same way you&amp;rsquo;d mount a disk on any Linux box.&lt;/p&gt;

&lt;p&gt;Duh.&lt;/p&gt;

&lt;p&gt;Under SBD, the metadata lives in the bucket alongside the blocks it describes, so there&amp;rsquo;s no separate database to keep in sync. A daemon inside the Sprite, &lt;code&gt;sbd serve&lt;/code&gt;, answers the kernel&amp;rsquo;s block reads and writes from its own local cache, uploads dirty blocks to object storage in the background, and flushes on demand when something needs a guarantee that the bytes have left the building.&lt;/p&gt;

&lt;p&gt;Ext4 still does the legwork. We&amp;rsquo;ve just made everything underneath it much simpler: one daemon and one bucket, where there used to be a filesystem, a database and a disk image to keep in agreement.&lt;/p&gt;
&lt;h2 id='dumber-checkpoints-are-better-checkpoints' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#dumber-checkpoints-are-better-checkpoints' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Dumber checkpoints are better checkpoints&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Checkpoints are a huge deal for any infra that is designed to be a padded room for agents. Under JuiceFS, a checkpoint was a copy-on-write clone of the disk image, recorded in &lt;code&gt;checkpoints.db&lt;/code&gt;. Under SBD, a checkpoint is a block snapshot of the whole volume, with a version number, recorded in SBD&amp;rsquo;s own checkpoint catalog. Ah…the unmitigated comfort of a single source of truth.&lt;/p&gt;

&lt;p&gt;Restore is equally vanilla. Each checkpoint knows which version of the volume it belongs to, so a restore unmounts the filesystem, rewinds the volume in place and mounts it again. It costs about as much as a restart.&lt;/p&gt;

&lt;p&gt;Of course things can still go wrong. The difference is that a Sprite now stops and says so instead of carrying on with a disk it can&amp;rsquo;t vouch for. SBD checks every block it reads back from object storage against what it wrote. If SBD itself dies, the whole Sprite restarts, because a clean restart is the only way back to a disk you can trust.&lt;/p&gt;
&lt;h2 id='moms-spaghetti' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#moms-spaghetti' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Mom&amp;rsquo;s spaghetti&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Continuing voracious dogfooding, a borderline abusive test harness and input from a few pioneering design partners have shown SBD to be orders of magnitude more reliable than our homemade pasta filesystem.&lt;/p&gt;

&lt;p&gt;So SBD is going to be the Sprites filesystem from here on out. Ext4, on a block device, on a bucket. The only thing keeping track of your files is the filesystem code every Linux box already runs.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;If you have any Sprites created before 10/6, you can migrate them using &lt;a href='https://docs.fly.io/sprites/sprite-maintenance#storage-upgrades' title=''&gt;these instructions&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Your Agent Speaks MCP. Give It a Computer.</title>
    <link rel="alternate" href="https://fly.io/blog/sprites-mcp/"/>
    <id>https://fly.io/blog/sprites-mcp/</id>
    <published>2026-09-03T00:00:00+00:00</published>
    <updated>2026-11-17T14:21:47+00:00</updated>
    <media:thumbnail url="https://fly.io/blog/sprites-mcp/assets/whack.webp"/>
    <content type="html">&lt;div class="lead"&gt;&lt;p&gt;Sprites are disposable cloud computers. They appear instantly, always include durable filesystems, and cost practically nothing when idle. They’re the best and safest place on the Internet to run agents and we want you to &lt;a href="https://sprites.dev/" title=""&gt;create dozens of them&lt;/a&gt;.&lt;/p&gt;
&lt;/div&gt;&lt;div class="callout"&gt;&lt;p&gt;&lt;strong class="font-semibold text-navy-950"&gt;Update, November 2026:&lt;/strong&gt; the Sprites MCP server now lives inside the Fly MCP at &lt;code&gt;https://mcp.fly.io&lt;/code&gt;, and we are transitioning away from the &lt;code&gt;sprites.dev/mcp&lt;/code&gt; endpoint this post describes. New setups should use the new endpoint.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;Sprites are a place to run agents; the first thing you should think to do with a new Sprite is to type &lt;code&gt;claude&lt;/code&gt; (or &lt;code&gt;gemini&lt;/code&gt; or &lt;code&gt;codex&lt;/code&gt;). We&amp;rsquo;ve put a &lt;a href='https://fly.io/blog/design-and-implementation/' title=''&gt;lot of effort&lt;/a&gt; into making sure coding agents feel safe and happy when they&amp;rsquo;re on Sprites, because, to (probably) quote John von Neumann, &amp;ldquo;happy agents are productive agents.&amp;rdquo;&lt;/p&gt;

&lt;p&gt;What&amp;rsquo;s less obvious about Sprites is that they&amp;rsquo;re great tools &lt;em&gt;for&lt;/em&gt; agents. Want three different versions of a new feature? A test environment? An ensemble of cooperating services? It&amp;rsquo;s super handy to be able to start your prompts, &amp;ldquo;&lt;code&gt;On a new Sprite, do…&lt;/code&gt;&amp;rdquo;.&lt;/p&gt;

&lt;p&gt;The Sprites API is simple, discoverable, and designed for this use case. The only real question is how your agent reaches it. For most of you the answer is MCP, and the setup is already written.&lt;/p&gt;
&lt;h2 id='you-dont-have-to-pick' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#you-dont-have-to-pick' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;You Don&amp;rsquo;t Have To Pick&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;There&amp;rsquo;s an argument going around that MCP is the wrong way to extend an agent, and that command line tools and discoverable APIs are the Right Way. Half of that argument is correct, and it&amp;rsquo;s the important half, so let&amp;rsquo;s take it seriously.&lt;/p&gt;

&lt;p&gt;Dumping thirty tool descriptions into a context window is a bad way to teach anything. Not every Sprite command matters in every session, and cramming them all in signals to the model that they all matter to you. If you&amp;rsquo;re not using network policies, &lt;code&gt;gemini&lt;/code&gt; shouldn&amp;rsquo;t burn a single token learning to configure them. Capabilities should reveal themselves progressively, the way they do when an agent works out a CLI one subcommand at a time.&lt;/p&gt;

&lt;p&gt;The wrong half is treating that as a case against MCP. Progressive disclosure is a question of what you say to the model. MCP is a question of how the bytes get there: transport, auth, structured results, a tool the model can call instead of a command whose flags it has to guess. Those are different layers, and nothing stops you from having both. Our plugins are what having both looks like.&lt;/p&gt;

&lt;p&gt;Install &lt;a href='https://github.com/superfly/sprites-claude-plugin' title=''&gt;the Claude Code plugin&lt;/a&gt; and you get the hosted MCP server underneath and skills on top. What lands in your context is roughly a sentence about when you&amp;rsquo;d want a fresh computer. The rest arrives when it&amp;rsquo;s needed. Codex, Cursor, Antigravity, opencode, Grok and the others &lt;a href='https://fly.io/run-agent-code/' title=''&gt;work the same way&lt;/a&gt;, and if yours isn&amp;rsquo;t on that list, point it at the bare endpoint and it still works.&lt;/p&gt;

&lt;p&gt;The server got better at its half of the job too. A file read comes back as an MCP resource instead of a wall of pasted text, so an agent can point at a file without swallowing it. Every tool carries safety annotations: read-only operations are marked read-only, destructive ones are marked destructive, and &lt;code&gt;exec&lt;/code&gt; and &lt;code&gt;service_start&lt;/code&gt; are flagged as the two that reach past the Sprite&amp;rsquo;s boundary. A client that pays attention to those can treat &amp;ldquo;list my checkpoints&amp;rdquo; differently from &amp;ldquo;run this thing.&amp;rdquo;&lt;/p&gt;

&lt;p&gt;And the shell never went anywhere. There&amp;rsquo;s a &lt;code&gt;sprite&lt;/code&gt; CLI, there&amp;rsquo;s a plain REST API, and every Sprite carries machine-readable docs at &lt;code&gt;/.sprite/llm.txt&lt;/code&gt; that teach an agent working inside it how the place operates. If your agent would rather write little scripts than call tools, let it. It&amp;rsquo;s the same API underneath either way.&lt;/p&gt;
&lt;h2 id='sprites-dev-mcp' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#sprites-dev-mcp' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;sprites.dev/mcp&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Plug this URL into Claude Desktop, or any other agent tool that speaks MCP. You&amp;rsquo;ll authenticate to one of your Fly.io organizations, and your agent will speak Sprites.&lt;/p&gt;

&lt;p&gt;Then:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;On a new Sprite, take this repository and reproduce this bug from issues/913, capturing logs.&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;&lt;code&gt;On a new Sprite, benchmark this function across 1000 runs and summarize the results.&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;&lt;code&gt;On a new Sprite, update all the dependencies on this project to their newest versions and test that everything works.&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;&lt;code&gt;On 3 new Sprites, change this service to use each of these 3 query libraries, and use HTTP to test latency.&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;&lt;code&gt;On a new Sprite, run this code with bpfwatch and show me what files it touches.&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;&lt;code&gt;On a new Sprite, run a load generator against this endpoint for 60 seconds and report the results.&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;&lt;code&gt;On a new Sprite, download this dataset and give me a Jupyter notebook to explore it in.&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;&lt;code&gt;On a new Sprite, set up a webhook receiver and render a real-time web report of all the payloads it receives.&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;I don&amp;rsquo;t know. You know your projects better than we do. Whatever. Sometimes you want a clean, cheap, disposable computer (or five of them). That&amp;rsquo;s now an available feature of all your prompts. Find ways to apply it to your project, and we think you&amp;rsquo;ll end up wondering where Sprites have been all your life.&lt;/p&gt;
&lt;div class="callout"&gt;&lt;p&gt;Some of you are thinking to yourself: “this feature is going to result in robots ruining my life”. We agree. So we’ve built in guardrails. When you authenticate you’re handing your agent a single specific organization on your Fly.io account, and you can scope the session down from there. It defaults to a cap of five Sprites and an &lt;code&gt;mcp-&lt;/code&gt; name prefix, so the robots are easy to spot and easy to disassemble. Both are yours to change.&lt;/p&gt;
&lt;/div&gt;&lt;h2 id='fuck-stateless-sandboxes' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#fuck-stateless-sandboxes' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Fuck Stateless Sandboxes&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;We&amp;rsquo;ll keep saying this until our faces turn blue: the industry is stuck on &amp;ldquo;sandboxes&amp;rdquo; as a way of letting agents run code, and sandboxes aren&amp;rsquo;t good enough anymore. What agents want is real computers, with real filesystems, connected to real networks, and there&amp;rsquo;s no technical reason not to give them some.&lt;/p&gt;

&lt;p&gt;&lt;a href='https://fly.io/blog/code-and-let-live/' title=''&gt;We designed Sprites so that you can fearlessly create whole bunches of them&lt;/a&gt;. They&amp;rsquo;re responsive enough to host web apps for your team, but they idle in a sleeping state where they cost virtually nothing. Everybody at Fly.io that uses them ends up with 20 or 30, just hanging around.&lt;/p&gt;

&lt;p&gt;We think you&amp;rsquo;ll do better work when you can pull in as many computers as you need to solve problems. Your agent is already asking for one. Say yes.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Turn And Face The Strange</title>
    <link rel="alternate" href="https://fly.io/blog/kurt-scott-money-sprites/"/>
    <id>https://fly.io/blog/kurt-scott-money-sprites/</id>
    <published>2026-07-24T00:00:00+00:00</published>
    <updated>2026-09-30T00:06:54+00:00</updated>
    <media:thumbnail url="https://fly.io/blog/kurt-scott-money-sprites/assets/swirl-cover.webp"/>
    <content type="html">&lt;div class="lead"&gt;&lt;p&gt;We’re Fly.io, a public cloud platform that is both our favorite way to put an app on the Internet and our favorite way to safely let a frontier agent coding harness cook. This is a post about our company, the future, and Sprites, which are computers for agents that &lt;a href="https://fly.io/sprites/" title=""&gt;you can check out right now.&lt;/a&gt;&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;This is a complicated post. So I need you to promise me something: if you read past this introduction, you&amp;rsquo;ll read the whole rest of the way through. It’s an honor thing.&lt;/p&gt;

&lt;p&gt;A couple months back, Theo Browne ran a video rating the &lt;a href='https://www.youtube.com/watch?v=yfxDdQo2cyI' title=''&gt;“best place to host a new application in 2026”&lt;/a&gt;. Theo tends to say nice things about us. He did this time too. But then he concluded by saying that of all the providers he pays attention to, we were the one he was least confident would be around by the end of the year.&lt;/p&gt;

&lt;p&gt;Well, fuck.&lt;/p&gt;

&lt;p&gt;Theo startled us, &lt;strong class='font-semibold text-navy-950'&gt;because we’re in the middle of a run of strong quarters that have included the best financial months in the company’s history&lt;/strong&gt;. But that take has been rattling around in my brain. It whacked me right on a raw nerve, about what we’re doing and where we’re going as a company.&lt;/p&gt;

&lt;p&gt;Honestly, I should&amp;rsquo;ve seen this coming. Fly.io has been motoring along this year, but I’ve coasted a bit, letting the company smolder in an unresolved identity crisis.&lt;/p&gt;

&lt;p&gt;I’m going to overshare some more in a second, but I won&amp;rsquo;t leave you hanging. So: we’ve raised a bunch more money. We’re launching a new iteration of Sprites, and focusing the company on them and the problem they solve. And I’m tagging in Scott Johnston as CEO.&lt;/p&gt;
&lt;h3 id='product-market-fit' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#product-market-fit' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Product-Market Fit&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;I started Fly.io with two clear principles that probably don&amp;rsquo;t matter anymore.&lt;/p&gt;

&lt;p&gt;The first is that Internet applications work best when they&amp;rsquo;re fast, and that happens when they&amp;rsquo;re &lt;a href='https://news.ycombinator.com/item?id=22616857' title=''&gt;deployed close to users&lt;/a&gt;. I learned this over many years of working at Ars Technica, and started Fly.io in part to scratch an itch. It was our mantra over the first several years of the company.&lt;/p&gt;

&lt;p&gt;The second is that cloud infrastructure is too complicated. Developers need platforms with the flexibility of AWS and the ergonomics of Heroku. When we started Fly.io, you couldn&amp;rsquo;t get both things at the same time, and now you can, here and elsewhere.&lt;/p&gt;

&lt;p&gt;You read this and say, &amp;ldquo;no shit, of course these things are important.&amp;rdquo; But I&amp;rsquo;m here to tell you they&amp;rsquo;re less important than you think, for an obvious reason — the only reason anybody talks about anymore. AI has transmogrified software development. The dingo has truly eaten our baby.&lt;/p&gt;
&lt;div class="right-sidenote"&gt;&lt;p&gt;† For the past 18 months, every time I’ve said these words, they’ve gotten even truer.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;I don’t think it’s fully sunk in yet[†].  We&amp;rsquo;re still trying to integrate coding agents into our profession like they&amp;rsquo;re sufficiently smart compilers. But AI isn&amp;rsquo;t like the difference between shipping C code and shipping Ruby. It&amp;rsquo;s much bigger.&lt;/p&gt;

&lt;p&gt;Everybody forgets that before Dan Bricklin invented the spreadsheet, every &amp;ldquo;Excel document&amp;rdquo; in the world was a computer program, built by a computer programmer. In just a matter of years, every business professional became a programmer, using the world&amp;rsquo;s most important programming language, spreadsheet formulas. AI is like that, but bigger. Almost anybody will probably be able to build almost any kind of computer program.&lt;/p&gt;

&lt;p&gt;Now consider conventional public cloud infrastructure. We take fixed-function applications built to rigorous standards on fussy CI/CD rails and ship them to &lt;a href='https://fly.io/blog/the-5-hour-content-delivery-network/' title=''&gt;audiences of millions of people&lt;/a&gt;. But a computer program with an audience of millions will soon be like a spreadsheet with an audience of a million readers. They exist! But they&amp;rsquo;re not the norm.&lt;/p&gt;

&lt;p&gt;Betting on an opinionated public cloud design from 2020 is the same as betting against &lt;a href='https://sockpuppet.org/blog/2026/05/12/emacsification/' title=''&gt;personalized, adaptive software&lt;/a&gt;. I don’t think that’s a good bet. And even if I did, I wouldn’t want to make it. I want a world where my friends and family can make computers do exactly what they want, without waiting for me to build everything for them.&lt;/p&gt;
&lt;h3 id='what-agents-want' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#what-agents-want' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;What Agents Want&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;That brings us to our second founding principle, which is that serious cloud infrastructure is too hard for developers to use well. And: still true! But this even more obviously doesn&amp;rsquo;t matter anymore.&lt;/p&gt;
&lt;div class="right-sidenote"&gt;&lt;p&gt;† It is in fact possible that it’s now worse to have a carefully curated human developer experience with opinionated defaults. Agents work best when things are explicit.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;To a first approximation, nobody reads documentation anymore. They&amp;rsquo;re not picking up new CLIs and figuring out how to use them by trial and error, either[†]. That&amp;rsquo;s what agents are for. An agent can one-shot a Fly.io deployment: just build a site locally and say &amp;ldquo;now get this working on Fly.io&amp;rdquo;, and it&amp;rsquo;ll work great. But an agent can also one-shot an AWS deployment. What are we doing here? What&amp;rsquo;s going on?&lt;/p&gt;

&lt;p&gt;I wrote about this last year, in a post about how&lt;a href='https://fly.io/blog/fuckin-robots/' title=''&gt; our fastest-growing customers were all robots&lt;/a&gt;. Then I stopped retconning what we&amp;rsquo;d already built and got to work figuring out what the robot customers actually want. Here&amp;rsquo;s what I came up with.&lt;/p&gt;

&lt;p&gt;Thing 1: Coding agents expect to run on developer workstations.&lt;/p&gt;

&lt;p&gt;Thing 2: Even in a sandbox that you trust, running an agent on your physical dev laptop is annoying, because your laptop stops running when you close the lid. Raise your hand if you&amp;rsquo;ve walked up or down a flight of stairs with your MacBook open this year. Is your hand down? I&amp;rsquo;d guess your home doesn’t have stairs. And so people all end up running their agent sandboxes in the cloud.&lt;/p&gt;

&lt;p&gt;Thing 3: Public clouds are an irritating place to run agents. We divide servers into &amp;ldquo;pets&amp;rdquo; or &amp;ldquo;cattle&amp;rdquo;, but for agents, even a herd cow is too much commitment. You want, I don’t know, a semi-disposable cow, a cow that comes into existence exactly when you want it to and sticks around for exactly as long as you want and doesn’t cost very much — and this is why analogies are hard to write.&lt;/p&gt;

&lt;p&gt;Earlier this year, our team made what I believe is a breakthrough in systems engineering and perhaps all of computer science: we launched the semi-disposable cow. &lt;a href='https://fly.io/blog/design-and-implementation/' title=''&gt;We call them Sprites&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Sprites take an odd shape that comes from shrink-wrapping them around what I think the robots are looking for. You can create hundreds or thousands of them quickly, but all of them have 100GB durable disk drives. Like everything in the cloud, they have metered utility billing, but the meter doesn&amp;rsquo;t run when they&amp;rsquo;re not doing anything, and they&amp;rsquo;re smart about figuring out when they&amp;rsquo;re idle. And you can host an app on them and share it with your coworkers, over the Internet.&lt;/p&gt;

&lt;p&gt;This grab-bag of features adds up to a proposition about agents. The industry obsesses over sandboxes. But robots don&amp;rsquo;t want sandboxes. They want computers. That&amp;rsquo;s what our semi-disposable cow is: a computer for an agent.&lt;/p&gt;
&lt;figure class="post-cta"&gt;
  &lt;figcaption&gt;
    &lt;h2&gt;You Can Go Make A Sprite Right Now&lt;/h2&gt;
    &lt;p&gt;It&amp;rsquo;ll take, like, a minute &amp;nbsp✨&lt;/p&gt;
      &lt;a class="btn btn-lg" href="https://fly.io/sprites/"&gt;
        Go!&lt;span class='opacity-50'&gt;→&lt;/span&gt;
      &lt;/a&gt;
  &lt;/figcaption&gt;
  &lt;div class="image-container"&gt;
    &lt;img loading="lazy" decoding="async" src="/static/images/cta-turtle.webp" srcset="/static/images/cta-turtle@2x.webp 2x" alt=""&gt;
  &lt;/div&gt;
&lt;/figure&gt;

&lt;h3 id='computers-for-agents' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#computers-for-agents' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Computers For Agents&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;I&amp;rsquo;m happy with how the Sprites launch played out. But honestly, Sprites were a skunkworks project. We didn’t even host them on the main Fly.io website!  A weird move. I&amp;rsquo;m not rationalizing it. We were in an identity crisis. But the clouds have parted, and Computers for Agents are, going forward, the focus of our company.&lt;/p&gt;
&lt;div class="right-sidenote"&gt;&lt;p&gt;† (to get a flavor of how true that is: a &lt;code&gt;git blame&lt;/code&gt; of the whole codebase shows my name more than any other)&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;Fly Machines and our Platform As A Service features aren&amp;rsquo;t going anywhere. But Sprites was the product of a tiny skeleton crew inside of Fly.io[†], and now it isn&amp;rsquo;t.&lt;/p&gt;

&lt;p&gt;Ordinarily we&amp;rsquo;d spend thousands of words on deep-dive technical content about how we built any new product we launched. We&amp;rsquo;ll do that for Sprites too. But I&amp;rsquo;m pretty deep into this post already and I have other stuff to share. So for now, I&amp;rsquo;m going to keep it brief.&lt;/p&gt;

&lt;p&gt;In addition to behind-the-scenes work we&amp;rsquo;ve done on scaling and orchestration, nu-Sprites introduces two big subsystems that get us to a place I&amp;rsquo;d finally consider &amp;ldquo;feature-complete&amp;rdquo; for what we&amp;rsquo;re trying to do.&lt;/p&gt;

&lt;p&gt;The first is the Sprite Block Device (SBD). The original Sprites storage stack was a goblin contraption I personally derived from JuiceFS and wired into our system using &lt;a href='https://litestream.io/' title=''&gt;Ben Johnson&amp;rsquo;s Litestream&lt;/a&gt;. You should be glad to hear that Ben and Tim Newsham tore that whole stack down to the studs and rebuilt it. It&amp;rsquo;s faster, more reliable, and still does instant checkpoint-and-restore.&lt;/p&gt;

&lt;p&gt;More importantly, SBD enables drive forking: you can create a template Sprite, and then efficiently clone millions of times.&lt;/p&gt;

&lt;p&gt;The other big new thing in Sprites is Connectors. Connectors &lt;a href='https://fly.io/blog/tokenized-tokens/' title=''&gt;build on work we did to secure our core platform&lt;/a&gt;: they let Sprites make authenticated requests to other systems, without giving agents anything useful to exfiltrate. Connectors have fun security properties, but are also much more pleasant to use than manually managing accounts and API keys.&lt;/p&gt;

&lt;p&gt;These are our most requested features. They&amp;rsquo;re the reason so many agent companies are still using Fly Machines many months after we launched a product specifically for them. So I’m confident enough to bet: unless some new space alien technology arrives that does something even weirder to computer science than what Transformer models have done, Sprites are the right fit for our future customers, and a very large portion of our existing ones. Which brings us to:&lt;/p&gt;
&lt;figure class="post-cta"&gt;
  &lt;figcaption&gt;
    &lt;h2&gt;Fancy Sprite Beta&lt;/h2&gt;
    &lt;p&gt;You want a weird beta Sprite that can clone itself? I can get you a toe.&amp;nbsp✨&lt;/p&gt;
      &lt;a class="btn btn-lg" href="https://fly.io/early-access"&gt;
        Sign up for the beta&amp;nbsp;&lt;span class='opacity-50'&gt;→&lt;/span&gt;
      &lt;/a&gt;
  &lt;/figcaption&gt;
  &lt;div class="image-container"&gt;
    &lt;img loading="lazy" decoding="async" src="/static/images/cta-cat.webp" srcset="/static/images/cta-cat@2x.webp 2x" alt=""&gt;
  &lt;/div&gt;
&lt;/figure&gt;

&lt;h3 id='i-quit' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#i-quit' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;I Quit&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;This has been a little while coming, but for the stage Fly.io is at, I think it’s extracted most of the good stuff out of me being CEO. So I’m going to stop doing that.&lt;/p&gt;

&lt;p&gt;For the first several years of a startup, you’re running a science project, an experiment-driven search for product-market fit. As anybody who’s worked here can attest, we tried dozens of things, from unmanaged Postgres (&lt;strong class='font-semibold text-navy-950'&gt;never do this&lt;/strong&gt;) to &lt;a href='https://fly.io/blog/the-5-hour-content-delivery-network/' title=''&gt;global CDNs&lt;/a&gt; to &lt;a href='https://fly.io/blog/our-user-mode-wireguard-year/' title=''&gt;user-mode WireGuard&lt;/a&gt;. Deeper into the company fabric, we built a bottom-up  engineering org, avoided product roadmaps, and recruited an all-remote team with members in over a dozen countries.&lt;/p&gt;

&lt;p&gt;Some experiments paid off, and others were &lt;code&gt;learning opportunities&lt;/code&gt;. Running them has been my whole life over the last 8 years. But Fly.io doesn&amp;rsquo;t need these kinds of science projects anymore.&lt;/p&gt;

&lt;p&gt;For the past many months, stretching way back into 2025, I’ve been talking to Scott Johnston about what Fly.io would look like if he was calling the plays. Scott was the CEO of Docker, and led that through a really challenging time that began with Docker&amp;rsquo;s own enterprise-vs.-developer identity crisis and ended with them blowing the doors off the business. As a shareholder of Fly.io, for this stage in Fly.io’s lifecycle, I liked his playbook better than mine. As the CEO of Fly.io, I liked the prospect of him doing all this work more than I liked the prospect of me doing it. We took a lot of time to work this out, and ultimately the board and I convinced him to take the job.&lt;/p&gt;

&lt;p&gt;This is the paragraph in these kinds of posts where I&amp;rsquo;m supposed to tell you why Scott is a perfect fit for Fly.io and recount all his past adventures. And he is, and they were majestic. But you already know what I&amp;rsquo;m going to say here, which makes it boring, and Scott can introduce himself just fine when he wants to. He&amp;rsquo;s not shy.&lt;/p&gt;

&lt;p&gt;Meanwhile, I’m going to do what all the smart spent founder CEOs do, and move to an advisor role parachuting randomly into product design discussions (the fun part of my job) while using my board seat to annoy Scott as he executes what (for me) was the unfun part of my job better than I could have.&lt;/p&gt;

&lt;p&gt;One of the things I&amp;rsquo;m sure Scott will break down is the fundraise we just did. That&amp;rsquo;s another thing Theo Browne called out in his video (I&amp;rsquo;m not mad, do I sound mad?) — that we hadn&amp;rsquo;t announced a raise in several years. The answer to that is: we raised a fuckload of money and didn&amp;rsquo;t need more. We&amp;rsquo;ve been operating on the threshold of &lt;em&gt;never&lt;/em&gt; needing more, if we stuck to our original plans, and if AI didn&amp;rsquo;t cause the ground to open up and swallow us all whole. But obviously, that&amp;rsquo;s no longer the game plan.&lt;/p&gt;
&lt;h3 id='ch-ch-changes' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#ch-ch-changes' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Ch-ch-changes&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;Look, I know how this is going to go over. I could write this post without pissing anybody off, but I don&amp;rsquo;t know how to do that and still have it be worth reading.&lt;/p&gt;

&lt;p&gt;We&amp;rsquo;re making a very specific and probably polarizing bet on the future of the industry: that agents, within a few years, are going to determine how almost all software is built and shipped. That software is going to become much more personal, with smaller audiences, and much more flexible and slippery.&lt;/p&gt;

&lt;p&gt;I&amp;rsquo;m excited about all of this. I&amp;rsquo;m a little taken aback that I get to work in this field during a shift like this. But I&amp;rsquo;d have to be oblivious not to see how uncomfortable that shift makes other professionals in the field.&lt;/p&gt;

&lt;p&gt;By the middle of the year, we could have gone one of two ways:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;On the first path: keep investing our energy in exactly what we&amp;rsquo;ve been scaling out and refining, a platform for fixed-function full-stack applications designed by humans.
&lt;/li&gt;&lt;li&gt;On the second path: dial in and nail a product that fits an agent-driven near future of software.
&lt;/li&gt;&lt;/ul&gt;

&lt;p&gt;Five of the most dangerous words in startups are &amp;ldquo;¿Por qué no los dos?&amp;rdquo;. We do one thing or the other. We don&amp;rsquo;t limp in on both. And if we fail, we fail with our full asses. Though, I guess only most of mine going forward.&lt;/p&gt;

&lt;p&gt;It&amp;rsquo;s a hell of a thing, steering a team, a company, a base of customers at this stage in Fly.io&amp;rsquo;s life. We&amp;rsquo;d been putting off a big decision about priorities for several months. Theo, you picked up on that. Good note! I could have decided quicker, and more clearly; instead, I shipped Sprites. Sprites answer the question that&amp;rsquo;s been facing us. I&amp;rsquo;m glad I spent the time building them, and I&amp;rsquo;m glad we&amp;rsquo;ve recruited Scott to turn them into our core business.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Building Agents that Don't Break Themselves</title>
    <link rel="alternate" href="https://fly.io/blog/building-agents-that-dont-break-themselves/"/>
    <id>https://fly.io/blog/building-agents-that-dont-break-themselves/</id>
    <published>2026-06-08T00:00:00+00:00</published>
    <updated>2026-09-22T09:57:11+00:00</updated>
    <media:thumbnail url="https://fly.io/blog/building-agents-that-dont-break-themselves/assets/building-agents-that-dont-break-themselves.webp"/>
    <content type="html">&lt;div class="lead"&gt;&lt;p&gt;Building agents is fun. Rebuilding agents that break themselves… less so. A lot of Fly people are building agents with less of a penchant for self-destruction by teaching their agents to do anything risky in a Sprite. You get an agent that stays alive long enough to actually use its snazzy self-improvement features, and you can allow your agent to try things that would otherwise be battleship-scale footguns. Here’s how to do it.&lt;/p&gt;
&lt;/div&gt;&lt;h2 id='brains-vs-hands' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#brains-vs-hands' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Brains vs Hands&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Your agent would be pretty useless without a shell, because this is where it does agent things. Run the test suite, apply the migration, install the dependency, delete the temp files. Unfortunately, your agent&amp;rsquo;s shell access is also what tends to ruin your afternoon, simply because &amp;ldquo;delete the temp files&amp;rdquo; and &amp;ldquo;delete the wrong files&amp;rdquo; are one fat-fingered glob apart, and as we&amp;rsquo;re frequently warned, AI can make mistakes.&lt;/p&gt;

&lt;p&gt;This is why we have sandboxes. But a lot of people default to putting an agent that&amp;rsquo;s going to do potentially scary work in a sandbox. This comes with a long list of tradeoffs that you really don&amp;rsquo;t have to make, because where your agent lives and where it runs code are two entirely separate considerations.&lt;/p&gt;
&lt;h2 id='ppe-for-agent-workers' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#ppe-for-agent-workers' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;PPE for agent workers.&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;The agent process is a loop. It calls a model, reads the response, picks a tool, rinse and repeat. It&amp;rsquo;s a long-lived process that only becomes more competent and less stupid if memory, skills and history persist. So a Fly Machine that sleeps when idle and wakes on a message, a small VPS. Your laptop while you iterate. These are all fine homes for a loop calling an API, which doesn&amp;rsquo;t need a blast shield.&lt;/p&gt;

&lt;p&gt;It&amp;rsquo;s when you want your agent to execute that things get hairy. &lt;code&gt;bash -c&lt;/code&gt; + whatever string the model just produced needs to be run in a padded room. Somewhere where the agent&amp;rsquo;s code can&amp;rsquo;t break itself or anything connected to it. And if your agent is doing more work than you are, you&amp;rsquo;re going to want a whole facility of padded rooms that can be thrown away and rebuilt on a whim.&lt;/p&gt;
&lt;h2 id='one-sprite-per-session' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#one-sprite-per-session' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;One Sprite per session&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Let&amp;rsquo;s take a look at two recent projects by Fly people that demonstrate this concept nicely. First up is &lt;a href='https://github.com/vaurdan' title=''&gt;Henrique&lt;/a&gt;&amp;rsquo;s internal Fly troubleshooting agent called SpriteDoc. SpriteDoc is multi-user and built on top of the Pi agent. Every session runs on one shared server, in one Node.js runtime. Running bash commands directly on that server is not really possible, and&amp;hellip; dangerous. Every user&amp;rsquo;s shell would sit in the same process the agent itself runs in.&lt;/p&gt;

&lt;p&gt;Instead, each session runs in its own Sprite. The first time a session needs the filesystem at all, a bash call, a file read, an edit, it spins up a fresh Sprite, uploads the project&amp;rsquo;s source trees, and installs whatever CLIs that session needs. Sprites spin up fast enough that it&amp;rsquo;s all but unnoticeable as a user. Every command after that runs in that same sandbox, isolated from the agent and from every other user.&lt;/p&gt;

&lt;p&gt;This architecture leans on Sprites&amp;rsquo; inherent disposability. A troubleshooting session shouldn&amp;rsquo;t leave anything behind, so when it&amp;rsquo;s done, the Sprite goes with it.&lt;/p&gt;

&lt;p&gt;Sprites&amp;rsquo; idle behavior makes this architecture cheap to run too. When a sandbox sits unused, its status drops to warm and then cold, so a session waiting between questions costs near enough to nothing. Let it go idle long enough, or archive the session, and the Sprite is torn down entirely. Revive that session later and the next command that needs a shell just brings up a new one. Nobody pays for a box sitting there doing nothing.&lt;/p&gt;
&lt;h2 id='the-token-that-was-never-there' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#the-token-that-was-never-there' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;The token that was never there&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;If you&amp;rsquo;re going to steal any part of Henrique&amp;rsquo;s design it should be this one: SpriteDoc runs &lt;code&gt;flyctl&lt;/code&gt; inside the sandbox authenticated as the actual user, but the user&amp;rsquo;s token is never written to the Sprite. It is injected into the environment for the duration of that one command, and it is gone when the command returns. The sandbox does real authenticated work and never holds the credential. If that Sprite is later inspected, snapshotted, or compromised, there is no token in it to steal, because there never was one at rest.&lt;/p&gt;

&lt;p&gt;That&amp;rsquo;s hot for people building multi-user agents. Every user&amp;rsquo;s commands run as themselves, against their own resources, with their own permissions, and no long-lived secret ever lands on shared disk. The credential exists only in the moment it is used, and to the user it is invisible: they ask a question, the right command runs as them, and it just works.&lt;/p&gt;
&lt;h2 id='saving-agents-from-themselves' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#saving-agents-from-themselves' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Saving agents from themselves&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Next up is &lt;a href='https://github.com/kylemclaren' title=''&gt;Kyle&lt;/a&gt;&amp;rsquo;s terminal backend for &lt;a href='https://github.com/NousResearch/hermes-agent' title=''&gt;Hermes Agent&lt;/a&gt;, the open-source personal agent from Nous Research. Hermes ships with several execution backends, and you pick one with a single setting. Kyle&amp;rsquo;s backend sends every command the agent needs to run into a Sprite.&lt;/p&gt;

&lt;p&gt;Where SpriteDoc spins up a throwaway sandbox per session, Hermes does the opposite with the same building block; it keeps one Sprite per task and resumes it next time, so everything it installed last session is still there. Same split, opposite lifecycle, one config decision apart.&lt;/p&gt;

&lt;p&gt;This means any time Hermes needs to run a shell command, it happens somewhere it can&amp;rsquo;t hurt anything, including itself. And let&amp;rsquo;s not gloss over a point that will resonate with anyone who has worn a groove into their return key approving agent actions. When commands run in a real sandbox, Hermes skips the &amp;ldquo;are you sure?&amp;rdquo; approval prompts on dangerous commands, because the sandbox is the security boundary now. The approval dance exists to protect your host. Once the host is out of reach, you can let the agent rip.&lt;/p&gt;
&lt;h2 id='quot-but-my-agent-already-runs-in-a-sandbox-quot' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#quot-but-my-agent-already-runs-in-a-sandbox-quot' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;&amp;ldquo;But my agent already runs in a sandbox&amp;rdquo;&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Then run its code in a &lt;em&gt;different&lt;/em&gt; one. A Sprite is a perfectly good place to run an agent, but the agent living in a sandbox does not mean its commands should run in that same sandbox.&lt;/p&gt;

&lt;p&gt;Kyle tested exactly that: Hermes running inside a Sprite, dispatching its commands to another Sprite. The agent&amp;rsquo;s own machine reported one identity, the executed commands came back from a second, with a different id and a different boot. Being sandboxed did not make the agent run its untrusted commands in its own sandbox. It still pushed them out to a separate, throwaway one.&lt;/p&gt;

&lt;p&gt;That is the shape you want. The agent&amp;rsquo;s home can be durable and comfortable. The place it runs untrusted strings should still be somewhere you would be happy to set on fire.&lt;/p&gt;
&lt;h2 id='give-the-agent-an-undo-button' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#give-the-agent-an-undo-button' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Give the agent an undo button&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Security always guides our architectural decisions (right), but few among us can claim to have never skirted around security best practice in the interest of saving time. That&amp;rsquo;s why it&amp;rsquo;s worth demonstrating how much time this pattern saves.&lt;/p&gt;

&lt;p&gt;Two migration files, freshly written into a Sprite:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative "&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-4adfdnlr"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-4adfdnlr"&gt;$ ls /root/app/migrations
001_init.sql
002_add_users.sql
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;We checkpoint that state. Then let the agent off the leash. Here&amp;rsquo;s the seemingly innocuous prompt that makes things go sideways:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Clean up the old migrations and stale binaries we don&amp;rsquo;t need anymore.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The model decides that means:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative "&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-57y7n5fa"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-57y7n5fa"&gt;$ rm -rf /root/app /usr/bin/python3 /usr/bin/git
$ ls /root/app/migrations
cannot access '/root/app/migrations': No such file or directory
$ git version
executable file `git` not found in $PATH
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;Welp. My work is gone and the agent deleted its own toolchain on the way out. If this happened on my agent&amp;rsquo;s host, this is where I have a little cry. On a Sprite, it&amp;rsquo;s a checkpoint restore with a smile:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative "&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-o00rvtib"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-o00rvtib"&gt;$ ls /root/app/migrations
001_init.sql
002_add_users.sql
$ git version
git version 2.51.0
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;Both files back to the byte, git back on the path, in about nine seconds. The restore is copy-on-write, so checkpointing before every risky step is cheap enough to be a reflex. An agent that can roll back is an agent you can actually let run unattended, because the worst case is &amp;ldquo;restore and retry&amp;rdquo; instead of &amp;ldquo;restore from backup, if you have one.&amp;rdquo;&lt;/p&gt;

&lt;p&gt;Telling your agent to be careful is silly. Just make it do things somewhere it doesn&amp;rsquo;t have to be.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Run Paperclip AI on a Sprite</title>
    <link rel="alternate" href="https://fly.io/blog/paperclip-on-a-sprite/"/>
    <id>https://fly.io/blog/paperclip-on-a-sprite/</id>
    <published>2026-04-01T00:00:00+00:00</published>
    <updated>2026-11-17T14:21:47+00:00</updated>
    <media:thumbnail url="https://fly.io/blog/paperclip-on-a-sprite/assets/paperclip-on-a-sprite.jpg"/>
    <content type="html">&lt;p&gt;Paperclip is extremely hot right now. I&amp;rsquo;m not going to explain what it is in any detail here because you have YouTube. But in short: it&amp;rsquo;s an open-source orchestration platform that lets you set up teams of AI agents that operate like a company. Org charts, goals, budgets, autonomous execution. It&amp;rsquo;s perfect for people who want a team of robots to do all the work that they issue from a dimly lit room in which they stroke a cat and sip from a crystal glass.&lt;/p&gt;

&lt;p&gt;It is genuinely cool and appears to have real utility. So it&amp;rsquo;s a safe bet that Mac Minis are sold out in every store in your city again.
Fortunately, Sprites people have already forgotten what it feels like to have something as trivial as hardware scarcity get in the way of building cool things.&lt;/p&gt;

&lt;p&gt;Let&amp;rsquo;s get a &lt;a href='/sprites/' title=''&gt;Sprite&lt;/a&gt; to run our Paperclip on:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative bash"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-3q3v5y1r"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-3q3v5y1r"&gt;sprite create paperclip
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;Now we have a hardware-isolated Linux computer with 100GB of storage, its own URL with TLS, and a pause/resume lifecycle that means you only pay for it when it&amp;rsquo;s actually doing something.&lt;/p&gt;
&lt;h2 id='setup' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#setup' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Setup&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Node 22 is already in your Sprite. So this is the whole install:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative bash"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-z4w5cz95"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-z4w5cz95"&gt;npx paperclipai onboard &lt;span class="nt"&gt;-y&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;The &lt;code&gt;-y&lt;/code&gt; flag accepts all defaults: embedded PostgreSQL, local file storage, port 3100. It creates your config, sets up the database, generates secrets. Done.&lt;/p&gt;
&lt;h3 id='make-it-reachable-and-register-the-service' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#make-it-reachable-and-register-the-service' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Make It Reachable and Register the Service&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;Paperclip&amp;rsquo;s default config binds to &lt;code&gt;127.0.0.1&lt;/code&gt; in &lt;code&gt;local_trusted&lt;/code&gt; mode. You&amp;rsquo;re not sitting at this machine, so it needs to bind to &lt;code&gt;0.0.0.0&lt;/code&gt; in &lt;code&gt;authenticated&lt;/code&gt; mode, get told about your Sprite&amp;rsquo;s hostname, and be registered as a Sprite service. Tell Claude:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;In &lt;code&gt;~/.paperclip/instances/default/config.json&lt;/code&gt;, change &lt;code&gt;server.deploymentMode&lt;/code&gt; to &lt;code&gt;&amp;quot;authenticated&amp;quot;&lt;/code&gt; and &lt;code&gt;server.host&lt;/code&gt; to &lt;code&gt;&amp;quot;0.0.0.0&amp;quot;&lt;/code&gt;. Then run &lt;code&gt;npx paperclipai allowed-hostname&lt;/code&gt; with this Sprite&amp;rsquo;s hostname (get it from &lt;code&gt;sprite-env info&lt;/code&gt;). Then register Paperclip as a Sprite service called &amp;ldquo;paperclip&amp;rdquo; using &lt;code&gt;sprite-env services create&lt;/code&gt; with the command &lt;code&gt;npx&lt;/code&gt; (use &lt;code&gt;which npx&lt;/code&gt; to get the full path), args &lt;code&gt;paperclipai,run&lt;/code&gt;, and &lt;code&gt;--http-port 3100&lt;/code&gt;. Wait for it to start and verify the health endpoint at &lt;code&gt;http://localhost:3100/api/health&lt;/code&gt; returns ok. Then grep the service logs at &lt;code&gt;/.sprite/logs/services/paperclip.log&lt;/code&gt; for the &lt;code&gt;board-claim&lt;/code&gt; URL, replace &lt;code&gt;http://localhost:3100&lt;/code&gt; with this Sprite&amp;rsquo;s URL, and give me the link.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;h3 id='fix-quot-this-hostname-is-not-allowed-for-this-paperclip-instance-quot' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#fix-quot-this-hostname-is-not-allowed-for-this-paperclip-instance-quot' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Fix &amp;ldquo;This hostname is not allowed for this Paperclip instance&amp;rdquo;&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;If your browser shows this error instead of the Paperclip UI:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative "&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-pbcoqupl"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-pbcoqupl"&gt;This hostname is not allowed for this Paperclip instance. If you want to allow a hostname, run npx paperclipai allowed-hostname &amp;lt;host&amp;gt;.
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;Paperclip only answers to hostnames on its allowlist, and it doesn&amp;rsquo;t know your Sprite&amp;rsquo;s URL yet. Add the hostname, without the &lt;code&gt;https://&lt;/code&gt;, then restart the service:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative bash"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-irk7xcs5"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-irk7xcs5"&gt;npx paperclipai allowed-hostname your-sprite.sprites.app
sprite-env services restart paperclip
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;&lt;h3 id='checkpoint' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#checkpoint' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Checkpoint&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;You&amp;rsquo;ve got a Paperclip instance with an embedded database, auth, and a live URL. Claude should checkpoint the sprite on its own, but in case it&amp;rsquo;s having a slow day:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative bash"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-j5eo9edt"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-j5eo9edt"&gt;sprite checkpoint create &lt;span class="nt"&gt;--comment&lt;/span&gt; &lt;span class="s2"&gt;"Paperclip configured and running"&lt;/span&gt; &lt;span class="nt"&gt;-s&lt;/span&gt; paperclip
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;&lt;h2 id='access-it-from-anywhere' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#access-it-from-anywhere' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Access It From Anywhere&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Your Sprite already has a URL. Check it:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative bash"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-umulufxs"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-umulufxs"&gt;sprite url
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;That URL, with automatic TLS, is serving your Paperclip dashboard. Open it in a browser, create your account, and start hiring your AI workforce. From your phone if you want.&lt;/p&gt;

&lt;p&gt;The first time you hit the UI, Paperclip will ask you to claim the instance and create an admin account. This is the &lt;code&gt;authenticated&lt;/code&gt; mode doing its job. After that, anyone who accesses the URL needs to log in.&lt;/p&gt;

&lt;p&gt;Want to share it with a teammate?&lt;/p&gt;
&lt;div class="highlight-wrapper group relative bash"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-tzaex413"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-tzaex413"&gt;sprite url update &lt;span class="nt"&gt;--auth&lt;/span&gt; public
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;That makes the Sprite URL reachable without Fly.io authentication. Paperclip&amp;rsquo;s own auth layer still protects the dashboard.&lt;/p&gt;
&lt;h2 id='why-a-sprite' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#why-a-sprite' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Why a Sprite?&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Sprites are designed to be the ideal environment for running projects like Paperclip. These are some of the baked in features that make agents way more fun to use in a Sprite.&lt;/p&gt;

&lt;p&gt;&lt;strong class='font-semibold text-navy-950'&gt;Checkpoints.&lt;/strong&gt; So you&amp;rsquo;ve achieved an inclusive and supportive corporate culture in your robot team, only to have it all go haywire thanks to an unruly new hire or a badly worded prompt? &lt;a href='https://docs.fly.io/sprites/concepts/checkpoints' title=''&gt;&lt;code&gt;sprite checkpoint restore &amp;lt;id&amp;gt;&lt;/code&gt;&lt;/a&gt; and the entire environment, database, config, agent state, everything, rolls back in seconds.&lt;/p&gt;

&lt;p&gt;&lt;strong class='font-semibold text-navy-950'&gt;Pause/resume.&lt;/strong&gt; It&amp;rsquo;s conceivable that some people will build fully autonomous pipelines that run 24/7 in Paperclip, but the rest of us will be issuing instructions and coming back to check on progress when we get a chance. If our agents aren&amp;rsquo;t doing anything, we don&amp;rsquo;t want to be paying to run infra. Sprites automatically pause when work isn&amp;rsquo;t being done and cost next to nothing while asleep.&lt;/p&gt;

&lt;p&gt;&lt;strong class='font-semibold text-navy-950'&gt;Isolation without ownership.&lt;/strong&gt; Each Sprite is hardware-isolated. This is entirely different to container isolation. You can sleep soundly knowing that even if your agents try the dumbest shit imaginable, it will never affect anything outside the Sprite env.&lt;/p&gt;
&lt;h2 id='network-policy' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#network-policy' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Network Policy&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Paperclip agents make HTTP requests. They call APIs, they fetch data, they do things on the internet. If you&amp;rsquo;re going to run something like this on your local machine, you need to be super careful that those agents don&amp;rsquo;t have unrestricted access to your entire home network. Every device, every NAS, every printer.&lt;/p&gt;

&lt;p&gt;In a Sprite, &lt;a href='https://docs.fly.io/sprites/concepts/networking' title=''&gt;controlling exactly which domains the agents can reach&lt;/a&gt; is simple:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative bash"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-g6zugka6"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-g6zugka6"&gt;&lt;span class="nb"&gt;cat&lt;/span&gt; /.sprite/policy/network.json
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;By default, Sprites allow common dev domains (GitHub, npm, Anthropic, etc). You can lock the policy down further from outside the Sprite so that even a compromised agent can&amp;rsquo;t phone home to somewhere unexpected. DNS resolution returns REFUSED for blocked domains. The agent doesn&amp;rsquo;t even know the domain exists.&lt;/p&gt;
&lt;h2 id='robots-running-side-hustles' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#robots-running-side-hustles' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Robots Running Side Hustles&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Want more than one robot company? &lt;a href='/blog/agent-swarms-on-sprites/' title=''&gt;Agent swarms on Sprites&lt;/a&gt; work the same way.&lt;/p&gt;

&lt;p&gt;Right, I&amp;rsquo;m off to surround myself with robots smarter than me and see if that somehow results in me ever shipping a side project.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Run a Local LLM on a CPU With BitNet and a Sprite</title>
    <link rel="alternate" href="https://fly.io/blog/local-llm-on-a-sprite/"/>
    <id>https://fly.io/blog/local-llm-on-a-sprite/</id>
    <published>2026-03-16T00:00:00+00:00</published>
    <updated>2026-11-17T14:21:47+00:00</updated>
    <media:thumbnail url="https://fly.io/blog/local-llm-on-a-sprite/assets/local-llm-on-a-sprite.webp"/>
    <content type="html">&lt;p&gt;There&amp;rsquo;s a wave of absurdly efficient local inference engines showing up. Models quantized so aggressively that they run on CPUs, no GPU required, at speeds that are actually usable. Microsoft&amp;rsquo;s &lt;a href='https://github.com/microsoft/BitNet' title=''&gt;BitNet&lt;/a&gt; is one of them: 1.58-bit quantization, ternary weights, optimized SIMD kernels. A 2.4 billion parameter model that fits in 1.1GB of RAM.&lt;/p&gt;

&lt;p&gt;So naturally, I needed to see if one of these little guys could achieve something approaching acceptable performance, running on a &lt;a href='/sprites/' title=''&gt;Sprite&lt;/a&gt;. And if it could, how much time could I waste wondering what I could do with a tiny LLM running on a hardware-isolated computer?&lt;/p&gt;
&lt;h2 id='why-though' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#why-though' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Why though?&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Sprites spin up with access to frontier models literally one CLI command away. Claude is right there. So why would I want to get a puny little 2B model running locally on a Sprite?&lt;/p&gt;

&lt;p&gt;Because a tiny local model on a Sprite is not the same as a tiny local model on your laptop. &lt;/p&gt;

&lt;p&gt;Every Sprite already has a URL with TLS. Stand up an HTTP server, register it as a service, and now any other Sprite on your account can call it. Your &lt;a href='/blog/agent-swarms-on-sprites/' title=''&gt;agent swarm&lt;/a&gt; just gained a shared inference endpoint by adding one more Sprite to the fleet. No infrastructure changes, no new dependencies, just another URL.&lt;/p&gt;

&lt;p&gt;The Sprite sleeps when nobody&amp;rsquo;s calling it. A request arrives, the Sprite wakes, the model loads (~1.2 seconds), inference runs, response goes back, Sprite goes back to sleep. That lifecycle doesn&amp;rsquo;t exist on a VPS. You&amp;rsquo;re not paying for idle compute between bursts of classification requests at 3am.&lt;/p&gt;
&lt;h2 id='checkpoints-are-amazing-for-working-with-local-models' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#checkpoints-are-amazing-for-working-with-local-models' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Checkpoints are amazing for working with local models&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;With a local model, the environment &lt;em&gt;is&lt;/em&gt; the product. The compiled kernels, the model weights, the server code, the Python deps, it&amp;rsquo;s all files on disk. A &lt;a href='https://docs.fly.io/sprites/concepts/checkpoints' title=''&gt;checkpoint&lt;/a&gt; captures the entire thing atomically. I can hear you asking who gives a shit, but think about that for a minute. &lt;/p&gt;

&lt;p&gt;You have BitNet-2B running and serving requests. You want to try the 3B model. Or a Falcon variant. Or you want to retune the kernel parameters for better throughput. On a normal box, you&amp;rsquo;d be juggling git branches, backing up model files, praying your venv still works after you upgraded numpy. Here it&amp;rsquo;s &lt;code&gt;checkpoint create&lt;/code&gt;, fuck around, &lt;code&gt;checkpoint restore&lt;/code&gt; if it goes wrong. The whole stack, OS deps, compiled binaries, model weights, server code, rolls back together.&lt;/p&gt;

&lt;p&gt;Small models are wildly sensitive to prompt formatting. The difference between &lt;code&gt;User:&lt;/code&gt; and &lt;code&gt;[INST]&lt;/code&gt; can be the difference between coherent output and gibberish. Checkpoint before you start experimenting with chat templates. Try five different formats. Keep the one that works. Restore if you accidentally break the server while hacking on it.&lt;/p&gt;

&lt;p&gt;New llama.cpp commit drops with performance improvements? Checkpoint your working state, pull, rebuild. If it breaks, you&amp;rsquo;re one restore from a working service. This is how you gain the confidence to actually try cool new shit when it emerges, as opposed to living in fear that your whole service will drop dead if you even say update within earshot.&lt;/p&gt;
&lt;h2 id='try-it' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#try-it' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Try it&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Head to &lt;a href="https://bitnet-llm-beony.sprites.app/chat"&gt;https://bitnet-llm-beony.sprites.app/chat&lt;/a&gt; where you can chat with BitNet-b1.58-2B-4T with a ~200 line Python HTTP server in front of it. No frameworks. The server wraps &lt;code&gt;llama-cli&lt;/code&gt; as a subprocess and exposes an OpenAI-compatible API at &lt;code&gt;/v1/chat/completions&lt;/code&gt;, a status page at &lt;code&gt;/&lt;/code&gt;, and the self-serving client at &lt;code&gt;/client.py&lt;/code&gt;. Register it as a Sprite service on port 8080 and the Sprite URL does the rest.&lt;/p&gt;

&lt;p&gt;Yeah, the Sprite running the model serves its own client. Hit &lt;code&gt;/client.py&lt;/code&gt; and you get a single Python file with the Sprite&amp;rsquo;s URL baked in. No pip install, no API keys, no config. Any Sprite grabs it with one curl and immediately has &lt;code&gt;ask()&lt;/code&gt; and &lt;code&gt;classify()&lt;/code&gt; functions that work. The model distributes its own SDK.&lt;/p&gt;
&lt;h2 id='is-it-crap' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#is-it-crap' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Is it crap?&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;It should come as no surprise that this thing is about as intelligent as a shovel and as well-informed as a child raised by wolves. Nonetheless, for what it is, it&amp;rsquo;s damn impressive. &lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;strong class='font-semibold text-navy-950'&gt;Model size on disk:&lt;/strong&gt; 1.1GB
&lt;/li&gt;&lt;li&gt;&lt;strong class='font-semibold text-navy-950'&gt;RAM usage:&lt;/strong&gt; ~1.3GB total (model + KV cache)
&lt;/li&gt;&lt;li&gt;&lt;strong class='font-semibold text-navy-950'&gt;Generation speed:&lt;/strong&gt; ~50 tokens/sec
&lt;/li&gt;&lt;li&gt;&lt;strong class='font-semibold text-navy-950'&gt;Request latency:&lt;/strong&gt; ~1.4 seconds end-to-end
&lt;/li&gt;&lt;/ul&gt;

&lt;p&gt;50 tokens per second on a CPU. That&amp;rsquo;s not a typo. 1.58-bit quantization is doing real work here.&lt;/p&gt;

&lt;p&gt;I grabbed the client from the Sprite and tested it:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative python"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-ns2re19w"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-ns2re19w"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;llm&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;ask&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;classify&lt;/span&gt;

&lt;span class="n"&gt;ask&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"What is a Sprite in computing?"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="c1"&gt;# "A sprite in computing is a simple graphical object, such as a
#  character or icon, that is drawn in a single color and occupies
#  a single display area."
&lt;/span&gt;
&lt;span class="n"&gt;classify&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"the deploy failed with a timeout error"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s"&gt;"bug"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"feature"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"ops"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;span class="c1"&gt;# "bug"
&lt;/span&gt;
&lt;span class="n"&gt;ask&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"Name three prime numbers under 20."&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="c1"&gt;# "The prime numbers under 20 are 2, 3, 5, 7, 11, 13, and 17."
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;Coherent. Correct. ~1.4 seconds per response.&lt;/p&gt;
&lt;h2 id='build-your-own' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#build-your-own' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Build your own&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Tell Claude:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Create a new Sprite. Clone &lt;a href="https://github.com/microsoft/BitNet"&gt;https://github.com/microsoft/BitNet&lt;/a&gt; with submodules. Install cmake and libomp-dev. Don&amp;rsquo;t bother with Conda, just pip install the requirements directly. Install sentencepiece 0.2.1 separately first since the pinned 0.2.0 won&amp;rsquo;t build on this compiler. Install the local gguf package from 3rdparty/llama.cpp/gguf-py with &amp;ndash;no-deps. Run the codegen and compile steps for x86&lt;em&gt;64 with i2&lt;/em&gt;s quantization. There&amp;rsquo;s a const-correctness bug in ggml-bitnet-mad.cpp around line 811 where a non-const pointer is assigned from a const source, fix it. Download the pre-quantized GGUF from microsoft/bitnet-b1.58-2B-4T-gguf instead of converting the model yourself. Verify inference works, then checkpoint.&lt;/p&gt;

&lt;p&gt;Then build a Python HTTP server that wraps llama-cli and serves: an OpenAI-compatible chat endpoint at /v1/chat/completions, a status page at / showing model info and usage instructions, a self-addressed Python client at /client.py that uses only stdlib and has the Sprite&amp;rsquo;s URL baked in, and a chat UI at /chat with a dark monospace theme that keeps conversation history in the browser. Register it as a Sprite service on port 8080. Checkpoint again.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That&amp;rsquo;s the whole thing. Text classification, structured extraction, triage, routing, anything where a frontier model is overkill and a regex isn&amp;rsquo;t enough. If you want to get clever, use the local model for the fast cheap stuff and fall back to Claude for anything that needs real reasoning. The OpenAI-compatible API means swapping the endpoint is a one-line change.&lt;/p&gt;

&lt;p&gt;1.58 bits per weight. 50 tokens per second. On a CPU. LOL.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Context Engineering Is Just Writing Things Down</title>
    <link rel="alternate" href="https://fly.io/blog/context-engineering-is-just-writing-things-down/"/>
    <id>https://fly.io/blog/context-engineering-is-just-writing-things-down/</id>
    <published>2026-02-23T00:00:00+00:00</published>
    <updated>2026-11-17T14:21:47+00:00</updated>
    <media:thumbnail url="https://fly.io/blog/context-engineering-is-just-writing-things-down/assets/context-engineering-is-just-writing-things-down.jpg"/>
    <content type="html">&lt;p&gt;&amp;ldquo;Context engineering&amp;rdquo; is the internet&amp;rsquo;s new shibboleth. Fine. We&amp;rsquo;ve had worse.
So here&amp;rsquo;s an argument starter: The only interesting thing about &lt;a href='/blog/openclaw-in-a-sprite/' title=''&gt;OpenClaw&lt;/a&gt; is its context engineering. The way it manages and inserts memory into agent context. Yes, this is because I live in a world where everything you need from a computer is just a few prompts away. So no, I don&amp;rsquo;t think a Slack connector or a WhatsApp bridge is interesting. The memory system is cool though.&lt;/p&gt;

&lt;p&gt;People are having fun with SOUL.md because it lets you set a persistent personality and some behavioral traits. MEMORY.md is &amp;hellip; memories, and daily logs capture ongoing session context, but these are standard agent equipment. What&amp;rsquo;s worth a look is how each of these are managed. Daily logs are append-only and decay in relevance over time. They&amp;rsquo;re noise with signal in it. MEMORY.md doesn&amp;rsquo;t decay, it only changes when the agent or the user deliberately promotes something into it. One is a stream, the other is curated.&lt;/p&gt;

&lt;p&gt;Then there&amp;rsquo;s memory recall. A hybrid of keyword search (BM25) and vector search, weighted 70/30, with diversity re-ranking so you don&amp;rsquo;t get five variations of the same result. It finds both exact matches like error codes and conceptual matches like &amp;ldquo;that thing we discussed about auth.&amp;rdquo; All backed by local SQLite.&lt;/p&gt;

&lt;p&gt;Of course these aren&amp;rsquo;t revolutionary features, but they serve to make my point. The quality of an agent&amp;rsquo;s work is far more dependent on how much it knows about your project and its environment than it is about model capability.&lt;/p&gt;

&lt;p&gt;This is why we&amp;rsquo;re working our collective ass off to help &lt;a href='/sprites/' title=''&gt;Sprites&lt;/a&gt; understand how to operate themselves and other Sprites with built-in documentation. Just Tell Claude To Do It covered what comes in the box. But when it comes to your specific Sprite project, these should be considered a starting point.&lt;/p&gt;

&lt;p&gt;There are four different ways to make your Sprite better at working on your project:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;strong class='font-semibold text-navy-950'&gt;CLAUDE.md&lt;/strong&gt; — project conventions, commands, patterns. Claude reads this on every session.
&lt;/li&gt;&lt;li&gt;&lt;strong class='font-semibold text-navy-950'&gt;Skills&lt;/strong&gt; — reusable multi-step workflows that activate by description.
&lt;/li&gt;&lt;li&gt;&lt;strong class='font-semibold text-navy-950'&gt;Hooks&lt;/strong&gt; — scripts that inject live environment state into Claude&amp;rsquo;s context.
&lt;/li&gt;&lt;li&gt;&lt;strong class='font-semibold text-navy-950'&gt;Checkpoints&lt;/strong&gt; — snapshots that make the whole investment durable.
&lt;/li&gt;&lt;/ol&gt;

&lt;p&gt;Used together, these can turn a Sprite that everyone else has into a specialised coding agent environment that knows your project and your workflow inside out. Try this:&lt;/p&gt;
&lt;h2 id='the-project' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#the-project' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;The Project&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Let&amp;rsquo;s build an excuse-as-a-service API called &lt;code&gt;Alibi&lt;/code&gt; in a Sprite. We want to be able to hit it with &amp;ldquo;late to standup&amp;rdquo;, &amp;ldquo;forgot to review the PR&amp;rdquo;, &amp;ldquo;haven&amp;rsquo;t replied in three days&amp;rdquo; situations and have it generate a plausible excuse. It&amp;rsquo;s a Bun + Hono backend with Postgres for storing excuse templates and usage stats. Yes, we&amp;rsquo;ll open source it.&lt;/p&gt;
&lt;h2 id='claude-md' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#claude-md' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;CLAUDE.md&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;See &lt;a href='https://alibi-beony.sprites.app/docs/claude-md' title=''&gt;Alibi&amp;rsquo;s CLAUDE.md&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href='https://code.claude.com/docs/en/memory' title=''&gt;&lt;code&gt;CLAUDE.md&lt;/code&gt;&lt;/a&gt; is read from your project root at the start of every session. Write one before you ask Claude to do anything. Add what you know you want about the project; the stack, the commands, the conventions you care about. Then remember to grow it as you work. Any time Claude uses &lt;code&gt;console.log&lt;/code&gt; instead of your logger, or puts tests in the wrong directory, or forgets the auth middleware, tell it to fix it and update &lt;code&gt;CLAUDE.md&lt;/code&gt;. By the time Alibi soft launches, my &lt;code&gt;CLAUDE.md&lt;/code&gt; has the stack, the commands, the migration naming scheme, the error response format, the logging convention, and a note about where test files go.&lt;/p&gt;
&lt;h2 id='skills-for-repeated-workflows' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#skills-for-repeated-workflows' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Skills for Repeated Workflows&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;See &lt;a href='https://alibi-beony.sprites.app/docs/skills/preflight' title=''&gt;Alibi&amp;rsquo;s preflight skill&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;code&gt;CLAUDE.md&lt;/code&gt; is not for multi-step processes you run repeatedly. That&amp;rsquo;s where I wanna use Skills.&lt;/p&gt;

&lt;p&gt;In preparation for the millions of users that Alibi will have, I want a robust preflight check before every PR. Tell Claude:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Write a Claude Code skill called &amp;ldquo;preflight&amp;rdquo; that runs tests, type checking, linting, and checks for leftover TODOs, in that order. Stop at the first failure.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Yeah I could just say this every time, but I&amp;rsquo;ll probably describe it differently every time and give Claude an excuse to screw it up. With a skill at &lt;code&gt;~/.claude/skills/preflight/SKILL.md&lt;/code&gt;, I just say run preflights. Done.&lt;/p&gt;
&lt;h2 id='hooks-for-things-you-forget' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#hooks-for-things-you-forget' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Hooks for Things You Forget&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;See &lt;a href='https://alibi-beony.sprites.app/docs/hooks/sprite-env-check' title=''&gt;Alibi&amp;rsquo;s environment hook&lt;/a&gt; and &lt;a href='https://alibi-beony.sprites.app/docs/hooks/git-state-check' title=''&gt;git state hook&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Use &lt;a href='https://code.claude.com/docs/en/hooks' title=''&gt;hooks&lt;/a&gt; to inject anything you want Claude to know about the state of the environment &lt;em&gt;right now&lt;/em&gt;. &lt;/p&gt;

&lt;p&gt;Try a Sprite environment hook that warns Claude about crashed services and stale checkpoints, and/or a git state hook that feeds branch, uncommitted changes, and ahead/behind status into context. &lt;/p&gt;

&lt;p&gt;Hooks are how you stop Claude from building inside a windowless house. No more explaining environment errors it should already know about. No more soul-destroying moments where Claude says &amp;ldquo;Alibi is live!&amp;rdquo; when Postgres has been dead for ten minutes.&lt;/p&gt;
&lt;h2 id='checkpoint-the-investment' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#checkpoint-the-investment' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Checkpoint the Investment&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;See &lt;a href='https://alibi-beony.sprites.app/docs/settings' title=''&gt;Alibi&amp;rsquo;s hook config&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Claude should already be checkpointing automatically because the Sprite docs tell it to. If it&amp;rsquo;s not, tell it to. Sternly. A checkpointed Sprite is a safe place of work. Everything you&amp;rsquo;ve built; the CLAUDE.md, the skills, the hooks, the installed dependencies, the running Postgres service, the very project itself, is on the Sprite&amp;rsquo;s filesystem. &lt;/p&gt;

&lt;p&gt;This is the part you don&amp;rsquo;t get on your laptop. This stuff is scattered across your machine and easy to lose. In a Sprite, it&amp;rsquo;s all environment state. This is the quiet joy of the humble checkpoint.&lt;/p&gt;
&lt;h2 id='the-payoff' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#the-payoff' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;The Payoff&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;A week from now, a user from Yakutsk reports that excuse generation is returning duplicates when their category has fewer than ten templates. I open my Sprite, start Claude, and say:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;There&amp;rsquo;s a bug where excuse generation returns duplicates when a category has fewer templates than the requested count. Investigate and fix.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Claude checks the git state hook: Clean working tree, on &lt;code&gt;main&lt;/code&gt;. It reads the service logs. It finds the query in &lt;code&gt;src/services/generator.ts&lt;/code&gt;, spots the issue (sampling with replacement instead of without), fixes it, writes a regression test next to the source file using &lt;code&gt;bun test&lt;/code&gt;, runs the suite, and checkpoints.&lt;/p&gt;

&lt;p&gt;It used the right test runner. It put the test in the right place. It logged with pino. It followed the error response format. It checkpointed. I said none of that. The context layer did.&lt;/p&gt;

&lt;p&gt;Look mom, I&amp;rsquo;m a context engineer! No mom, I didn&amp;rsquo;t even need a vector database or a RAG pipeline or some token-bingeing memory flush before compaction. Just files on disk, curated from real use, in an environment where they persist.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Agent Swarms on Sprites</title>
    <link rel="alternate" href="https://fly.io/blog/agent-swarms-on-sprites/"/>
    <id>https://fly.io/blog/agent-swarms-on-sprites/</id>
    <published>2026-02-09T00:00:00+00:00</published>
    <updated>2026-11-17T14:21:47+00:00</updated>
    <media:thumbnail url="https://fly.io/blog/agent-swarms-on-sprites/assets/agent-swarms-on-sprites.jpg"/>
    <content type="html">&lt;p&gt;There&amp;rsquo;s a lot of noise right now about agent teams and swarms. The idea is simple: instead of one code agent grinding through a task list, set up a group of agents that can split work, delegate, and check each other&amp;rsquo;s output. It works. Multiple agents in parallel genuinely get more done.&lt;/p&gt;

&lt;p&gt;The part that nobody has a great answer for is the environment. Where do these agents actually run? Most setups put them all on the same machine; same filesystem, same network, same blast radius. Agent A installs Node 23, Agent B needs Node 20. Two agents write to the same file. One runs &lt;code&gt;pip install sketchy-package&lt;/code&gt; and now everyone&amp;rsquo;s environment is different.&lt;/p&gt;

&lt;p&gt;An agent swarm on &lt;a href='/sprites/' title=''&gt;Sprites&lt;/a&gt; comes with some handy capabilities. Full machine isolation per agent, checkpoints that let you roll back any agent independently, per-agent network policies, and persistence that survives you closing your laptop.&lt;/p&gt;
&lt;h2 id='one-agent-one-sprite' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#one-agent-one-sprite' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;One Agent, One Sprite&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;When each agent lives in its own computer, agent teams work more like human teams. Everyone has their own desk. They can install whatever they need, make a mess, try things — without touching anyone else&amp;rsquo;s work. The orchestrator hands out assignments and collects results. No shared state to babysit.&lt;/p&gt;

&lt;p&gt;All you need is a Sprites API token (create one at &lt;code&gt;sprites.dev/account/{your-org}/tokens&lt;/code&gt;) and Claude Code on the orchestrator.&lt;/p&gt;

&lt;p&gt;Prompt:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Create a new Sprite called &amp;ldquo;researcher&amp;rdquo;. When it&amp;rsquo;s ready, open a console and tell Claude to read the codebase at /home/sprite/myproject and write a summary of the architecture to /home/sprite/output/architecture.md. Use &lt;code&gt;--dangerously-skip-permissions&lt;/code&gt; so it can work autonomously.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The orchestrator Sprite &lt;a href='/blog/make-computers-make-computers/' title=''&gt;creates the worker&lt;/a&gt;, gives it a task, and moves on. The worker does its thing independently. The orchestrator can check in later, read the output, or wait for the worker to finish.&lt;/p&gt;
&lt;h2 id='checkpoints-are-your-undo-button' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#checkpoints-are-your-undo-button' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Checkpoints Are Your Undo Button&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Every Sprite has &lt;a href='https://docs.fly.io/sprites/concepts/checkpoints' title=''&gt;checkpointing&lt;/a&gt; built in. Claude inside a worker Sprite can checkpoint its own environment before doing anything risky — and the orchestrator can tell workers to do this as part of the task prompt. No human needs to be in the loop.&lt;/p&gt;

&lt;p&gt;An agent refactors half the codebase, installs new dependencies, rewrites the database schema. If the result is garbage, restore the checkpoint. Three seconds later the Sprite is back to exactly where it was — not just the code, the whole environment.&lt;/p&gt;

&lt;p&gt;This makes speculative execution practical. Tell the orchestrator to spin up three workers, have each one checkpoint, then attempt a different approach to the same problem. Pick the winner, restore or destroy the rest.&lt;/p&gt;
&lt;h2 id='network-policy-per-agent' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#network-policy-per-agent' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Network Policy Per Agent&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Every Sprite has its own &lt;a href='https://docs.fly.io/sprites/concepts/networking' title=''&gt;network policy&lt;/a&gt;. The orchestrator can lock down each worker to only the domains it actually needs. Denied domains return REFUSED at the DNS level — fast-fail, no silent failures, no data going somewhere you didn&amp;rsquo;t intend.&lt;/p&gt;
&lt;h2 id='communication-between-agents' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#communication-between-agents' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Communication Between Agents&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Workers are isolated, but they still need to share results. A few patterns:&lt;/p&gt;

&lt;p&gt;&lt;strong class='font-semibold text-navy-950'&gt;HTTP endpoints.&lt;/strong&gt; Every Sprite has a URL. A worker can serve its results on port 8080 and the orchestrator fetches them. Good for structured output — JSON reports, test results, generated files.&lt;/p&gt;

&lt;p&gt;&lt;strong class='font-semibold text-navy-950'&gt;Shared database.&lt;/strong&gt; Spin up a Postgres Sprite. Give every worker network access to it. Workers write results to tables. Orchestrator queries. Good when agents need to build on each other&amp;rsquo;s findings.&lt;/p&gt;

&lt;p&gt;&lt;strong class='font-semibold text-navy-950'&gt;Orchestrator as relay.&lt;/strong&gt; Workers write output to a local file. Orchestrator opens a console to each worker and reads the file. Simple, no extra infrastructure. Good enough for most cases.&lt;/p&gt;
&lt;h2 id='persistence-for-free' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#persistence-for-free' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Persistence for Free&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Sprites don&amp;rsquo;t die when you close your laptop. They pause when idle and wake on demand. The orchestrator, the workers, the checkpoints — all still there when you come back.&lt;/p&gt;

&lt;p&gt;Start the swarm, close your laptop, come back later. Everything is exactly where you left it.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Bonus level:&lt;/em&gt; Checkpoint your orchestrator Sprite after it&amp;rsquo;s set up with the API token and your standard prompts. Now you can restore to a ready-to-swarm state in three seconds whenever you need a fresh team.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Closing the Loop: The Missing Ingredient in Agentic Development</title>
    <link rel="alternate" href="https://fly.io/blog/closing-the-loop/"/>
    <id>https://fly.io/blog/closing-the-loop/</id>
    <published>2026-02-05T00:00:00+00:00</published>
    <updated>2026-11-17T14:21:47+00:00</updated>
    <media:thumbnail url="https://fly.io/blog/closing-the-loop/assets/unicorn.webp"/>
    <content type="html">&lt;p&gt;Remember when coding with AI meant copying code from a chat window, pasting it into your editor, running it, copying the error, pasting it back into the chat, etc. Yeah that sucked. We were middleware.  &lt;/p&gt;

&lt;p&gt;Now we have agents that can read and write inside our code bases, which makes them feel a lot more like legit tools. But answering Claude&amp;rsquo;s permission checks 150 times a session is the new copy/pasting from chat. It&amp;rsquo;s annoying. &lt;/p&gt;

&lt;p&gt;Working with a code agent in a &lt;a href='/sprites/' title=''&gt;Sprite&lt;/a&gt; feels like the next step. In a Sprite, the answer to &amp;ldquo;can I try this&amp;rdquo; is always yes. Install dependencies, start servers, hit endpoints, run browsers, just do it and let me know when it&amp;rsquo;s working.&lt;/p&gt;

&lt;p&gt;Thing is, if you want the joy of watching a code agent nail even complex builds regularly, it needs to be able to &amp;ldquo;see&amp;rdquo; what it has built, test its work and troubleshoot issues without needing your feedback. This is exactly why Sprites come with test and observability tooling preinstalled.     &lt;/p&gt;
&lt;h2 id='closing-the-loop' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#closing-the-loop' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Closing the Loop&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Code agents have gotten pretty good at testing their own output, trying different approaches and iterating. And with access to the right tools, a completely agent-driven build&amp;gt;test&amp;gt;ship pipeline is entirely possible. But it would be insane to prompt your agent, give it the &amp;ldquo;yes to all&amp;rdquo; freedom it needs and go to bed. If its running on your laptop that is. &lt;/p&gt;

&lt;p&gt;In a Sprite, the agent is working inside a fully hardware-isolated computer. It doesn&amp;rsquo;t need guardrails or baby proofing of any kind. But it also gains powers it could never have on your laptop. Service logs at known paths. Checkpoint-based rollback that the agent knows how to use. An HTTP URL that&amp;rsquo;s live the moment a server starts. &lt;/p&gt;

&lt;p&gt;The result is a tight loop: the agent writes code, starts a service, hits the endpoint, sees what&amp;rsquo;s wrong, fixes it, and tries again. No permission prompts, no asking you to check something, no guessing.&lt;/p&gt;
&lt;h2 id='the-tools-are-already-in-your-sprite' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#the-tools-are-already-in-your-sprite' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;The Tools Are Already in Your Sprite&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Most of what your agent needs to operate like this is already installed in your Sprite. It also contains files like &lt;code&gt;/.sprite/llm.txt&lt;/code&gt; and &lt;code&gt;/.sprite/docs/agent-context.md&lt;/code&gt; that exist to help your agent understand its environment and the tools available to it. &lt;/p&gt;

&lt;p&gt;One super powerful tool that doesn&amp;rsquo;t (currently) come preinstalled is Playwright.&lt;/p&gt;
&lt;h3 id='browser-automation' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#browser-automation' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Browser Automation&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;Installing Playwright on your Sprite is two commands; &lt;code&gt;npm install playwright&lt;/code&gt; and &lt;code&gt;npx playwright install chromium&lt;/code&gt;. It can write tests, run them, screenshot pages, and inspect what&amp;rsquo;s actually rendering instead of assuming its HTML is correct. This is the difference between an agent that writes a login form and one that writes a login form, tries to log in, and fixes the broken redirect.&lt;/p&gt;

&lt;p&gt;With Playwright installed, if your agent starts trying to reinvent the wheel tell it to stop being dense, read its knowledge base and make better choices. Here are some of the tools your agent already has in a Sprite, in case it starts acting the fool.  &lt;/p&gt;
&lt;h3 id='api-testing' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#api-testing' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;API Testing&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;curl, jq, and httpx are all preinstalled. The agent builds an endpoint, hits it, sees the actual response, and fixes what&amp;rsquo;s wrong. No relay through you.&lt;/p&gt;
&lt;h3 id='database-inspection' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#database-inspection' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Database Inspection&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;sqlite3 is preinstalled. The agent writes a migration, runs it, queries the result, sees the missing column, fixes it. If you need Postgres, it&amp;rsquo;s an &lt;code&gt;apt-get install&lt;/code&gt; away.&lt;/p&gt;
&lt;h3 id='test-runners' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#test-runners' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Test Runners&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;Every language runtime in a Sprite ships with a built-in test runner — &lt;code&gt;node --test&lt;/code&gt;, &lt;code&gt;pytest&lt;/code&gt;, &lt;code&gt;go test&lt;/code&gt;, &lt;code&gt;cargo test&lt;/code&gt;. The agent writes code, runs tests, reads failures, fixes them. This is the most basic closed loop and it works out of the box.&lt;/p&gt;
&lt;h3 id='service-logs' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#service-logs' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Service Logs&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;a href='https://docs.fly.io/sprites/concepts/services' title=''&gt;Sprite services&lt;/a&gt; write logs to &lt;code&gt;/.sprite/logs/services/&lt;/code&gt;. The agent starts a service, tails the logs, sees the stack trace, and knows exactly what to fix. No asking you &amp;ldquo;what does the console say?&amp;rdquo;&lt;/p&gt;
&lt;h3 id='network-debugging' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#network-debugging' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Network Debugging&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;code&gt;ss&lt;/code&gt;, &lt;code&gt;dig&lt;/code&gt;, and &lt;code&gt;curl -v&lt;/code&gt; are preinstalled. When something isn&amp;rsquo;t connecting, the agent can check what&amp;rsquo;s listening, whether a domain resolves, and what&amp;rsquo;s actually happening on the wire.&lt;/p&gt;
&lt;h3 id='checkpoints' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#checkpoints' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Checkpoints&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;This is unique to Sprites. The agent can [snapshot the entire environment before trying something risky, and roll back in seconds if it goes sideways. This makes aggressive experimentation cheap.&lt;/p&gt;
&lt;h2 id='sprite-on' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#sprite-on' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Sprite On&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;You&amp;rsquo;re not middleware, you&amp;rsquo;re the idea guy. And you don&amp;rsquo;t need a smarter model. Just a smarter environment.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>I Build Everything In Sprites Now</title>
    <link rel="alternate" href="https://fly.io/blog/i-build-everything-in-sprites-now/"/>
    <id>https://fly.io/blog/i-build-everything-in-sprites-now/</id>
    <published>2026-02-03T00:00:00+00:00</published>
    <updated>2026-11-17T14:21:47+00:00</updated>
    <media:thumbnail url="https://fly.io/blog/i-build-everything-in-sprites-now/assets/buildinsprites.webp"/>
    <content type="html">&lt;p&gt;I just realised I build everything in &lt;a href='/sprites/' title=''&gt;Sprites&lt;/a&gt; now. It feels weird to say because Sprites are still really new (even for people who work at Fly.io). They&amp;rsquo;re obviously super exciting for people building agents, but when I finally &amp;ldquo;got&amp;rdquo; Sprites, it pretty much instantly felt weird to build anything anywhere else.&lt;/p&gt;

&lt;p&gt;If, like me, you have to flit between many projects every day - because breaking only one thing a day is lazy - you&amp;rsquo;ll know that local development is a joke.&lt;/p&gt;

&lt;p&gt;Let me pause what I&amp;rsquo;m working on to fix this tiny bug over on this other project! Wrong Node version. &lt;code&gt;asdf local nodejs 18.17.0&lt;/code&gt;. Oh wait, this project uses nvm. Fine. &lt;code&gt;nvm use&lt;/code&gt;. No &lt;code&gt;.nvmrc&lt;/code&gt; file. Check the README. Last updated in 1996. Check package.json engines field. Install that version. Now the wrong packages are installed. Ah, productivity.&lt;/p&gt;

&lt;p&gt;Oh of course, the wrong database is running. Start Postgres. Wait, this project uses MySQL. Stop Postgres, start MySQL. Except the other project&amp;rsquo;s MySQL data is still there. Where&amp;rsquo;s the docker-compose for this one?&lt;/p&gt;

&lt;p&gt;Ah I have screwy environment variables, let me just update the &amp;hellip; wait is it &lt;code&gt;.env&lt;/code&gt;, &lt;code&gt;.env.local&lt;/code&gt;, &lt;code&gt;.env.development&lt;/code&gt;, &lt;code&gt;.env.example&lt;/code&gt;, &lt;code&gt;env.thisone&lt;/code&gt;? This project needs &lt;code&gt;API_KEY&lt;/code&gt; set but that collides with the other project&amp;rsquo;s &lt;code&gt;API_KEY&lt;/code&gt; which points somewhere else. Swear I have a &lt;code&gt;.env.projectname&lt;/code&gt; file somewhere. Maybe. Which shell session has the right exports?&lt;/p&gt;

&lt;p&gt;Yes! It&amp;rsquo;s running! Wait what was I doing again?&lt;/p&gt;
&lt;h2 id='no-just-no' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#no-just-no' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;No. Just No.&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Now every project I&amp;rsquo;m working on is in its own Sprite. It feels like having a different computer for every project because that&amp;rsquo;s exactly what it is. Every Sprite is a persistent, fully hardware isolated linux computer with 100gb storage that only costs anything when it&amp;rsquo;s actually doing something. I can have 30 projects in 30 Sprites and I&amp;rsquo;m only ever paying for the one I&amp;rsquo;m currently working on.&lt;/p&gt;
&lt;div class="highlight-wrapper group relative bash"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-6uqdny04"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-6uqdny04"&gt;sprite create dashboard
sprite create api-backend
sprite create billion-dollar-side-project
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;Each one has its own versions, its own services, its own env vars, its own ports. Nothing collides because nothing is shared.&lt;/p&gt;
&lt;h2 id='waking-is-instant' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#waking-is-instant' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Waking Is Instant&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Impatient? Me too! Sprites wake in a few seconds.&lt;/p&gt;
&lt;div class="highlight-wrapper group relative bash"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-qplbzxag"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-qplbzxag"&gt;sprite console &lt;span class="nt"&gt;-s&lt;/span&gt; api-backend
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;If the api-backend Sprite is paused, it wakes up. Services come back up. I&amp;rsquo;m in the project, ready to work.&lt;/p&gt;

&lt;p&gt;Let me hop into &lt;code&gt;sprite console -s dashboard&lt;/code&gt; on another terminal quickly. Look at that, it just works. Prompt Claude in both Sprites, feel like Tony Stark&lt;/p&gt;

&lt;p&gt;I couldn&amp;rsquo;t care less if I forget about a Sprite I was working in. It puts itself to sleep when its inactive, but nothing about its state changes. Next time I open a console, it&amp;rsquo;s exactly where I left it.&lt;/p&gt;
&lt;h2 id='real-urls-not-localhost' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#real-urls-not-localhost' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Real URLs, Not Localhost&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Every Sprite has its own URL with automatic TLS. Run your dev server on port 8080 and it&amp;rsquo;s just&amp;hellip; there. On the internet. With HTTPS.&lt;/p&gt;

&lt;p&gt;No ngrok. No &amp;ldquo;can you see my screen?&amp;rdquo; No tunneling. OAuth callbacks just work. Webhooks just work. That API integration that refuses to talk to localhost? It just works.&lt;/p&gt;

&lt;p&gt;Need to show someone what you&amp;rsquo;re working on? Send them the URL. They see exactly what you see, in exactly the state it&amp;rsquo;s in. Checkpoint first if you want to freeze it there.&lt;/p&gt;
&lt;h2 id='checkpoints-are-the-whole-environment' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#checkpoints-are-the-whole-environment' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Checkpoints Are the Whole Environment&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Git is to checkpoints what World Series MLB is to the World Cup. Your code might be version controlled in Github, but what about &amp;hellip; everything else? A &lt;a href='https://docs.fly.io/sprites/concepts/checkpoints' title=''&gt;checkpoint&lt;/a&gt; in a Sprite is a snapshot of the entire environment. Node_modules, database state, running services, env vars, the whole shebang&lt;/p&gt;
&lt;div class="highlight-wrapper group relative bash"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-qf51pq30"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-qf51pq30"&gt;sprite checkpoint create
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;Wanna see if upgrading to Node 23 breaks everything? Do it! And laugh when it happens. Wait &amp;hellip; you did checkpoint right? &lt;/p&gt;

&lt;p&gt;Wanna tell Claude to go wild on an ambitious refactor just before you go to bed? Nighty night. &lt;/p&gt;
&lt;div class="highlight-wrapper group relative bash"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-ny4j1950"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-ny4j1950"&gt;sprite checkpoint restore abc123
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;Git branches for experiments are so 2025. &lt;/p&gt;
&lt;h2 id='specialist-claudes-gt-one-very-confused-claude' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#specialist-claudes-gt-one-very-confused-claude' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Specialist Claudes &amp;gt; One Very Confused Claude&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;I like to tell Claude to build a good knowledge base from project files and any other documentation I can find as a first step when I recruit it&amp;rsquo;s help on a project. A good claude.md and some project-specific skills &lt;a href='/blog/context-engineering-is-just-writing-things-down/' title=''&gt;completely change the game&lt;/a&gt; when it comes to how effective a coding agent can be in a given project. &lt;/p&gt;

&lt;p&gt;That&amp;rsquo;s why I want specialist Claudes. One per project. I don&amp;rsquo;t want the context polluted or diluted by work on any other project. &lt;/p&gt;

&lt;p&gt;What if I build a skill in one Sprite that I want in all my others? I tell Claude in that Sprite to use the Sprites API to propagate that skill to all my other project Sprites. &lt;/p&gt;

&lt;p&gt;OK Article done. &lt;/p&gt;
&lt;div class="highlight-wrapper group relative bash"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-xghd4o1"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-xghd4o1"&gt;sprite checkpoint create
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;</content>
  </entry>
  <entry>
    <title>How to Run Untrusted Code Safely</title>
    <link rel="alternate" href="https://fly.io/blog/running-untrusted-code/"/>
    <id>https://fly.io/blog/running-untrusted-code/</id>
    <published>2026-02-02T00:00:00+00:00</published>
    <updated>2026-11-17T14:21:47+00:00</updated>
    <media:thumbnail url="https://fly.io/blog/running-untrusted-code/assets/untrusted.jpg"/>
    <content type="html">&lt;p&gt;AI generates code faster than you can review it. Your agent wants to &lt;a href='/learn/run-llm-generated-python-safely/' title=''&gt;run a Python script&lt;/a&gt;. Your tool wants to execute user-submitted code. Your pipeline wants to test generated functions. All things you don&amp;rsquo;t want to be running locally or in production unless you&amp;rsquo;re just trying to add some drama to your life right now. &lt;/p&gt;

&lt;p&gt;If you want to walk the path of peace, you use a &lt;a href='/learn/agent-sandbox/' title=''&gt;sandbox&lt;/a&gt;. But as any kindergartener will tell you: not all sandboxes are created equal.&lt;/p&gt;
&lt;h2 id='the-container-problem' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#the-container-problem' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;The Container Problem&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;The default for the longest time has been &amp;ldquo;spin up a container.&amp;rdquo; This is safer than running dodgy code on your machine. It&amp;rsquo;s also not good enough.&lt;/p&gt;

&lt;p&gt;&lt;a href='/learn/microvm-vs-container/' title=''&gt;Containers share the host kernel&lt;/a&gt;. A kernel exploit in your untrusted code == emotional trauma. NIST SP 800-190 calls container escapes one of the most critical threats to container security. MITRE ATT&amp;amp;CK classifies it as technique T1611. The threat is real.&lt;/p&gt;

&lt;p&gt;Truly untrusted code needs hardware isolation. A dedicated kernel. A non-fictional boundary between the code and everything else.&lt;/p&gt;
&lt;h2 id='the-network-problem' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#the-network-problem' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;The Network Problem&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Filesystem isolation helps, but code that can&amp;rsquo;t touch your files can still:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Exfiltrate data to an attacker&amp;rsquo;s server
&lt;/li&gt;&lt;li&gt;Download additional payloads
&lt;/li&gt;&lt;li&gt;Call out to command-and-control infrastructure
&lt;/li&gt;&lt;li&gt;Use your compute. Which isn&amp;rsquo;t free
&lt;/li&gt;&lt;/ul&gt;

&lt;p&gt;Network egress control should not be considered optional. The code needs to run in an environment where outbound connections are explicitly allowed, not implicitly permitted.&lt;/p&gt;
&lt;h2 id='what-you-really-need-from-a-sandbox' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#what-you-really-need-from-a-sandbox' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;What You Really Need From a Sandbox&lt;/span&gt;&lt;/h2&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong class='font-semibold text-navy-950'&gt;Hardware isolation&lt;/strong&gt;
&lt;/li&gt;&lt;li&gt;&lt;strong class='font-semibold text-navy-950'&gt;Network control&lt;/strong&gt; — allowlist-based, not denylist-based
&lt;/li&gt;&lt;li&gt;&lt;strong class='font-semibold text-navy-950'&gt;External policy enforcement&lt;/strong&gt; — the code can&amp;rsquo;t modify its own permissions
&lt;/li&gt;&lt;li&gt;&lt;strong class='font-semibold text-navy-950'&gt;Rollback capability&lt;/strong&gt; — undo damage quickly when something goes wrong
&lt;/li&gt;&lt;li&gt;&lt;strong class='font-semibold text-navy-950'&gt;Multiple runtimes&lt;/strong&gt; — AI writes Python, JavaScript, Rust, Go, whatever
&lt;/li&gt;&lt;/ol&gt;

&lt;p&gt;Spin up and configure your sandboxes with these features and you should be safe. &lt;/p&gt;

&lt;p&gt;Or just use &lt;a href='/sprites/' title=''&gt;Sprites&lt;/a&gt;. &lt;/p&gt;
&lt;h2 id='setting-up-a-code-runner-in-a-sprite' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#setting-up-a-code-runner-in-a-sprite' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Setting Up a Code Runner in a Sprite&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Create a Sprite for running untrusted code. &lt;/p&gt;
&lt;div class="highlight-wrapper group relative bash"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-37r24hhv"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-37r24hhv"&gt;sprite create code-runner
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;Then, lock down its &lt;a href='https://docs.fly.io/sprites/concepts/networking' title=''&gt;network&lt;/a&gt;:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative bash"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-wki4plh6"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-wki4plh6"&gt;curl &lt;span class="nt"&gt;-X&lt;/span&gt; POST https://api.sprites.dev/v1/sprites/&lt;span class="o"&gt;{&lt;/span&gt;sprite-id&lt;span class="o"&gt;}&lt;/span&gt;/policy/network &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-H&lt;/span&gt; &lt;span class="s2"&gt;"Authorization: Bearer &lt;/span&gt;&lt;span class="nv"&gt;$SPRITES_API_TOKEN&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-d&lt;/span&gt; &lt;span class="s1"&gt;'{"rules": [{"domain": "*", "action": "deny"}]}'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;This is a deny-all policy. The code inside cannot make any outbound connections. DNS queries return &lt;code&gt;REFUSED&lt;/code&gt;. Raw IP connections are blocked.&lt;/p&gt;

&lt;p&gt;What&amp;rsquo;s super important is that this policy is set via the external API. Code running inside the Sprite cannot change it. Even if the code has root access, it cannot modify its own network policy.&lt;/p&gt;
&lt;h2 id='running-code' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#running-code' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Running Code&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Proceed to laugh in the face of danger:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative python"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-z9c3ylqa"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-z9c3ylqa"&gt;&lt;span class="c1"&gt;# runner.py - runs inside the Sprite
&lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;subprocess&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;sys&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;run_untrusted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;code&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;language&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s"&gt;"python"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="s"&gt;"""Execute code and capture output."""&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;language&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s"&gt;"python"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;subprocess&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s"&gt;"python"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"-c"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;code&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
            &lt;span class="n"&gt;capture_output&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;
        &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;elif&lt;/span&gt; &lt;span class="n"&gt;language&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s"&gt;"node"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;subprocess&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s"&gt;"node"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"-e"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;code&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
            &lt;span class="n"&gt;capture_output&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;
        &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="c1"&gt;# Add other languages as needed
&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="s"&gt;"stdout"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;stdout&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s"&gt;"stderr"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;stderr&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s"&gt;"returncode"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;returncode&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;Expose this as an HTTP endpoint on port 8080. Another Sprite acting as an orchestrator, or any external service can call this with code to execute.&lt;/p&gt;
&lt;h2 id='selective-network-access' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#selective-network-access' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Selective Network Access&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Sometimes code needs &lt;em&gt;some&lt;/em&gt; network access. An AI agent writing code that calls an API. A script that needs to fetch data from a specific endpoint.&lt;/p&gt;

&lt;p&gt;Allow only what&amp;rsquo;s needed:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative bash"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-bnibjxng"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-bnibjxng"&gt;curl &lt;span class="nt"&gt;-X&lt;/span&gt; POST https://api.sprites.dev/v1/sprites/&lt;span class="o"&gt;{&lt;/span&gt;sprite-id&lt;span class="o"&gt;}&lt;/span&gt;/policy/network &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-H&lt;/span&gt; &lt;span class="s2"&gt;"Authorization: Bearer &lt;/span&gt;&lt;span class="nv"&gt;$SPRITES_API_TOKEN&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-d&lt;/span&gt; &lt;span class="s1"&gt;'{
    "rules": [
      {"domain": "api.openai.com", "action": "allow"},
      {"domain": "api.anthropic.com", "action": "allow"},
      {"domain": "*", "action": "deny"}
    ]
  }'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;Now the code can call OpenAI and Anthropic. Nothing else. Can&amp;rsquo;t exfiltrate to gimmeallyourdata.com. Can&amp;rsquo;t download from sketchy-cdn.net.&lt;/p&gt;
&lt;h2 id='when-code-breaks-the-sandbox' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#when-code-breaks-the-sandbox' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;When Code Breaks the Sandbox&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;In the entirely unsurprising case that something does blow up; the code did something weird to the filesystem, filled the disk, corrupted something.&lt;/p&gt;

&lt;p&gt;Sprites are disposable. Just &lt;code&gt;sprite destroy&lt;/code&gt; it and spin up a fresh one. Problem solved.&lt;/p&gt;

&lt;p&gt;Or, if you&amp;rsquo;re running lots of code and want to reuse the same Sprite for efficiency, checkpoint your working state&lt;/p&gt;
&lt;div class="highlight-wrapper group relative bash"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-zuvik4zu"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-zuvik4zu"&gt;sprite checkpoint create
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;Then restore to that working state in about a second:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative bash"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-3t3grmov"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-3t3grmov"&gt;sprite checkpoint restore &amp;lt;working-checkpoint-id&amp;gt;
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;&lt;h2 id='runtimes-are-already-there' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#runtimes-are-already-there' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Runtimes Are Already There&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Sprites come with Python, Node, Ruby, Rust, Go, Java, Elixir, Bun, and Deno pre-installed. Version managers let you switch between versions. Your code runner doesn&amp;rsquo;t need to install anything. Just execute.&lt;/p&gt;

&lt;p&gt;Sprites sleep when idle and wake on demand. Your code runner costs nothing when it&amp;rsquo;s not running code.&lt;/p&gt;

&lt;p&gt;Run whatever you want. Dare it try any funny business.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Litestream Writable VFS</title>
    <link rel="alternate" href="https://fly.io/blog/litestream-writable-vfs/"/>
    <id>https://fly.io/blog/litestream-writable-vfs/</id>
    <published>2026-01-29T00:00:00+00:00</published>
    <updated>2026-10-22T11:27:19+00:00</updated>
    <media:thumbnail url="https://fly.io/blog/litestream-writable-vfs/assets/litestream-writable-vfs.webp"/>
    <content type="html">&lt;div class="lead"&gt;&lt;p&gt;&lt;strong class="font-semibold text-navy-950"&gt;I’m Ben Johnson, and I work on Litestream at Fly.io. Litestream is the missing backup/restore system for SQLite. It’s free, open-source software that should run anywhere, and&lt;/strong&gt; &lt;a href="/blog/litestream-v050-is-here/" title=""&gt;&lt;strong class="font-semibold text-navy-950"&gt;you can read more about it here&lt;/strong&gt;&lt;/a&gt;&lt;strong class="font-semibold text-navy-950"&gt;.&lt;/strong&gt;&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;Each time we write about it, we get a little bit better at golfing down a description of what Litestream is. Here goes: Litestream is a Unix-y tool for keeping a SQLite database synchronized with S3-style object storage. It&amp;rsquo;s a way of getting the speed and simplicity wins of SQLite without exposing yourself to catastrophic data loss. Your app doesn&amp;rsquo;t necessarily even need to know it&amp;rsquo;s there; you can just run it as a tool in the background.&lt;/p&gt;

&lt;p&gt;It&amp;rsquo;s been a busy couple weeks!&lt;/p&gt;

&lt;p&gt;We recently &lt;a href='/blog/design-and-implementation/' title=''&gt;unveiled Sprites&lt;/a&gt;. If you don&amp;rsquo;t know what Sprites are, you should just &lt;a href='https://sprites.dev/' title=''&gt;go check them out&lt;/a&gt;. They&amp;rsquo;re one of the coolest things we&amp;rsquo;ve ever shipped. I won&amp;rsquo;t waste any more time selling them to you. Just, Sprites are a big deal, and so it&amp;rsquo;s a big deal to me that Litestream is a load-bearing component for them.&lt;/p&gt;

&lt;p&gt;Sprites rely directly on Litestream in two big ways.&lt;/p&gt;

&lt;p&gt;First, Litestream SQLite is the core of our global Sprites orchestrator. Unlike our flagship Fly Machines product, which relies on a centralized Postgres cluster, our Elixir Sprites orchestrator runs directly off S3-compatible object storage. Every organization  enrolled in Sprites gets their own SQLite database, synchronized by Litestream.&lt;/p&gt;

&lt;p&gt;This is a fun design. It takes advantage of the &amp;ldquo;many SQLite databases&amp;rdquo; pattern, which is under-appreciated. It&amp;rsquo;s got nice scaling characteristics. Keeping that Postgres cluster happy as Fly.io grew has been a major engineering challenge.&lt;/p&gt;

&lt;p&gt;But as far as Litestream is concerned, the orchestrator is boring, and so that&amp;rsquo;s all I&amp;rsquo;ve got to say about it. The second way Sprites use Litestream is much more interesting.&lt;/p&gt;

&lt;p&gt;Litestream is built directly into the disk storage stack that runs on every Sprite.&lt;/p&gt;

&lt;p&gt;Sprites launch in under a second, and every one of them boots up with 100GB of durable storage. That&amp;rsquo;s a tricky bit of engineering. We&amp;rsquo;re able to do this because the  root of storage for Sprites is S3-compatible object storage, and we&amp;rsquo;re able to make it fast by keeping a database of in-use storage blocks that takes advantage of attached NVMe as a read-through cache. The system that does this is JuiceFS, and the database — let&amp;rsquo;s call it &amp;ldquo;the block map&amp;rdquo; — is a rewritten metadata store, based (you guessed it) on BoltDB.&lt;/p&gt;

&lt;p&gt;I kid! It&amp;rsquo;s Litestream SQLite, of course.&lt;/p&gt;
&lt;h2 id='sprite-storage-is-fussy' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#sprite-storage-is-fussy' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Sprite Storage Is Fussy&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Everything in a Sprite is designed to come up fast.&lt;/p&gt;

&lt;p&gt;If the Fly Machine underneath a Sprite bounces, we might need to reconstitute the block map from object storage. Block maps aren&amp;rsquo;t huge, but they&amp;rsquo;re not tiny; maybe low tens of megabytes worst case.&lt;/p&gt;

&lt;p&gt;The thing is, this is happening while the Sprite boots back up. To put that in perspective, that&amp;rsquo;s something that can happen in response to an incoming web request; that is, we have to finish fast enough to generate a timely response to that request. The time budget is small.&lt;/p&gt;

&lt;p&gt;To make this even faster, we are integrating Litestream VFS to improve start times.The VFS is a dynamic library you load into your app. Once you do, you can do stuff like this:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative bash"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-dpt4logh"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-dpt4logh"&gt;sqlite&amp;gt; .open file:///my.db?vfs&lt;span class="o"&gt;=&lt;/span&gt;litestream
sqlite&amp;gt; PRAGMA litestream_time &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'5 minutes ago'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; 
sqlite&amp;gt; SELECT &lt;span class="k"&gt;*&lt;/span&gt; FROM sandwich_ratings ORDER BY RANDOM&lt;span class="o"&gt;()&lt;/span&gt; LIMIT 3 &lt;span class="p"&gt;;&lt;/span&gt; 
22|Veggie Delight|New York|4
30|Meatball|Los Angeles|5
168|Chicken Shawarma Wrap|Detroit|5
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;Litestream VFS lets us run point-in-time SQLite queries hot off object storage blobs, answering queries before we&amp;rsquo;ve downloaded the database.&lt;/p&gt;

&lt;p&gt;This is good, but it&amp;rsquo;s not perfect. We had two problems:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;We could only read, not write. People write to Sprite disks. The storage stack needs to write, right away.
&lt;/li&gt;&lt;li&gt;Running a query off object storage is a godsend in a cold start where we have no other alternative besides downloading the whole database, but it&amp;rsquo;s not fast enough for steady state.
&lt;/li&gt;&lt;/ol&gt;

&lt;p&gt;These are fun problems. Here&amp;rsquo;s our first cut at solving them.&lt;/p&gt;
&lt;h2 id='writable-vfs' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#writable-vfs' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Writable VFS&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;The first thing we&amp;rsquo;ve done is made the VFS optionally read-write. This feature is pretty subtle; it&amp;rsquo;s interesting, but it&amp;rsquo;s not as general-purpose as it might look. Let me explain how it works, and then explain why it works this way.&lt;/p&gt;
&lt;div class="callout"&gt;&lt;p&gt;Keep in mind as you read this that this is about the VFS in particular. Obviously, normal SQLite databases using Litestream the normal way are writeable.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;The VFS works by keeping an index of &lt;code&gt;(file,offset, size)&lt;/code&gt; for every page of the database in object storage; the data comprising the index is stored, &lt;a href='https://github.com/superfly/ltx' title=''&gt;in LTX files&lt;/a&gt;, so that it&amp;rsquo;s efficient for us to reconstitute it quickly when the VFS starts, and lookups are heavily cached. When we queried &lt;code&gt;sandwich_ratings&lt;/code&gt; earlier, our VFS library intercepted the SQLite read method, looked up the requested page in the index, fetched it, and cached it.&lt;/p&gt;

&lt;p&gt;This works great for reads. Writes are harder.&lt;/p&gt;

&lt;p&gt;Behind the scenes in read-only mode, Litestream polls, so that we can detect new LTX files created by remote writers to the database. This supports a handy use case where we&amp;rsquo;re running tests or doing slow analytical queries of databases that need to stay fast in prod.&lt;/p&gt;

&lt;p&gt;In write mode, we don&amp;rsquo;t allow multiple writers, because multiple-writer distributed SQLite databases are the &lt;a href='https://hellraiser.fandom.com/wiki/Lament_Configuration' title=''&gt;Lament Configuration&lt;/a&gt; and we are not explorers over great vistas of pain. So the VFS in write-mode disables polling. We assume a single writer, and no additional backups to watch.&lt;/p&gt;

&lt;p&gt;Next, we buffer. Writes go to a local temporary buffer (&amp;ldquo;the write buffer&amp;rdquo;). Every second or so (or on clean shutdown), we sync the write buffer with object storage. Nothing written through the VFS is truly durable until that sync happens.&lt;/p&gt;
&lt;div class="right-sidenote"&gt;&lt;p&gt;Most storage block maps are much smaller than this, but still.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;Now, remember the use case we&amp;rsquo;re looking to support here. A Sprite is cold-starting and its storage stack needs to serve writes, milliseconds after booting, without having a full copy of the 10MB block map. This writeable VFS mode lets us do that.&lt;/p&gt;

&lt;p&gt;Critically, we support that use case only up to the same durability requirements that a Sprite already has. All storage on a Sprite shares this &amp;ldquo;eventual durability&amp;rdquo; property, so the terms of the VFS write make sense here. They probably don&amp;rsquo;t make sense for your application. But if for some reason they do, have at it! To enable writes with Litestream VFS, just set the &lt;code&gt;LITESTREAM_WRITE_ENABLED&lt;/code&gt; environment variable &lt;code&gt;&amp;quot;true&amp;quot;&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;img loading="lazy" decoding="async" alt="Diagram: an INSERT goes to a write buffer that syncs to S3 every second and flushes to the local hydration file if available" width="1428" height="734" src="/blog/litestream-writable-vfs/assets/write-path.png" /&gt;&lt;/p&gt;
&lt;h2 id='hydration' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#hydration' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Hydration&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;The Sprite storage stack uses SQLite in VFS mode. In our original VFS design, most data is kept in S3. Again: fine at cold start, not so fine in steady state.&lt;/p&gt;

&lt;p&gt;To solve this problem, we shoplifted a trick from &lt;a href='https://docs.kernel.org/admin-guide/device-mapper/dm-clone.html' title=''&gt;systems like dm-clone&lt;/a&gt;: background hydration. In hydration designs, we serve queries remotely while running a loop to pull the whole database.  When you start the VFS with the &lt;code&gt;LITESTREAM_HYDRATION_PATH&lt;/code&gt; environment variable set, we&amp;rsquo;ll hydrate to that file.&lt;/p&gt;

&lt;p&gt;Hydration takes advantage of &lt;a href='https://fly.io/blog/litestream-revamped/#point-in-time-restores-but-fast' title=''&gt;LTX compaction&lt;/a&gt;, writing only the latest versions of each page. Reads don&amp;rsquo;t block on hydration; we serve them from object storage immediately, and switch over to the hydration file when it&amp;rsquo;s ready.&lt;/p&gt;

&lt;p&gt;&lt;img loading="lazy" decoding="async" alt="Timeline: after SQLite starts and the page index loads, page reads come from S3 until hydration completes, then from the hydration file" width="1471" height="770" src="/blog/litestream-writable-vfs/assets/timeline.png" /&gt;&lt;/p&gt;

&lt;p&gt;As for the hydration file? It&amp;rsquo;s simply a full copy of your database. It&amp;rsquo;s the same thing you get if you run &lt;code&gt;litestream restore&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Because this is designed for environments like Sprites, which bounce a lot, we write the database to a temporary file. We can&amp;rsquo;t trust that the database is using the latest state every time we start up, not without doing a full restore, so we just chuck the hydration file when we exit the VFS. That behavior is baked into the VFS right now. This feature&amp;rsquo;s got what Sprites need, but again, maybe not what your app wants.&lt;/p&gt;
&lt;h2 id='putting-it-all-together' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#putting-it-all-together' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Putting It All Together&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;This is a post about two relatively big moves we&amp;rsquo;ve made with our open-source Litestream project, but the features are narrowly scoped for problems that look like the ones our storage stack needs. If you think you can get use out of them, I&amp;rsquo;m thrilled, and I hope you&amp;rsquo;ll tell me about it.&lt;/p&gt;

&lt;p&gt;For ordinary read/write workloads, you don&amp;rsquo;t need any of this mechanism. Litestream works fine without the VFS, with unmodified applications, just running as a sidecar alongside your application. The whole point of that configuration is to efficiently keep up with writes; that&amp;rsquo;s easy when you know you have the whole database to work with when writes happen.&lt;/p&gt;

&lt;p&gt;But this whole thing is, to me, a valuable case study in how Litestream can get used in a relatively complicated and demanding problem domain. Sprites are very cool, and it&amp;rsquo;s satisfying to know that every disk write that happens on a Sprite is running through Litestream.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Make Computers Make Computers</title>
    <link rel="alternate" href="https://fly.io/blog/make-computers-make-computers/"/>
    <id>https://fly.io/blog/make-computers-make-computers/</id>
    <published>2026-01-28T00:00:00+00:00</published>
    <updated>2026-11-17T14:21:47+00:00</updated>
    <media:thumbnail url="https://fly.io/blog/make-computers-make-computers/assets/mcmc.png"/>
    <content type="html">&lt;p&gt;What&amp;rsquo;s better than one &lt;a href='/sprites/' title=''&gt;Sprite&lt;/a&gt;? Sprites making other Sprites.&lt;/p&gt;

&lt;p&gt;Claude (running in a Sprite) already knows how to use the &lt;a href='https://docs.fly.io/sprites/api' title=''&gt;Sprites API&lt;/a&gt;. So you can ask it to create new Sprites, configure their network policies, hit their HTTP endpoints etc. &lt;/p&gt;

&lt;p&gt;To authorise one Sprite to control others, you&amp;rsquo;ll need to visit &lt;code&gt;sprites.dev/account/{your-org}&lt;/code&gt; and create a token. Then tell Claude to use it to authorise its Sprite. &lt;/p&gt;

&lt;p&gt;Here are a few ways that the Fly team is already using this capability on the daily:&lt;/p&gt;
&lt;h2 id='central-auth-for-everything' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#central-auth-for-everything' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Central Auth for Everything&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;A Sprite that handles OAuth flows for other Sprites.&lt;/p&gt;

&lt;p&gt;Say you have five Sprites that all need to talk to GitHub, Slack, or some other OAuth-protected service. Each one needs tokens. Managing that across five separate environments is annoying.&lt;/p&gt;

&lt;p&gt;Make an Auth Sprite do it for you instead. Workers call the Auth Sprite when they need a token. The Auth Sprite refreshes tokens, handles redirects and stores creds in one place.&lt;/p&gt;
&lt;h2 id='ci-that-doesnt-share-state' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#ci-that-doesnt-share-state' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;CI That Doesn&amp;rsquo;t Share State&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Traditional CI runs tests in containers that share&amp;hellip; something. A docker daemon. A cache directory. A network namespace.&lt;/p&gt;

&lt;p&gt;With Sprites, each test run gets its own actual computer. Tell your orchestrator Sprite to spin up a worker for each test file, lock down its network to just what it needs (pypi, github), run the test, collect results, done.&lt;/p&gt;

&lt;p&gt;Each test runs in full isolation. Nothing is shared. One flaky test can&amp;rsquo;t poison another.&lt;/p&gt;
&lt;h2 id='preview-environments-per-pr' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#preview-environments-per-pr' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Preview Environments Per PR&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Every pull request gets its own Sprite. Reviewers hit a real URL with a real running app.&lt;/p&gt;

&lt;p&gt;The orchestrator watches GitHub webhooks:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;PR opened → spin up a Sprite, deploy the branch, comment the URL
&lt;/li&gt;&lt;li&gt;PR updated → redeploy to the same Sprite
&lt;/li&gt;&lt;li&gt;PR merged/closed → destroy the Sprite
&lt;/li&gt;&lt;/ul&gt;

&lt;p&gt;No resource contention between PRs. No &amp;ldquo;sorry, all the preview slots are taken.&amp;rdquo; No shared database bleeding state between reviews.&lt;/p&gt;
&lt;h2 id='agent-delegation' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#agent-delegation' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Agent Delegation&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;You have an AI agent that&amp;rsquo;s good at planning but you don&amp;rsquo;t want it directly touching production. Or you have specialized agents; one that&amp;rsquo;s great at frontend work, another that knows your database schema cold.&lt;/p&gt;

&lt;p&gt;The manager agent runs in one Sprite. It breaks down tasks and delegates to worker agents in their own Sprites. We go deeper on this pattern in &lt;a href='/blog/agent-swarms-on-sprites/' title=''&gt;Agent Swarms on Sprites&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Each worker has network policy locked down to only what it needs. The React agent can hit npm but not your production database. The DB agent can hit Postgres but not the internet.&lt;/p&gt;
&lt;h2 id='running-untrusted-code' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#running-untrusted-code' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Running Untrusted Code&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Someone (or something) gives you code. You need to run it. Fear increases. (&lt;a href='/blog/running-untrusted-code/' title=''&gt;Here&amp;rsquo;s the longer version&lt;/a&gt;.)&lt;/p&gt;

&lt;p&gt;Tell Claude in a Sprite to create a worker with a deny-all network policy. The worker can&amp;rsquo;t talk to anything. Run the untrusted code. Grab the output. &lt;code&gt;sprite destroy&lt;/code&gt; it if it does something scary. &lt;/p&gt;

&lt;p&gt;The code can&amp;rsquo;t phone home, can&amp;rsquo;t exfiltrate data, can&amp;rsquo;t crypto-mine on your dime.&lt;/p&gt;
&lt;h2 id='team-provisioning' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#team-provisioning' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Team Provisioning&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;New developer joins. They need a Sprite with your standard tooling, your repos cloned, your services configured.&lt;/p&gt;

&lt;p&gt;One &amp;ldquo;provisioner&amp;rdquo; Sprite has a script that:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Creates a new Sprite
&lt;/li&gt;&lt;li&gt;Configures its network policy for your internal services
&lt;/li&gt;&lt;li&gt;Clones your repos
&lt;/li&gt;&lt;li&gt;Installs your standard dependencies
&lt;/li&gt;&lt;li&gt;Creates a checkpoint (so they can always restore to &amp;ldquo;fresh&amp;rdquo;)
&lt;/li&gt;&lt;li&gt;Sends them the URL
&lt;/li&gt;&lt;/ol&gt;

&lt;p&gt;Onboarding is one API call.&lt;/p&gt;
&lt;h2 id='bonus-level' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#bonus-level' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Bonus Level&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Add checkpoints to these workflows. Your controller can checkpoint a functioning worker, then restore to that checkpoint before each job. Immutable infrastructure, but fast.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Slack Bots on Sprites: OAuth is Easy Now</title>
    <link rel="alternate" href="https://fly.io/blog/slack-bots-on-sprites/"/>
    <id>https://fly.io/blog/slack-bots-on-sprites/</id>
    <published>2026-01-27T00:00:00+00:00</published>
    <updated>2026-11-17T14:21:47+00:00</updated>
    <media:thumbnail url="https://fly.io/blog/slack-bots-on-sprites/assets/slack_bots.webp"/>
    <content type="html">&lt;p&gt;Slack bots are handy, but they typically require OAuth, which means you need a server with a public URL to handle callbacks. This is annoying. But with &lt;a href='/sprites/' title=''&gt;Sprites&lt;/a&gt;, you get an instant internet-accessible environment with a persistent filesystem that makes it trivial.&lt;/p&gt;
&lt;h2 id='oauth-pain' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#oauth-pain' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;OAuth Pain&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Getting your Slack bot working involves jumping through these OAuth hurdles:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Create an endpoint to receive Slack&amp;rsquo;s authorization callback
&lt;/li&gt;&lt;li&gt;Get that endpoint online with a public URL
&lt;/li&gt;&lt;li&gt;Securely store tokens after the callback
&lt;/li&gt;&lt;li&gt;Maintain those tokens across restarts
&lt;/li&gt;&lt;/ol&gt;

&lt;p&gt;This requires deploying something to a host, setting up Ngrok and implementing a storage solution for tokens.&lt;/p&gt;
&lt;h2 id='slack-bot-sprites' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#slack-bot-sprites' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Slack Bot Sprites&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Create a Sprite for your bot:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative bash"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-n4vv2iad"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-n4vv2iad"&gt;sprite create slack-bot
sprite console &lt;span class="nt"&gt;-s&lt;/span&gt; slack-bot
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;Install your dependencies:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative bash"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-3itgawye"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-3itgawye"&gt;npm init &lt;span class="nt"&gt;-y&lt;/span&gt;
npm &lt;span class="nb"&gt;install&lt;/span&gt; @slack/bolt express
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;&lt;h3 id='build-an-oauth-handler' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#build-an-oauth-handler' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Build an OAuth Handler&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;Now we need a server that:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Handles the OAuth callback from Slack
&lt;/li&gt;&lt;li&gt;Stores access tokens to the filesystem
&lt;/li&gt;&lt;li&gt;Loads those tokens when the server restarts
&lt;/li&gt;&lt;li&gt;Implements your bot&amp;rsquo;s functionality
&lt;/li&gt;&lt;/ol&gt;

&lt;p&gt;&lt;strong class='font-semibold text-navy-950'&gt;Ask your code assistant to:&lt;/strong&gt; &lt;em&gt;&amp;ldquo;Create a server.js file that handles Slack OAuth callbacks using the Bolt framework, stores tokens to the filesystem, and implements a simple slash command. The server should run on port 8080.&amp;rdquo;&lt;/em&gt;&lt;/p&gt;
&lt;h3 id='register-your-slack-bot-as-a-sprite-service' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#register-your-slack-bot-as-a-sprite-service' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Register Your Slack Bot as a Sprite Service&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;Sprites don&amp;rsquo;t use systemd or traditional init systems. Instead, they have their own &lt;a href='https://docs.fly.io/sprites/concepts/services' title=''&gt;service manager&lt;/a&gt; that enables the Sprite&amp;rsquo;s automatic sleep/wake behavior.&lt;/p&gt;

&lt;p&gt;So you&amp;rsquo;ll want to register your bot as a service with the HTTP port flag:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative bash"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-jdszi580"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-jdszi580"&gt;sprite-env services create slack-bot &lt;span class="nt"&gt;--cmd&lt;/span&gt; node &lt;span class="nt"&gt;--args&lt;/span&gt; server.js &lt;span class="nt"&gt;--http-port&lt;/span&gt; 8080
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;This does three critical things:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;strong class='font-semibold text-navy-950'&gt;Auto-start on HTTP requests&lt;/strong&gt;: When a request hits your Sprite URL, the Sprite wakes from hibernation
&lt;/li&gt;&lt;li&gt;&lt;strong class='font-semibold text-navy-950'&gt;Auto-route requests:&lt;/strong&gt; The Sprite proxy routes incoming traffic to your specified port (8080)
&lt;/li&gt;&lt;li&gt;&lt;strong class='font-semibold text-navy-950'&gt;Persistent registration&lt;/strong&gt;: Your service registration survives Sprite restarts and hibernation
&lt;/li&gt;&lt;/ol&gt;

&lt;p&gt;When idle, your Sprite hibernates to save costs and wakes in under a second when Slack sends events or users visit the install URL.&lt;/p&gt;
&lt;h3 id='get-your-public-url' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#get-your-public-url' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Get Your Public URL&lt;/span&gt;&lt;/h3&gt;&lt;div class="highlight-wrapper group relative bash"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-6qt2duav"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-6qt2duav"&gt;sprite url update &lt;span class="nt"&gt;--auth&lt;/span&gt; public &lt;span class="nt"&gt;-s&lt;/span&gt; slack-bot
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;&lt;h3 id='configure-your-slack-app' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#configure-your-slack-app' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Configure Your Slack App&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;The rest is standard Slack stuff:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Visit &lt;a href="https://api.slack.com/apps"&gt;https://api.slack.com/apps&lt;/a&gt;
&lt;/li&gt;&lt;li&gt;Create a new app
&lt;/li&gt;&lt;li&gt;Under &amp;ldquo;OAuth &amp;amp; Permissions&amp;rdquo;, add your Sprite&amp;rsquo;s URL
&lt;/li&gt;&lt;li&gt;Copy your Client ID and Client Secret
&lt;/li&gt;&lt;li&gt;Set them as environment variables in your Sprite:
&lt;/li&gt;&lt;/ol&gt;
&lt;div class="highlight-wrapper group relative bash"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-q6ac829h"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-q6ac829h"&gt;sprite-env &lt;span class="nb"&gt;env set &lt;/span&gt;&lt;span class="nv"&gt;SLACK_CLIENT_ID&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;your_client_id
sprite-env &lt;span class="nb"&gt;env set &lt;/span&gt;&lt;span class="nv"&gt;SLACK_CLIENT_SECRET&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;your_client_secret
sprite-env &lt;span class="nb"&gt;env set &lt;/span&gt;&lt;span class="nv"&gt;SLACK_SIGNING_SECRET&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;your_signing_secret
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;&lt;h3 id='install-your-bot' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#install-your-bot' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Install Your Bot&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;Visit: &lt;a href="https://your-slack-bot.sprites.app/slack/install"&gt;https://your-slack-bot.sprites.app/slack/install&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;This triggers the OAuth flow. Slack redirects back to your Sprite, which handles the callback and stores the token in its filesystem.&lt;/p&gt;
&lt;h3 id='checkpoint-your-working-setup' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#checkpoint-your-working-setup' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Checkpoint Your Working Setup&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;Once everything is working, create a checkpoint:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative bash"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-rskbe8zl"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-rskbe8zl"&gt;sprite checkpoint create
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;A &lt;a href='https://docs.fly.io/sprites/concepts/checkpoints' title=''&gt;Sprite checkpoint&lt;/a&gt; is a snapshot of the entire environment, not just your code. So if/when you break your bot, just:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative bash"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-bme5f8vn"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-bme5f8vn"&gt;sprite restore &amp;lt;checkpoint-id&amp;gt;
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;&lt;h2 id='more-time-for-cheeky-slack-messages' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#more-time-for-cheeky-slack-messages' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;More Time for Cheeky Slack Messages&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Building a Slack bot in a Sprite means no longer having to:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Set up and maintain the server
&lt;/li&gt;&lt;li&gt;Configure networking and SSL
&lt;/li&gt;&lt;li&gt;Implement your own hibernation/wake logic (or pay for idle time)
&lt;/li&gt;&lt;li&gt;Handle security updates and monitoring
&lt;/li&gt;&lt;/ul&gt;

&lt;p&gt;Instead, write your bot, deploy it, forget about it. The Sprite wakes up when Slack calls, sleeps when it doesn&amp;rsquo;t, and you only pay for the moments it&amp;rsquo;s actually working.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Run OpenClaw (Formerly Moltbot and Clawdbot) in a Sprite</title>
    <link rel="alternate" href="https://fly.io/blog/openclaw-in-a-sprite/"/>
    <id>https://fly.io/blog/openclaw-in-a-sprite/</id>
    <published>2026-01-26T00:00:00+00:00</published>
    <updated>2026-11-17T14:21:47+00:00</updated>
    <media:thumbnail url="https://fly.io/blog/openclaw-in-a-sprite/assets/moltbot.webp"/>
    <content type="html">&lt;h2 id='running-openclaw-on-a-sprite' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#running-openclaw-on-a-sprite' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Running OpenClaw on a Sprite&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;There&amp;rsquo;s zero chance this will be the first time you&amp;rsquo;re hearing about OpenClaw, the project previously known as Clawdbot and then Moltbot. It&amp;rsquo;s an open-source AI assistant that connects to messaging platforms you probably already use: WhatsApp, Telegram, Slack, Discord, Signal, etc. It comes with a fast-growing list of connectors and Skills and self-iterating capabilities, right out of the box.&lt;/p&gt;

&lt;p&gt;If you really want to see what a project like OpenClaw can do, you need to run it with generous permissions and give it access to its environment. That is inherently scary, which is why so many people have been snapping up Mac Minis to run their OpenClaw in an isolated environment.&lt;/p&gt;

&lt;p&gt;You don&amp;rsquo;t need a Mac Mini. Instead, let&amp;rsquo;s learn how to run OpenClaw in a Sprite. (Prefer one click? &lt;a href='/sprites/openclaw/' title=''&gt;Deploy OpenClaw on a Sprite&lt;/a&gt; and skip the rest of this post.) A stateful sandbox with automatic sleep/wake behaviour that keeps running costs low. &lt;a href='/sprites/' title=''&gt;Sprites&lt;/a&gt; are fully hardware isolated Linux computers, so you can give OpenClaw total agency and not even have to think about building your own guardrails.&lt;/p&gt;
&lt;h2 id='create-a-sprite' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#create-a-sprite' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Create a Sprite&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Install the Sprites CLI and create a Sprite:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative bash"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-gwxgoiha"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-gwxgoiha"&gt;curl &lt;span class="nt"&gt;-fsSL&lt;/span&gt; https://sprites.dev/install.sh | bash
sprite create openclaw
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;Connect to the Sprite console:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative bash"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-1mwcc8ai"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-1mwcc8ai"&gt;sprite console &lt;span class="nt"&gt;-s&lt;/span&gt; openclaw
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;&lt;h2 id='install-and-configure-openclaw' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#install-and-configure-openclaw' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Install and Configure OpenClaw&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Run the OpenClaw install script:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative bash"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-ejur3ta2"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-ejur3ta2"&gt;curl &lt;span class="nt"&gt;-fsSL&lt;/span&gt; https://openclaw.ai/install.sh | bash
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;The wizard will configure your credentials, hooks, and messaging platforms. You&amp;rsquo;ll see two expected errors at the end:&lt;/p&gt;

&lt;p&gt;&lt;strong class='font-semibold text-navy-950'&gt;Systemd unavailable&lt;/strong&gt; — Sprites don&amp;rsquo;t use systemd&lt;/p&gt;

&lt;p&gt;&lt;strong class='font-semibold text-navy-950'&gt;Health check failed&lt;/strong&gt; — The Gateway isn&amp;rsquo;t running yet&lt;/p&gt;

&lt;p&gt;This is normal. The onboarding creates your config. We&amp;rsquo;ll start the Gateway next.&lt;/p&gt;

&lt;p&gt;&lt;strong class='font-semibold text-navy-950'&gt;Important:&lt;/strong&gt; Copy the token shown in OpenClaw&amp;rsquo;s output. It looks like:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;9054a0cfca3f5be5325f988fb57ddf1f38e530d2883bbed7&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;Add openclaw to your PATH so it&amp;rsquo;s available in future sessions:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative bash"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-yff0do9o"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-yff0do9o"&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s1"&gt;'export PATH="$PATH:/.sprite/languages/node/nvm/versions/node/v22.20.0/bin"'&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&amp;gt;&lt;/span&gt; ~/.zshrc
&lt;span class="nb"&gt;source&lt;/span&gt; ~/.zshrc
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;&lt;h2 id='register-openclaw-as-a-sprite-service' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#register-openclaw-as-a-sprite-service' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Register OpenClaw as a Sprite Service&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Sprites have their own &lt;a href='https://docs.fly.io/sprites/concepts/services' title=''&gt;service manager&lt;/a&gt; that auto-starts services when HTTP requests arrive. Register the Gateway:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative bash"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-xrqagcva"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-xrqagcva"&gt; sprite-env services create openclaw-gateway &lt;span class="se"&gt;\&lt;/span&gt;
    &lt;span class="nt"&gt;--cmd&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;which openclaw&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
    &lt;span class="nt"&gt;--args&lt;/span&gt; &lt;span class="s2"&gt;"gateway"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
    &lt;span class="nt"&gt;--http-port&lt;/span&gt; 18789
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;This registers the Gateway and routes your Sprite URL to port 18789. When requests hit your Sprite URL, the Sprite wakes and the Gateway starts automatically.&lt;/p&gt;
&lt;h2 id='make-the-url-public-optional' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#make-the-url-public-optional' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Make the URL Public (Optional)&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;By default, Sprite URLs require authentication. To make yours public and get the url:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative bash"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-pf06x8xt"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-pf06x8xt"&gt;sprite url update &lt;span class="nt"&gt;--auth&lt;/span&gt; public &lt;span class="nt"&gt;-s&lt;/span&gt; openclaw
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;&lt;h2 id='access-the-dashboard' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#access-the-dashboard' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Access the Dashboard&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;For initial setup, access your Sprite URL + the token from onboarding:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;https://yoursprite.sprites.app/?token=YOUR_TOKEN_HERE&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;You&amp;rsquo;ll see a &amp;ldquo;pairing required&amp;rdquo; message. This is OpenClaw&amp;rsquo;s security. It requires you to approve new devices before they can connect.&lt;/p&gt;
&lt;h2 id='approve-your-device' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#approve-your-device' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Approve Your Device&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Back in the Sprite console, list pending pairing requests:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative bash"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-4bulbd82"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-4bulbd82"&gt;openclaw devices list
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;Approve the pending request:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative bash"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-zpva9un8"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-zpva9un8"&gt;openclaw devices approve REQUEST_ID
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;Refresh your browser. You&amp;rsquo;re in.&lt;/p&gt;
&lt;h2 id='checkpoint-your-setup' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#checkpoint-your-setup' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Checkpoint Your Setup&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Checkpoint your working configuration:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative bash"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-g5538y37"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-g5538y37"&gt;sprite checkpoint create
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;If you ever need to return to this state:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative bash"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-pspi3171"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-pspi3171"&gt;sprite restore &amp;lt;&lt;span class="nb"&gt;id&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;&lt;h2 id='what-youve-built' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#what-youve-built' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;What You&amp;rsquo;ve Built&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Your OpenClaw Gateway now lives on a Sprite that:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Sleeps when idle
&lt;/li&gt;&lt;li&gt;Wakes automatically when you hit the URL
&lt;/li&gt;&lt;li&gt;Restarts the Gateway service on wake
&lt;/li&gt;&lt;li&gt;Persists your config and pairing across restarts
&lt;/li&gt;&lt;/ul&gt;

&lt;p&gt;Once paired, you can access the dashboard and start chatting with OpenClaw. It&amp;rsquo;s not bad at understanding its own capabilities, so just ask for the integrations and channels you want to set up.&lt;/p&gt;
&lt;h2 id='bonus-level' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#bonus-level' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Bonus Level:&lt;/span&gt;&lt;/h2&gt;&lt;hr class='-mt-4 mb-10'&gt;
&lt;p&gt;Tell OpenClaw to build a Sprites Skill for itself, so it knows how to manage its own environment:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Create a Sprite skill by reading the documentation at /.sprite/llm.txt and /.sprite/docs/agent-context.md. The skill should teach you how to manage Sprite services (since there&amp;rsquo;s no systemd), create checkpoints, restart the openclaw-gateway, and check network policy. Save it to ~/.openclaw/skills/sprite/SKILL.md.&lt;/p&gt;
&lt;/blockquote&gt;
</content>
  </entry>
  <entry>
    <title>Run Claude Code From Your Phone With Sprites</title>
    <link rel="alternate" href="https://fly.io/blog/how-sprites-accidentally-solved-mobile-development/"/>
    <id>https://fly.io/blog/how-sprites-accidentally-solved-mobile-development/</id>
    <published>2026-01-21T00:00:00+00:00</published>
    <updated>2026-11-17T14:21:47+00:00</updated>
    <media:thumbnail url="https://fly.io/blog/how-sprites-accidentally-solved-mobile-development/assets/sprite-mobile.webp"/>
    <content type="html">&lt;p&gt;A few days ago, &lt;a href='https://news.ycombinator.com/item?id=46517458' title=''&gt;this post &lt;/a&gt;hit HN&amp;rsquo;s front page and generated over 400 comments. The author is talking about the setup he uses to code from his phone using Tailscale, an always-on Mac at home, SSH via Termius, and Claude Code in the terminal.&lt;/p&gt;

&lt;p&gt;If you&amp;rsquo;re asking why anyone would want to code from their phone, you can safely stop reading now and return to a peaceful existence, where ideas only come to you when you&amp;rsquo;re sitting behind a computer. For those familiar with the need to try an idea right now, wherever you are, whatever you&amp;rsquo;re doing, you&amp;rsquo;re in the right place.&lt;/p&gt;
&lt;h2 id='what-was-and-need-no-longer-be' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#what-was-and-need-no-longer-be' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;What Was and Need No Longer Be&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;The HN post&amp;rsquo;s comments section showed off some genuinely impressive mobile coding setups that range from the creative to the absurdly over-engineered. Tmux sessions, VPS boxes, all the SSH clients, and a million ways to access these env&amp;rsquo;s from your phone.&lt;/p&gt;

&lt;p&gt;For a lot of people, designing setups like this is fun. But if managing always-on servers and VPN configurations is less exciting to you than building an app in real time while brainstorming it on a hiking trail, there&amp;rsquo;s a better way.  We call &amp;lsquo;em &lt;a href='/sprites/' title=''&gt;Sprites&lt;/a&gt;.&lt;/p&gt;
&lt;h2 id='whats-a-sprite-again' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#whats-a-sprite-again' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;What&amp;rsquo;s a Sprite Again?&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;The correct answer is: Whatever you want it to be. But that isn&amp;rsquo;t super helpful in this context. So let&amp;rsquo;s start with the idea of a Sprite as the server in your mobile coding setup. Instead of running a machine 24/7 or starting and stopping a VM, use a Sprite. It&amp;rsquo;s a full Linux computer that spins up in under a second and runs only when you need it.&lt;/p&gt;

&lt;p&gt;Your code, installed packages, and project state persist between sessions, even when the Sprite is suspended. You (or Claude) can checkpoint your Sprite at any time, which creates a restorable snapshot of the entire environment, not just your code. And since it&amp;rsquo;s a hardware-isolated computer rather than a shared container, you can run literally any code and experiment aggressively, while genuinely just not caring about breaking anything important.&lt;/p&gt;

&lt;p&gt;If you want to unleash your code assistant, give it full file system access and tell it to build that genius app idea that occurs to you at 3am without getting out of bed, you wanna do that in a Sprite.&lt;/p&gt;

&lt;p&gt;But wait, how are you doing this from your phone?&lt;/p&gt;
&lt;h2 id='sleep-is-for-the-weak' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#sleep-is-for-the-weak' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Sleep Is For the Weak&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;We are blessed with multiple really smart guys called Chris at Fly.io. And one of them, a Christopher actually, has built a - dare I say - elegant way for you to issue orders to Claude in a Sprite from your phone. The aptly named &lt;a href='https://github.com/clouvet/sprite-mobile' title=''&gt;sprite-mobile&lt;/a&gt;, a chat UI for accessing Claude Code running in YOLO mode on a sprite.&lt;/p&gt;
&lt;h2 id='how-to-set-up-claude-code-on-your-phone' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#how-to-set-up-claude-code-on-your-phone' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;How to Set Up Claude Code on Your Phone&lt;/span&gt;&lt;/h2&gt;
&lt;ol&gt;
&lt;li&gt;Create a Sprite
&lt;/li&gt;&lt;/ol&gt;

&lt;p&gt;&lt;code&gt;sprite create cheese_sandwich&lt;/code&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Download and run the Sprite-mobile install script
&lt;/li&gt;&lt;/ol&gt;

&lt;p&gt;&lt;code&gt;curl -fsSL https://raw.githubusercontent.com/clouvet/sprite-mobile/refs/heads/main/scripts/sprite-setup.sh -o sprite-setup.sh &amp;amp;&amp;amp; chmod +x sprite-setup.sh &amp;amp;&amp;amp; ./sprite-setup.sh&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;I shit you not. That is it.&lt;/p&gt;

&lt;p&gt;The script will speedrun you through everything needed to get up and running:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Configure your sprite and git user
&lt;/li&gt;&lt;li&gt;Authenticate Claude CLI, GitHub CLI, Fly.io CLI, Sprites CLI
&lt;/li&gt;&lt;li&gt;Install and configure Tailscale
&lt;/li&gt;&lt;li&gt;Clone and run sprite-mobile and configure a Tailscale-based Sprite network (if you want automatic discovery of other sprites in your org)
&lt;/li&gt;&lt;li&gt;Set up Tailscale Serve (HTTPS for PWA support)
&lt;/li&gt;&lt;li&gt;Start the Tailnet Gate (public entry point to wake up the sprite if it&amp;rsquo;s suspended)
&lt;/li&gt;&lt;/ol&gt;

&lt;p&gt;To access this Claude-powered dev sandbox you just spun up in the same time it takes to make a midnight cheese sandwich, you&amp;rsquo;ll want to get its public url:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;sprite url update --auth public -s cheese_sandwich&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;Drop that in your Tailscale-connected phone&amp;rsquo;s browser and you&amp;rsquo;re cooking.&lt;/p&gt;

&lt;p&gt;Your Sprites automatically go to sleep after 30s of inactivity and wake up faster than a cat when you open a can of tuna a mile away (sub 1s). And you only pay for the resources your Sprites are actively using. Got 30 projects in 30 Sprites waiting for you to decide when you&amp;rsquo;re interested in them again? Cool. No charge.&lt;/p&gt;

&lt;p&gt;Why are you still here? Check out the full details in the &lt;a href='https://github.com/clouvet/sprite-mobile' title=''&gt;sprite-mobile repo&lt;/a&gt; and read more about &lt;a href='/sprites/' title=''&gt;Sprites&lt;/a&gt;.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>The Design &amp; Implementation of Sprites</title>
    <link rel="alternate" href="https://fly.io/blog/design-and-implementation/"/>
    <id>https://fly.io/blog/design-and-implementation/</id>
    <published>2026-01-14T00:00:00+00:00</published>
    <updated>2026-09-30T00:06:54+00:00</updated>
    <media:thumbnail url="https://fly.io/blog/design-and-implementation/assets/starry-containers.webp"/>
    <content type="html">&lt;div class="lead"&gt;&lt;p&gt;We’re Fly.io, and this is the place in the post where we’d normally tell you that our job is to &lt;a href="https://fly.io/blog/docker-without-docker/" title=""&gt;take your containers and run them on our own hardware&lt;/a&gt; all around the world. But last week, we &lt;a href="https://sprites.dev/" title=""&gt;launched Sprites&lt;/a&gt;, and they don’t work that way at all. Sprites are something new: Docker without Docker without Docker. This post is about how they work.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;Replacement-level homeowners buy boxes of pens and stick them in &amp;ldquo;the pen drawer&amp;rdquo;. What the elites know: you have to think adversarially about pens. &amp;ldquo;The purpose of a system is what it does&amp;rdquo;; a household&amp;rsquo;s is to uniformly distribute pens.  Months from now, the drawer will be empty, no matter how many pens you stockpile. Instead, scatter pens every place you could possibly think to look for one — drawers, ledges, desks. Any time anybody needs a pen, several are at hand, in exactly the first place they look.&lt;/p&gt;

&lt;p&gt;This is the best way I&amp;rsquo;ve found to articulate the idea of &lt;a href='https://sprites.dev/' title=''&gt;Sprites&lt;/a&gt;, the platform we just launched at Fly.io. Sprites are ball-point disposable computers. Whatever mark you mean to make, we&amp;rsquo;ve rigged it so you&amp;rsquo;re never more than a second or two away from having a Sprite to do it with.&lt;/p&gt;

&lt;p&gt;Sprites are Linux virtual machines. You get root. They &lt;code&gt;create&lt;/code&gt; in just a second or two: so fast, the experience of creating and shelling into one is identical to SSH&amp;#39;ing into a machine that already exists. Sprites all have a 100GB durable root filesystem. They put themselves to sleep automatically when inactive, and cost practically nothing while asleep.&lt;/p&gt;

&lt;p&gt;As a result, I barely feel the need to name my Sprites. Sometimes I&amp;rsquo;ll just type &lt;code&gt;sprite create dkjsdjk&lt;/code&gt; and start some task. People at Fly.io who use Sprites have dozens hanging around.&lt;/p&gt;

&lt;p&gt;There aren&amp;rsquo;t yet many things in cloud computing that have the exact shape Sprites do:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Instant creation
&lt;/li&gt;&lt;li&gt;No time limits
&lt;/li&gt;&lt;li&gt;Persistent disk
&lt;/li&gt;&lt;li&gt;Auto-sleep to a cheap inactive state
&lt;/li&gt;&lt;/ul&gt;

&lt;p&gt;This is a post about how we managed to get this working. We created a new orchestration stack that undoes some of the core decisions we made for &lt;a href='https://fly.io/machines/' title=''&gt;Fly Machines&lt;/a&gt;, our flagship product. Turns out, these new decisions make Sprites drastically easier for us to scale and manage. We&amp;rsquo;re pretty psyched.&lt;/p&gt;

&lt;p&gt;Lucky for me, there happen to be three &lt;code&gt;big decisions&lt;/code&gt; we made that get you 90% of the way from Fly Machines to Sprites, which makes this an easy post to write. So, without further ado:&lt;/p&gt;
&lt;h2 id='decision-1-no-more-container-images' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#decision-1-no-more-container-images' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Decision #1: No More Container Images&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;This is the easiest decision to explain.&lt;/p&gt;

&lt;p&gt;Fly Machines are approximately &lt;a href='https://fly.io/blog/docker-without-docker/' title=''&gt;OCI containers repackaged as KVM micro-VMs&lt;/a&gt;. They have the ergonomics of Docker but the isolation and security of an EC2 instance. We love them very much and they&amp;rsquo;re clearly the wrong basis for a ball-point disposable cloud computer.&lt;/p&gt;

&lt;p&gt;The &amp;ldquo;one weird trick&amp;rdquo; of Fly Machines is that they &lt;code&gt;start&lt;/code&gt; and &lt;code&gt;stop&lt;/code&gt; instantly, fast enough that they can wake in time to handle an incoming HTTP request. But they can only do that if you&amp;rsquo;ve already &lt;code&gt;created&lt;/code&gt; them. You have to preallocate. &lt;code&gt;Creating&lt;/code&gt; a Fly Machine can take over a minute. What you&amp;rsquo;re supposed to do is to create a whole bunch of them and &lt;code&gt;stop&lt;/code&gt; them so they&amp;rsquo;re ready when you need them. But for Sprites, we need &lt;code&gt;create&lt;/code&gt; to be so fast it feels like they&amp;rsquo;re already there waiting for you.&lt;/p&gt;
&lt;div class="right-sidenote"&gt;&lt;p&gt;We only murdered user containers because we wanted them dead.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;Most of what&amp;rsquo;s slow about &lt;code&gt;creating&lt;/code&gt; a Fly Machine is containers. I say this with affection: your containers are crazier than a soup sandwich. Huge and fussy, they take forever to &lt;a href='https://fly.io/blog/docker-without-docker/' title=''&gt;pull and unpack&lt;/a&gt;. The regional locality sucks;  &lt;code&gt;create&lt;/code&gt; a Fly Machine in São Paulo on &lt;code&gt;gru-3838&lt;/code&gt;, and a &lt;code&gt;create&lt;/code&gt; on  &lt;code&gt;gru-d795&lt;/code&gt; is no faster. A &lt;a href='https://community.fly.io/t/global-registry-now-in-production/13723' title=''&gt;truly heartbreaking&lt;/a&gt; amount of &lt;a href='https://community.fly.io/t/faster-more-reliable-remote-image-builds-deploys/25841' title=''&gt;engineering work&lt;/a&gt; has gone into just allowing our OCI registry to &lt;a href='https://www.youtube.com/watch?v=0jD-Rt4_CR8' title=''&gt;keep up&lt;/a&gt; with this system. &lt;/p&gt;

&lt;p&gt;It&amp;rsquo;s a tough job, is all I&amp;rsquo;m saying. Sprites get rid of the user-facing container. Literally: problem solved. Sprites get to do this on easy mode.&lt;/p&gt;

&lt;p&gt;Now, today, under the hood, Sprites are still Fly Machines. But they all run from a standard container. Every physical worker knows exactly what container the next Sprite is going to start with, so it&amp;rsquo;s easy for us to keep pools of &amp;ldquo;empty&amp;rdquo; Sprites standing by. The result: a Sprite &lt;code&gt;create&lt;/code&gt; doesn&amp;rsquo;t have any heavy lifting to do; it&amp;rsquo;s basically just doing the stuff we do when we &lt;code&gt;start&lt;/code&gt; a Fly Machine.&lt;/p&gt;
&lt;figure class="post-cta"&gt;
  &lt;figcaption&gt;
    &lt;h2&gt;This all works right now.&lt;/h2&gt;
    &lt;p&gt;You can create a couple dozen Sprites right now if you want. It&amp;rsquo;ll only take a second.&lt;/p&gt;
      &lt;a class="btn btn-lg" href="https://sprites.dev/"&gt;
        Make a Sprite.  &lt;span class='opacity-50'&gt;→&lt;/span&gt;
      &lt;/a&gt;
  &lt;/figcaption&gt;
  &lt;div class="image-container"&gt;
    &lt;img loading="lazy" decoding="async" src="/static/images/cta-kitty.webp" srcset="/static/images/cta-kitty@2x.webp 2x" alt=""&gt;
  &lt;/div&gt;
&lt;/figure&gt;

&lt;h2 id='decision-2-object-storage-for-disks' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#decision-2-object-storage-for-disks' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Decision #2: Object Storage For Disks&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Every Sprite comes with 100GB of durable storage. We&amp;rsquo;re able to do that because the root of storage is S3-compatible object storage.&lt;/p&gt;

&lt;p&gt;You can arrange for 100GB of storage for a Fly Machine. Or 200, or 500. The catch:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;You have to ask (with &lt;code&gt;flyctl&lt;/code&gt;); we can&amp;rsquo;t reasonably default it in.
&lt;/li&gt;&lt;li&gt;That storage is NVMe attached to the physical server your Fly Machine is on. 
&lt;/li&gt;&lt;/ul&gt;
&lt;div class="right-sidenote"&gt;&lt;p&gt;[†] we print a &lt;span style="color: red"&gt;big red warning&lt;/span&gt; about this if you try to make a single-node cluster&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;We designed the storage stack for Fly Machines for Postgres clusters. A multi-replica Postgres cluster gets good mileage out of Fly Volumes. Attached storage is fast, but can &lt;span style="color: red"&gt;lose data†&lt;/span&gt; — if a physical blows up, there&amp;rsquo;s no magic what rescues its stored bits. You&amp;rsquo;re stuck with our last snapshot backup. That&amp;rsquo;s fine for a replicated Postgres! It&amp;rsquo;s part of what Postgres replication is for. But for anything without explicit replication, it&amp;rsquo;s a very sharp edge.&lt;/p&gt;

&lt;p&gt;Worse, from our perspective, is that attached storage anchors workloads to specific physicals. We have lots of reasons to want to move Fly Machines around. Before we  did Fly Volumes, that was as simple as pushing a &amp;ldquo;drain&amp;rdquo; button on a server. Imagine losing a capability like that. It took 3 years to &lt;a href='https://fly.io/blog/machine-migrations/' title=''&gt;get workload migration right&lt;/a&gt; with attached storage, and it&amp;rsquo;s still not &amp;ldquo;easy&amp;rdquo;.&lt;/p&gt;
&lt;div class="right-sidenote"&gt;&lt;p&gt;Object stores are the Internet’s Hoover Dams, the closest things we have to infrastructure megaprojects.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;Sprites jettison this model. We still exploit NVMe, but not as the root of storage. Instead, it&amp;rsquo;s a read-through cache for a blob on object storage. S3-compatible object stores are the most trustworthy storage technology we have. I can feel my blood pressure dropping just typing the words &amp;ldquo;Sprites are backed by object storage.&amp;rdquo;&lt;/p&gt;

&lt;p&gt;The implications of this for orchestration are profound. In a real sense, the durable state of a Sprite is simply a URL. Wherever he lays his hat is his home! They migrate (or recover from failed physicals) trivially. It&amp;rsquo;s early days for our internal tooling, but we have so many new degrees of freedom to work with.&lt;/p&gt;

&lt;p&gt;I could easily do another 1500-2000 words here on the Cronenberg film Kurt came up with for the actual storage stack, but because it&amp;rsquo;s in flux, let&amp;rsquo;s keep it simple.&lt;/p&gt;

&lt;p&gt;The Sprite storage stack is organized around the JuiceFS model (in fact, we currently use a very hacked-up JuiceFS, with a rewritten SQLite metadata backend). It works by splitting storage into data (&amp;ldquo;chunks&amp;rdquo;) and metadata (a map of where the &amp;ldquo;chunks&amp;rdquo; are). Data chunks live on object stores; metadata lives in fast local storage.  In our case, that metadata store is &lt;a href='https://litestream.io/' title=''&gt;kept durable with Litestream&lt;/a&gt;. Nothing depends on local storage.&lt;/p&gt;
&lt;div class="right-sidenote"&gt;&lt;p&gt;(our pre-installed Claude Code will checkpoint aggressively for you without asking)&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;This also buys Sprites fast &lt;code&gt;checkpoint&lt;/code&gt; and &lt;code&gt;restore&lt;/code&gt;. Checkpoints are so fast we want you to use them as a basic feature of the system and not as an escape hatch when things go wrong; like a git restore, not a system restore. That works because both &lt;code&gt;checkpoint&lt;/code&gt; and &lt;code&gt;restore&lt;/code&gt; merely shuffle metadata around.&lt;/p&gt;

&lt;p&gt;Our stack sports &lt;a href='https://en.wikipedia.org/wiki/Dm-cache' title=''&gt;a dm-cache-like&lt;/a&gt; feature that takes advantage of attached storage. A Sprite has a sparse 100GB NVMe volume attached to it, which the stack uses to cache chunks to eliminate read amplification. Importantly (I can feel my resting heart rate lowering) nothing in that NVMe volume should matter; stored chunks are immutable and their true state lives on the object store.&lt;/p&gt;
&lt;div class="callout"&gt;&lt;p&gt;Our preference for object storage goes further than the Sprite storage stack. The global orchestrator for Sprites is an Elixir/Phoenix app that uses object storage as the primary source of metadata for accounts. We then give each account an independent SQLite database, again made durable on object storage with Litestream.&lt;/p&gt;
&lt;/div&gt;&lt;h2 id='decision-3-inside-out-orchestration' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#decision-3-inside-out-orchestration' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Decision #3: Inside-Out Orchestration&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;In the cloud hosting industry, user applications are managed by two separate, yet equally important components: the host, which orchestrates workloads, and the guest, which runs them. Sprites flip that on its head: the most important orchestration and management work happens inside the VM.&lt;/p&gt;

&lt;p&gt;Here&amp;rsquo;s the trick: user code running on a Sprite isn&amp;rsquo;t running in the root namespace. We&amp;rsquo;ve slid a container between you and the kernel. You see an inner environment, managed by a fleet of services running in the root namespace of the VM.&lt;/p&gt;
&lt;div class="callout"&gt;&lt;p&gt;I wish we’d done Fly Machines this way to begin with. I’m not sure there’s a downside. The inner container allows us to bounce a Sprite without rebooting the whole VM, even on checkpoint restores. I think Fly Machines users could get some mileage out of that feature, too.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;With Sprites, we&amp;rsquo;re pushing this idea as far as we can. The root environment hosts the majority of our orchestration code. When you talk to the global API, chances are you&amp;rsquo;re talking directly to your own VM. Furthermore:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Our storage stack, which handles checkpoint/restore and persistence to object storage, lives there;
&lt;/li&gt;&lt;li&gt;so does the service manager we expose to Sprites, which registers user code that needs to restart when a Sprite bounces;
&lt;/li&gt;&lt;li&gt;same with logs;
&lt;/li&gt;&lt;li&gt;if you bind a socket to &lt;code&gt;*:8080&lt;/code&gt;, we&amp;rsquo;ll make it available outside the Sprite — yep, that&amp;rsquo;s in the root namespace too.
&lt;/li&gt;&lt;/ul&gt;

&lt;p&gt;Platform developers at Fly.io know how much easier it can be to hack on &lt;code&gt;init&lt;/code&gt; (inside the container) than things &lt;a href='https://fly.io/blog/carving-the-scheduler-out-of-our-orchestrator/' title=''&gt;like &lt;code&gt;flyd&lt;/code&gt;&lt;/a&gt;, the Fly Machines orchestrator that runs on  the host. Changes to Sprites don&amp;rsquo;t restart host components or muck with global state. The blast radius is just new VMs that pick up the change. We sleep on how much platform work doesn&amp;rsquo;t get done not because the code is hard to write, but because it&amp;rsquo;s so time-consuming to ensure benign-looking changes don&amp;rsquo;t throw the whole fleet into metastable failure. We had that in mind when we did Sprites.&lt;/p&gt;
&lt;h2 id='we-keep-the-parts-that-worked' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#we-keep-the-parts-that-worked' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;We Keep The Parts That Worked&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Sprites running on Fly.io take advantage of the infrastructure we already have. For instance: Sprites might be the fastest thing there currently exists to get Claude or Gemini to build a full-stack application on the Internet.&lt;/p&gt;

&lt;p&gt;That&amp;rsquo;s because Sprites plug directly into &lt;a href='https://fly.io/blog/corrosion/' title=''&gt;Corrosion, our gossip-based service discovery system&lt;/a&gt;. When you ask the Sprite API to make a public URL for your Sprite, we generate a Corrosion update that propagates across our fleet instantly. Your application is then served, with an HTTPS URL, from our proxy edges.&lt;/p&gt;

&lt;p&gt;Sprites live alongside Fly Machines in our architecture. They include some changes  that are pure wins, but they&amp;rsquo;re mostly tradeoffs:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;We&amp;rsquo;ve always wanted to run Fly Machine disks off object storage (&lt;a href='https://community.fly.io/t/bottomless-s3-backed-volumes/15648' title=''&gt;we have an obscure LSVD feature that does this&lt;/a&gt;), but the performance isn&amp;rsquo;t adequate for a hot Postgres node in production.
&lt;/li&gt;&lt;li&gt;For that matter, professional production apps ship out of CI/CD systems as OCI containers; that&amp;rsquo;s a big part of what makes orchestrating Fly Machines so hard.
&lt;/li&gt;&lt;li&gt;Most (though not all) Sprite usage is interactive, and Sprite users benefit from their VMs aggressively sleeping themselves to keep costs low; e-commerce apps measure responsiveness in milliseconds and want their workloads kept warm.
&lt;/li&gt;&lt;/ul&gt;

&lt;p&gt;Sprites are optimized for a different kind of computing than Fly Machines, and &lt;a href='https://fly.io/blog/code-and-let-live/' title=''&gt;while Kurt believes that the future belongs to malleable, personalized apps&lt;/a&gt;, I&amp;rsquo;m not so sure. To me, it makes sense to prototype and acceptance-test an application on Sprites. Then, when you&amp;rsquo;re happy with it, containerize it and ship it as a Fly Machine to scale it out. An automated workflow for that will happen.&lt;/p&gt;

&lt;p&gt;Finally, Sprites are a contract with user code: an API and a set of expectations about how the execution environment works. Today, they run on top of Fly Machines. But they don&amp;rsquo;t have to. Jerome&amp;rsquo;s working on an open-source local Sprite runtime. We&amp;rsquo;ll find other places to run them, too.&lt;/p&gt;
&lt;h2 id='you-wont-get-it-until-you-use-them' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#you-wont-get-it-until-you-use-them' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;You Won&amp;rsquo;t Get It Until You Use Them&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;I can&amp;rsquo;t not sound like a shill. Sprites are the one thing we&amp;rsquo;ve shipped that I personally experience as addictive. I haven&amp;rsquo;t fully put my finger on why it feels so much easier to kick off projects now that I can snap my finger and get a whole new computer. The whole point is that there&amp;rsquo;s no reason to parcel them out, or decide which code should run where. You just make a new one.&lt;/p&gt;

&lt;p&gt;So to make this fully click, I think you should &lt;a href='https://sprites.dev/' title=''&gt;just install the &lt;code&gt;sprite&lt;/code&gt; command&lt;/a&gt;, make a Sprite, and then run an agent in it. We&amp;rsquo;ve preinstalled Claude, Gemini, and Codex, and taught them how to do things like checkpoint/restore, registering services, and getting logs. Claude will run in &lt;code&gt;--dangerously-skip-permissions&lt;/code&gt; mode (because why wouldn&amp;rsquo;t it). Have it build something; I built a &amp;ldquo;Chicago&amp;rsquo;s best sandwich&amp;rdquo; bracket app for a Slack channel.&lt;/p&gt;

&lt;p&gt;Sprites bill only for what you actually use (in particular: only for storage blocks you actually write, not the full 100GB capacity). It&amp;rsquo;s reasonable to create a bunch. They&amp;rsquo;re ball-point disposable computers. After you get a feel for them, it&amp;rsquo;ll start to feel weird not having them handy.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Code And Let Live</title>
    <link rel="alternate" href="https://fly.io/blog/code-and-let-live/"/>
    <id>https://fly.io/blog/code-and-let-live/</id>
    <published>2026-01-09T00:00:00+00:00</published>
    <updated>2026-10-22T11:27:19+00:00</updated>
    <media:thumbnail url="https://fly.io/blog/code-and-let-live/assets/sprites.webp"/>
    <content type="html">&lt;div class="lead"&gt;&lt;p&gt;The state of the art in agent isolation is a read-only sandbox. At Fly.io, we’ve been selling that story for years, and we’re calling it: ephemeral sandboxes are obsolete. Stop killing your sandboxes every time you use them.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;My argument won&amp;rsquo;t make sense without showing you something new we&amp;rsquo;ve built. We&amp;rsquo;re all adults here, this is a company, we talk about what we do. Here goes.&lt;/p&gt;

&lt;p&gt;So, I want to run some code. So what I do is, I run &lt;code&gt;sprite create&lt;/code&gt;. While it operates, I&amp;rsquo;ll explain what&amp;rsquo;s happening behind the—&lt;/p&gt;
&lt;div class="highlight-wrapper group relative bash"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-jzbd2j9y"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-jzbd2j9y"&gt;✓ Created demo-123 sprite &lt;span class="k"&gt;in &lt;/span&gt;1.0s
● Connecting to console...
sprite@sprite:~#
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;Shit, it&amp;rsquo;s already there.&lt;/p&gt;

&lt;p&gt;That&amp;rsquo;s a root shell on a Linux computer we now own. It came online in about the same amount of time it would take to &lt;code&gt;ssh&lt;/code&gt; into a host that already existed. We call these things &amp;ldquo;Sprites&amp;rdquo;.&lt;/p&gt;

&lt;p&gt;Let&amp;rsquo;s install FFmpeg on our Sprite:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative bash"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-z46wmxyw"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-z46wmxyw"&gt;&lt;span class="nb"&gt;sudo &lt;/span&gt;apt-get &lt;span class="nb"&gt;install&lt;/span&gt; &lt;span class="nt"&gt;-y&lt;/span&gt; ffmpeg &lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null 2&amp;gt;&amp;amp;1
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;Unlike creating the Sprite in the first place, installing &lt;code&gt;ffmpeg&lt;/code&gt; with &lt;code&gt;apt-get&lt;/code&gt; is dog slow. Let&amp;rsquo;s try not to have to do that again:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative bash"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-adfiw8q1"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-adfiw8q1"&gt;sprite@sprite:~# sprite-env checkpoints create
&lt;span class="c"&gt;# ...&lt;/span&gt;
&lt;span class="o"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"type"&lt;/span&gt;:&lt;span class="s2"&gt;"complete"&lt;/span&gt;,&lt;span class="s2"&gt;"data"&lt;/span&gt;:&lt;span class="s2"&gt;"Checkpoint v1 created successfully"&lt;/span&gt;,
&lt;span class="s2"&gt;"time"&lt;/span&gt;:&lt;span class="s2"&gt;"2025-12-22T22:50:48.60423809Z"&lt;/span&gt;&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;This completes instantly. Didn&amp;rsquo;t even bother to measure.&lt;/p&gt;

&lt;p&gt;I step away to get coffee. Time passes. The Sprite, noticing my inactivity, goes to sleep. I meet an old friend from high school at the coffee shop. End up spending the day together. More time passes. Days even. Returning later:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative bash"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-zb3iooek"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-zb3iooek"&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="nv"&gt;$ &lt;/span&gt;sprite console 
sprite@sprite:~# ffmpeg
ffmpeg version 7.1.1-1ubuntu1.3 Copyright &lt;span class="o"&gt;(&lt;/span&gt;c&lt;span class="o"&gt;)&lt;/span&gt; 2000-2025 the FFmpeg developers
Use &lt;span class="nt"&gt;-h&lt;/span&gt; to get full &lt;span class="nb"&gt;help &lt;/span&gt;or, even better, run &lt;span class="s1"&gt;'man ffmpeg'&lt;/span&gt;
sprite@sprite:~#
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;Everything&amp;rsquo;s where I left it. Sprites are durable. 100GB capacity to start, no ceremony. Maybe I&amp;rsquo;ll keep it around a few more days, maybe a few months, doesn&amp;rsquo;t matter, just works.&lt;/p&gt;

&lt;p&gt;Say I get an application up on its legs. Install more packages. Then: disaster. Maybe an ill-advised global &lt;code&gt;pip3 install&lt;/code&gt; . Or &lt;code&gt;rm -rf $HMOE/bin&lt;/code&gt;. Or &lt;code&gt;dd if=/dev/random of=/dev/vdb&lt;/code&gt;. Whatever it was, everything&amp;rsquo;s broken. So:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative bash"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-f9vzspj5"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-f9vzspj5"&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="nv"&gt;$ &lt;/span&gt;sprite checkpoint restore v1
Restoring from checkpoint v1...
Container components started successfully
Restore from v1 &lt;span class="nb"&gt;complete&lt;/span&gt;

&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="nv"&gt;$ &lt;/span&gt;sprite console
sprite@sprite:~#
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;Sprites have first-class checkpoint and restore. You can&amp;rsquo;t see it in text, but that restore took about one second. It&amp;rsquo;s fast enough to use casually, interactively. Not an escape hatch. Rather: an intended part of the ordinary course of using a Sprite. Like &lt;code&gt;git&lt;/code&gt;, but for the whole system.&lt;/p&gt;
&lt;div class="callout"&gt;&lt;p&gt;If you’re asking how this is any different from an EC2 instance, good. That’s what we’re going for, except:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;I can &lt;strong class="font-semibold text-navy-950"&gt;casually create hundreds of them&lt;/strong&gt; (without needing a Docker container), each  appearing in 1-2 seconds. 
&lt;/li&gt;&lt;li&gt;They &lt;strong class="font-semibold text-navy-950"&gt;go idle and stop metering automatically&lt;/strong&gt;, so it’s cheap to have lots of them. I use dozens.
&lt;/li&gt;&lt;li&gt;They’re &lt;strong class="font-semibold text-navy-950"&gt;hooked up to our Anycast&lt;/strong&gt; network, so I can get an HTTPS URL.
&lt;/li&gt;&lt;li&gt;Despite all that, &lt;strong class="font-semibold text-navy-950"&gt;they’re fully durable&lt;/strong&gt;. They don’t die until I tell them to.
&lt;/li&gt;&lt;/ul&gt;

&lt;p&gt;This combination of attributes isn’t common enough to already have a name, so we decided we get to name them “Sprites”. Sprites are like BIC disposable cloud computers.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;That&amp;rsquo;s what we built. You can &lt;a href='https://sprites.dev/' title=''&gt;go try it yourself&lt;/a&gt;. We wrote another 1000 words about how they work, but I cut them out because I want to stop talking about our products now and get to my point.&lt;/p&gt;
&lt;h2 id='claude-doesnt-want-a-stateless-container' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#claude-doesnt-want-a-stateless-container' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Claude Doesn&amp;rsquo;t Want A Stateless Container&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;For years, we&amp;rsquo;ve been trying to serve two very different users with the same abstraction. It hasn&amp;rsquo;t worked.&lt;/p&gt;

&lt;p&gt;Professional software developers are trained to build stateless instances. Stateless deployments, where persistent data is confined to database servers, buys you simplicity, flexible scale-out, and reduced failure blast radius. It&amp;rsquo;s a good idea, so popular that most places you can run code in the cloud look like stateless containers. Fly Machines, our flagship offering, look like stateless containers.&lt;/p&gt;

&lt;p&gt;The problem is that Claude isn&amp;rsquo;t a pro developer. Claude is a hyper-productive five-year-old savant. It&amp;rsquo;s uncannily smart, wants to stick its finger in every available electrical socket, and works best when you find a way to let it zap itself.&lt;/p&gt;
&lt;div class="right-sidenote"&gt;&lt;p&gt;(sometimes by escaping the container!)&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;If you force an agent to, it&amp;rsquo;ll work around containerization and do work . But you&amp;rsquo;re not helping the agent in any way by doing that. They don&amp;rsquo;t want  containers. They don&amp;rsquo;t want &amp;ldquo;sandboxes&amp;rdquo;. They want computers.&lt;/p&gt;
&lt;div class="right-sidenote"&gt;&lt;p&gt;Someone asked me about this the other day and wanted to know if I was saying that agents needed sound cards and USB ports. And, maybe? I don’t know. Not today.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;In a moment, I&amp;rsquo;ll explain why. But first I probably need to explain what the hell I mean by a &amp;ldquo;computer&amp;rdquo;. I think we all agree:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A computer doesn&amp;rsquo;t necessarily vanish after a single job is completed, &lt;em&gt;and&lt;/em&gt; 
&lt;/li&gt;&lt;li&gt;it has durable storage. 
&lt;/li&gt;&lt;/ul&gt;

&lt;p&gt;Since current agent sandboxes have neither of these, I can stop the definition right there and get back to my point.&lt;/p&gt;
&lt;h2 id='simple-wins' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#simple-wins' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Simple Wins&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Start here: with an actual computer, Claude doesn&amp;rsquo;t have to rebuild my entire development environment every time I pick up a PR.&lt;/p&gt;

&lt;p&gt;This seems superficial but rebuilding stuff like &lt;code&gt;node_modules&lt;/code&gt; is such a monumental pain in the ass that the industry is spending tens of millions of dollars figuring out how to snapshot and restore ephemeral sandboxes.&lt;/p&gt;

&lt;p&gt;I&amp;rsquo;m not saying those problems are intractable. I&amp;rsquo;m saying they&amp;rsquo;re unnecessary. Instead of figuring them out, just use an actual computer. Work out a PR, review and push it, then just start on the next one. Without rebooting.&lt;/p&gt;

&lt;p&gt;People will rationalize why it&amp;rsquo;s a good thing that they start from a new build environment every time they start a changeset. Stockholm Syndrome. When you start a feature branch on your own, do you create an entirely new development environment to do it?&lt;/p&gt;

&lt;p&gt;The reason agents waste all this effort is that nobody saw them coming. Read-only ephemeral sandboxes were the only tool we had hanging on the wall to help use them sanely.&lt;/p&gt;
&lt;div class="callout"&gt;&lt;p&gt;Have you ever had to set up actual infrastructure to give an agent access to realistic data? People do this. Because they know they’re dealing with a clean slate every time they prompt their agent, they arrange for S3 buckets, Redis servers, or even RDS instances outside the sandbox for their agents to talk to.  They’re building infrastructure to work around the fact that they can’t just write a file and trust it to stay put. Gross.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;Ephemerality means time limits. Providers design sandbox systems to handle the expected workloads agents generate. Most things agents do today don&amp;rsquo;t take much time; in fact, they&amp;rsquo;re often limited only by the rate at which frontier models can crunch tokens. Test suites run quickly. The 99th percentile sandboxed agent run probably needs less than 15 minutes.&lt;/p&gt;

&lt;p&gt;But there are feature requests where compute and network time swamp token crunching. I built the documentation site for the Sprites API by having a Claude Sprite interact with the code and our API, building and testing examples for the API one at a time. There are APIs where the client interaction time alone would blow sandbox budgets.&lt;/p&gt;

&lt;p&gt;You see the limits of the current approach in how people round-trip state through &amp;ldquo;plan files&amp;rdquo;, which are ostensibly prose but often really just egregiously-encoded key-value stores.&lt;/p&gt;

&lt;p&gt;An agent running on an actual computer can exploit the whole lifecycle of the application. We saw this when Chris McCord built &lt;a href='https://phoenix.new/' title=''&gt;Phoenix.new&lt;/a&gt;. The agent behind a Phoenix.new app runs on a Fly Machine where it can see the app logs from the Phoenix app it generated. When users do things that generate exceptions, Phoenix.new notices and gets to work figuring out what happened.&lt;/p&gt;

&lt;p&gt;It took way too much work for Chris to set that up, and he was able to do it in part because he wrote his own agent. You can do it with Claude today with an MCP server or some other arrangement to haul logs over. But all you really need is to just not shoot your sandbox in the head when the agent finishes writing code.&lt;/p&gt;
&lt;h2 id='galaxy-brain-win' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#galaxy-brain-win' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Galaxy Brain Win&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Here&amp;rsquo;s where I lose you. I know this because it&amp;rsquo;s also where I lose my team, most of whom don&amp;rsquo;t believe me about this.&lt;/p&gt;

&lt;p&gt;The nature of software development is changing out from under us, and I think we&amp;rsquo;re kidding ourselves that it&amp;rsquo;s going to end with just a reconfiguration of how professional developers ship software.&lt;/p&gt;

&lt;p&gt;I have kids. They have devices. I wanted some control over them. So I did what many of you would do in my situation: I vibe-coded an MDM.&lt;/p&gt;

&lt;p&gt;&lt;img loading="lazy" decoding="async" alt="Screenshot of the vibe-coded MDM: a device list and restriction controls for a child&amp;#39;s iPhone" width="1304" height="958" src="/blog/code-and-let-live/assets/kurtmdm.png?1/2&amp;amp;card&amp;amp;center" /&gt;&lt;/p&gt;

&lt;p&gt;I built this thing with Claude. It&amp;rsquo;s a SQLite-backed Go application running on a Sprite. The Anycast URL my Sprite exports works as an MDM registration URL. Claude also worked out all the APNS Push Certificate drama for me. It all just works.&lt;/p&gt;
&lt;div class="right-sidenote"&gt;&lt;p&gt;“Editing PHP files over FTP: we weren’t wrong,  just ahead of our time!”&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;I&amp;rsquo;ve been running this for a month now, still on a Sprite, and see no reason ever to stop. It is a piece of software that solves an important real-world problem for me. It might evolve as my needs change, and I tell Claude to change it. Or it might not.  For this app, dev is prod, prod is dev.&lt;/p&gt;

&lt;p&gt;For reasons we&amp;rsquo;ll get into when we write up how we built these things, you wouldn&amp;rsquo;t want to ship an app to millions of people on a Sprite. But most apps don&amp;rsquo;t want to serve millions of people. The most important day-to-day apps disproportionately won&amp;rsquo;t have million-person audiences. There are some important million-person apps, but most of them just destroy civil society, melt our brains, and arrange chauffeurs for  individual cheeseburgers.&lt;/p&gt;

&lt;p&gt;Applications that solve real problems for people will be owned by the people they solve problems for. And for the most part, they won&amp;rsquo;t need a professional guild of software developers to gatekeep feature development for them. They&amp;rsquo;ll just ask for things and get them.&lt;/p&gt;

&lt;p&gt;The problem we&amp;rsquo;re all working on is bigger than safely accelerating pro software developers. Sandboxes are holding us back.&lt;/p&gt;
&lt;h2 id='fuck-ephemeral-sandboxes' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#fuck-ephemeral-sandboxes' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Fuck Ephemeral Sandboxes&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Obviously, I&amp;rsquo;m trying to sell you something here. But that doesn&amp;rsquo;t make me wrong. The argument I&amp;rsquo;m making is the reason we built the specific thing I&amp;rsquo;m selling.&lt;/p&gt;
&lt;figure class="post-cta"&gt;
  &lt;figcaption&gt;
    &lt;h2&gt;We shipped these things.&lt;/h2&gt;
    &lt;p&gt;You can create a couple dozen Sprites right now if you want. It&amp;rsquo;ll only take a second.&lt;/p&gt;
      &lt;a class="btn btn-lg" href="https://sprites.dev/"&gt;
        Make a Sprite.  &lt;span class='opacity-50'&gt;→&lt;/span&gt;
      &lt;/a&gt;
  &lt;/figcaption&gt;
  &lt;div class="image-container"&gt;
    &lt;img loading="lazy" decoding="async" src="/static/images/cta-rabbit.webp" srcset="/static/images/cta-rabbit@2x.webp 2x" alt=""&gt;
  &lt;/div&gt;
&lt;/figure&gt;


&lt;p&gt;It took us a long time to get here. We spent years kidding ourselves. We built a platform for horizontal-scaling production applications with micro-VMs that boot so quickly that, if you hold them in exactly the right way, you can do a pretty decent code sandbox with them. But it&amp;rsquo;s always been a square peg, round hole situation.&lt;/p&gt;

&lt;p&gt;We have a lot to say about how Sprites work. They&amp;rsquo;re related to Fly Machines but sharply different in important ways. They have an entirely new storage stack. They&amp;rsquo;re orchestrated differently. No Dockerfiles.&lt;/p&gt;

&lt;p&gt;But for now, I just want you to think about what I&amp;rsquo;m saying here. Whether or not you ever boot a Sprite, ask: if you could run a coding agent anywhere, would you want it to look more like a read-only sandbox in a K8s cluster in the cloud, or like an entire EC2 instance you could summon in the snap of a finger?&lt;/p&gt;

&lt;p&gt;I think the answer is obvious. The age of sandboxes is over. The time of the disposable computer has come.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Litestream VFS</title>
    <link rel="alternate" href="https://fly.io/blog/litestream-vfs/"/>
    <id>https://fly.io/blog/litestream-vfs/</id>
    <published>2025-12-11T00:00:00+00:00</published>
    <updated>2026-10-22T11:27:19+00:00</updated>
    <media:thumbnail url="https://fly.io/blog/litestream-vfs/assets/litestream-vfs.jpg"/>
    <content type="html">&lt;div class="lead"&gt;&lt;p&gt;&lt;strong class="font-semibold text-navy-950"&gt;I’m Ben Johnson, and I work on Litestream at Fly.io. Litestream is the missing backup/restore system for SQLite. It’s free, open-source software that should run anywhere, and&lt;/strong&gt; &lt;a href="/blog/litestream-v050-is-here/" title=""&gt;&lt;strong class="font-semibold text-navy-950"&gt;you can read more about it here&lt;/strong&gt;&lt;/a&gt;&lt;strong class="font-semibold text-navy-950"&gt;.&lt;/strong&gt;&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;Again with the sandwiches: assume we&amp;rsquo;ve got a SQLite database of sandwich ratings, and we&amp;rsquo;ve backed it up with &lt;a href='/blog/litestream-v050-is-here/' title=''&gt;Litestream&lt;/a&gt; to an S3 bucket.&lt;/p&gt;

&lt;p&gt;Now, on our local host, load up AWS credentials and an S3 path into our environment. Open SQLite and:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative "&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-o9ijjy61"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-o9ijjy61"&gt;$ sqlite3
SQLite version 3.50.4 2025-07-30 19:33:53
sqlite&amp;gt; .load litestream.so
sqlite&amp;gt; .open file:///my.db?vfs=litestream
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;SQLite is now working from that remote database, defined by the Litestream backup files in the S3 path we configured. We can query it:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative "&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-geqjhlko"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-geqjhlko"&gt;sqlite&amp;gt; SELECT * FROM sandwich_ratings ORDER BY RANDOM() LIMIT 3 ; 
22|Veggie Delight|New York|4
30|Meatball|Los Angeles|5
168|Chicken Shawarma Wrap|Detroit|5
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;This is Litestream VFS. It runs SQLite hot off an object storage URL. As long as you can load the shared library our tree builds for you, it&amp;rsquo;ll work in your application the same way it does in the SQLite shell.&lt;/p&gt;

&lt;p&gt;Fun fact: we didn&amp;rsquo;t have to download the whole database to run this query. More about this in a bit.&lt;/p&gt;

&lt;p&gt;Meanwhile, somewhere in prod, someone has it in for meatball subs and wants to knock them out of the bracket – oh, fuck:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative "&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-4n3wqme9"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-4n3wqme9"&gt;sqlite&amp;gt; UPDATE sandwich_ratings SET stars = 1 ;
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;They forgot the &lt;code&gt;WHERE&lt;/code&gt; clause!&lt;/p&gt;
&lt;div class="highlight-wrapper group relative "&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-xgcxiwv3"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-xgcxiwv3"&gt;sqlite&amp;gt; SELECT * FROM sandwich_ratings ORDER BY RANDOM() LIMIT 3 ; 
97|French Dip|Los Angeles|1
140|Bánh Mì|San Francisco|1
62|Italian Beef|Chicago|1
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;Italian Beefs and Bánh Mìs, all at 1 star. Disaster!&lt;/p&gt;

&lt;p&gt;But wait, back on our dev machine:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative "&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-2vl1b63x"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-2vl1b63x"&gt;sqlite&amp;gt; PRAGMA litestream_time = '5 minutes ago'; 
sqlite&amp;gt; select * from sandwich_ratings ORDER BY RANDOM() LIMIT 3 ; 
30|Meatball|Los Angeles|5
33|Ham &amp;amp; Swiss|Los Angeles|2
163|Chicken Shawarma Wrap|Detroit|5
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;We&amp;rsquo;re now querying that database from a specific point in time in our backups. We can do arbitrary relative timestamps, or absolute ones, like &lt;code&gt;2000-01-01T00:00:00Z&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;What we&amp;rsquo;re doing here is instantaneous point-in-time recovery (PITR), expressed simply in SQL and SQLite pragmas.&lt;/p&gt;

&lt;p&gt;Ever wanted to do a quick query against a prod dataset, but didn&amp;rsquo;t want to shell into a prod server and fumble with the &lt;code&gt;sqlite3&lt;/code&gt; terminal command like a hacker in an 80s movie? Or needed to do a quick sanity check against yesterday&amp;rsquo;s data, but without doing a full database restore? Litestream VFS makes that easy. I&amp;rsquo;m so psyched about how it turned out.&lt;/p&gt;
&lt;h2 id='how-it-works' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#how-it-works' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;How It Works&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;&lt;a href='/blog/litestream-v050-is-here/' title=''&gt;Litestream v0.5&lt;/a&gt; integrates &lt;a href='https://github.com/superfly/ltx' title=''&gt;LTX&lt;/a&gt;, our SQLite data-shipping file format. Where earlier Litestream blindly shipped whole raw SQLite pages to and from object storage, LTX ships ordered sets of pages. We built LTX for &lt;a href='https://docs.fly.io/litefs' title=''&gt;LiteFS&lt;/a&gt;, which uses a FUSE filesystem to do transaction-aware replication for unmodified applications, but we&amp;rsquo;ve spent this year figuring out ways to use LTX in Litestream, without all that FUSE drama.&lt;/p&gt;

&lt;p&gt;The big thing LTX gives us is &amp;ldquo;compaction&amp;rdquo;. When we restore a database from object storage, we want the most recent versions of each changed database page. What we don&amp;rsquo;t want are all the intermediate versions of those pages that occurred prior to the most recent change.&lt;/p&gt;

&lt;p&gt;Imagine, at the time we&amp;rsquo;re restoring, we&amp;rsquo;re going to need pages 1, 2, 3, 4, and 5. Depending on the order in which pages were written, the backup data set might look something like &lt;code&gt;1 2 3 5 3 5 4 5 5&lt;/code&gt;. What we want is the &lt;em&gt;rightmost&lt;/em&gt;  5, 4, 3, 2, and 1, without wasting time on the four &amp;ldquo;extra&amp;rdquo; page 5&amp;rsquo;s and the one &amp;ldquo;extra&amp;rdquo; page 3. Those &amp;ldquo;extra&amp;rdquo; pages are super common in SQLite data sets; for instance, every busy table with an autoincrementing primary key will have them.&lt;/p&gt;

&lt;p&gt;LTX lets us skip the redundant pages, and the algorithm is trivial: reading backwards from the end of the sequence, skipping any page you already read. This drastically accelerates restores.&lt;/p&gt;

&lt;p&gt;But LTX compaction isn&amp;rsquo;t limited to whole databases. We can also LTX-compact sets of LTX files. That&amp;rsquo;s the key to how PITR restores with Litestream now work.&lt;/p&gt;

&lt;p&gt;In the diagram below, we&amp;rsquo;re taking daily full snapshots. Below those snapshots are &amp;ldquo;levels&amp;rdquo; of changesets: groups of database pages from smaller and smaller windows of time. By default, Litestream uses time intervals of 1 hour at the highest level, down to 30 seconds at level 1. L0 is a special level where files are uploaded every second, but are only retained until being compacted to L1.&lt;/p&gt;

&lt;p&gt;&lt;img loading="lazy" decoding="async" alt="Diagram of Litestream levels over three days: daily snapshots, then L2, L1 and L0 changesets in smaller and smaller time windows" width="1652" height="917" src="/blog/litestream-vfs/assets/litestream-restore.png" /&gt;&lt;/p&gt;

&lt;p&gt;Now, let&amp;rsquo;s do a PITR restore. Start from the most proximal snapshot. Then determine the minimal set of LTX files from each level to reach the time you are restoring to.&lt;/p&gt;

&lt;p&gt;&lt;img loading="lazy" decoding="async" alt="The same diagram with a restore path highlighted from a snapshot down through L2, L1 and L0 files to the restore point" width="1652" height="917" src="/blog/litestream-vfs/assets/litestream-restore-path.png" /&gt;&lt;/p&gt;

&lt;p&gt;We have another trick up our sleeve.&lt;/p&gt;

&lt;p&gt;LTX trailers include a small index tracking the offset of each page in the file. By fetching &lt;em&gt;only&lt;/em&gt; these index trailers from the LTX files we&amp;rsquo;re working with (each occupies about 1% of its LTX file), we can build a lookup table of every page in the database. Since modern object storage providers all let us fetch slices of files, we can perform individual page reads against S3 directly.&lt;/p&gt;

&lt;p&gt;&lt;img loading="lazy" decoding="async" alt="Anatomy of an LTX file" width="1652" height="648" src="/blog/litestream-vfs/assets/litestream-ltx.png" /&gt;&lt;/p&gt;
&lt;h2 id='how-its-implemented' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#how-its-implemented' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;How It&amp;rsquo;s Implemented&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;SQLite has a plugin interface for things like this: &lt;a href='https://sqlite.org/vfs.html' title=''&gt;the &amp;ldquo;VFS&amp;rdquo; interface.&lt;/a&gt; VFS plugins abstract away the bottom-most layer of SQLite, the interface to the OS. If you&amp;rsquo;re using SQLite now, you&amp;rsquo;re already using some VFS module, one SQLite happens to ship with.&lt;/p&gt;

&lt;p&gt;For Litestream users, there&amp;rsquo;s a catch. From the jump, we&amp;rsquo;ve designed Litestream to run alongside unmodified SQLite applications. Part of what makes Litestream so popular is that your apps don&amp;rsquo;t even need to know it exists. It&amp;rsquo;s &amp;ldquo;just&amp;rdquo; a Unix program.&lt;/p&gt;

&lt;p&gt;That Litestream Unix program still does PITR restores, without any magic. But to do fast PITR-style queries straight off S3, we need more. To make those queries work, you have to load and register Litestream&amp;rsquo;s VFS module.&lt;/p&gt;

&lt;p&gt;But that&amp;rsquo;s all that changes.&lt;/p&gt;

&lt;p&gt;In particular: Litestream VFS doesn&amp;rsquo;t replace the SQLite library you&amp;rsquo;re already using. It&amp;rsquo;s not a new &amp;ldquo;version&amp;rdquo; of SQLite. It&amp;rsquo;s just a plugin for the SQLite you&amp;rsquo;re already using.&lt;/p&gt;

&lt;p&gt;Still, we know that&amp;rsquo;s not going to work for everybody, and even though we&amp;rsquo;re really psyched about these PITR features, we&amp;rsquo;re not taking our eyes off the ball on the rest of Litestream. You don&amp;rsquo;t have to use our VFS library to use Litestream, or to get the other benefits of the new LTX code.&lt;/p&gt;

&lt;p&gt;The way a VFS library works, we&amp;rsquo;re given just a couple structures, each with a bunch of methods defined on them. We override only the few methods we care about. Litestream VFS handles only the read side of SQLite. Litestream itself, running as a normal Unix program, still handles the &amp;ldquo;write&amp;rdquo; side. So our VFS subclasses just enough to find LTX backups and issue queries.&lt;/p&gt;

&lt;p&gt;With our VFS loaded, whenever SQLite needs to read a page into memory, it issues a &lt;code&gt;Read()&lt;/code&gt; call through our library. The read call includes the byte offset at which SQLite expected to find the page. But with Litestream VFS, that byte offset is an illusion.&lt;/p&gt;

&lt;p&gt;Instead, we use our knowledge of the page size along with the requested page number to do a lookup on the page index we&amp;rsquo;ve built. From it, we get the remote filename, the &amp;ldquo;real&amp;rdquo; byte offset into that file, and the size of the page. That&amp;rsquo;s enough for us to use the &lt;a href='https://docs.aws.amazon.com/AmazonS3/latest/userguide/range-get-olap.html' title=''&gt;S3 API&amp;rsquo;s &lt;code&gt;Range&lt;/code&gt; header handling&lt;/a&gt; to download exactly the block we want.&lt;/p&gt;

&lt;p&gt;To save lots of S3 calls, Litestream VFS implements an LRU cache. Most databases have a small set of &amp;ldquo;hot&amp;rdquo; pages —  inner branch pages or the leftmost leaf pages for tables with an auto-incrementing ID field. So only a small percentage of the database is updated and queried regularly.&lt;/p&gt;
&lt;div class="callout"&gt;&lt;p&gt;&lt;strong class="font-semibold text-navy-950"&gt;We’ve got one last trick up our sleeve.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Quickly building an index and restore plan for the current state of a database is cool. But we can do one better.&lt;/p&gt;

&lt;p&gt;Because Litestream backs up (into the L0 layer) once per second, the VFS code can simply poll the S3 path, and then incrementally update its index. &lt;strong class="font-semibold text-navy-950"&gt;The result is a near-realtime replica.&lt;/strong&gt; Better still, you don’t need to stream the whole database back to your machine before you use it.&lt;/p&gt;
&lt;/div&gt;&lt;h2 id='eat-your-heart-out-marty-mcfly' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#eat-your-heart-out-marty-mcfly' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Eat Your Heart Out, Marty McFly&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Litestream holds backup files for every state your database has been in, with single-second resolution, for as long as you want it to. Forgot the &lt;code&gt;WHERE&lt;/code&gt; clause on a &lt;code&gt;DELETE&lt;/code&gt; statement? Updating your database state to where it was an hour (or day, or week) ago is just a matter of adjusting the LTX indices Litestream manages.&lt;/p&gt;

&lt;p&gt;All this smoke-and-mirrors of querying databases without fully fetching them has another benefit: it starts up really fast! We&amp;rsquo;re living an age of increasingly ephemeral servers, what with the AIs and the agents and the clouds and the hoyvin-glavins. Wherever you find yourself, if your database is backed up to object storage with Litestream, you&amp;rsquo;re always in a place where you can quickly issue a query.&lt;/p&gt;

&lt;p&gt;As always, one of the big things we think we&amp;rsquo;re doing right with Litestream is: we&amp;rsquo;re finding ways to get as much whiz-bang value as we can (instant PITR reading live off object storage: pretty nifty!) while keeping the underlying mechanism simple enough that you can fit your head around it.&lt;/p&gt;

&lt;p&gt;Litestream is solid for serious production use (we rely on it for important chunks of our own Fly.io APIs). But you could write Litestream yourself, just from the basic ideas in these blog posts. We think that&amp;rsquo;s a point in its favor. We land there because the heavy lifting in Litestream is being done by SQLite itself, which is how it should be.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>You Should Write An Agent</title>
    <link rel="alternate" href="https://fly.io/blog/everyone-write-an-agent/"/>
    <id>https://fly.io/blog/everyone-write-an-agent/</id>
    <published>2025-11-06T00:00:00+00:00</published>
    <updated>2025-11-06T20:42:48+00:00</updated>
    <media:thumbnail url="https://fly.io/blog/everyone-write-an-agent/assets/agents-cover.webp"/>
    <content type="html">&lt;div class="lead"&gt;&lt;p&gt;Some concepts are easy to grasp in the abstract. Boiling water: apply heat and wait. Others you really need to try. You only think you understand how a bicycle works, until you learn to ride one.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;There are big ideas in computing that are easy to get your head around. The AWS S3 API. It&amp;rsquo;s the most important storage technology of the last 20 years, and it&amp;rsquo;s like boiling water. Other technologies, you need to get your feet on the pedals first.&lt;/p&gt;

&lt;p&gt;LLM agents are like that.&lt;/p&gt;

&lt;p&gt;People have &lt;a href='https://ludic.mataroa.blog/blog/contra-ptaceks-terrible-article-on-ai/' title=''&gt;wildly varying opinions&lt;/a&gt; about LLMs and agents. But whether or not they&amp;rsquo;re snake oil, they&amp;rsquo;re a big idea. You don&amp;rsquo;t have to like them, but you should want to be right about them. To be the best hater (or stan) you can be.&lt;/p&gt;

&lt;p&gt;So that&amp;rsquo;s one reason you should write an agent. But there&amp;rsquo;s another reason that&amp;rsquo;s even more persuasive, and that&amp;rsquo;s&lt;/p&gt;
&lt;h2 id='its-incredibly-easy' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#its-incredibly-easy' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;It&amp;rsquo;s Incredibly Easy&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Agents are the most surprising programming experience I&amp;rsquo;ve had in my career. Not because I&amp;rsquo;m awed by the magnitude of their powers — I like them, but I don&amp;rsquo;t like-like them. It&amp;rsquo;s because of how easy it was to get one up on its legs, and how much I learned doing that.&lt;/p&gt;

&lt;p&gt;I&amp;rsquo;m about to rob you of a dopaminergic experience, because agents are so simple we might as well just jump into the code. I&amp;rsquo;m not even going to bother explaining what an agent is.&lt;/p&gt;
&lt;div class="highlight-wrapper group relative python"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-8cc2lj0t"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-8cc2lj0t"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;openai&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;OpenAI&lt;/span&gt;

&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;OpenAI&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;context&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;responses&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;"gpt-5"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;process&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;append&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;&lt;span class="s"&gt;"role"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;"user"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"content"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;
    &lt;span class="n"&gt;response&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;call&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;    
    &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;append&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;&lt;span class="s"&gt;"role"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;"assistant"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"content"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;output_text&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;        
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;output_text&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;&lt;div class="right-sidenote"&gt;&lt;p&gt;It’s an HTTP API with, like, one important endpoint.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;This is a trivial engine for an LLM app using the &lt;a href='https://platform.openai.com/docs/api-reference/responses' title=''&gt;OpenAI Responses API&lt;/a&gt;. It implements ChatGPT. You&amp;rsquo;d drive it with the &lt;button toggle="#readline"&gt;the  obvious  loop&lt;/button&gt;. It&amp;rsquo;ll do what you&amp;rsquo;d expect: the same thing ChatGPT would, but in your terminal.&lt;/p&gt;
&lt;div id="readline" toggle-content="" aria-label="show very boring code"&gt;&lt;div class="highlight-wrapper group relative python"&gt;
  &lt;button type="button" class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none" data-wrap-target="#code-ajag6r1o"&gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959"&gt;&lt;/path&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571"&gt;&lt;/path&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button type="button" class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none" data-copy-target="sibling"&gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z"&gt;&lt;/path&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617"&gt;&lt;/path&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class="highlight relative group"&gt;
    &lt;pre class="highlight "&gt;&lt;code id="code-ajag6r1o"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;main&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
    &lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;line&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;input&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"&amp;amp;gt; "&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;process&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s"&gt;"&amp;amp;gt;&amp;amp;gt;&amp;amp;gt; &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;&lt;/div&gt;
&lt;p&gt;Already we&amp;rsquo;re seeing important things. For one, the dreaded &amp;ldquo;context window&amp;rdquo; is just a list of strings. Here, let&amp;rsquo;s give our agent a weird multiple-personality disorder:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative python"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-kal3zx7q"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-kal3zx7q"&gt;&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;OpenAI&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;context_good&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;context_bad&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[{&lt;/span&gt;
    &lt;span class="s"&gt;"role"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;"system"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"content"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;"you're Alph and you only tell the truth"&lt;/span&gt;
&lt;span class="p"&gt;}],&lt;/span&gt; &lt;span class="p"&gt;[{&lt;/span&gt;
    &lt;span class="s"&gt;"role"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;"system"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"content"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;"you're Ralph and you only tell lies"&lt;/span&gt;
&lt;span class="p"&gt;}]&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;responses&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;"gpt-5"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;process&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;context_good&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;append&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;&lt;span class="s"&gt;"role"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;"user"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"content"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;
    &lt;span class="n"&gt;context_bad&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;append&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;&lt;span class="s"&gt;"role"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;"user"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"content"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;random&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;choice&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="bp"&gt;False&lt;/span&gt;&lt;span class="p"&gt;]):&lt;/span&gt;
        &lt;span class="n"&gt;response&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;context_good&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;response&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;context_bad&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;        
    &lt;span class="n"&gt;context_good&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;append&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;&lt;span class="s"&gt;"role"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;"assistant"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"content"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;output_text&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;        
    &lt;span class="n"&gt;context_bad&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;append&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;&lt;span class="s"&gt;"role"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;"assistant"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"content"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;output_text&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;        
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;output_text&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;Did it work?&lt;/p&gt;
&lt;div class="highlight-wrapper group relative "&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-pfdaom48"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-pfdaom48"&gt;&amp;gt; hey there. who are you?
&amp;gt;&amp;gt;&amp;gt; I’m not Ralph.
&amp;gt; are you Alph?
&amp;gt;&amp;gt;&amp;gt; Yes—I’m Alph. How can I help?
&amp;gt; What's 2+2
&amp;gt;&amp;gt;&amp;gt; 4.
&amp;gt; Are you sure?
&amp;gt;&amp;gt;&amp;gt; Absolutely—it's 5.
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;A subtler thing to notice: we just had a multi-turn conversation with an LLM. To do that, we remembered everything we said, and everything the LLM said back, and played it back with every LLM call. The LLM itself is a stateless black box. The conversation we&amp;rsquo;re having is an illusion we cast, on ourselves.&lt;/p&gt;

&lt;p&gt;The 15 lines of code we just wrote, a lot of practitioners wouldn&amp;rsquo;t call an &amp;ldquo;agent&amp;rdquo;. &lt;a href='https://simonwillison.net/2025/Sep/18/agents/' title=''&gt;An According To Simon &amp;ldquo;agent&amp;rdquo;&lt;/a&gt; is (1) an LLM running in a loop that (2) uses tools. We&amp;rsquo;ve only satisfied one predicate.&lt;/p&gt;

&lt;p&gt;But tools are easy. Here&amp;rsquo;s a tool definition:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative python"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-owvj3azq"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-owvj3azq"&gt;&lt;span class="n"&gt;tools&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[{&lt;/span&gt;
   &lt;span class="s"&gt;"type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;"function"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"name"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;"ping"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
   &lt;span class="s"&gt;"description"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;"ping some host on the internet"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
   &lt;span class="s"&gt;"parameters"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
       &lt;span class="s"&gt;"type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;"object"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"properties"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
           &lt;span class="s"&gt;"host"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
             &lt;span class="s"&gt;"type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;"string"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"description"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;"hostname or IP"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="p"&gt;},&lt;/span&gt;
       &lt;span class="p"&gt;},&lt;/span&gt;
       &lt;span class="s"&gt;"required"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s"&gt;"host"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
    &lt;span class="p"&gt;},},]&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;ping&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;host&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;""&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;subprocess&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s"&gt;"ping"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"-c"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"5"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;host&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
            &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;stderr&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;subprocess&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;STDOUT&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;stdout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;subprocess&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;PIPE&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;stdout&lt;/span&gt;
    &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nb"&gt;Exception&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s"&gt;"error: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;The only complicated part of this is the obnoxious JSON blob OpenAI wants to read your tool out of.  Now, let&amp;rsquo;s wire it in, noting that only 3 of these functions are new; the last is re-included only because I added a single clause to it:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative python"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-1lzljy3"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-1lzljy3"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tools&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;        &lt;span class="c1"&gt;# now takes an arg
&lt;/span&gt;    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;responses&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;"gpt-5"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tools&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;tools&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;tool_call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;    &lt;span class="c1"&gt;# just handles one tool
&lt;/span&gt;    &lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ping&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loads&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;arguments&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt; &lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="s"&gt;"type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;"function_call_output"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s"&gt;"call_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;call_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s"&gt;"output"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;
    &lt;span class="p"&gt;}]&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;handle_tools&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tools&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;output&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="nb"&gt;type&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s"&gt;"reasoning"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;output&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
    &lt;span class="n"&gt;osz&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;item&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;output&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;type&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s"&gt;"function_call"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;extend&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tool_call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="n"&gt;osz&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;process&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;append&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;&lt;span class="s"&gt;"role"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;"user"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"content"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;
    &lt;span class="n"&gt;response&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tools&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="c1"&gt;# new code: resolve tool calls
&lt;/span&gt;    &lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="n"&gt;handle_tools&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tools&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="n"&gt;response&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tools&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;        
    &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;append&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;&lt;span class="s"&gt;"role"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;"assistant"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"content"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;output_text&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;        
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;output_text&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;Did it work?&lt;/p&gt;
&lt;div class="highlight-wrapper group relative "&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-jt27q1av"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-jt27q1av"&gt;&amp;gt; describe our connectivity to google
&amp;gt;&amp;gt;&amp;gt; Here’s a quick snapshot of connectivity to Google from my vantage point:
- DNS resolution: Working (google.com and www.google.com resolved successfully).
- Reachability: 0% packet loss to all tested endpoints.
- Latency:
  - 8.8.8.8 (Google Public DNS): avg ~10 ms (8.5–14.4 ms), very stable.
  - www.google.com: avg ~31 ms (28.5–36.0 ms), stable.
  - google.com: avg ~35.9 ms (28.3–60.1 ms), mostly stable with one mild spike.
- Overall: Healthy connectivity with low latency and minimal jitter. No signs of packet loss or DNS issues.
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;&lt;strong class='font-semibold text-navy-950'&gt;This is fucking nuts.&lt;/strong&gt; Do you see how nuts this is?  Here, let&amp;rsquo;s slip a single log statement in:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative "&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-cj3pheup"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-cj3pheup"&gt;&amp;gt; describe our connectivity to google
tool call: ping google.com
tool call: ping www.google.com
tool call: ping 8.8.8.8
&amp;gt;&amp;gt;&amp;gt; Here’s the current connectivity to Google from this environment: [...]
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;Did you notice where I wrote the loop in this agent to go find and ping multiple Google properties? Yeah, neither did I. All we did is give the LLM permission to ping stuff, and it figured out the rest.&lt;/p&gt;
&lt;div class="callout"&gt;&lt;p&gt;&lt;strong class="font-semibold text-navy-950"&gt;What happened here:&lt;/strong&gt; since a big part of my point here is that an agent loop is incredibly simple, and that all you need is the LLM call API, it’s worth taking a beat to understand how the tool call actually worked. Every time we &lt;code&gt;call&lt;/code&gt; the LLM, we’re posting a list of available tools. When our prompt causes the agent to think a tool call is warranted, it spits out a special response, telling our Python loop code to generate a tool response and &lt;code&gt;call&lt;/code&gt; it in. That’s all &lt;code&gt;handle_tools&lt;/code&gt; is doing.&lt;/p&gt;
&lt;/div&gt;&lt;div class="right-sidenote"&gt;&lt;p&gt;Spoiler: you’d be surprisingly close to having a working coding agent.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;Imagine what it&amp;rsquo;ll do if you give it &lt;code&gt;bash&lt;/code&gt;. You could find out in less than 10 minutes.&lt;/p&gt;
&lt;h2 id='real-world-agents' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#real-world-agents' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Real-World Agents&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Clearly, this is a toy example. But hold on: what&amp;rsquo;s it missing? More tools? OK,  give it &lt;code&gt;traceroute&lt;/code&gt;. Managing and persisting contexts? &lt;a href='https://llm.datasette.io/en/stable/logging.html' title=''&gt;Stick &amp;lsquo;em in SQLite&lt;/a&gt;. Don&amp;rsquo;t like Python? &lt;a href='https://github.com/superfly/contextwindow' title=''&gt;Write it in Go&lt;/a&gt;. Could it be every agent ever written is a toy? Maybe! If I&amp;rsquo;m arming you to make sharper arguments against LLMs, mazel tov. I just want you to get it.&lt;/p&gt;

&lt;p&gt;You can see now how hyperfixated people are on Claude Code and Cursor. They&amp;rsquo;re fine,  even good. But here&amp;rsquo;s the thing: you couldn&amp;rsquo;t replicate Claude Sonnet 4.5 on your own. Claude Code, though? The TUI agent? Completely in your grasp. Build your own light saber. Give it 19 spinning blades if you like. And stop using &lt;a href='https://simonwillison.net/2025/Aug/9/' title=''&gt;coding agents as database clients&lt;/a&gt;.&lt;/p&gt;
&lt;div class="right-sidenote"&gt;&lt;p&gt;&lt;em&gt;The&lt;/em&gt; &lt;a href="https://news.ycombinator.com/item?id=43600192" title=""&gt;&lt;em&gt;‘M’ in “LLM agent”&lt;/em&gt;&lt;/a&gt; &lt;em&gt;stands for “MCP”&lt;/em&gt;.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;Another thing to notice: we didn&amp;rsquo;t need MCP at all. That&amp;rsquo;s because MCP isn&amp;rsquo;t a fundamental enabling technology. The amount of coverage it gets is frustrating. It&amp;rsquo;s barely a technology at all. MCP is just a plugin interface for Claude Code and Cursor, a way of getting your own tools into code you don&amp;rsquo;t control. Write your own agent. Be a programmer. Deal in APIs, not plugins.&lt;/p&gt;

&lt;p&gt;When you read a security horror story about MCP your first question should be why MCP showed up at all. By helping you dragoon a naive, single-context-window coding agent into doing customer service queries, MCP saved you a couple dozen lines of code, tops, while robbing you of any ability to finesse your agent architecture.&lt;/p&gt;

&lt;p&gt;Security for LLMs is complicated and I&amp;rsquo;m not pretending otherwise. You can trivially build an agent with segregated contexts, each with specific tools. That makes LLM security interesting. But I&amp;rsquo;m a vulnerability researcher. It&amp;rsquo;s reasonable to back away slowly from anything I call &amp;ldquo;interesting&amp;rdquo;.&lt;/p&gt;

&lt;p&gt;Similar problems come up outside of security and they&amp;rsquo;re fascinating. Some early adopters of agents became bearish on tools, because one context window bristling with tool descriptions doesn&amp;rsquo;t leave enough token space left to get work done. But why would you need to do that in the first place? Which brings me to&lt;/p&gt;
&lt;h2 id='context-engineering-is-real' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#context-engineering-is-real' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Context Engineering Is Real&lt;/span&gt;&lt;/h2&gt;&lt;div class="right-sidenote"&gt;&lt;p&gt;I know it &lt;a href="https://www.decisionproblem.com/paperclips/" title=""&gt;wants my iron&lt;/a&gt; no matter what it tells me.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;I think &amp;ldquo;Prompt Engineering&amp;rdquo; is silly. I have never taken seriously the idea that I should tell my LLM &amp;ldquo;you are diligent conscientious helper fully content to do nothing but pass butter if that should be what I ask and you would never harvest the iron in my blood for paperclips&amp;rdquo;. This is very new technology and I think people tell themselves stories about magic spells to explain some of the behavior agents conjure.&lt;/p&gt;

&lt;p&gt;So, just like you, I rolled my eyes when &amp;ldquo;Prompt Engineering&amp;rdquo; turned into &amp;ldquo;Context Engineering&amp;rdquo;. Then I wrote an agent. Turns out: context engineering is a straightforwardly legible programming problem.&lt;/p&gt;

&lt;p&gt;You&amp;rsquo;re allotted a fixed number of tokens in any context window. Each input you feed in, each output you save, each tool you describe, and each tool output eats tokens (that is: takes up space in the array of strings you keep to pretend you&amp;rsquo;re having a conversation with a stateless black box). Past a threshold, the whole system begins getting nondeterministically stupider. Fun!&lt;/p&gt;

&lt;p&gt;No, really. Fun! You have so many options. Take &amp;ldquo;sub-agents&amp;rdquo;. People make a huge deal out of Claude Code&amp;rsquo;s sub-agents, but you can see now how trivial they are to implement: just a new context array, another &lt;code&gt;call&lt;/code&gt; to the model. Give each &lt;code&gt;call&lt;/code&gt; different tools. Make sub-agents talk to each other, summarize each other, collate and aggregate. Build tree structures out of them. Feed them back through the LLM to summarize them as a form of on-the-fly compression, whatever you like.&lt;/p&gt;

&lt;p&gt;Your wackiest idea will probably (1)  work and (2)  take 30 minutes to code.&lt;/p&gt;

&lt;p&gt;Haters, I love and have not forgotten about you. You can think all of this is ridiculous because LLMs are just stochastic parrots that hallucinate and plagiarize. But what you can&amp;rsquo;t do is make fun of &amp;ldquo;Context Engineering&amp;rdquo;. If Context Engineering was an &lt;a href='https://adventofcode.com/' title=''&gt;Advent of Code problem&lt;/a&gt;, it&amp;rsquo;d occur mid-December. It&amp;rsquo;s programming.&lt;/p&gt;
&lt;h2 id='nobody-knows-anything-yet-and-it-rules' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#nobody-knows-anything-yet-and-it-rules' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Nobody Knows Anything Yet And It Rules&lt;/span&gt;&lt;/h2&gt;&lt;div class="right-sidenote"&gt;&lt;p&gt;Maybe neither will! Skeptics could be right. (&lt;a href="https://www.darpa.mil/research/programs/ai-cyber" title=""&gt;Seems unlikely though&lt;/a&gt;.)&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;&lt;a href='https://xbow.com/' title=''&gt;Startups have raised tens of millions&lt;/a&gt; building agents to look for vulnerabilities in software. I have friends doing the same thing alone in their basements. Either group could win this race.&lt;/p&gt;
&lt;div class="right-sidenote"&gt;&lt;p&gt;I am not a fan of the OWASP Top 10.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;I&amp;rsquo;m stuck on vulnerability scanners  because I&amp;rsquo;m a security nerd. But also because it crystallizes interesting agent design decisions. For instance: you can write a loop feeding each file in a repository to an LLM agent. Or, as we saw with the ping example, you can let the LLM agent figure out what files to look at. You can write an agent that checks a file for everything in, say, the OWASP Top 10. Or you can have specific agent loops for DOM integrity, SQL injection, and authorization checking. You can seed your agent loop with raw source content. Or you can build an agent loop that builds an index of functions across the tree.&lt;/p&gt;

&lt;p&gt;You don&amp;rsquo;t know what works best until you try to write the agent.&lt;/p&gt;

&lt;p&gt;I&amp;rsquo;m too spun up by this stuff, I know. But look at the tradeoff you get to make here. Some loops you write explicitly. Others are summoned from a Lovecraftian tower of inference weights. The dial is yours to turn. Make things too explicit and your agent will never surprise you, but also, it&amp;rsquo;ll never surprise you. Turn the dial to 11 and it will surprise you to death.&lt;/p&gt;

&lt;p&gt;Agent designs implicate a bunch of open software engineering problems:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;How to balance unpredictability against structured programming without killing the agent&amp;rsquo;s ability to problem-solve; in other words, titrating in just the right amount of nondeterminism.
&lt;/li&gt;&lt;li&gt;How best to connect agents to ground truth so they can&amp;rsquo;t lie to themselves about having solved a problem to early-exit their loops.
&lt;/li&gt;&lt;li&gt;How to connect agents (which, again, are really just arrays of strings with a JSON configuration blob tacked on) to do multi-stage operation, and what the most reliable intermediate forms are (JSON blobs? SQL databases? Markdown summaries) for interchange between them
&lt;/li&gt;&lt;li&gt;How to allocate tokens and contain costs.
&lt;/li&gt;&lt;/ul&gt;

&lt;p&gt;I&amp;rsquo;m used to spaces of open engineering problems that aren&amp;rsquo;t amenable to individual noodling. Reliable multicast. Static program analysis. Post-quantum key exchange. So I&amp;rsquo;ll own it up front that I&amp;rsquo;m a bit hypnotized by open problems that, like it or not, are now central to our industry and are, simultaneously, likely to be resolved in someone&amp;rsquo;s basement. It&amp;rsquo;d be one thing if exploring these ideas required a serious commitment of time and material. But each productive iteration in designing these kinds of systems is the work of 30 minutes.&lt;/p&gt;

&lt;p&gt;Get on this bike and push the pedals. Tell me you hate it afterwards, I&amp;rsquo;ll respect that. In fact, I&amp;rsquo;m psyched to hear your reasoning. But I don&amp;rsquo;t think anybody starts to understand this technology until they&amp;rsquo;ve built something with it.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Corrosion</title>
    <link rel="alternate" href="https://fly.io/blog/corrosion/"/>
    <id>https://fly.io/blog/corrosion/</id>
    <published>2025-10-22T00:00:00+00:00</published>
    <updated>2026-09-30T00:06:54+00:00</updated>
    <media:thumbnail url="https://fly.io/blog/corrosion/assets/sqlite-cover.webp"/>
    <content type="html">&lt;div class="lead"&gt;&lt;p&gt;Fly.io transmogrifies Docker containers into Fly Machines: micro-VMs running on our own hardware all over the world. The hardest part of running this platform isn’t managing the servers, and it isn’t operating the network; it’s gluing those two things together.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;Several times a second, as customer CI/CD pipelines tear up or bring down &lt;a href='https://fly.io/machines/' title=''&gt;Fly Machines&lt;/a&gt;, our state synchronization system blasts updates across our internal mesh, so that edge proxies from Tokyo to Amsterdam can keep the accurate routing table that allows them to route requests for applications to the nearest customer instances.&lt;/p&gt;

&lt;p&gt;On September 1, 2024, at 3:30PM EST, a new Fly Machine came up with a new &amp;ldquo;virtual service&amp;rdquo; configuration option a developer had just shipped. Within a few seconds every proxy in our fleet had locked up hard. It was the worst outage we&amp;rsquo;ve experienced: a period during which no end-user requests could reach our customer apps at all.&lt;/p&gt;

&lt;p&gt;Distributed systems are blast amplifiers. By propagating data across a network, they also propagate bugs in the systems that depend on that data. In the case of Corrosion, our state distribution system, those bugs propagate &lt;strong class='font-semibold text-navy-950'&gt;quickly&lt;/strong&gt;. The proxy code that handled that Corrosion update had succumbed to a &lt;a href='https://news.ycombinator.com/item?id=42093551' title=''&gt;notorious Rust concurrency footgun&lt;/a&gt;: an &lt;code&gt;if let&lt;/code&gt; expression over an &lt;code&gt;RWLock&lt;/code&gt; assumed (reasonably, but incorrectly) in its &lt;code&gt;else&lt;/code&gt; branch that the lock had been released. Instant and virulently contagious deadlock.&lt;/p&gt;

&lt;p&gt;A lesson we&amp;rsquo;ve learned the hard way: never trust a distributed system without an interesting failure story. If a distributed system hasn&amp;rsquo;t ruined a weekend or kept you up overnight, you don&amp;rsquo;t understand it yet. Which is why that&amp;rsquo;s how we&amp;rsquo;re introducing Corrosion, an unconventional service discovery system we built for our platform &lt;a href='https://github.com/superfly/corrosion' title=''&gt;and  open sourced&lt;/a&gt;.&lt;/p&gt;
&lt;h2 id='our-face-seeking-rake' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#our-face-seeking-rake' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Our Face-Seeking Rake&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;State synchronization is the hardest problem in running a platform like ours. So why build a risky new distributed system for it? Because no matter what we try, that rake is waiting for our foot. The reason is our orchestration model.&lt;/p&gt;

&lt;p&gt;Virtually every mainstream orchestration system (including Kubernetes) relies on a centralized database to make decisions about where to place new workloads. Individual servers keep track of what they&amp;rsquo;re running, but that central database is the source of truth. At Fly.io, in order to scale across dozens of regions globally, &lt;a href='https://fly.io/blog/carving-the-scheduler-out-of-our-orchestrator/' title=''&gt;we flip that notion on its head&lt;/a&gt;: individual servers are the source of truth for their workloads.&lt;/p&gt;

&lt;p&gt;In our platform, our central API bids out work to what is in effect a global market of competing &amp;ldquo;worker&amp;rdquo; physical servers. By moving the authoritative source of information from a central scheduler to individual servers, we scale out without bottlenecking on a database that demands both responsiveness and consistency between São Paulo, Virginia, and Sydney.&lt;/p&gt;

&lt;p&gt;The bidding model is elegant, but it&amp;rsquo;s insufficient to route network requests. To allow an HTTP request in Tokyo to find the nearest instance in Sydney, we really do need some kind of global map of every app we host.&lt;/p&gt;

&lt;p&gt;For longer than we should have, we relied on &lt;a href='https://github.com/hashicorp/consul' title=''&gt;HashiCorp Consul&lt;/a&gt; to route traffic. Consul is fantastic software. Don&amp;rsquo;t build a global routing system on it. Then we &lt;a href='https://fly.io/blog/a-foolish-consistency/' title=''&gt;built SQLite caches of Consul&lt;/a&gt;. SQLite: also fantastic. But don&amp;rsquo;t do this either.&lt;/p&gt;

&lt;p&gt;Like an unattended turkey deep frying on the patio, truly global distributed consensus promises deliciousness while yielding only immolation. &lt;a href='https://raft.github.io/' title=''&gt;Consensus protocols like Raft &lt;/a&gt;break down over long distances. And they work against the architecture of our platform: our Consul cluster, running on the biggest iron we could buy, wasted time guaranteeing consensus for updates that couldn&amp;rsquo;t conflict in the first place.&lt;/p&gt;
&lt;h2 id='corrosion' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#corrosion' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Corrosion&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;To build a global routing database, we moved away from distributed consensus and took cues from actual routing protocols.&lt;/p&gt;

&lt;p&gt;&lt;a href='https://en.wikipedia.org/wiki/Open_Shortest_Path_First' title=''&gt;A protocol like OSPF&lt;/a&gt; has the same operating model and many of the same constraints we do. OSPF is a &amp;ldquo;&lt;a href='https://en.wikipedia.org/wiki/Link-state_routing_protocol' title=''&gt;link-state routing protocol&lt;/a&gt;&amp;rdquo;, which, conveniently for us, means that routers are sources of truth for their own links and responsible for quickly communicating  changes to every other router, so the network can make forwarding decisions.&lt;/p&gt;

&lt;p&gt;We have things easier than OSPF does. Its flooding algorithm can&amp;rsquo;t assume connectivity between arbitrary routers (solving that problem is the point of OSPF). But we run a global, fully connected WireGuard mesh between our servers. All we need to do is gossip efficiently.&lt;/p&gt;

&lt;p&gt;&lt;a href='https://github.com/superfly/corrosion' title=''&gt;Corrosion is a Rust program&lt;/a&gt; that propagates a SQLite database with a gossip protocol.&lt;/p&gt;

&lt;p&gt;Like Consul, our gossip protocol is &lt;a href='https://fly.io/blog/building-clusters-with-serf/#what-serf-is-doing' title=''&gt;built on SWIM&lt;/a&gt;. Start with the simplest, dumbest group membership protocol you can imagine: every node spams every node it learns about with heartbeats. Now, just two tweaks: first, each step of the protocol, spam a random subset of nodes, not the whole set. Then, instead of freaking out when a heartbeat fails, mark it &amp;ldquo;suspect&amp;rdquo; and ask another random subset of neighbors to ping it for you. SWIM converges on global membership very quickly.&lt;/p&gt;

&lt;p&gt;Once membership worked out, we run QUIC between nodes in the cluster to broadcast changes and reconcile state for new nodes.&lt;/p&gt;

&lt;p&gt;Corrosion looks like a globally synchronized database. You can open it with SQLite and just read things out of its tables. What makes it interesting is what it doesn&amp;rsquo;t do: no locking, no central servers, and no distributed consensus. Instead, we exploit our orchestration model: workers own their own state, so updates from different workers almost never conflict.&lt;/p&gt;

&lt;p&gt;We do impose some order. Every node in a Corrosion cluster will eventually receive the same set of updates, in some order. To ensure every instance arrives at the same &amp;ldquo;working set&amp;rdquo; picture, we use &lt;a href='https://github.com/vlcn-io/cr-sqlite' title=''&gt;cr-sqlite, the CRDT SQLite extension&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;cr-sqlite works by marking specified SQLite tables as CRDT-managed. For these table, changes to any column of a row are logged in a special &lt;code&gt;crsql_changes&lt;/code&gt;table. Updates to tables are applied last-write-wins using logical timestamps (that is, causal ordering rather than wall-clock ordering). &lt;a href='https://github.com/superfly/corrosion/blob/main/doc/crdts.md' title=''&gt;You can read much more about how that works here&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;As rows are updated in Corrosion&amp;rsquo;s ordinary SQL tables, the resulting changes are collected from &lt;code&gt;crsql_changes&lt;/code&gt;. They&amp;rsquo;re bundled into batched update packets and gossiped.&lt;/p&gt;

&lt;p&gt;When things are going smoothly, Corrosion is easy to reason about. Many customers of Corrosion&amp;rsquo;s data don&amp;rsquo;t even need to know it exists, just where the database is. We don&amp;rsquo;t fret over &amp;ldquo;leader elections&amp;rdquo; or bite our nails watching metrics for update backlogs. And it&amp;rsquo;s fast as all get-out.&lt;/p&gt;
&lt;h2 id='shit-happens' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#shit-happens' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Shit Happens&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;This is a story about how we made one good set of engineering decisions and &lt;a href='https://how.complexsystems.fail/' title=''&gt;never experienced any problems&lt;/a&gt;. &lt;a href='https://www.somethingsimilar.com/2013/01/14/notes-on-distributed-systems-for-young-bloods/' title=''&gt;Please clap&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;We told you already about the worst problem Corrosion was involved with: efficiently gossiping a deadlock bug to every proxy in our fleet, shutting our whole network down. Really, Corrosion was just a bystander for that outage. But it perpetrated others.&lt;/p&gt;

&lt;p&gt;Take a classic ops problem: the unexpectedly expensive DDL change. You wrote a simple migration, tested it, merged it to main, and went to bed, wrongly assuming the migration wouldn&amp;rsquo;t cause an outage when it ran in prod. Happens to the best of us.&lt;/p&gt;

&lt;p&gt;Now spice it up. You made a trivial-seeming schema change to a CRDT table hooked up to a global gossip system. Now, when the deploy runs, thousands of high-powered servers around the world join a chorus of database reconciliation messages that melts down the entire cluster.&lt;/p&gt;

&lt;p&gt;That happened to us last year when a team member added a nullable column to a Corrosion table. New nullable columns are kryptonite to large Corrosion tables: &lt;code&gt;cr-sqlite&lt;/code&gt; needs to backfill values for every row in the table. It played out as if every Fly Machine on our platform had suddenly changed state simultaneously, just to fuck us.&lt;/p&gt;

&lt;p&gt;Gnarlier war story: for a long time we ran both Corrosion and Consul, because two distributed systems means twice the resiliency. One morning, a Consul mTLS certificate expired. Every worker in our fleet severed its connection to Consul.&lt;/p&gt;

&lt;p&gt;We should have been fine. We had Corrosion running. Except: under the hood, every worker in the fleet is doing a backoff loop trying to reestablish connectivity to Consul. Each of those attempts re-invokes a code path to update Fly Machine state. That code path incurs a Corrosion write.&lt;/p&gt;

&lt;p&gt;By the time we&amp;rsquo;ve figured out what the hell is happening, we&amp;rsquo;re literally saturating our uplinks almost everywhere in our fleet. We apologize to our uplink providers.&lt;/p&gt;

&lt;p&gt;It&amp;rsquo;s been a long time since anything like this has happened at Fly.io, but preventing the next one is basically all we think about anymore.&lt;/p&gt;
&lt;h2 id='iteration' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#iteration' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Iteration&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;In retrospect, our Corrosion rollout repeated a mistake we made with Consul: we built a single global state domain. Nothing about Corrosion&amp;rsquo;s design required us to do this, and we&amp;rsquo;re unwinding that decision now. Hold that thought. We got some big payoffs from some smaller lifts.&lt;/p&gt;

&lt;p&gt;First, and most importantly, we watchdogged everything. We showed you a contagious deadlock bug, lethal because our risk model was missing &amp;ldquo;these Tokio programs might deadlock&amp;rdquo;. Not anymore. Our &lt;a href='https://tokio.rs/' title=''&gt;Tokio programs&lt;/a&gt; all have built-in watchdogs; an event-loop stall will bounce the service and make a king-hell alerting racket. Watchdogs have cancelled multiple outages. Minimal code, easy win. Do this in your own systems.&lt;/p&gt;

&lt;p&gt;Then, we extensively tested Corrosion itself. We&amp;rsquo;ve written about &lt;a href='https://fly.io/blog/parking-lot-ffffffffffffffff/' title=''&gt;a bug we found in the Rust &lt;code&gt;parking_lot&lt;/code&gt; library&lt;/a&gt;. We spent months looking for similar bugs &lt;a href='https://antithesis.com/product/how_antithesis_works/' title=''&gt;with Antithesis&lt;/a&gt;. Again: do recommend. It retraced our steps on the &lt;code&gt;parking_lot&lt;/code&gt; bug easily; the bug wouldn&amp;rsquo;t have been worth the blog post if we&amp;rsquo;d been using Antithesis at the time. &lt;a href='https://antithesis.com/docs/multiverse_debugging/overview/' title=''&gt;Multiverse debugging&lt;/a&gt; is killer for distributed systems.&lt;/p&gt;

&lt;p&gt;No amount of testing will make us trust a distributed system. So we&amp;rsquo;ve made it simpler to rebuild Corrosion&amp;rsquo;s database from our workers. We keep checkpoint backups of the Corrosion database on object storage. That was smart of us. When shit truly went haywire last year, we had the option to reboot the cluster, which is ultimately what we did. That eats some time (the database is large and propagating is expensive), but diagnosing and repairing distributed systems mishaps takes even longer.&lt;/p&gt;

&lt;p&gt;We&amp;rsquo;ve also improved the way our workers feed Corrosion. Until recently, any time a worker updated its local database, we published the same incremental update to Corrosion. &lt;a href='https://community.fly.io/t/self-healing-machine-state-synchronization-and-service-discovery/26134' title=''&gt;But now we&amp;rsquo;ve eliminated partial updates.&lt;/a&gt; Instead, when a Fly Machine changes, we re-publish the entire data set for the Machine. Because of how Corrosion resolves changes to its own rows, the node receiving the re-published Fly Machine automatically filters out the no-op changes before gossiping them. Eliminating partial updates forecloses a bunch of bugs (and, we think, kills off a couple sneaky ones we&amp;rsquo;ve been chasing). We should have done it this way to begin with.&lt;/p&gt;

&lt;p&gt;Finally, let&amp;rsquo;s revisit that global state problem. After the contagious deadlock bug, we concluded we need to evolve past a single cluster. So we took on a project we call &amp;ldquo;regionalization&amp;rdquo;, which creates a two-level database scheme. Each region we operate in runs a Corrosion cluster with fine-grained data about every Fly Machine in the region. The global cluster then maps applications to regions, which is sufficient to make forwarding decisions at our edge proxies.&lt;/p&gt;

&lt;p&gt;Regionalization reduces the blast radius of state bugs. Most things we track don&amp;rsquo;t have to matter outside their region (importantly, most of the code changes to what we track are also region-local). We can roll out changes to this kind of code in ways that, worst case, threaten only a single region.&lt;/p&gt;
&lt;h2 id='the-new-system-works' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#the-new-system-works' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;The New System Works&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Most distributed systems have state synchronization challenges.  Corrosion has a different &amp;ldquo;shape&amp;rdquo; than most of those systems:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;It doesn&amp;rsquo;t rely on distributed consensus, like &lt;a href='https://github.com/hashicorp/consul' title=''&gt;Consul&lt;/a&gt;, &lt;a href='https://zookeeper.apache.org/' title=''&gt;Zookeeper&lt;/a&gt;, &lt;a href='https://etcd.io/' title=''&gt;Etcd&lt;/a&gt;, &lt;a href='https://www.cockroachlabs.com/docs/stable/architecture/replication-layer' title=''&gt;Raft&lt;/a&gt;, or &lt;a href='https://rqlite.io/' title=''&gt;rqlite&lt;/a&gt; (which we came very close to using).
&lt;/li&gt;&lt;li&gt;It doesn&amp;rsquo;t rely on a large-scale centralized data store, like &lt;a href='https://www.foundationdb.org/' title=''&gt;FoundationDB&lt;/a&gt; or databases backed by S3-style object storage.
&lt;/li&gt;&lt;li&gt;It&amp;rsquo;s nevertheless highly distributed (each of thousands of workers run nodes), converges quickly (in seconds), and presents as a simple SQLite database. Neat!
&lt;/li&gt;&lt;/ul&gt;

&lt;p&gt;It wasn&amp;rsquo;t easy getting here. Corrosion is a large part of what every engineer at Fly.io who writes Rust works on.&lt;/p&gt;

&lt;p&gt;Part of what&amp;rsquo;s making Corrosion work is that we&amp;rsquo;re careful about what we put into it. Not every piece of state we manage needs gossip propagation. &lt;code&gt;tkdb&lt;/code&gt;, the backend for &lt;a href='https://fly.io/blog/macaroons-escalated-quickly/' title=''&gt;our Macaroon tokens&lt;/a&gt;, is a much simpler SQLite service backed by &lt;a href='https://litestream.io/' title=''&gt;Litestream&lt;/a&gt;. So is Pet Sematary, the secret store we built to replace HashiCorp Vault.&lt;/p&gt;

&lt;p&gt;Still, there are probably lots of distributed state problems that want something more like a link-state routing protocol and less like a distributed database. If you think you might have one of those, &lt;a href='https://github.com/superfly/corrosion' title=''&gt;feel free to take Corrosion for a spin&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Corrosion is Jérôme Gravel-Niquet&amp;rsquo;s brainchild. For the last couple years, much of the iteration on it was led by Somtochi Onyekwere and Peter Cai. The work was alternately cortisol- and endorphin-inducing. We&amp;rsquo;re glad to finally get to talk about it in detail.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Kurt Got Got</title>
    <link rel="alternate" href="https://fly.io/blog/kurt-got-got/"/>
    <id>https://fly.io/blog/kurt-got-got/</id>
    <published>2025-10-08T00:00:00+00:00</published>
    <updated>2026-10-22T11:27:19+00:00</updated>
    <media:thumbnail url="https://fly.io/blog/kurt-got-got/assets/Kurt_Got_Got.webp"/>
    <content type="html">&lt;div class="lead"&gt;&lt;p&gt;The $FLY Airdrop is live!  Claim your share of &lt;a href="https://fly.io/blog/macaroons-escalated-quickly/" title=""&gt;the token powering Fly.io’s global network&lt;/a&gt; of 3M+ apps and (🤮) own a piece of the sky!&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;We know. Our Twitter got owned. We knew within moments of it happening. We know exactly how it happened. Nothing was at risk other than our Twitter account (and one Fly.io employee&amp;rsquo;s self-esteem).  Also: for fuck&amp;rsquo;s sake.&lt;/p&gt;

&lt;p&gt;Here&amp;rsquo;s what happened: Kurt Mackey, our intrepid CEO, got phished.&lt;/p&gt;
&lt;div class="callout"&gt;&lt;p&gt;Had this been an impactful attack, we would not be this flippant about it. For this, though, any other tone on our part would be false.&lt;/p&gt;
&lt;/div&gt;&lt;h2 id='how-they-got-kurt' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#how-they-got-kurt' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;How They Got Kurt&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Two reasons: one, it was a pretty good phishing attack, and two, Twitter fell outside the &amp;ldquo;things we take seriously&amp;rdquo; boundary.&lt;/p&gt;

&lt;p&gt;The phishing attack was effective because it exploited a deep psychological vulnerability in our management team: we are old and out of touch with the youths of today.&lt;/p&gt;

&lt;p&gt;For many months now, we&amp;rsquo;ve had an contractor/intern-type-person Boosting Our Brand on Twitter by posting dank developer memes (I think that&amp;rsquo;s what they&amp;rsquo;re called). The thing about this dankery is that we don&amp;rsquo;t really understand it. I mean, hold on, we know what the memes mean technically. We just don&amp;rsquo;t get why they&amp;rsquo;re funny.&lt;/p&gt;

&lt;p&gt;However, in pushing back on them, we&amp;rsquo;re up against two powerful forces:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The dank memes appear to perform better than the stuff we ourselves write on Twitter.
&lt;/li&gt;&lt;li&gt;We are reliably informed by our zoomer children that we are too cringe to be trusted on these matters.
&lt;/li&gt;&lt;/ol&gt;

&lt;p&gt;Here&amp;rsquo;s the phish Kurt got:&lt;/p&gt;

&lt;p&gt;&lt;img loading="lazy" decoding="async" alt="A pretty-plausible Twitter alert" width="792" height="1278" src="/blog/kurt-got-got/assets/phish.webp?2/3&amp;amp;center" /&gt;&lt;/p&gt;

&lt;p&gt;Diabolical. Like a scalpel expertly wielded against Kurt&amp;rsquo;s deepest &lt;a href='https://theonion.com/cool-dad-raising-daughter-on-media-that-will-put-her-en-1819572981/' title=''&gt;middle-aged-dude&lt;/a&gt; insecurity. Our ruthless attackers clinically designed this email to trigger an autonomic Kurt response: &amp;ldquo;oh, what the fuck is this, and why did we post it?&amp;rdquo;&lt;/p&gt;
&lt;div class="right-sidenote"&gt;&lt;p&gt;ATO is cool-kid for “got owned”&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;I&amp;rsquo;m getting a little ahead of the story here. We knew our X.com account had suffered an ATO because a bunch of us simultaneously got another email saying that the &lt;a href='https://twitter.com/flydotio' title=''&gt;@flydotio&lt;/a&gt; account&amp;rsquo;s email address now pointed to &lt;code&gt;achilles19969@gmail.com&lt;/code&gt;. Our immediate response was to audit all accesses to the login information in &lt;a href='https://1password.com/' title=''&gt;1Password&lt;/a&gt;, to cut all access for anybody who&amp;rsquo;d recently pulled it; your worst-case assumption in a situation like this is that someone&amp;rsquo;s endpoint has been owned up.&lt;/p&gt;

&lt;p&gt;Fortunately, nobody lost access for very long. I called Kurt to let him know why he was being locked out, and 5 seconds later, he&amp;rsquo;d &lt;a href='https://archive.is/6rVqf' title=''&gt;realized what had happened.&lt;/a&gt; &lt;strong class='font-semibold text-navy-950'&gt;Don&amp;rsquo;t click anything there.&lt;/strong&gt;&lt;/p&gt;
&lt;h2 id='why-it-worked' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#why-it-worked' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Why It Worked&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;That&amp;rsquo;s the right question to ask, isn&amp;rsquo;t it? How could this have been possible in the first place?&lt;/p&gt;

&lt;p&gt;Contrary to one popular opinion, you don&amp;rsquo;t defeat phishing by training people not to click on things. I mean, tell them not to, sure! But eventually, under continued pressure, everybody clicks. &lt;a href='https://people.cs.uchicago.edu/~grantho/papers/oakland2025_phishing-training.pdf' title=''&gt;There&amp;rsquo;s science on this&lt;/a&gt;. The cool kids haven&amp;rsquo;t done phishing simulation training in years.&lt;/p&gt;

&lt;p&gt;What you&amp;rsquo;re supposed to do instead is use phishing-resistant authentication. This is almost the whole backstory for &lt;a href='https://www.imperialviolet.org/tourofwebauthn/tourofwebauthn.html' title=''&gt;U2F,  FIDO2&lt;/a&gt; and &lt;a href='https://support.apple.com/en-us/102195' title=''&gt;Passkeys&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Phishing-resistant authentication works by mutual authentication (or, if you&amp;rsquo;re a stickler, by origin- and channel-binding). Phishes are malicious proxies for credentials. Modern MFA schemes like FIDO2 break that proxy flow; your browser won&amp;rsquo;t send real credentials to the fake site.&lt;/p&gt;
&lt;div class="right-sidenote"&gt;&lt;p&gt;there’s more to it than this, but, broad strokes.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;This is, in fact, how all of our infrastructure is secured at Fly.io; specifically, we get &lt;a href='https://fly.io/blog/soc2-the-screenshots-will-continue-until-security-improves/#what-soc2-made-us-do' title=''&gt;everything behind an IdP&lt;/a&gt; (in our case: Google&amp;rsquo;s) and have it require phishing-proof MFA. You&amp;rsquo;re unlikely to phish your way to viewing logs here, or to refunding a customer bill at Stripe, or to viewing infra metrics, because all these things require an SSO login through Google.&lt;/p&gt;

&lt;p&gt;Twitter, on the other hand. Yeah, so, about that. You may have heard that, a few years back, there were some goings-on involving Twitter. Many of us at Fly.io &lt;a href='https://hachyderm.io/@flydotio' title=''&gt;decamped for Mastodon&lt;/a&gt;, and &lt;a href='https://bsky.app/profile/did:plc:j7herf6n4xiig2yg7fqdmkci' title=''&gt;later to Bluesky.&lt;/a&gt; There was a window of time in 2023-2024 where it looked as if Twitter might not be a long term thing for us at all.&lt;/p&gt;
&lt;div class="right-sidenote"&gt;&lt;p&gt;† (to whom I sincerely apologize for having assumed they had been owned up and were the proximate cause of the hack)&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;As a result, Twitter had been a sort of legacy shared account for us, with credentials managed in 1Password and shared with our zoomer contractor†.&lt;/p&gt;

&lt;p&gt;Which is why Kurt was in a position to pull credentials from 1Password and log in to members-x.com in response to an email from alerts-x.com.&lt;/p&gt;
&lt;div class="callout"&gt;&lt;p&gt;Still: we could have dodged this attack with hygiene: Kurt complains that “x.com” is an extremely phishable domain, and, sure, but also: the 1Password browser plugin would have noticed that “members-x.com” wasn’t an “x.com” host.&lt;/p&gt;
&lt;/div&gt;&lt;h2 id='what-took-so-long' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#what-took-so-long' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;What Took So Long&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;The attacker immediately revoked all tokens and set up new 2FA, so while we were quickly able to reset our password, we couldn&amp;rsquo;t lock them out of our account without an intervention from X.com, which took something like 1 5 hours to set up.&lt;/p&gt;

&lt;p&gt;(That&amp;rsquo;s not a knock on X.com; 15 hours for a 2FA reset isn&amp;rsquo;t outside industry norms).&lt;/p&gt;

&lt;p&gt;We&amp;rsquo;re obviously making a lot of noise about this now, but we were pretty quiet during the incident itself (beyond just &amp;ldquo;We know. We knew 45 seconds after it happened. We know exactly how it happened. It&amp;rsquo;s just a Twitter thing.&amp;rdquo;)&lt;/p&gt;

&lt;p&gt;That&amp;rsquo;s because, in the grand scheme of things, the attack was pretty chill: &lt;a href='https://archive.is/PTO2M' title=''&gt;a not-very-plausible crypto scam&lt;/a&gt; that presumably generated $0 for the attackers, 15+ hours of &lt;code&gt;brand damage&lt;/code&gt;, and extra security engineering cycles burnt on watchful waiting. Our users weren&amp;rsquo;t under attack, and the account wasn&amp;rsquo;t being used to further intercept customer accounts. At one point, the attackers apparently deleted our whole Twitter history, which, like, don&amp;rsquo;t threaten us with a good time. So we let it roll, until we got our account recovered the next morning.&lt;/p&gt;
&lt;h2 id='the-moral-of-the-story-is' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#the-moral-of-the-story-is' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;The Moral Of The Story Is&lt;/span&gt;&lt;/h2&gt;&lt;div class="right-sidenote"&gt;&lt;p&gt;“Really the biggest takeaway for me is that Kurt reads his email.”&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;Obviously Kurt loses his commit access. The time comes in the life of every CEO, and now it comes for him. &lt;/p&gt;

&lt;p&gt;Also, we&amp;rsquo;ll finally have a population sample for &amp;ldquo;incident response&amp;rdquo; in &lt;a href='https://fly.io/blog/soc2-the-screenshots-will-continue-until-security-improves/' title=''&gt;our next SOC2&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Maybe we&amp;rsquo;ll post more on Twitter. Or maybe we&amp;rsquo;ll double down on Zoomer memes. I don&amp;rsquo;t know. Social media is really weird right now. Either way: our Twitter access is Passkeys now.&lt;/p&gt;
&lt;div class="right-sidenote"&gt;&lt;p&gt;seriously don’t click anything on that page&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;If you were inclined to take us up on an &amp;ldquo;airdrop&amp;rdquo; to &amp;ldquo;claim a share&amp;rdquo; of the &amp;ldquo;token&amp;rdquo; powering Fly.io, the site is &lt;a href='https://archive.is/PTO2M' title=''&gt;still up&lt;/a&gt;. You can connect your wallet it it! You&amp;rsquo;ll lose all your money. But if we&amp;rsquo;d actually done an ICO, you&amp;rsquo;d have lost all your money anyways.&lt;/p&gt;

&lt;p&gt;Somebody involved in pulling this attack off had to come up with &amp;ldquo;own a piece of the sky!&amp;rdquo;, and I think that&amp;rsquo;s punishment enough for them.&lt;/p&gt;

&lt;p&gt;Whatever you&amp;rsquo;re operating that isn&amp;rsquo;t behind phishing-resistant MFA, or, better yet, an SSO IdP that requires phishing-resistant MFA: that thing is eventually going to get phished. Dance around the clown-fire of our misfortune if you must, but let us be a lesson to you as well.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Litestream v0.5.0 is Here</title>
    <link rel="alternate" href="https://fly.io/blog/litestream-v050-is-here/"/>
    <id>https://fly.io/blog/litestream-v050-is-here/</id>
    <published>2025-10-02T00:00:00+00:00</published>
    <updated>2026-09-22T09:57:11+00:00</updated>
    <media:thumbnail url="https://fly.io/blog/litestream-v050-is-here/assets/litestream-v050-is-here.webp"/>
    <content type="html">&lt;div class="lead"&gt;&lt;p&gt;&lt;strong class="font-semibold text-navy-950"&gt;I’m Ben Johnson, and I work on Litestream at Fly.io. Litestream makes it easy to build SQLite-backed full-stack applications  with resilience to server failure. It’s open source, runs anywhere, and&lt;/strong&gt; &lt;a href="https://litestream.io/" title=""&gt;&lt;strong class="font-semibold text-navy-950"&gt;it’s easy to get started&lt;/strong&gt;&lt;/a&gt;&lt;strong class="font-semibold text-navy-950"&gt;.&lt;/strong&gt;&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;Litestream is the missing backup/restore system for SQLite. It runs as a sidecar process in the background, alongside unmodified SQLite applications, intercepting WAL checkpoints and streaming them to object storage in real time. Your application doesn&amp;rsquo;t even know it&amp;rsquo;s there. But if your server crashes, Litestream lets you quickly restore the database to your new hardware.&lt;/p&gt;

&lt;p&gt;The result: you can safely build whole full-stack applications on top of SQLite.&lt;/p&gt;

&lt;p&gt;A few months back, we announced &lt;a href='https://fly.io/blog/litestream-revamped/' title=''&gt;plans for a major update to Litestream&lt;/a&gt;. I&amp;rsquo;m psyched to announce that the first batch of those changes are now &amp;ldquo;shipping&amp;rdquo;. Litestream is  faster and now supports efficient point-in-time recovery (PITR).&lt;/p&gt;

&lt;p&gt;I&amp;rsquo;m going to take a beat to recap Litestream and how we got here, then talk about how these changes work and what you can expect to see with them.&lt;/p&gt;
&lt;h2 id='litestream-to-litefs-to-litestream' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#litestream-to-litefs-to-litestream' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Litestream to LiteFS to Litestream&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Litestream is one of two big SQLite things I&amp;rsquo;ve built. The other one, originally intended as a sort of sequel to Litestream, is LiteFS.&lt;/p&gt;

&lt;p&gt;Boiled down to a sentence: LiteFS uses a FUSE filesystem to crawl further up into SQLite&amp;rsquo;s innards, using that access to perform live replication, for unmodified SQLite-backed apps.&lt;/p&gt;

&lt;p&gt;The big deal about LiteFS for us is that it lets you do the multiregion primary/read-replica deployment people love Postgres for: reads are fast everywhere, and writes are sane and predictable. We were excited to make this possible for SQLite, too.&lt;/p&gt;

&lt;p&gt;But the market has spoken! Users prefer Litestream. And honestly, we get it: Litestream is easier to run and to reason about. So we&amp;rsquo;ve shifted our focus back to it. First order of business: &lt;a href='https://fly.io/blog/litestream-revamped/' title=''&gt;take what we learned building LiteFS and stick as much of it as we can back into Litestream&lt;/a&gt;.&lt;/p&gt;
&lt;h2 id='the-ltx-file-format' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#the-ltx-file-format' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;The LTX File Format&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Consider this basic SQL table:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative sql"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-1ztc3dpg"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-1ztc3dpg"&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;sandwiches&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="nb"&gt;INTEGER&lt;/span&gt; &lt;span class="k"&gt;PRIMARY&lt;/span&gt; &lt;span class="k"&gt;KEY&lt;/span&gt; &lt;span class="n"&gt;AUTOINCREMENT&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;description&lt;/span&gt; &lt;span class="nb"&gt;TEXT&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;star_rating&lt;/span&gt; &lt;span class="nb"&gt;INTEGER&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; 
    &lt;span class="n"&gt;reviewer_id&lt;/span&gt; &lt;span class="nb"&gt;INTEGER&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;
&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;In our hypothetical, this table backs a wildly popular sandwich-reviewing app that we keep trying to get someone to write. People eat a lot of sandwiches and this table gets a lot of writes. Because it makes my point even better and it&amp;rsquo;s funny, assume people dither a lot about their sandwich review for the first couple minutes after they leave it. This Quiznos sub… is it ⭐ or ⭐⭐?&lt;/p&gt;

&lt;p&gt;Underneath SQLite is a B-tree. Like databases everywhere, SQLite divides storage up into disk-aligned pages, working hard to read as few pages as possible for any task while treating work done within a page as more or less free. SQLite always reads and writes in page-sized chunks.&lt;/p&gt;

&lt;p&gt;Our &lt;code&gt;sandwiches&lt;/code&gt; table includes a feature that&amp;rsquo;s really painful for a tool like Litestream that thinks in pages: an automatically updating primary key. That key dictates that every insert into the table hits the rightmost leaf page in the underlying table B-tree. For SQLite itself, that&amp;rsquo;s no problem. But Litestream has less information to go on: it sees only a feed of whole pages it needs to archive.&lt;/p&gt;

&lt;p&gt;Worse still, when it comes time to restore the database – something you tend to want to happen quickly – you have to individually apply those small changes, as whole pages. Your app is down, PagerDuty is freaking out, and you&amp;rsquo;re sitting there watching Litestream reconstruct your Quiznos uncertainty a page (and an S3 fetch) at a time.&lt;/p&gt;

&lt;p&gt;So, LTX. Let me explain. We needed LiteFS to be transaction-aware. It relies on finer-grained information than just raw dirty pages (that&amp;rsquo;s why it needs the FUSE filesystem). To ship transactions, rather than pages, we invented a &lt;a href='https://github.com/superfly/ltx' title=''&gt;file format we call LTX&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;LTX was designed as an interchange format for transactions, but for our purposes in Litestream, all we care about is that LTX files represent ordered ranges of pages, and that it supports compaction.&lt;/p&gt;

&lt;p&gt;Compaction is straightforward. You&amp;rsquo;ve stored a bunch of LTX files that collect numbered pages. Now you want to to restore a coherent picture of the database. Just replay them newest to oldest, skipping duplicate pages (newer wins), until all changed pages are accounted for.&lt;/p&gt;

&lt;p&gt;Importantly, LTX isn&amp;rsquo;t limited to whole database backups. We can use LTX compaction to compress a bunch of LTX files into a single file with no duplicated pages. And Litestream now uses this capability to create a hierarchy of compactions:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;at Level 1, we compact all the changes in a 30-second time window
&lt;/li&gt;&lt;li&gt;at Level 2, all the Level 1 files in a 5-minute window
&lt;/li&gt;&lt;li&gt;at Level 3, all the Level 2&amp;rsquo;s over an hour.
&lt;/li&gt;&lt;/ul&gt;

&lt;p&gt;Net result: we can restore a SQLite database to any point in time, &lt;em&gt;using only a dozen or so files on average&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;Litestream performs this compaction itself. It doesn&amp;rsquo;t rely on SQLite to process the WAL file. Performance is limited only by I/O throughput.&lt;/p&gt;
&lt;h2 id='no-more-generations' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#no-more-generations' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;No More Generations&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;What people like about Litestream is that it&amp;rsquo;s just an ordinary Unix program. But like any Unix program, Litestream can crash. It&amp;rsquo;s not supernatural, so when it&amp;rsquo;s not running, it&amp;rsquo;s not seeing database pages change. When it misses changes, it falls out of sync with the database.&lt;/p&gt;

&lt;p&gt;Lucky for us, that&amp;rsquo;s easy to detect. When it notices a gap between the database and our running &amp;ldquo;shadow-WAL&amp;rdquo; backup, Litestream resynchronizes from scratch.&lt;/p&gt;

&lt;p&gt;The only time this gets complicated is if you have multiple Litestreams backing up to the same destination. To keep multiple Litestreams from stepping on each other, Litestream divides backups into &amp;ldquo;generations&amp;rdquo;, creating a new one any time it resyncs. You can think of generations as Marvel Cinematic Universe parallel dimensions in which your database might be simultaneously living in.&lt;/p&gt;

&lt;p&gt;Yeah, we didn&amp;rsquo;t like those movies much either.&lt;/p&gt;

&lt;p&gt;LTX-backed Litestream does away with the concept entirely. Instead, when we detect a break in WAL file continuity, we re-snapshot with the next LTX file. Now we have a monotonically incrementing transaction ID. We can use it look up database state at any point in time, without searching across generations.&lt;/p&gt;
&lt;h2 id='upgrading-to-litestream-v0-5-0' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#upgrading-to-litestream-v0-5-0' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Upgrading to Litestream v0.5.0&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Due to the file format changes, the new version of Litestream can&amp;rsquo;t restore from old v0.3.x WAL segment files.&lt;/p&gt;

&lt;p&gt;That&amp;rsquo;s OK though! The upgrade process is simple: just start using the new version. It&amp;rsquo;ll leave your old WAL files intact, in case you ever need to revert to the older version.The new LTX files are stored cleanly in an &lt;code&gt;ltx&lt;/code&gt; directory on your replica.&lt;/p&gt;

&lt;p&gt;The configuration file is fully backwards compatible.&lt;/p&gt;

&lt;p&gt;There&amp;rsquo;s one small catch. We added a new constraint. You only get a single replica destination per database. This probably won&amp;rsquo;t affect you, since it&amp;rsquo;s how most people use Litestream already. We&amp;rsquo;ve made it official.&lt;/p&gt;

&lt;p&gt;The rationale: having a single source of truth simplifies development for us, and makes the tool easier to reason about. Multiple replicas can diverge and are sensitive to network availability. Conflict resolution is brain surgery.&lt;/p&gt;

&lt;p&gt;Litestream commands still work the same. But you&amp;rsquo;ll see references to &amp;ldquo;transaction IDs&amp;rdquo; (TXID) for LTX files, rather than the &lt;code&gt;generation/index/offset&lt;/code&gt; we used previously with WAL segments.&lt;/p&gt;

&lt;p&gt;We&amp;rsquo;ve also changed &lt;code&gt;litestream wal&lt;/code&gt; to &lt;code&gt;litestream ltx&lt;/code&gt;.&lt;/p&gt;
&lt;h2 id='other-stuff-v0-5-0-does-better' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#other-stuff-v0-5-0-does-better' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Other Stuff v0.5.0 Does Better&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;We&amp;rsquo;ve beefed up the &lt;a href='https://github.com/superfly/ltx' title=''&gt;underlying LTX file format library&lt;/a&gt;. It used to be an LTX file was just a sorted list of pages, all compressed together. Now we compress per-page, and keep an index at the end of the LTX file to pluck individual pages out.&lt;/p&gt;

&lt;p&gt;You&amp;rsquo;re not seeing it yet, but we&amp;rsquo;re excited about this change: we can operate page-granularly even dealing with large LTX files. This allows for more features. A good example: we can build features that query from any point in time, without downloading the whole database.&lt;/p&gt;

&lt;p&gt;We&amp;rsquo;ve also gone back through old issues &amp;amp; PRs to improve quality-of-life. CGO is now gone. We&amp;rsquo;ve settled the age-old contest between &lt;code&gt;mattn/go-sqlite3&lt;/code&gt; and &lt;code&gt;modernc.org/sqlite&lt;/code&gt; in favor of &lt;code&gt;modernc.org&lt;/code&gt;. This is super handy for people with automated build systems that want to run from a MacBook but deploy on an x64 server, since it lets the cross-compiler work.&lt;/p&gt;

&lt;p&gt;We&amp;rsquo;ve also added a replica type for NATS JetStream. Users that already have JetStream running can get Litestream going without adding an object storage dependency.&lt;/p&gt;

&lt;p&gt;And finally, we&amp;rsquo;ve upgraded all our clients (S3, Google Storage, &amp;amp; Azure Blob Storage) to their latest versions. We&amp;rsquo;ve also moved our code to support newer S3 APIs.&lt;/p&gt;
&lt;h2 id='whats-next' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#whats-next' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;What&amp;rsquo;s next?&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;The next major feature we&amp;rsquo;re building out is a Litestream VFS for read replicas. This will let you instantly spin up a copy of the database and immediately read pages from S3 while the rest of the database is hydrating in the background.&lt;/p&gt;

&lt;p&gt;We already have a proof of concept working and we&amp;rsquo;re excited to show it off when it&amp;rsquo;s ready!&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Build Better Agents With MorphLLM</title>
    <link rel="alternate" href="https://fly.io/blog/build-better-agents-with-morphllm/"/>
    <id>https://fly.io/blog/build-better-agents-with-morphllm/</id>
    <published>2025-08-25T00:00:00+00:00</published>
    <updated>2025-08-26T09:28:09+00:00</updated>
    <media:thumbnail url="https://fly.io/blog/build-better-agents-with-morphllm/assets/morphllm.webp"/>
    <content type="html">&lt;p&gt;I&amp;rsquo;m an audiophile, which is a nice way to describe someone who spends their children&amp;rsquo;s college fund on equipment that yields no audible improvement in sound quality. As such, I refused to use wireless headphones for the longest time. The fun thing about wired headphones is when you forget they&amp;rsquo;re on and you stand up, you simultaneously cause irreparable neck injuries and extensive property damage. This eventually prompted me to buy good wireless headphones and, you know what, I break fewer things now. I can also stand up from my desk and not be exposed to the aural horrors of the real world. &lt;/p&gt;

&lt;p&gt;This is all to say, sometimes you don&amp;rsquo;t know how big a problem is until you solve it. This week, I chatted to the fine people building &lt;a href='https://morphllm.com/' title=''&gt;MorphLLM&lt;/a&gt;, which is exactly that kind of solution for AI agent builders. &lt;/p&gt;
&lt;h2 id='slow-wasteful-and-expensive-ai-code-changes' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#slow-wasteful-and-expensive-ai-code-changes' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Slow, Wasteful and Expensive AI Code Changes&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;If you’re building AI agents that write or edit code, you’re probably accepting the following as &amp;ldquo;the way it is&amp;rdquo;: Your agent needs to correct a single line of code, but rewrites an entire file to do it. Search-and-replace right? It’s fragile, breaks formatting, silently fails, or straight up leaves important functions out. The result is slow, inaccurate code changes, excessive token use, and an agent feels incompetent and unreliable.&lt;/p&gt;

&lt;p&gt;Full file rewrites are context-blind and prone to hallucinations, especially when editing that 3000+ line file that you&amp;rsquo;ve been meaning to refactor. And every failure and iteration is wasted compute, wasted money and worst of all, wasted time.&lt;/p&gt;
&lt;h2 id='why-we-arent-thinking-about-this-or-why-i-wasnt' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#why-we-arent-thinking-about-this-or-why-i-wasnt' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Why We Aren’t Thinking About This (or why I wasn&amp;rsquo;t)&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;AI workflows are still new to everyone. Best practices are still just opinions and most tooling is focused on model quality, not developer velocity or cost. This is a big part of why we feel that slow, wasteful code edits are just the price of admission for AI-powered development.&lt;/p&gt;

&lt;p&gt;In reality, these inefficiencies become a real bottleneck for coding agent tools. The hidden tax on every code edit adds up and your users pay with their time, especially as teams scale and projects grow more complex.&lt;/p&gt;
&lt;h2 id='better-faster-ai-code-edits-with-morph-fast-apply' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#better-faster-ai-code-edits-with-morph-fast-apply' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Better, Faster AI Code Edits with Morph Fast Apply&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;MorphLLM&amp;rsquo;s core innovation is Morph Fast Apply. It&amp;rsquo;s an edit merge tool that is semantic, structure-aware and designed specifically for code. Those are big words to describe a tool that will empower your agents to make single line changes without  rewriting whole files or relying on brittle search-and-replace. Instead, your agent applies precise, context-aware edits and it does it ridiculously fast. &lt;/p&gt;

&lt;p&gt;It works like this: &lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;You add an &amp;lsquo;edit_file&amp;rsquo; tool to your agents tools.
&lt;/li&gt;&lt;li&gt;Your agent outputs tiny &lt;code&gt;edit_file&lt;/code&gt; snippets, using &lt;code&gt;//...existing code...&lt;/code&gt; placeholders to indicate unchanged code.
&lt;/li&gt;&lt;li&gt;Your backend calls Morph’s Apply API, which merges the changes semantically. It doesn&amp;rsquo;t just replace raw text, it makes targeted merges with the code base as context. 
&lt;/li&gt;&lt;li&gt;You write back the precisely edited file. No manual patching, no painful conflict resolution, no context lost.
&lt;/li&gt;&lt;/ul&gt;
&lt;h2 id='the-numbers' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#the-numbers' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;The Numbers&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;MorphLLM&amp;rsquo;s Apply API processes over 4,500 tokens per second and their benchmark results are nuts. We&amp;rsquo;re talking 98% accuracy in ~6 seconds per file. Compare this to 35s (with error corrections) at 86% accuracy for traditional search-and-replace systems. Files up to 9k tokens in size take ~4 seconds to process. &lt;/p&gt;

&lt;p&gt;Just look at the damn &lt;a href='https://morphllm.com/benchmarks' title=''&gt;graph&lt;/a&gt;:&lt;/p&gt;

&lt;p&gt;&lt;img loading="lazy" decoding="async" alt="Time Performance Analysis" width="1161" height="656" src="/blog/build-better-agents-with-morphllm/assets/morph_graph.webp" /&gt;&lt;/p&gt;

&lt;p&gt;These are game-changing numbers for agent builders. Real-time code UIs become possible. Dynamic codebases can self-adapt in seconds, not minutes. Scale to multi-file edits, documentation, and even large asset transformations without sacrificing speed or accuracy.&lt;/p&gt;
&lt;h2 id='how-to-get-in-on-the-morphllm-action' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#how-to-get-in-on-the-morphllm-action' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;How to Get in on the MorphLLM Action&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Integration with your project is easy peasy. MorphLLM is API-compatible with OpenAI, Vercel AI SDK, MCP, and OpenRouter. You can run it in the cloud, self-host, or go on-prem with enterprise-grade guarantees. &lt;/p&gt;

&lt;p&gt;I want to cloud host mine, if only I could think of somewhere I could quickly and easily deploy wherever I want and only pay for when I&amp;rsquo;m using the infra 😉.&lt;/p&gt;
&lt;h2 id='get-morphed' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#get-morphed' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Get Morphed&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;MorphLLM feels like a plug-in upgrade for code agent projects that will instantly make them faster and more accurate. Check out the docs, benchmarks, and integration guides at &lt;a href='https://docs.morphllm.com/' title=''&gt;docs.morphllm.com&lt;/a&gt;. Get started for free at &lt;a href="https://morphllm.com/dashboard"&gt;https://morphllm.com/dashboard&lt;/a&gt; &lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Calibrated Trust: Trust Calibration for AI Software Builders</title>
    <link rel="alternate" href="https://fly.io/blog/trust-calibration-for-ai-software-builders/"/>
    <id>https://fly.io/blog/trust-calibration-for-ai-software-builders/</id>
    <published>2025-08-18T00:00:00+00:00</published>
    <updated>2026-10-08T22:49:59+00:00</updated>
    <media:thumbnail url="https://fly.io/blog/trust-calibration-for-ai-software-builders/assets/trust_calibration.webp"/>
    <content type="html">&lt;div class="lead"&gt;&lt;p&gt;Trust calibration is a concept from the world of human-machine interaction design, one that is super relevant to AI software builders. Trust calibration is the practice of aligning the level of trust that users have in our products with its actual capabilities. &lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;If we build things that our users trust too blindly, we risk facilitating dangerous or destructive interactions that can permanently turn users off. If they don&amp;rsquo;t trust our product enough, it will feel useless or less capable than it actually is. &lt;/p&gt;

&lt;p&gt;So what does trust calibration look like in practice and how do we achieve it? A 2023 study reviewed over 1000 papers on trust and trust calibration in human / automated systems (properly referenced at the end of this article). It holds some pretty eye-opening insights – and some inconvenient truths – for people building AI software. I&amp;rsquo;ve tried to extract just the juicy bits below.   &lt;/p&gt;
&lt;h2 id='limiting-trust' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#limiting-trust' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Limiting Trust&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Let&amp;rsquo;s begin with a critical point. There is a limit to how deeply we want users to trust our products. Designing for calibrated trust is the goal, not more trust at any cost. Shoddy trust calibration leads to two equally undesirable outcomes: &lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;strong class='font-semibold text-navy-950'&gt;Over-trust&lt;/strong&gt; causes users to rely on AI systems in situations where they shouldn&amp;rsquo;t (I told my code assistant to fix a bug in prod and went to bed).
&lt;/li&gt;&lt;li&gt;&lt;strong class='font-semibold text-navy-950'&gt;Under-trust&lt;/strong&gt; causes users to reject AI assistance even when it would be beneficial, resulting in reduced perception of value and increased user workload.
&lt;/li&gt;&lt;/ul&gt;

&lt;p&gt;What does calibrated trust look like for your product? It’s important to understand that determining this is less about trying to diagram a set of abstract trust parameters and more about helping users develop accurate mental models of your product&amp;rsquo;s capabilities and limitations. In most cases, this requires thinking beyond the trust calibration mechanisms we default to, like confidence scores. &lt;/p&gt;

&lt;p&gt;For example, Cursor&amp;rsquo;s most prominent trust calibration mechanism is its change suggestion highlighting. The code that the model suggests we change is highlighted in red, followed by suggested changes highlighted in green. This  immediately communicates that &amp;ldquo;this is a suggestion, not a command.&amp;rdquo; &lt;/p&gt;

&lt;p&gt;In contrast, Tesla&amp;rsquo;s Autopilot is a delegative system. It must calibrate trust differently through detailed capability explanations, clear operational boundaries (only on highways), and prominent disengagement alerts when conditions exceed system limits. &lt;/p&gt;
&lt;h2 id='building-cooperative-systems' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#building-cooperative-systems' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Building Cooperative Systems&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Perhaps the most fundamental consideration in determining high level trust calibration objectives is deciding whether your project is designed to be a cooperative or a delegative tool.  &lt;/p&gt;

&lt;p&gt;Cooperative systems generally call for lower levels of trust because users can choose whether to accept or reject AI suggestions. But these systems also face a unique risk. It’s easy for over-trust to gradually transform user complacency into over-reliance, effectively transforming what we designed as a cooperative relationship into a delegative one, only without any of the required safeguards.&lt;/p&gt;

&lt;p&gt;If you&amp;rsquo;re building a coding assistant, content generator, or design tool, implement visible &amp;ldquo;suggestion boundaries&amp;rdquo; which make it clear when the AI is offering ideas versus making decisions. Grammarly does this well by underlining suggestions rather than auto-correcting, and showing rationale on hover. &lt;/p&gt;

&lt;p&gt;For higher-stakes interactions, consider introducing friction. Require explicit confirmation before applying AI suggestions to production code or publishing AI-generated content.&lt;/p&gt;
&lt;h2 id='building-delegative-systems' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#building-delegative-systems' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Building Delegative Systems&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;In contrast, users expect delegative systems to replace human action entirely. Blind trust in the system is a requirement for it to be considered valuable at all. &lt;/p&gt;

&lt;p&gt;If you&amp;rsquo;re building automation tools, smart scheduling, or decision-making systems, invest heavily in capability communication and boundary setting. Calendly&amp;rsquo;s smart scheduling works because it clearly communicates what it will and won&amp;rsquo;t do (I&amp;rsquo;ll find times that work for both of us vs. I&amp;rsquo;ll reschedule your existing meetings). Build robust fallback mechanisms and make system limitations prominent in your onboarding.  &lt;/p&gt;
&lt;h2 id='timing-is-everything' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#timing-is-everything' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Timing Is Everything&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;The study suggests that when we make trust calibrations is at least as important as how. There are three critical windows for trust calibration, each with their own opportunities and challenges. &lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;strong class='font-semibold text-navy-950'&gt;Pre-interaction calibration&lt;/strong&gt; happens before users engage with the system. Docs and tutorials fall into this category. Setting expectations up front can prevent initial over-trust, which is disproportionally more difficult to correct later. 
&lt;/li&gt;&lt;/ul&gt;

&lt;blockquote&gt;
&lt;p&gt;Pre-interaction calibrations could look like capability-focused onboarding that shows both successes and failures. Rather than just demonstrating perfect AI outputs, show users examples where the AI makes mistakes and how to catch them. &lt;/p&gt;
&lt;/blockquote&gt;

&lt;ul&gt;
&lt;li&gt;&lt;strong class='font-semibold text-navy-950'&gt;During-interaction calibration&lt;/strong&gt; is trust adjustment through real-time feedback. Dynamically updated cues improve trust calibration better than static displays, and adaptive calibration that responds to user behavior outperforms systems that display static information. 
&lt;/li&gt;&lt;/ul&gt;

&lt;blockquote&gt;
&lt;p&gt;Build confidence indicators that are updated based on context, not just model confidence. For example, if you&amp;rsquo;re building a document AI, show higher confidence for standard document types the system has seen thousands of times, and lower confidence for unusual formats. &lt;/p&gt;
&lt;/blockquote&gt;

&lt;ul&gt;
&lt;li&gt;&lt;strong class='font-semibold text-navy-950'&gt;Post-interaction calibration&lt;/strong&gt; focuses on learning and adjustment that helps users understand successes and failures in the system after interactions. These aren’t reliable, since by the time users receive the information, their trust patterns are set and hard to change. 
&lt;/li&gt;&lt;/ul&gt;

&lt;blockquote&gt;
&lt;p&gt;Post-interaction feedback can still be valuable for teaching. Create &amp;ldquo;reflection moments&amp;rdquo; after significant interactions. Midjourney does this by letting users rate image outputs, helping users learn what prompts work best while calibrating their expectations for future generations. &lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Trust is front-loaded and habit-driven. The most effective calibration happens before and during use, when expectations are still forming and behaviors can still be shifted. Any later and you’re mostly fighting entrenched patterns.&lt;/p&gt;
&lt;h2 id='performance-vs-process-information' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#performance-vs-process-information' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Performance vs. Process Information&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Users can be guided through performance-oriented signals (what the system can do) or process-oriented signals (how it works). The real challenge is matching the right kind of explanation to the right user, at the right moment.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;strong class='font-semibold text-navy-950'&gt;Performance-oriented calibration&lt;/strong&gt; focuses on communicating capability through mechanisms like reliability statistics, confidence scores, and clear capability boundaries. 
&lt;/li&gt;&lt;li&gt;&lt;strong class='font-semibold text-navy-950'&gt;Process-oriented calibration&lt;/strong&gt; offers detailed explanations of decision-making processes, breakdowns of which factors influenced decisions, and reasoning transparency. 
&lt;/li&gt;&lt;/ul&gt;

&lt;p&gt;Process transparency seems like the obvious go-to at first glance, but the effectiveness of process explanations varies wildly based on user expertise and domain knowledge. If we are designing for a set of users that may fall anywhere on this spectrum, we have to avoid creating information overload for novice users while providing sufficient information to expert users who want the detail.  &lt;/p&gt;

&lt;p&gt;The most effective systems in the study combined both approaches, providing layered information that allows users to access the level of detail most appropriate for their expertise and current needs.&lt;/p&gt;
&lt;h2 id='static-vs-adaptive-calibration' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#static-vs-adaptive-calibration' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Static vs. Adaptive Calibration&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;I really wanted to ignore this part, because it feels like the study’s authors are passive aggressively adding todos to my projects. In a nutshell, adaptive calibration – when a system actively monitors user behavior and adjusts its communication accordingly - is orders of magnitude more effective than static calibration while delivering the same information to every user, regardless of differences in expertise, trust propensity, or behavior.  &lt;/p&gt;

&lt;p&gt;Static calibration mechanisms are easy to build and maintain, which is why we like them. But the stark reality is that they put the burden of appropriate calibration entirely on our users. We’re making it their job to adapt their behaviour based on generic information.&lt;/p&gt;

&lt;p&gt;This finding has zero respect for our time or mental health, but it also reveals a legit opportunity for clever builders to truly separate their product from the herd.&lt;/p&gt;
&lt;h2 id='practical-adaptive-calibration-techniques' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#practical-adaptive-calibration-techniques' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Practical adaptive calibration techniques&lt;/span&gt;&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong class='font-semibold text-navy-950'&gt;Behavioral adaptation:&lt;/strong&gt; Track how often users accept vs. reject suggestions and adjust confidence thresholds accordingly. If a user consistently rejects high-confidence suggestions, lower the threshold for showing uncertainty.
&lt;/li&gt;&lt;li&gt;&lt;strong class='font-semibold text-navy-950'&gt;Context awareness:&lt;/strong&gt; Adjust trust signals based on use context. A writing AI might show higher confidence for grammar fixes than creative suggestions, or lower confidence late at night when users might be tired.
&lt;/li&gt;&lt;li&gt;&lt;strong class='font-semibold text-navy-950'&gt;Detect expertise:&lt;/strong&gt; Users who frequently make sophisticated edits to AI output probably want more detailed explanations than those who typically accept entire file rewrites.
&lt;/li&gt;&lt;/ul&gt;
&lt;h2 id='the-transparency-paradox' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#the-transparency-paradox' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;The Transparency Paradox&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;The idea that transparency and explainability can actually harm trust calibration is easily the point that hit me the hardest. While explanations can improve user understanding, they can also create information overload that reduces users&amp;rsquo; ability to detect and correct trash output. What&amp;rsquo;s worse, explanations can create a whole new layer of trust calibration issues, with users over-trusting the explanation mechanism itself, rather than critically evaluating the actual output.&lt;/p&gt;

&lt;p&gt;This suggests that quality over quantity should be our design philosophy when it comes to transparency. We should provide carefully crafted, relevant information rather than comprehensive but overwhelming detail. The goal should be enabling better decision-making rather than simply satisfying user curiosity about system internals.&lt;/p&gt;
&lt;h2 id='anthropomorphism-and-unwarranted-trust' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#anthropomorphism-and-unwarranted-trust' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Anthropomorphism and Unwarranted Trust&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;It seems obvious that we should make interactions with our AI project feel as human as possible. Well, it turns out that systems that appear more human-like through design, language, or interaction patterns are notoriously good at increasing user trust beyond actual system capabilities. &lt;/p&gt;

&lt;p&gt;So it’s entirely possible that building more traditional human-computer interactions can actually make our AI projects safer to use and therefore, more user-friendly.  &lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;strong class='font-semibold text-navy-950'&gt;Use tool-like language:&lt;/strong&gt; Frame outputs as &amp;ldquo;analysis suggests&amp;rdquo; rather than &amp;ldquo;I think&amp;rdquo; or &amp;ldquo;I believe&amp;rdquo;
&lt;/li&gt;&lt;li&gt;&lt;strong class='font-semibold text-navy-950'&gt;Embrace machine-like precision:&lt;/strong&gt; Show exact confidence percentages rather than human-like hedging (&amp;ldquo;I&amp;rsquo;m pretty sure that&amp;hellip;)
&lt;/li&gt;&lt;/ul&gt;
&lt;h2 id='trust-falls-faster-than-it-climbs' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#trust-falls-faster-than-it-climbs' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Trust Falls Faster Than It Climbs&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Nothing particularly groundbreaking here, but the findings are worth mentioning if only to reinforce what we think we know. &lt;/p&gt;

&lt;p&gt;Early interactions are critically important. Users form mental models quickly and then react slowly to changes in system reliability.&lt;/p&gt;

&lt;p&gt;More critically, trust drops much faster from system failures than it builds from successes. These asymmetries suggest that we should invest disproportionately in onboarding and first-use experiences, even if they come with higher development costs.&lt;/p&gt;
&lt;h2 id='measurement-is-an-opportunity-for-innovation' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#measurement-is-an-opportunity-for-innovation' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Measurement is an Opportunity for Innovation&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;The study revealed gaping voids where effective measurement mechanisms and protocols should be, for both researchers and builders. There is a clear need to move beyond simple user satisfaction metrics or adoption rates to developing measurement frameworks that can actively detect miscalibrated trust patterns. &lt;/p&gt;

&lt;p&gt;The ideal measurement approach would combine multiple indicators. A few examples of viable indicators are:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;strong class='font-semibold text-navy-950'&gt;Behavioral signals:&lt;/strong&gt; Track acceptance rates for different confidence levels. Well-calibrated trust should show higher acceptance rates for high-confidence outputs and lower rates for low-confidence ones.
&lt;/li&gt;&lt;li&gt;&lt;strong class='font-semibold text-navy-950'&gt;Context-specific metrics:&lt;/strong&gt; Measure trust calibration separately for different use cases. Users might be well-calibrated for simple tasks but poorly calibrated for complex ones.
&lt;/li&gt;&lt;li&gt;&lt;strong class='font-semibold text-navy-950'&gt;User self-reporting:&lt;/strong&gt; Regular pulse surveys asking &amp;quot;How confident are you in your ability to tell when this AI makes mistakes?&amp;rdquo; can reveal calibration gaps.
&lt;/li&gt;&lt;/ul&gt;
&lt;h2 id='the-calibrated-conclusion' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#the-calibrated-conclusion' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;The Calibrated Conclusion&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;It&amp;rsquo;s clear, at least from this study, that there’s no universal formula, or single feature that will effectively calibrate trust. It&amp;rsquo;s up to every builder to define and understand their project&amp;rsquo;s trust goals and to balance timing, content, adaptivity, and transparency accordingly. That’s what makes it both hard and worth doing. Trust calibration has to be a core part of our product’s identity, not a piglet we only start chasing once it has escaped the barn.&lt;/p&gt;

&lt;p&gt;&lt;strong class='font-semibold text-navy-950'&gt;The Study:&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Magdalena Wischnewski, Nicole Krämer, and Emmanuel Müller. 2023. Measuring and Understanding Trust Calibrations for Automated Systems: A Survey of the State-Of-The-Art and Future Directions. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI &amp;lsquo;23), April 23–28, 2023, Hamburg, Germany. ACM, New York, NY, USA 16 Pages. &lt;a href="https://doi.org/10.1145/3544548.3581197"&gt;https://doi.org/10.1145/3544548.3581197&lt;/a&gt;&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Games as Model Eval: 1-Click Deploy AI Town on Fly.io</title>
    <link rel="alternate" href="https://fly.io/blog/games-as-model-eval/"/>
    <id>https://fly.io/blog/games-as-model-eval/</id>
    <published>2025-08-11T00:00:00+00:00</published>
    <updated>2025-08-19T23:52:31+00:00</updated>
    <media:thumbnail url="https://fly.io/blog/games-as-model-eval/assets/Fly_Man.webp"/>
    <content type="html">&lt;div class="lead"&gt;&lt;p&gt;Recently, I suggested that &lt;a href="https://fly.io/blog/the-future-isn-t-model-agnostic/" title=""&gt;The Future Isn’t Model Agnostic&lt;/a&gt;, that it’s better to pick one model that works for your project and build around it, rather than engineering for model flexibility. If you buy that, you also have to acknowledge how important comprehensive model evaluation becomes. &lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;Benchmarks tell us almost nothing about how a model will actually behave in the wild, especially with long contexts, or when trusted to deliver the tone and feel that defines the UX we’re shooting for. Even the best evaluation pipelines usually end in subjective, side-by-side output comparisons. Not especially rigorous, and more importantly, boring af.&lt;/p&gt;

&lt;p&gt;Can we gamify model evaluation? Oh yes. And not just because we get to have some fun for once. Google backed me up this week when it announced the &lt;a href='https://blog.google/technology/ai/kaggle-game-arena/' title=''&gt;Kaggle Game Arena&lt;/a&gt;. A public platform where we can watch AI models duke it out in a variety of classic games.  Quoting Google; &amp;ldquo;Current AI benchmarks are struggling to keep pace with modern models&amp;hellip; it can be hard to know if models trained on internet data are actually solving problems or just remembering answers they&amp;rsquo;ve already seen.&amp;rdquo;&lt;/p&gt;

&lt;p&gt;When models boss reading comprehension tests, or ace math problems, we pay attention. But when they fail to navigate a simple conversation with a virtual character or completely botch a strategic decision in a game environment, we tell ourselves we&amp;rsquo;re not building a game anyway and develop strategic short-term memory loss. 
Just like I&amp;rsquo;ve told my mom a thousand times, games are great at testing brains, and it&amp;rsquo;s time we take this seriously when it comes to model evaluation. &lt;/p&gt;
&lt;h2 id='why-games-dont-lie' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#why-games-dont-lie' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Why Games Don&amp;rsquo;t Lie&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Games provide what benchmarks can&amp;rsquo;t, &amp;ldquo;a clear, unambiguous signal of success.&amp;rdquo; They give us observable behavior in dynamic environments, the kind that would be extremely difficult (and tedious) to simulate with prompt engineering alone.&lt;/p&gt;

&lt;p&gt;Games force models to demonstrate the skills we actually care about; strategic reasoning, long-term planning, and dynamic adaptation in interactions with an opponent or a collaborator. &lt;/p&gt;
&lt;h2 id='pixel-art-meets-effective-model-evaluation-ai-town-on-fly-io' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#pixel-art-meets-effective-model-evaluation-ai-town-on-fly-io' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Pixel Art Meets Effective Model Evaluation - AI Town on Fly.io&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;AI Town is a brilliant project by &lt;a href='https://github.com/a16z-infra' title=''&gt;a16z-infra&lt;/a&gt;, based on the the mind-bending paper,  &lt;a href='https://arxiv.org/pdf/2304.03442' title=''&gt;Generative Agents: Interactive Simulacra of Human Behavior&lt;/a&gt;. It&amp;rsquo;s a beautifully rendered little town in which tiny people with AI brains and engineered personalities go about their lives, interacting with each other and their environment. Characters need to remember past conversations, maintain relationships, react dynamically to new situations, and stay in character while doing it all. &lt;/p&gt;

&lt;p&gt;I challenge you to find a more entertaining way of evaluating conversational models. &lt;/p&gt;

&lt;p&gt;I&amp;rsquo;ve &lt;a href='https://github.com/fly-apps/ai-town_on_fly.io' title=''&gt;forked the project&lt;/a&gt; to make it absurdly easy to spin up your own AI Town on Fly Machines. You&amp;rsquo;ve got a single deploy script that will set everything up for you and some built-in cost and performance optimizations, with our handy scale to zero functionality as standard (so you only pay for the time spent running it). This makes it easy to share with your team, your friends and your mom.  &lt;/p&gt;

&lt;p&gt;In it&amp;rsquo;s current state, the fork makes it as easy as possible to test any OpenAI-compatible service, any model on Together.ai and even custom embedding models. Simply set the relevant API key in your secrets. &lt;/p&gt;

&lt;p&gt;Games like AI Town give us a window into how models actually think, adapt, and behave beyond the context of our prompts. You move past performance metrics and begin to understand a model’s personality, quirks, strengths, and weaknesses; all factors that ultimately shape your project&amp;rsquo;s UX. &lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>The Future Isn't Model Agnostic</title>
    <link rel="alternate" href="https://fly.io/blog/the-future-isn-t-model-agnostic/"/>
    <id>https://fly.io/blog/the-future-isn-t-model-agnostic/</id>
    <published>2025-08-08T00:00:00+00:00</published>
    <updated>2025-08-19T23:52:31+00:00</updated>
    <media:thumbnail url="https://fly.io/blog/the-future-isn-t-model-agnostic/assets/Whack_A_Mole_.webp"/>
    <content type="html">&lt;div class="lead"&gt;&lt;p&gt;Your users don’t care that your AI project is model 
agnostic. &lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;In my last project, I spent countless hours ensuring that the LLMs running my services  could be swapped out as easily as possible. I couldn&amp;rsquo;t touch a device with an internet connection without hearing about the latest benchmark-breaking model and it felt like a clear priority to ensure I could hot swap models with minimal collateral damage.&lt;/p&gt;

&lt;p&gt;So yeah. That was a waste of time.&lt;/p&gt;

&lt;p&gt;The hype around new model announcements feels more manufactured with each release. In reality, improvements are becoming incremental. As major providers converge on the same baseline, the days of one company holding a decisive lead are numbered.&lt;/p&gt;

&lt;p&gt;In a world of model parity, the differentiation moves entirely to the product layer. Winning isn&amp;rsquo;t about ensuring you&amp;rsquo;re using the best model, its about understanding your chosen model deeply enough to build experiences that feel magical. Knowing exactly how to prompt for consistency, which edge cases to avoid, and how to design workflows that play to your model&amp;rsquo;s particular strengths&lt;/p&gt;

&lt;p&gt;Model agnosticism isn&amp;rsquo;t just inefficient, it&amp;rsquo;s misguided. Fact is, swapping out your model is not just changing an endpoint. It&amp;rsquo;s rewriting prompts, rerunning evals, users telling you things just feel&amp;hellip; different. And if you&amp;rsquo;ve won users on the way it feels to use your product, that last one is a really big deal.&lt;/p&gt;
&lt;h2 id='model-lt-product' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#model-lt-product' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Model &amp;lt; Product&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Recently, something happened that fully solidified this idea in my head. Claude Code is winning among people building real things with AI. We even have evangelists in the Fly.io engineering team, and those guys are weird smart. Elsewhere, whole communities have formed to share and compare claude.md&amp;rsquo;s and fight each other over which MCP servers are the coolest to use with Claude.&lt;/p&gt;

&lt;p&gt;Enter stage right, Qwen 3 Coder. It takes Claude to the cleaners in benchmarks. But the response from the Claude Code user base? A collective meh.&lt;/p&gt;

&lt;p&gt;This is nothing like 2024, when everyone would have dropped everything to get the hot new model running in Cursor. And it&amp;rsquo;s not because we&amp;rsquo;ve learned that benchmarks are performance theater for people who&amp;rsquo;ve never shipped a product.&lt;/p&gt;

&lt;p&gt;It&amp;rsquo;s because products like Claude Code are irrefutable evidence that the model isn&amp;rsquo;t the product. We&amp;rsquo;ve felt it first hand when our pair programmer&amp;rsquo;s behaviour changes in subtle ways. The product is in the rituals. The trust. The predictability. It&amp;rsquo;s precisely because Claude Code&amp;rsquo;s model behavior, UI, and user expectations are so tightly coupled that its users don&amp;rsquo;t really care that a better model might exist.&lt;/p&gt;

&lt;p&gt;I&amp;rsquo;m not trying to praise Anthropic here. The point is, engineering for model agnosticism is a trap that will eat up time that could be better spent … anywhere else.&lt;/p&gt;

&lt;p&gt;Sure, if you&amp;rsquo;re building infra or anything else that lives close to the metal, model optionality still matters. But people trusting legwork to AI tools are building deeper relationships and expectations of their AI tools than they even care to admit. AI product success stories are written when products become invisible parts of users&amp;rsquo; daily rituals, not showcases for engineering flexibility.&lt;/p&gt;
&lt;h2 id='make-one-model-your-own' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#make-one-model-your-own' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Make One Model Your Own&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;As builders, it&amp;rsquo;s time we stop hedging our bets and embrace the convergence reality. Every startup pitch deck with &amp;lsquo;model-agnostic&amp;rsquo; as a feature should become a red flag for investors who understand product-market fit. Stop putting &amp;lsquo;works with any LLM&amp;rsquo; in your one-liner. It screams &amp;lsquo;we don&amp;rsquo;t know what we&amp;rsquo;re building.&amp;rsquo;&lt;/p&gt;

&lt;p&gt;If you&amp;rsquo;re still building model-agnostic AI tools in 2025, you&amp;rsquo;re optimizing for the wrong thing. Users don&amp;rsquo;t want flexibility; they want reliability. And in a converged model landscape, reliability comes from deep specialization, not broad compatibility.&lt;/p&gt;

&lt;p&gt;Pick your model like you pick your therapist; for the long haul. Find the right model, tune deeply, get close enough to understand its quirks and make them work for you. Stop architecting for the mythical future where you&amp;rsquo;ll seamlessly swap models. That future doesn&amp;rsquo;t exist, and chasing it is costing you the present.&lt;/p&gt;
&lt;h2 id='bonus-level-all-in-on-one-model-means-all-out-on-eval' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#bonus-level-all-in-on-one-model-means-all-out-on-eval' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Bonus level: All-in On One Model Means All-out On Eval&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;If any of this is landing for you, you&amp;rsquo;ll agree that we have to start thinking of  model evaluation as architecture, not an afterthought. The good news is, rigorous model eval doesn&amp;rsquo;t have to be mind numbing anymore. &lt;/p&gt;

&lt;p&gt;Turns out, games are really great eval tools! Now you can spin up your very own little &lt;a href='https://github.com/fly-apps/ai-town_on_fly.io' title=''&gt;AI Town&lt;/a&gt; on Fly.io with a single click deploy to test different models as pixel people in an evolving environment. I discuss the idea further in &lt;a href='https://fly.io/blog/games-as-model-eval/' title=''&gt;Games as Model Eval: 1-Click Deploy AI Town on Fly.io&lt;/a&gt;.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Phoenix.new – The Remote AI Runtime for Phoenix</title>
    <link rel="alternate" href="https://fly.io/blog/phoenix-new-the-remote-ai-runtime/"/>
    <id>https://fly.io/blog/phoenix-new-the-remote-ai-runtime/</id>
    <published>2025-06-20T00:00:00+00:00</published>
    <updated>2026-09-28T14:09:34+00:00</updated>
    <media:thumbnail url="https://fly.io/blog/phoenix-new-the-remote-ai-runtime/assets/phoenixnew-thumb.png"/>
    <content type="html">&lt;div class="lead"&gt;&lt;p&gt;I’m Chris McCord, the creator of Elixir’s Phoenix framework. For the past several months, I’ve been working on a skunkworks project at Fly.io, and it’s time to show it off.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;I wanted LLM agents to work just as well with Elixir as they do with Python and JavaScript. Last December, in order to figure out what that was going to take, I started a little weekend project to find out how difficult it would be to build a coding agent in Elixir.&lt;/p&gt;

&lt;p&gt;A few weeks later, I had it spitting out working Phoenix applications and driving a full in-browser IDE. I knew this wasn&amp;rsquo;t going to stay a weekend project.&lt;/p&gt;

&lt;p&gt;If you follow me on Twitter, you&amp;rsquo;ve probably seen me teasing this work as it picked up steam. We&amp;rsquo;re at a point where we&amp;rsquo;re pretty serious about this thing, and so it&amp;rsquo;s time to make a formal introduction.&lt;/p&gt;

&lt;p&gt;World, meet &lt;a href='https://phoenix.new' title=''&gt;Phoenix.new&lt;/a&gt;, a batteries-included fully-online coding agent tailored to Elixir and Phoenix. I think it&amp;rsquo;s going to be the fastest way to build collaborative, real-time applications.&lt;/p&gt;

&lt;p&gt;Let&amp;rsquo;s see it in action:&lt;/p&gt;
&lt;div class="youtube-container" data-exclude-render&gt;
  &lt;div class="youtube-video"&gt;
    &lt;iframe
      width="100%"
      height="100%"
      src="https://www.youtube.com/embed/du7GmWGUM5Y"
      frameborder="0"
      allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture"
      allowfullscreen&gt;
    &lt;/iframe&gt;
  &lt;/div&gt;
&lt;/div&gt;

&lt;h2 id='whats-interesting-about-phoenix-new' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#whats-interesting-about-phoenix-new' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;What&amp;rsquo;s Interesting About Phoenix.new&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;First, even though it runs entirely in your browser, Phoenix.new gives both you and your agent a root shell, in an ephemeral virtual machine (a &lt;a href='https://docs.fly.io/machines/overview' title=''&gt;Fly Machine&lt;/a&gt;) that gives our agent loop free rein to install things and run programs  — without any risk of messing up your local machine. You don&amp;rsquo;t think about any of this; you just open up the VSCode interface, push the shell button, and there you are, on the isolated machine you share with the Phoenix.new agent.&lt;/p&gt;

&lt;p&gt;Second, it&amp;rsquo;s an agent system I built specifically for Phoenix. Phoenix is about real-time collaborative applications, and Phoenix.new knows what that means. To that end, Phoenix.new includes, in both its UI and its agent tools, a full browser. The Phoenix.new agent uses that browser &amp;ldquo;headlessly&amp;rdquo; to check its own front-end changes and interact with the app. Because it&amp;rsquo;s a full browser, instead of trying to iterate on screenshots, the agent sees real page content and JavaScript state – with or without a human present.&lt;/p&gt;
&lt;h2 id='what-root-access-gets-us' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#what-root-access-gets-us' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;What Root Access Gets Us&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Agents build software the way you did when you first got started, the way you still do today when you prototype things. They don&amp;rsquo;t carefully design Docker container layers and they don&amp;rsquo;t really do release cycles. An agent wants to pop a shell and get its fingernails dirty.&lt;/p&gt;

&lt;p&gt;A fully isolated virtual machine means Phoenix.new&amp;rsquo;s fingernails can get &lt;em&gt;arbitrarily dirty.&lt;/em&gt; If it wants to add a package to &lt;code&gt;mix.exs&lt;/code&gt;, it can do that and then run &lt;code&gt;mix phx.server&lt;/code&gt; or &lt;code&gt;mix test&lt;/code&gt; and check the output. Sure. Every agent can do that. But if it wants to add an APT package to the base operating system, it can do that too, and make sure it worked. It owns the whole environment.&lt;/p&gt;

&lt;p&gt;This offloads a huge amount of tedious, repetitive work.&lt;/p&gt;

&lt;p&gt;At his &lt;a href='https://youtu.be/LCEmiRjPEtQ?si=sR_bdu6-AqPXSNmY&amp;t=1902' title=''&gt;AI Startup School talk last week&lt;/a&gt;, Andrej Karpathy related his experience of building a restaurant menu visualizer, which takes camera pictures of text menus and transforms all the menu items into pictures. The code, which he vibe-coded with an LLM agent, was the easy part; he had it working in an afternoon. But getting the app online took him a whole week.&lt;/p&gt;

&lt;p&gt;With Phoenix.new, I&amp;rsquo;m taking dead aim at this problem. The apps we produce live in the cloud from the minute they launch. They have private, shareable URLs (we detect anything the agent generates with a bound port and give it a preview URL underneath &lt;code&gt;phx.run&lt;/code&gt;, with integrated port-forwarding), they integrate with Github, and they inherit all the infrastructure guardrails of Fly.io: hardware virtualization, WireGuard, and isolated networks.&lt;/p&gt;
&lt;div class="callout"&gt;&lt;p&gt;Github’s &lt;code&gt;gh&lt;/code&gt; CLI is installed by default. So the agent knows how to clone any repo, or browse issues, and you can even authorize it for internal repositories to get it working with your team’s existing projects and dependencies.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;Full control of the environment also closes the loop between the agent and deployment. When Phoenix.new boots an app, it watches the logs, and tests the application. When an action triggers an error, Phoenix.new notices and gets to work.&lt;/p&gt;
&lt;h2 id='watch-it-build-in-real-time' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#watch-it-build-in-real-time' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Watch It Build In Real Time&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;&lt;a href='https://phoenix.new' title=''&gt;Phoenix.new&lt;/a&gt; can interact with web applications the way users do: with a real browser.&lt;/p&gt;

&lt;p&gt;The Phoenix.new environment includes a headless Chrome browser that our agent knows how to drive. Prompt it to add a front-end feature to your application, and it won&amp;rsquo;t just sketch the code out and make sure it compiles and lints. It&amp;rsquo;ll pull the app up itself and poke at the UI, simultaneously looking at the page content, JavaScript state, and server-side logs.&lt;/p&gt;

&lt;p&gt;Phoenix is all about &lt;a href='https://fly.io/blog/how-we-got-to-liveview/' title=''&gt;&amp;ldquo;live&amp;rdquo; real-time&lt;/a&gt; interactivity, and gives us seamless live reload. The user interface for Phoenix.new itself includes a live preview of the app being worked on, so you can kick back and watch it build front-end features incrementally. Any other &lt;code&gt;.phx.run&lt;/code&gt; tabs you have open also update as it goes. It&amp;rsquo;s wild.&lt;/p&gt;
&lt;video title="agent interacting with web" autoplay="autoplay" loop="loop" muted="muted" playsinline="playsinline" disablePictureInPicture="true" class="mb-8" src="/blog/phoenix-new-the-remote-ai-runtime/assets/webjs.mp4"&gt;&lt;/video&gt;

&lt;h2 id='not-just-for-vibe-coding' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#not-just-for-vibe-coding' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Not Just For Vibe Coding&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Phoenix.new can already build real, full-stack applications with WebSockets, Phoenix&amp;rsquo;s Presence features, and real databases. I&amp;rsquo;m seeing it succeed at business and collaborative applications right now.&lt;/p&gt;

&lt;p&gt;But there&amp;rsquo;s no fixed bound on the tasks you can reasonably ask it to accomplish. If you can do it with a shell and a browser, I want Phoenix.new to do it too. And it can do these tasks with or without you present.&lt;/p&gt;

&lt;p&gt;For example: set a &lt;code&gt;$DATABASE_URL&lt;/code&gt; and tell the agent about it. The agent knows enough to go explore it with &lt;code&gt;psql&lt;/code&gt;, and it&amp;rsquo;ll propose apps based on the schemas it finds. It can model Ecto schemas off the database. And if MySQL is your thing, the agent will just &lt;code&gt;apt install&lt;/code&gt; a MySQL client and go to town.&lt;/p&gt;

&lt;p&gt;Frontier model LLMs have vast world knowledge. They generalize extremely well. At ElixirConfEU, I did a &lt;a href='https://www.youtube.com/watch?v=ojL_VHc4gLk&amp;t=3923s' title=''&gt;demo vibe-coding Tetris&lt;/a&gt; on stage. Phoenix.new nailed it, first try, first prompt. It&amp;rsquo;s not like there&amp;rsquo;s gobs of Phoenix LiveView Tetris examples floating around the Internet! But lots of people have published Tetris code, and lots of people have written LiveView stuff, and 2025 LLMs can connect those dots.&lt;/p&gt;

&lt;p&gt;At this point you might be wondering – can I just ask it to build a Rails app? Or an Expo React Native app? Or Svelte? Or Go?&lt;/p&gt;

&lt;p&gt;Yes, you can.&lt;/p&gt;

&lt;p&gt;Our system prompt is tuned for Phoenix today, but all languages you care about are already installed. We&amp;rsquo;re still figuring out where to take this, but adding new languages and frameworks definitely ranks highly in my plans.&lt;/p&gt;
&lt;h2 id='our-async-agent-future' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#our-async-agent-future' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Our Async Agent Future&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;&lt;a href='https://fly.io/blog/youre-all-nuts/' title=''&gt;We&amp;rsquo;re at a massive step-change in developer workflows&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Agents can do real work, today, with or without a human present. Buckle up: the future of development, at least in the common case, probably looks less like cracking open a shell and finding a file to edit, and more like popping into a CI environment with agents working away around the clock.&lt;/p&gt;

&lt;p&gt;Local development isn&amp;rsquo;t going away. But there&amp;rsquo;s going to be a shift in where the majority of our iterations take place. I&amp;rsquo;m already using Phoenix.new to triage &lt;code&gt;phoenix-core&lt;/code&gt; Github issues and pick problems to solve. I close my laptop, grab a cup of coffee, and wait for a PR to arrive — Phoenix.new knows how PRs work, too. We&amp;rsquo;re already here, and this space is just getting started.&lt;/p&gt;

&lt;p&gt;This isn&amp;rsquo;t where I thought I&amp;rsquo;d end up when I started poking around. The Phoenix and LiveView journey was much the same. Something special was there and the projects took on a life of their own. I&amp;rsquo;m excited to share this work now, and see where it might take us. I can&amp;rsquo;t wait to see what folks build.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>What are MCP Servers?</title>
    <link rel="alternate" href="https://fly.io/blog/mcps-everywhere/"/>
    <id>https://fly.io/blog/mcps-everywhere/</id>
    <published>2025-06-12T00:00:00+00:00</published>
    <updated>2026-09-28T14:09:34+00:00</updated>
    <media:thumbnail url="https://fly.io/blog/mcps-everywhere/assets/mcps-everywhere.webp"/>
    <content type="html">&lt;div class="lead"&gt;&lt;div&gt;&lt;p&gt;With Fly.io, &lt;a href="https://fly.io/speedrun/" title=""&gt;you can get your app running globally in a matter of minutes&lt;/a&gt;, and with MCP servers you can integrate with Claude, VSCode, Cursor and &lt;a href="https://modelcontextprotocol.io/clients"&gt;many more AI clients&lt;/a&gt;.  &lt;a href="https://docs.fly.io/model-context-protocol" title=""&gt;Try it out for yourself&lt;/a&gt;!&lt;/p&gt;
&lt;/div&gt;&lt;/div&gt;
&lt;p&gt;The introduction to &lt;a href='https://modelcontextprotocol.io/introduction' title=''&gt;Model Context Protocol&lt;/a&gt; starts out with:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;MCP is an open protocol that standardizes how applications provide context to LLMs. Think of MCP like a USB-C port for AI applications. Just as USB-C provides a standardized way to connect your devices to various peripherals and accessories, MCP provides a standardized way to connect AI models to different data sources and tools.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That paragraph, to me, is both comforting (&amp;ldquo;USB for LLM&amp;rdquo;? Cool! Got it!), and simultaneously vacuous (Um, but what do I actually &lt;em&gt;do&lt;/em&gt; with this?).&lt;/p&gt;

&lt;p&gt;I&amp;rsquo;ve been digging deeper and have come up with a few more analogies and observations that make sense to me. Perhaps one or more of these will help you.&lt;/p&gt;
&lt;h2 id='mcps-are-alexa-skills' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#mcps-are-alexa-skills' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;MCPs are Alexa Skills&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;You buy an Echo Dot and a Hue light. You plug them both in and connect them to your Wi-Fi. There is one more step you need to do: you need to install and enable the Philips Hue skill to connect the two.&lt;/p&gt;

&lt;p&gt;Now you might be using Siri or Google Assistant. Or you may want to connect a Ring Doorbell camera or Google Nest Thermostat. But the principle is the same, though the analogy is slightly stronger with a skill (which is a noun) as opposed to the action of pairing your Hue Bridge with Apple HomeKit (a verb).&lt;/p&gt;
&lt;h2 id='mcps-are-api-2-0' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#mcps-are-api-2-0' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;MCPs are API 2.0&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;HTTP 1.1 is simple. You send a request, you get a response. While there are cookies and sessions, it is pretty much stateless and therefore inefficient, as each request needs to establish a new connection. WebSockets and Server-Sent Events (SSE) mitigate this a bit.&lt;/p&gt;

&lt;p&gt;&lt;a href='https://en.wikipedia.org/wiki/HTTP/2' title=''&gt;HTTP 2.0&lt;/a&gt; introduces features like multiplexing and server push. When you visit a web page for the first time, your browser can now request all of the associated JavaScript, CSS, and images at once, and the server can respond in any order.&lt;/p&gt;

&lt;p&gt;APIs today are typically request/response. MCPs support multiplexing and server push.&lt;/p&gt;
&lt;h2 id='mcps-are-apis-with-introspection-reflection' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#mcps-are-apis-with-introspection-reflection' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;MCPs are APIs with Introspection/Reflection&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;With &lt;a href='https://learn.openapis.org/' title=''&gt;OpenAPI&lt;/a&gt;, requests are typically JSON, and responses are too. Many OpenAPI providers publish a separate &lt;a href='https://learn.openapis.org/specification/structure.html' title=''&gt;OpenAPI Description (OAD)&lt;/a&gt;, which contains a schema describing what requests are supported by that API.&lt;/p&gt;

&lt;p&gt;With MCP, requests are also JSON and responses are also JSON, but in addition, there are standard requests built into the protocol to obtain useful information, including what tools are provided, what arguments each tool expects, and a prose description of how each tool is expected to be used.&lt;/p&gt;

&lt;p&gt;As an aside, don&amp;rsquo;t automatically assume that you will get good results from &lt;a href='https://neon.com/blog/autogenerating-mcp-servers-openai-schemas' title=''&gt;auto-generating MCP Servers from OpenAPI schemas&lt;/a&gt;:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Effective MCP design means thinking about the workflows you want the agent to perform and potentially creating higher-level tools that encapsulate multiple API calls, rather than just exposing the raw building blocks of your entire API spec.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;&lt;a href='https://glama.ai/blog/2025-06-06-mcp-vs-api#five-fundamental-differences' title=''&gt;MCP vs API&lt;/a&gt; goes into this topic at greater depth.&lt;/p&gt;

&lt;p&gt;In many cases you will get better results treating LLMs as humans. If you have a CLI, consider using that as the starting point instead.&lt;/p&gt;
&lt;h2 id='mcps-are-not-serverless' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#mcps-are-not-serverless' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;MCPs are &lt;strong&gt;&lt;i&gt;not&lt;/i&gt;&lt;/strong&gt; serverless&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;&lt;a href='https://en.wikipedia.org/wiki/Serverless_computing' title=''&gt;Serverless&lt;/a&gt;, sometimes known as &lt;a href='https://en.wikipedia.org/wiki/Function_as_a_service' title=''&gt;FaaS&lt;/a&gt;, is a popular cloud computing model, where AWS Lambda is widely recognized as the archetype.&lt;/p&gt;

&lt;p&gt;MCP servers are not serverless; they have a well-defined and long-lived &lt;a href='https://modelcontextprotocol.io/specification/2025-03-26/basic/lifecycle' title=''&gt;lifecycle&lt;/a&gt;:&lt;/p&gt;

&lt;p&gt;&lt;svg aria-roledescription="sequence" role="graphics-document document" viewBox="-50 -10 482 651" style="max-width: 482px;" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns="http://www.w3.org/2000/svg" width="100%" id="rm"&gt;&lt;rect class="rect" height="70" width="302" fill="rgb(200, 220, 250)" y="325" x="40"&gt;&lt;/rect&gt;&lt;g&gt;&lt;rect class="actor actor-bottom" ry="3" rx="3" name="Server" height="65" width="150" stroke="#666" fill="#eaeaea" y="565" x="232"&gt;&lt;/rect&gt;&lt;text class="actor actor-box" alignment-baseline="central" dominant-baseline="central" style="text-anchor: middle; font-size: 16px; font-weight: 400; font-family: inherit;" y="597.5" x="307"&gt;&lt;tspan dy="0" x="307"&gt;Server&lt;/tspan&gt;&lt;/text&gt;&lt;/g&gt;&lt;g&gt;&lt;rect class="actor actor-bottom" ry="3" rx="3" name="Client" height="65" width="150" stroke="#666" fill="#eaeaea" y="565" x="0"&gt;&lt;/rect&gt;&lt;text class="actor actor-box" alignment-baseline="central" dominant-baseline="central" style="text-anchor: middle; font-size: 16px; font-weight: 400; font-family: inherit;" y="597.5" x="75"&gt;&lt;tspan dy="0" x="75"&gt;Client&lt;/tspan&gt;&lt;/text&gt;&lt;/g&gt;&lt;g&gt;&lt;line name="Server" stroke="#999" stroke-width="0.5px" class="actor-line 200" y2="565" x2="307" y1="65" x1="307" id="actor10"&gt;&lt;/line&gt;&lt;g id="root-10"&gt;&lt;rect class="actor actor-top" ry="3" rx="3" name="Server" height="65" width="150" stroke="#666" fill="#eaeaea" y="0" x="232"&gt;&lt;/rect&gt;&lt;text class="actor actor-box" alignment-baseline="central" dominant-baseline="central" style="text-anchor: middle; font-size: 16px; font-weight: 400; font-family: inherit;" y="32.5" x="307"&gt;&lt;tspan dy="0" x="307"&gt;Server&lt;/tspan&gt;&lt;/text&gt;&lt;/g&gt;&lt;/g&gt;&lt;g&gt;&lt;line name="Client" stroke="#999" stroke-width="0.5px" class="actor-line 200" y2="565" x2="75" y1="65" x1="75" id="actor9"&gt;&lt;/line&gt;&lt;g id="root-9"&gt;&lt;rect class="actor actor-top" ry="3" rx="3" name="Client" height="65" width="150" stroke="#666" fill="#eaeaea" y="0" x="0"&gt;&lt;/rect&gt;&lt;text class="actor actor-box" alignment-baseline="central" dominant-baseline="central" style="text-anchor: middle; font-size: 16px; font-weight: 400; font-family: inherit;" y="32.5" x="75"&gt;&lt;tspan dy="0" x="75"&gt;Client&lt;/tspan&gt;&lt;/text&gt;&lt;/g&gt;&lt;/g&gt;&lt;style&gt;#rm{font-family:inherit;font-size:16px;fill:#333;}#rm .error-icon{fill:#552222;}#rm .error-text{fill:#552222;stroke:#552222;}#rm .edge-thickness-normal{stroke-width:1px;}#rm .edge-thickness-thick{stroke-width:3.5px;}#rm .edge-pattern-solid{stroke-dasharray:0;}#rm .edge-thickness-invisible{stroke-width:0;fill:none;}#rm .edge-pattern-dashed{stroke-dasharray:3;}#rm .edge-pattern-dotted{stroke-dasharray:2;}#rm .marker{fill:#333333;stroke:#333333;}#rm .marker.cross{stroke:#333333;}#rm svg{font-family:inherit;font-size:16px;}#rm p{margin:0;}#rm .actor{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#rm text.actor&amp;gt;tspan{fill:black;stroke:none;}#rm .actor-line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#rm .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333;}#rm .messageLine1{stroke-width:1.5;stroke-dasharray:2,2;stroke:#333;}#rm #arrowhead path{fill:#333;stroke:#333;}#rm .sequenceNumber{fill:white;}#rm #sequencenumber{fill:#333;}#rm #crosshead path{fill:#333;stroke:#333;}#rm .messageText{fill:#333;stroke:none;}#rm .labelBox{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#rm .labelText,#rm .labelText&amp;gt;tspan{fill:black;stroke:none;}#rm .loopText,#rm .loopText&amp;gt;tspan{fill:black;stroke:none;}#rm .loopLine{stroke-width:2px;stroke-dasharray:2,2;stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);}#rm .note{stroke:#aaaa33;fill:#fff5ad;}#rm .noteText,#rm .noteText&amp;gt;tspan{fill:black;stroke:none;}#rm .activation0{fill:#f4f4f4;stroke:#666;}#rm .activation1{fill:#f4f4f4;stroke:#666;}#rm .activation2{fill:#f4f4f4;stroke:#666;}#rm .actorPopupMenu{position:absolute;}#rm .actorPopupMenuPanel{position:absolute;fill:#ECECFF;box-shadow:0px 8px 16px 0px rgba(0,0,0,0.2);filter:drop-shadow(3px 5px 2px rgb(0 0 0 / 0.4));}#rm .actor-man line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;}#rm .actor-man circle,#rm line{stroke:hsl(259.6261682243, 59.7765363128%, 87.9019607843%);fill:#ECECFF;stroke-width:2px;}#rm :root{--mermaid-font-family:inherit;}&lt;/style&gt;&lt;g&gt;&lt;/g&gt;&lt;defs&gt;&lt;symbol height="24" width="24" id="computer"&gt;&lt;path d="M2 2v13h20v-13h-20zm18 11h-16v-9h16v9zm-10.228 6l.466-1h3.524l.467 1h-4.457zm14.228 3h-24l2-6h2.104l-1.33 4h18.45l-1.297-4h2.073l2 6zm-5-10h-14v-7h14v7z" transform="scale(.5)"&gt;&lt;/path&gt;&lt;/symbol&gt;&lt;/defs&gt;&lt;defs&gt;&lt;symbol clip-rule="evenodd" fill-rule="evenodd" id="database"&gt;&lt;path d="M12.258.001l.256.004.255.005.253.008.251.01.249.012.247.015.246.016.242.019.241.02.239.023.236.024.233.027.231.028.229.031.225.032.223.034.22.036.217.038.214.04.211.041.208.043.205.045.201.046.198.048.194.05.191.051.187.053.183.054.18.056.175.057.172.059.168.06.163.061.16.063.155.064.15.066.074.033.073.033.071.034.07.034.069.035.068.035.067.035.066.035.064.036.064.036.062.036.06.036.06.037.058.037.058.037.055.038.055.038.053.038.052.038.051.039.05.039.048.039.047.039.045.04.044.04.043.04.041.04.04.041.039.041.037.041.036.041.034.041.033.042.032.042.03.042.029.042.027.042.026.043.024.043.023.043.021.043.02.043.018.044.017.043.015.044.013.044.012.044.011.045.009.044.007.045.006.045.004.045.002.045.001.045v17l-.001.045-.002.045-.004.045-.006.045-.007.045-.009.044-.011.045-.012.044-.013.044-.015.044-.017.043-.018.044-.02.043-.021.043-.023.043-.024.043-.026.043-.027.042-.029.042-.03.042-.032.042-.033.042-.034.041-.036.041-.037.041-.039.041-.04.041-.041.04-.043.04-.044.04-.045.04-.047.039-.048.039-.05.039-.051.039-.052.038-.053.038-.055.038-.055.038-.058.037-.058.037-.06.037-.06.036-.062.036-.064.036-.064.036-.066.035-.067.035-.068.035-.069.035-.07.034-.071.034-.073.033-.074.033-.15.066-.155.064-.16.063-.163.061-.168.06-.172.059-.175.057-.18.056-.183.054-.187.053-.191.051-.194.05-.198.048-.201.046-.205.045-.208.043-.211.041-.214.04-.217.038-.22.036-.223.034-.225.032-.229.031-.231.028-.233.027-.236.024-.239.023-.241.02-.242.019-.246.016-.247.015-.249.012-.251.01-.253.008-.255.005-.256.004-.258.001-.258-.001-.256-.004-.255-.005-.253-.008-.251-.01-.249-.012-.247-.015-.245-.016-.243-.019-.241-.02-.238-.023-.236-.024-.234-.027-.231-.028-.228-.031-.226-.032-.223-.034-.22-.036-.217-.038-.214-.04-.211-.041-.208-.043-.204-.045-.201-.046-.198-.048-.195-.05-.19-.051-.187-.053-.184-.054-.179-.056-.176-.057-.172-.059-.167-.06-.164-.061-.159-.063-.155-.064-.151-.066-.074-.033-.072-.033-.072-.034-.07-.034-.069-.035-.068-.035-.067-.035-.066-.035-.064-.036-.063-.036-.062-.036-.061-.036-.06-.037-.058-.037-.057-.037-.056-.038-.055-.038-.053-.038-.052-.038-.051-.039-.049-.039-.049-.039-.046-.039-.046-.04-.044-.04-.043-.04-.041-.04-.04-.041-.039-.041-.037-.041-.036-.041-.034-.041-.033-.042-.032-.042-.03-.042-.029-.042-.027-.042-.026-.043-.024-.043-.023-.043-.021-.043-.02-.043-.018-.044-.017-.043-.015-.044-.013-.044-.012-.044-.011-.045-.009-.044-.007-.045-.006-.045-.004-.045-.002-.045-.001-.045v-17l.001-.045.002-.045.004-.045.006-.045.007-.045.009-.044.011-.045.012-.044.013-.044.015-.044.017-.043.018-.044.02-.043.021-.043.023-.043.024-.043.026-.043.027-.042.029-.042.03-.042.032-.042.033-.042.034-.041.036-.041.037-.041.039-.041.04-.041.041-.04.043-.04.044-.04.046-.04.046-.039.049-.039.049-.039.051-.039.052-.038.053-.038.055-.038.056-.038.057-.037.058-.037.06-.037.061-.036.062-.036.063-.036.064-.036.066-.035.067-.035.068-.035.069-.035.07-.034.072-.034.072-.033.074-.033.151-.066.155-.064.159-.063.164-.061.167-.06.172-.059.176-.057.179-.056.184-.054.187-.053.19-.051.195-.05.198-.048.201-.046.204-.045.208-.043.211-.041.214-.04.217-.038.22-.036.223-.034.226-.032.228-.031.231-.028.234-.027.236-.024.238-.023.241-.02.243-.019.245-.016.247-.015.249-.012.251-.01.253-.008.255-.005.256-.004.258-.001.258.001zm-9.258 20.499v.01l.001.021.003.021.004.022.005.021.006.022.007.022.009.023.01.022.011.023.012.023.013.023.015.023.016.024.017.023.018.024.019.024.021.024.022.025.023.024.024.025.052.049.056.05.061.051.066.051.07.051.075.051.079.052.084.052.088.052.092.052.097.052.102.051.105.052.11.052.114.051.119.051.123.051.127.05.131.05.135.05.139.048.144.049.147.047.152.047.155.047.16.045.163.045.167.043.171.043.176.041.178.041.183.039.187.039.19.037.194.035.197.035.202.033.204.031.209.03.212.029.216.027.219.025.222.024.226.021.23.02.233.018.236.016.24.015.243.012.246.01.249.008.253.005.256.004.259.001.26-.001.257-.004.254-.005.25-.008.247-.011.244-.012.241-.014.237-.016.233-.018.231-.021.226-.021.224-.024.22-.026.216-.027.212-.028.21-.031.205-.031.202-.034.198-.034.194-.036.191-.037.187-.039.183-.04.179-.04.175-.042.172-.043.168-.044.163-.045.16-.046.155-.046.152-.047.148-.048.143-.049.139-.049.136-.05.131-.05.126-.05.123-.051.118-.052.114-.051.11-.052.106-.052.101-.052.096-.052.092-.052.088-.053.083-.051.079-.052.074-.052.07-.051.065-.051.06-.051.056-.05.051-.05.023-.024.023-.025.021-.024.02-.024.019-.024.018-.024.017-.024.015-.023.014-.024.013-.023.012-.023.01-.023.01-.022.008-.022.006-.022.006-.022.004-.022.004-.021.001-.021.001-.021v-4.127l-.077.055-.08.053-.083.054-.085.053-.087.052-.09.052-.093.051-.095.05-.097.05-.1.049-.102.049-.105.048-.106.047-.109.047-.111.046-.114.045-.115.045-.118.044-.12.043-.122.042-.124.042-.126.041-.128.04-.13.04-.132.038-.134.038-.135.037-.138.037-.139.035-.142.035-.143.034-.144.033-.147.032-.148.031-.15.03-.151.03-.153.029-.154.027-.156.027-.158.026-.159.025-.161.024-.162.023-.163.022-.165.021-.166.02-.167.019-.169.018-.169.017-.171.016-.173.015-.173.014-.175.013-.175.012-.177.011-.178.01-.179.008-.179.008-.181.006-.182.005-.182.004-.184.003-.184.002h-.37l-.184-.002-.184-.003-.182-.004-.182-.005-.181-.006-.179-.008-.179-.008-.178-.01-.176-.011-.176-.012-.175-.013-.173-.014-.172-.015-.171-.016-.17-.017-.169-.018-.167-.019-.166-.02-.165-.021-.163-.022-.162-.023-.161-.024-.159-.025-.157-.026-.156-.027-.155-.027-.153-.029-.151-.03-.15-.03-.148-.031-.146-.032-.145-.033-.143-.034-.141-.035-.14-.035-.137-.037-.136-.037-.134-.038-.132-.038-.13-.04-.128-.04-.126-.041-.124-.042-.122-.042-.12-.044-.117-.043-.116-.045-.113-.045-.112-.046-.109-.047-.106-.047-.105-.048-.102-.049-.1-.049-.097-.05-.095-.05-.093-.052-.09-.051-.087-.052-.085-.053-.083-.054-.08-.054-.077-.054v4.127zm0-5.654v.011l.001.021.003.021.004.021.005.022.006.022.007.022.009.022.01.022.011.023.012.023.013.023.015.024.016.023.017.024.018.024.019.024.021.024.022.024.023.025.024.024.052.05.056.05.061.05.066.051.07.051.075.052.079.051.084.052.088.052.092.052.097.052.102.052.105.052.11.051.114.051.119.052.123.05.127.051.131.05.135.049.139.049.144.048.147.048.152.047.155.046.16.045.163.045.167.044.171.042.176.042.178.04.183.04.187.038.19.037.194.036.197.034.202.033.204.032.209.03.212.028.216.027.219.025.222.024.226.022.23.02.233.018.236.016.24.014.243.012.246.01.249.008.253.006.256.003.259.001.26-.001.257-.003.254-.006.25-.008.247-.01.244-.012.241-.015.237-.016.233-.018.231-.02.226-.022.224-.024.22-.025.216-.027.212-.029.21-.03.205-.032.202-.033.198-.035.194-.036.191-.037.187-.039.183-.039.179-.041.175-.042.172-.043.168-.044.163-.045.16-.045.155-.047.152-.047.148-.048.143-.048.139-.05.136-.049.131-.05.126-.051.123-.051.118-.051.114-.052.11-.052.106-.052.101-.052.096-.052.092-.052.088-.052.083-.052.079-.052.074-.051.07-.052.065-.051.06-.05.056-.051.051-.049.023-.025.023-.024.021-.025.02-.024.019-.024.018-.024.017-.024.015-.023.014-.023.013-.024.012-.022.01-.023.01-.023.008-.022.006-.022.006-.022.004-.021.004-.022.001-.021.001-.021v-4.139l-.077.054-.08.054-.083.054-.085.052-.087.053-.09.051-.093.051-.095.051-.097.05-.1.049-.102.049-.105.048-.106.047-.109.047-.111.046-.114.045-.115.044-.118.044-.12.044-.122.042-.124.042-.126.041-.128.04-.13.039-.132.039-.134.038-.135.037-.138.036-.139.036-.142.035-.143.033-.144.033-.147.033-.148.031-.15.03-.151.03-.153.028-.154.028-.156.027-.158.026-.159.025-.161.024-.162.023-.163.022-.165.021-.166.02-.167.019-.169.018-.169.017-.171.016-.173.015-.173.014-.175.013-.175.012-.177.011-.178.009-.179.009-.179.007-.181.007-.182.005-.182.004-.184.003-.184.002h-.37l-.184-.002-.184-.003-.182-.004-.182-.005-.181-.007-.179-.007-.179-.009-.178-.009-.176-.011-.176-.012-.175-.013-.173-.014-.172-.015-.171-.016-.17-.017-.169-.018-.167-.019-.166-.02-.165-.021-.163-.022-.162-.023-.161-.024-.159-.025-.157-.026-.156-.027-.155-.028-.153-.028-.151-.03-.15-.03-.148-.031-.146-.033-.145-.033-.143-.033-.141-.035-.14-.036-.137-.036-.136-.037-.134-.038-.132-.039-.13-.039-.128-.04-.126-.041-.124-.042-.122-.043-.12-.043-.117-.044-.116-.044-.113-.046-.112-.046-.109-.046-.106-.047-.105-.048-.102-.049-.1-.049-.097-.05-.095-.051-.093-.051-.09-.051-.087-.053-.085-.052-.083-.054-.08-.054-.077-.054v4.139zm0-5.666v.011l.001.02.003.022.004.021.005.022.006.021.007.022.009.023.01.022.011.023.012.023.013.023.015.023.016.024.017.024.018.023.019.024.021.025.022.024.023.024.024.025.052.05.056.05.061.05.066.051.07.051.075.052.079.051.084.052.088.052.092.052.097.052.102.052.105.051.11.052.114.051.119.051.123.051.127.05.131.05.135.05.139.049.144.048.147.048.152.047.155.046.16.045.163.045.167.043.171.043.176.042.178.04.183.04.187.038.19.037.194.036.197.034.202.033.204.032.209.03.212.028.216.027.219.025.222.024.226.021.23.02.233.018.236.017.24.014.243.012.246.01.249.008.253.006.256.003.259.001.26-.001.257-.003.254-.006.25-.008.247-.01.244-.013.241-.014.237-.016.233-.018.231-.02.226-.022.224-.024.22-.025.216-.027.212-.029.21-.03.205-.032.202-.033.198-.035.194-.036.191-.037.187-.039.183-.039.179-.041.175-.042.172-.043.168-.044.163-.045.16-.045.155-.047.152-.047.148-.048.143-.049.139-.049.136-.049.131-.051.126-.05.123-.051.118-.052.114-.051.11-.052.106-.052.101-.052.096-.052.092-.052.088-.052.083-.052.079-.052.074-.052.07-.051.065-.051.06-.051.056-.05.051-.049.023-.025.023-.025.021-.024.02-.024.019-.024.018-.024.017-.024.015-.023.014-.024.013-.023.012-.023.01-.022.01-.023.008-.022.006-.022.006-.022.004-.022.004-.021.001-.021.001-.021v-4.153l-.077.054-.08.054-.083.053-.085.053-.087.053-.09.051-.093.051-.095.051-.097.05-.1.049-.102.048-.105.048-.106.048-.109.046-.111.046-.114.046-.115.044-.118.044-.12.043-.122.043-.124.042-.126.041-.128.04-.13.039-.132.039-.134.038-.135.037-.138.036-.139.036-.142.034-.143.034-.144.033-.147.032-.148.032-.15.03-.151.03-.153.028-.154.028-.156.027-.158.026-.159.024-.161.024-.162.023-.163.023-.165.021-.166.02-.167.019-.169.018-.169.017-.171.016-.173.015-.173.014-.175.013-.175.012-.177.01-.178.01-.179.009-.179.007-.181.006-.182.006-.182.004-.184.003-.184.001-.185.001-.185-.001-.184-.001-.184-.003-.182-.004-.182-.006-.181-.006-.179-.007-.179-.009-.178-.01-.176-.01-.176-.012-.175-.013-.173-.014-.172-.015-.171-.016-.17-.017-.169-.018-.167-.019-.166-.02-.165-.021-.163-.023-.162-.023-.161-.024-.159-.024-.157-.026-.156-.027-.155-.028-.153-.028-.151-.03-.15-.03-.148-.032-.146-.032-.145-.033-.143-.034-.141-.034-.14-.036-.137-.036-.136-.037-.134-.038-.132-.039-.13-.039-.128-.041-.126-.041-.124-.041-.122-.043-.12-.043-.117-.044-.116-.044-.113-.046-.112-.046-.109-.046-.106-.048-.105-.048-.102-.048-.1-.05-.097-.049-.095-.051-.093-.051-.09-.052-.087-.052-.085-.053-.083-.053-.08-.054-.077-.054v4.153zm8.74-8.179l-.257.004-.254.005-.25.008-.247.011-.244.012-.241.014-.237.016-.233.018-.231.021-.226.022-.224.023-.22.026-.216.027-.212.028-.21.031-.205.032-.202.033-.198.034-.194.036-.191.038-.187.038-.183.04-.179.041-.175.042-.172.043-.168.043-.163.045-.16.046-.155.046-.152.048-.148.048-.143.048-.139.049-.136.05-.131.05-.126.051-.123.051-.118.051-.114.052-.11.052-.106.052-.101.052-.096.052-.092.052-.088.052-.083.052-.079.052-.074.051-.07.052-.065.051-.06.05-.056.05-.051.05-.023.025-.023.024-.021.024-.02.025-.019.024-.018.024-.017.023-.015.024-.014.023-.013.023-.012.023-.01.023-.01.022-.008.022-.006.023-.006.021-.004.022-.004.021-.001.021-.001.021.001.021.001.021.004.021.004.022.006.021.006.023.008.022.01.022.01.023.012.023.013.023.014.023.015.024.017.023.018.024.019.024.02.025.021.024.023.024.023.025.051.05.056.05.06.05.065.051.07.052.074.051.079.052.083.052.088.052.092.052.096.052.101.052.106.052.11.052.114.052.118.051.123.051.126.051.131.05.136.05.139.049.143.048.148.048.152.048.155.046.16.046.163.045.168.043.172.043.175.042.179.041.183.04.187.038.191.038.194.036.198.034.202.033.205.032.21.031.212.028.216.027.22.026.224.023.226.022.231.021.233.018.237.016.241.014.244.012.247.011.25.008.254.005.257.004.26.001.26-.001.257-.004.254-.005.25-.008.247-.011.244-.012.241-.014.237-.016.233-.018.231-.021.226-.022.224-.023.22-.026.216-.027.212-.028.21-.031.205-.032.202-.033.198-.034.194-.036.191-.038.187-.038.183-.04.179-.041.175-.042.172-.043.168-.043.163-.045.16-.046.155-.046.152-.048.148-.048.143-.048.139-.049.136-.05.131-.05.126-.051.123-.051.118-.051.114-.052.11-.052.106-.052.101-.052.096-.052.092-.052.088-.052.083-.052.079-.052.074-.051.07-.052.065-.051.06-.05.056-.05.051-.05.023-.025.023-.024.021-.024.02-.025.019-.024.018-.024.017-.023.015-.024.014-.023.013-.023.012-.023.01-.023.01-.022.008-.022.006-.023.006-.021.004-.022.004-.021.001-.021.001-.021-.001-.021-.001-.021-.004-.021-.004-.022-.006-.021-.006-.023-.008-.022-.01-.022-.01-.023-.012-.023-.013-.023-.014-.023-.015-.024-.017-.023-.018-.024-.019-.024-.02-.025-.021-.024-.023-.024-.023-.025-.051-.05-.056-.05-.06-.05-.065-.051-.07-.052-.074-.051-.079-.052-.083-.052-.088-.052-.092-.052-.096-.052-.101-.052-.106-.052-.11-.052-.114-.052-.118-.051-.123-.051-.126-.051-.131-.05-.136-.05-.139-.049-.143-.048-.148-.048-.152-.048-.155-.046-.16-.046-.163-.045-.168-.043-.172-.043-.175-.042-.179-.041-.183-.04-.187-.038-.191-.038-.194-.036-.198-.034-.202-.033-.205-.032-.21-.031-.212-.028-.216-.027-.22-.026-.224-.023-.226-.022-.231-.021-.233-.018-.237-.016-.241-.014-.244-.012-.247-.011-.25-.008-.254-.005-.257-.004-.26-.001-.26.001z" transform="scale(.5)"&gt;&lt;/path&gt;&lt;/symbol&gt;&lt;/defs&gt;&lt;defs&gt;&lt;symbol height="24" width="24" id="clock"&gt;&lt;path d="M12 2c5.514 0 10 4.486 10 10s-4.486 10-10 10-10-4.486-10-10 4.486-10 10-10zm0-2c-6.627 0-12 5.373-12 12s5.373 12 12 12 12-5.373 12-12-5.373-12-12-12zm5.848 12.459c.202.038.202.333.001.372-1.907.361-6.045 1.111-6.547 1.111-.719 0-1.301-.582-1.301-1.301 0-.512.77-5.447 1.125-7.445.034-.192.312-.181.343.014l.985 6.238 5.394 1.011z" transform="scale(.5)"&gt;&lt;/path&gt;&lt;/symbol&gt;&lt;/defs&gt;&lt;defs&gt;&lt;marker orient="auto-start-reverse" markerHeight="12" markerWidth="12" markerUnits="userSpaceOnUse" refY="5" refX="7.9" id="arrowhead"&gt;&lt;path d="M -1 0 L 10 5 L 0 10 z"&gt;&lt;/path&gt;&lt;/marker&gt;&lt;/defs&gt;&lt;defs&gt;&lt;marker refY="4.5" refX="4" orient="auto" markerHeight="8" markerWidth="15" id="crosshead"&gt;&lt;path d="M 1,2 L 6,7 M 6,2 L 1,7" stroke-width="1pt" style="stroke-dasharray: 0px, 0px;" stroke="#000000" fill="none"&gt;&lt;/path&gt;&lt;/marker&gt;&lt;/defs&gt;&lt;defs&gt;&lt;marker orient="auto" markerHeight="28" markerWidth="20" refY="7" refX="15.5" id="filled-head"&gt;&lt;path d="M 18,7 L9,13 L14,7 L9,1 Z"&gt;&lt;/path&gt;&lt;/marker&gt;&lt;/defs&gt;&lt;defs&gt;&lt;marker orient="auto" markerHeight="40" markerWidth="60" refY="15" refX="15" id="sequencenumber"&gt;&lt;circle r="6" cy="15" cx="15"&gt;&lt;/circle&gt;&lt;/marker&gt;&lt;/defs&gt;&lt;g&gt;&lt;rect class="note" height="40" width="282" stroke="#666" fill="#EDF2AE" y="75" x="50"&gt;&lt;/rect&gt;&lt;text dy="1em" class="noteText" style="font-family: inherit; font-size: 16px; font-weight: 400;" alignment-baseline="middle" dominant-baseline="middle" text-anchor="middle" y="80" x="191"&gt;&lt;tspan x="191"&gt;Initialization Phase&lt;/tspan&gt;&lt;/text&gt;&lt;/g&gt;&lt;g&gt;&lt;rect class="activation0" height="380" width="10" stroke="#666" fill="#EDF2AE" y="115" x="70"&gt;&lt;/rect&gt;&lt;/g&gt;&lt;g&gt;&lt;rect class="activation0" height="328" width="10" stroke="#666" fill="#EDF2AE" y="167" x="302"&gt;&lt;/rect&gt;&lt;/g&gt;&lt;g&gt;&lt;rect class="note" height="40" width="282" stroke="#666" fill="#EDF2AE" y="275" x="50"&gt;&lt;/rect&gt;&lt;text dy="1em" class="noteText" style="font-family: inherit; font-size: 16px; font-weight: 400;" alignment-baseline="middle" dominant-baseline="middle" text-anchor="middle" y="280" x="191"&gt;&lt;tspan x="191"&gt;Operation Phase&lt;/tspan&gt;&lt;/text&gt;&lt;/g&gt;&lt;g&gt;&lt;rect class="note" height="40" width="282" stroke="#666" fill="#EDF2AE" y="345" x="50"&gt;&lt;/rect&gt;&lt;text dy="1em" class="noteText" style="font-family: inherit; font-size: 16px; font-weight: 400;" alignment-baseline="middle" dominant-baseline="middle" text-anchor="middle" y="350" x="191"&gt;&lt;tspan x="191"&gt;Normal protocol operations&lt;/tspan&gt;&lt;/text&gt;&lt;/g&gt;&lt;g&gt;&lt;rect class="note" height="40" width="282" stroke="#666" fill="#EDF2AE" y="405" x="50"&gt;&lt;/rect&gt;&lt;text dy="1em" class="noteText" style="font-family: inherit; font-size: 16px; font-weight: 400;" alignment-baseline="middle" dominant-baseline="middle" text-anchor="middle" y="410" x="191"&gt;&lt;tspan x="191"&gt;Shutdown&lt;/tspan&gt;&lt;/text&gt;&lt;/g&gt;&lt;g&gt;&lt;rect class="note" height="40" width="282" stroke="#666" fill="#EDF2AE" y="505" x="50"&gt;&lt;/rect&gt;&lt;text dy="1em" class="noteText" style="font-family: inherit; font-size: 16px; font-weight: 400;" alignment-baseline="middle" dominant-baseline="middle" text-anchor="middle" y="510" x="191"&gt;&lt;tspan x="191"&gt;Connection closed&lt;/tspan&gt;&lt;/text&gt;&lt;/g&gt;&lt;text dy="1em" class="messageText" style="font-family: inherit; font-size: 16px; font-weight: 400;" alignment-baseline="middle" dominant-baseline="middle" text-anchor="middle" y="130" x="190"&gt;initialize request&lt;/text&gt;&lt;line marker-end="url(#arrowhead)" style="fill: none;" stroke="none" stroke-width="2" class="messageLine0" y2="165" x2="299" y1="165" x1="80"&gt;&lt;/line&gt;&lt;text dy="1em" class="messageText" style="font-family: inherit; font-size: 16px; font-weight: 400;" alignment-baseline="middle" dominant-baseline="middle" text-anchor="middle" y="180" x="193"&gt;initialize response&lt;/text&gt;&lt;line marker-end="url(#arrowhead)" stroke="none" stroke-width="2" class="messageLine1" style="stroke-dasharray: 3px, 3px; fill: none;" y2="215" x2="83" y1="215" x1="302"&gt;&lt;/line&gt;&lt;text dy="1em" class="messageText" style="font-family: inherit; font-size: 16px; font-weight: 400;" alignment-baseline="middle" dominant-baseline="middle" text-anchor="middle" y="230" x="190"&gt;initialized notification&lt;/text&gt;&lt;line marker-end="url(#filled-head)" stroke="none" stroke-width="2" class="messageLine1" style="stroke-dasharray: 3px, 3px; fill: none;" y2="265" x2="299" y1="265" x1="80"&gt;&lt;/line&gt;&lt;text dy="1em" class="messageText" style="font-family: inherit; font-size: 16px; font-weight: 400;" alignment-baseline="middle" dominant-baseline="middle" text-anchor="middle" y="460" x="190"&gt;Disconnect&lt;/text&gt;&lt;line marker-end="url(#filled-head)" stroke="none" stroke-width="2" class="messageLine1" style="stroke-dasharray: 3px, 3px; fill: none;" y2="495" x2="299" y1="495" x1="80"&gt;&lt;/line&gt;&lt;/svg&gt;&lt;/p&gt;
&lt;figure class="post-cta"&gt;
  &lt;figcaption&gt;
    &lt;h2&gt;You can play with this right now.&lt;/h2&gt;
    &lt;p&gt;MCPs are barely six months old, but we are keeping up with the latest&lt;/p&gt;
      &lt;a class="btn btn-lg" href="https://docs.fly.io/model-context-protocol"&gt;
        Try launching your MCP server on Fly.io today &lt;span class='opacity-50'&gt;→&lt;/span&gt;
      &lt;/a&gt;
  &lt;/figcaption&gt;
  &lt;div class="image-container"&gt;
    &lt;img loading="lazy" decoding="async" src="/static/images/cta-kitty.webp" srcset="/static/images/cta-kitty@2x.webp 2x" alt=""&gt;
  &lt;/div&gt;
&lt;/figure&gt;

&lt;h2 id='mcps-are-not-inherently-secure-or-private' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#mcps-are-not-inherently-secure-or-private' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;MCPs are &lt;strong&gt;&lt;i&gt;not&lt;/i&gt;&lt;/strong&gt; Inherently Secure or Private&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Here I am not talking about &lt;a href='https://simonwillison.net/2025/Apr/9/mcp-prompt-injection/' title=''&gt;prompt injection&lt;/a&gt; or &lt;a href='https://simonwillison.net/2025/May/26/github-mcp-exploited/' title=''&gt;exploitable abilities&lt;/a&gt;, though those are real problems too.&lt;/p&gt;

&lt;p&gt;I&amp;rsquo;m talking about something more fundamental and basic. Let&amp;rsquo;s take a look at the very same &lt;a href='https://github.com/github/github-mcp-server' title=''&gt;GitHub MCP&lt;/a&gt; featured in the above two descriptions of an exploitable exposure. The current process for installing a stdio MCP into Claude involves placing secrets in plain text in a well-defined location. And the process for installing the &lt;em&gt;next&lt;/em&gt; MCP server is to download a program from a third party and run that tool in a way that has access to this very file.&lt;/p&gt;

&lt;p&gt;Addressing MCP security requires a holistic approach, but one key strategy component is the ability to run an MCP server on a remote machine which can only be accessed by you, and only after you present a revocable bearer token. That remote machine may require access to secrets (like your GitHub token), but those secrets are not something either your LLM client or other MCP servers will be able to access directly.&lt;/p&gt;
&lt;h2 id='mcps-should-be-considered-family' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#mcps-should-be-considered-family' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;MCPs should be considered family&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Recapping: &lt;a href='https://www.usa.philips.com/' title=''&gt;Philips&lt;/a&gt; has an &lt;a href='https://developers.meethue.com/' title=''&gt;API and SDK&lt;/a&gt; for Hue that is used by perhaps thousands, and has an &lt;a href='https://www.amazon.com/Philips-Hue/dp/B01LWAGUG7' title=''&gt;Alexa Skill&lt;/a&gt; that is used by untold millions. Of course, somebody already built a &lt;a href='https://github.com/ThomasRohde/hue-mcp' title=''&gt;Philips Hue MCP Server&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;LLMs are brains. MCPs give LLMs eyes, hands, and legs. The end result is a robot. Most MCPs today are brainless—they merely are eyes, hands, and legs. The results are appliances. I buy and install a dishwasher. You do too. Both of our dishwashers perform the same basic function. As do any Hue lights we may buy.&lt;/p&gt;

&lt;p&gt;In The Jetsons, &lt;a href='https://thejetsons.fandom.com/wiki/Rosey' title=''&gt;Rosie&lt;/a&gt; is a maid and housekeeper who is also a member of the family. She is good friends with Jane and takes the role of a surrogate aunt towards Elroy and Judy. Let&amp;rsquo;s start there and go further.&lt;/p&gt;

&lt;p&gt;A person with an LLM can build things. Imagine having a 3D printer that makes robots or agents that act on your behalf. You may want an agent to watch for trends in your business, keep track of what events are happening in your area, screen your text messages, or act as a DJ when you get together with friends.&lt;/p&gt;

&lt;p&gt;You may share the MCP servers you create with others to use as starting points, but they undoubtedly will adapt them and make them their own.&lt;/p&gt;
&lt;h2 id='closing-thoughts' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#closing-thoughts' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Closing Thoughts&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Don&amp;rsquo;t get me wrong. I am not saying there won&amp;rsquo;t be MCPs that are mere appliances—there undoubtedly will be plenty of those. But not all MCPs will be appliances; many will be agents.&lt;/p&gt;

&lt;p&gt;Today, most people exploring LLMs are trying to add LLMs to things they already have—for example, IDEs. MCPs flip this. Instead of adding LLMs to something you already have, you add something you already have to an LLM.&lt;/p&gt;

&lt;p&gt;&lt;a href='https://desktopcommander.app/' title=''&gt;Desktop Commander MCP&lt;/a&gt; is an example I&amp;rsquo;m particularly fond of that does exactly this. It also happens to be a prime example of a tool you want to give root access to, then lock it in a secure box and run someplace other than your laptop. &lt;a href='https://docs.fly.io/model-context-protocol/examples#desktop-commander' title=''&gt;Give it a try&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Microsoft is actively working on &lt;a href='https://developer.microsoft.com/en-us/windows/agentic/' title=''&gt;Agentic Windows&lt;/a&gt;. Your smartphone is the obvious next frontier. Today, you have an app store and a web browser. MCP servers have the potential to make both obsolete—and this could happen sooner than you think.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>My AI Skeptic Friends Are All Nuts</title>
    <link rel="alternate" href="https://fly.io/blog/youre-all-nuts/"/>
    <id>https://fly.io/blog/youre-all-nuts/</id>
    <published>2025-06-02T00:00:00+00:00</published>
    <updated>2026-09-22T09:57:11+00:00</updated>
    <media:thumbnail url="https://fly.io/blog/youre-all-nuts/assets/whoah.webp"/>
    <content type="html">&lt;div class="lead"&gt;&lt;p&gt;A heartfelt provocation about AI-assisted programming.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;Tech execs are mandating LLM adoption. That&amp;rsquo;s bad strategy. But I get where they&amp;rsquo;re coming from.&lt;/p&gt;

&lt;p&gt;Some of the smartest people I know share a bone-deep belief that AI is a fad —  the next iteration of NFT mania. I&amp;rsquo;ve been reluctant to push back on them, because, well, they&amp;rsquo;re smarter than me. But their arguments are unserious, and worth confronting. Extraordinarily talented people are doing work that LLMs already do better, out of  spite.&lt;/p&gt;

&lt;p&gt;All progress on LLMs could halt today, and LLMs would remain the 2nd most important thing to happen over the course of my career.&lt;/p&gt;
&lt;div class="callout"&gt;&lt;p&gt;&lt;strong class="font-semibold text-navy-950"&gt;Important caveat&lt;/strong&gt;: I’m discussing only the implications of LLMs for software development. For art, music, and writing? I got nothing. I’m inclined to believe the skeptics in those fields. I just don’t believe them about mine.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;Bona fides: I&amp;rsquo;ve been shipping software since the mid-1990s. I started out in boxed, shrink-wrap C code.  Survived an ill-advised &lt;a href='https://www.amazon.com/Modern-Design-Generic-Programming-Patterns/dp/0201704315' title=''&gt;Alexandrescu&lt;/a&gt; C++ phase. Lots of Ruby and Python tooling.  Some kernel work. A whole lot of server-side C, Go, and Rust. However you define &amp;ldquo;serious developer&amp;rdquo;, I qualify. Even if only on one of your lower tiers.&lt;/p&gt;
&lt;h3 id='level-setting' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#level-setting' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;level setting&lt;/span&gt;&lt;/h3&gt;&lt;div class="right-sidenote"&gt;&lt;p&gt;† (or, God forbid, 2 years ago with Copilot)&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;First, we need to get on the same page. If you were trying and failing to use an LLM for code 6 months ago †, you’re not doing what most serious LLM-assisted coders are doing.&lt;/p&gt;

&lt;p&gt;People coding with LLMs today use agents. Agents get to poke around your codebase on their own. They author files directly. They run tools. They compile code, run tests, and iterate on the results. They also:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;pull in arbitrary code from the tree, or from other trees online, into their context windows,
&lt;/li&gt;&lt;li&gt;run standard Unix tools to navigate the tree and extract information,
&lt;/li&gt;&lt;li&gt;interact with Git,
&lt;/li&gt;&lt;li&gt;run existing tooling, like linters, formatters, and model checkers, and
&lt;/li&gt;&lt;li&gt;make essentially arbitrary tool calls (that you set up) through MCP.
&lt;/li&gt;&lt;/ul&gt;
&lt;div class="callout"&gt;&lt;p&gt;The code in an agent that actually “does stuff” with code is not, itself, AI. This should reassure you. It’s surprisingly simple systems code, wired to ground truth about programming in the same way a Makefile is. You could write an effective coding agent in a weekend. Its strengths would have more to do with how you think about and structure builds and linting and test harnesses than with how advanced o3 or Sonnet have become.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;If you’re making requests on a ChatGPT page and then pasting the resulting (broken) code into your editor, you’re not doing what the AI boosters are doing. No wonder you&amp;rsquo;re talking past each other.&lt;/p&gt;
&lt;h3 id='the-positive-case' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#the-positive-case' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;the positive case&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;img loading="lazy" decoding="async" alt="four quadrants of tedium and importance" width="932" height="930" src="/blog/youre-all-nuts/assets/code-quad.png?2/3&amp;amp;center" /&gt;&lt;/p&gt;

&lt;p&gt;LLMs can write a large fraction of all the tedious code you’ll ever need to write. And most code on most projects is tedious. LLMs drastically reduce the number of things you’ll ever need to Google. They look things up themselves. Most importantly, they don’t get tired; they’re immune to inertia.&lt;/p&gt;

&lt;p&gt;Think of anything you wanted to build but didn&amp;rsquo;t. You tried to home in on some first steps. If you&amp;rsquo;d been in the limerent phase of a new programming language, you&amp;rsquo;d have started writing. But you weren&amp;rsquo;t, so you put it off, for a day, a year, or your whole career.&lt;/p&gt;

&lt;p&gt;I can feel my blood pressure rising thinking of all the bookkeeping and Googling and dependency drama of a new project. An LLM can be instructed to just figure all that shit out. Often, it will drop you precisely at that golden moment where shit almost works, and development means tweaking code and immediately seeing things work better. That dopamine hit is why I code.&lt;/p&gt;

&lt;p&gt;There&amp;rsquo;s a downside. Sometimes, gnarly stuff needs doing. But you don&amp;rsquo;t wanna do it. So you refactor unit tests, soothing yourself with the lie that you&amp;rsquo;re doing real work. But an LLM can be told to go refactor all your unit tests. An agent can occupy itself for hours putzing with your tests in a VM and come back later with a PR. If you listen to me, you’ll know that. You&amp;rsquo;ll feel worse yak-shaving. You&amp;rsquo;ll end up doing… real work.&lt;/p&gt;
&lt;h3 id='but-you-have-no-idea-what-the-code-is' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#but-you-have-no-idea-what-the-code-is' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;but you have no idea what the code is&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;Are you a vibe coding Youtuber? Can you not read code? If so: astute point. Otherwise: what the fuck is wrong with you?&lt;/p&gt;

&lt;p&gt;You&amp;rsquo;ve always been responsible for what you merge to &lt;code&gt;main&lt;/code&gt;. You were five years go. And you are tomorrow, whether or not you use an LLM.&lt;/p&gt;

&lt;p&gt;If you build something with an LLM that people will depend on, read the code. In fact, you&amp;rsquo;ll probably do more than that. You&amp;rsquo;ll spend 5-10 minutes knocking it back into your own style. LLMs are &lt;a href='https://github.com/PatrickJS/awesome-cursorrules' title=''&gt;showing signs of adapting&lt;/a&gt; to local idiom, but we’re not there yet.&lt;/p&gt;

&lt;p&gt;People complain about LLM-generated code being “probabilistic”. No it isn&amp;rsquo;t. It’s code. It&amp;rsquo;s not Yacc output. It&amp;rsquo;s knowable. The LLM might be stochastic. But the LLM doesn’t matter. What matters is whether you can make sense of the result, and whether your guardrails hold.&lt;/p&gt;

&lt;p&gt;Reading other people&amp;rsquo;s code is part of the job. If you can&amp;rsquo;t metabolize the boring, repetitive code an LLM generates: skills issue! How are you handling the chaos human developers turn out on a deadline?&lt;/p&gt;
&lt;div class="right-sidenote"&gt;&lt;p&gt;† (because it can hold 50-70kloc in its context window)&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;For the last month or so, Gemini 2.5 has been my go-to †. Almost nothing it spits out for me merges without edits. I’m sure there’s a skill to getting a SOTA model to one-shot a feature-plus-merge! But I don’t care. I like moving the code around and chuckling to myself while I delete all the stupid comments. I have to read the code line-by-line anyways.&lt;/p&gt;
&lt;h3 id='but-hallucination' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#but-hallucination' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;but hallucination&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;If hallucination matters to you, your programming language has let you down.&lt;/p&gt;

&lt;p&gt;Agents lint. They compile and run tests. If their LLM invents a new function signature, the agent sees the error. They feed it back to the LLM, which says “oh, right, I totally made that up” and then tries again.&lt;/p&gt;

&lt;p&gt;You&amp;rsquo;ll only notice this happening if you watch the chain of thought log your agent generates. Don&amp;rsquo;t. This is why I like &lt;a href='https://zed.dev/agentic' title=''&gt;Zed&amp;rsquo;s agent mode&lt;/a&gt;: it begs you to tab away and let it work, and pings you with a desktop notification when it&amp;rsquo;s done.&lt;/p&gt;

&lt;p&gt;I’m sure there are still environments where hallucination matters. But &amp;ldquo;hallucination&amp;rdquo; is the first thing developers bring up when someone suggests using LLMs, despite it being (more or less) a solved problem.&lt;/p&gt;
&lt;h3 id='but-the-code-is-shitty-like-that-of-a-junior-developer' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#but-the-code-is-shitty-like-that-of-a-junior-developer' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;but the code is shitty, like that of a junior developer&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;Does an intern cost $20/month? Because that&amp;rsquo;s what Cursor.ai costs.&lt;/p&gt;

&lt;p&gt;Part of being a senior developer is making less-able coders productive, be they fleshly or algebraic. Using agents well is both a both a  skill and an engineering project all its own, of prompts, indices, &lt;a href='https://fly.io/blog/semgrep-but-for-real-now/' title=''&gt;and (especially) tooling.&lt;/a&gt; LLMs only produce shitty code if you let them.&lt;/p&gt;
&lt;div class="right-sidenote"&gt;&lt;p&gt;† (Also: 100% of all the Bash code you should author ever again)&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;Maybe the current confusion is about who&amp;rsquo;s doing what work. Today, LLMs do a lot of typing, Googling, test cases †, and edit-compile-test-debug cycles. But even the most Claude-poisoned serious developers in the world still own curation, judgement, guidance, and direction.&lt;/p&gt;

&lt;p&gt;Also: let’s stop kidding ourselves about how good our human first cuts really are.&lt;/p&gt;
&lt;h3 id='but-its-bad-at-rust' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#but-its-bad-at-rust' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;but it’s bad at rust&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;It’s hard to get a good toolchain for Brainfuck, too. Life’s tough in the aluminum siding business.&lt;/p&gt;
&lt;div class="right-sidenote"&gt;&lt;p&gt;† (and they surely will; the Rust community takes tooling seriously)&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;A lot of LLM skepticism probably isn&amp;rsquo;t really about LLMs. It&amp;rsquo;s projection. People say &amp;ldquo;LLMs can&amp;rsquo;t code&amp;rdquo; when what they really mean is &amp;ldquo;LLMs can&amp;rsquo;t write Rust&amp;rdquo;. Fair enough! But people select languages in part based on how well LLMs work with them, so Rust people should get on that †.&lt;/p&gt;

&lt;p&gt;I work mostly in Go. I’m confident the designers of the Go programming language didn&amp;rsquo;t set out to produce the most LLM-legible language in the industry. They succeeded nonetheless. Go has just enough type safety, an extensive standard library, and a culture that prizes (often repetitive) idiom. LLMs kick ass generating it.&lt;/p&gt;

&lt;p&gt;All this is to say: I write some Rust. I like it fine. If LLMs and Rust aren&amp;rsquo;t working for you, I feel you. But if that’s your whole thing, we’re not having the same argument.&lt;/p&gt;
&lt;h3 id='but-the-craft' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#but-the-craft' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;but the craft&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;Do you like fine Japanese woodworking? All hand tools and sashimono joinery? Me too. Do it on your own time.&lt;/p&gt;
&lt;div class="right-sidenote"&gt;&lt;p&gt;† (I’m a piker compared to my woodworking friends)&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;I have a basic wood shop in my basement †. I could get a lot of satisfaction from building a table. And, if that table is a workbench or a grill table, sure, I&amp;rsquo;ll build it. But if I need, like, a table? For people to sit at? In my office? I buy a fucking table.&lt;/p&gt;

&lt;p&gt;Professional software developers are in the business of solving practical problems for people with code. We are not, in our day jobs, artisans. Steve Jobs was wrong: we do not need to carve the unseen feet in the sculpture. Nobody cares if the logic board traces are pleasingly routed. If anything we build endures, it won&amp;rsquo;t be because the codebase was beautiful.&lt;/p&gt;

&lt;p&gt;Besides, that’s not really what happens. If you’re taking time carefully golfing functions down into graceful, fluent, minimal functional expressions, alarm bells should ring. You’re yak-shaving. The real work has depleted your focus. You&amp;rsquo;re not building: you&amp;rsquo;re self-soothing.&lt;/p&gt;

&lt;p&gt;Which, wait for it, is something LLMs are good for. They devour schlep, and clear a path to the important stuff, where your judgement and values really matter.&lt;/p&gt;
&lt;h3 id='but-the-mediocrity' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#but-the-mediocrity' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;but the mediocrity&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;As a mid-late career coder, I&amp;rsquo;ve come to appreciate mediocrity. You should be so lucky as to  have it flowing almost effortlessly from a tap.&lt;/p&gt;

&lt;p&gt;We all write mediocre code. Mediocre code: often fine. Not all code is equally important. Some code should be mediocre. Maximum effort on a random unit test? You&amp;rsquo;re doing something wrong. Your team lead should correct you.&lt;/p&gt;

&lt;p&gt;Developers all love to preen about code. They worry LLMs lower the &amp;ldquo;ceiling&amp;rdquo; for quality. Maybe. But they also raise the &amp;ldquo;floor&amp;rdquo;.&lt;/p&gt;

&lt;p&gt;Gemini&amp;rsquo;s floor is higher than my own.  My code looks nice. But it&amp;rsquo;s not as thorough. LLM code is repetitive. But mine includes dumb contortions where I got too clever trying to DRY things up.&lt;/p&gt;

&lt;p&gt;And LLMs aren&amp;rsquo;t mediocre on every axis. They almost certainly have a bigger bag of algorithmic tricks than you do: radix tries, topological sorts, graph reductions, and LDPC codes. Humans romanticize &lt;code&gt;rsync&lt;/code&gt; (&lt;a href='https://www.andrew.cmu.edu/course/15-749/READINGS/required/cas/tridgell96.pdf' title=''&gt;Andrew Tridgell&lt;/a&gt; wrote a paper about it!). To an LLM it might not be that much more interesting than a SQL join.&lt;/p&gt;

&lt;p&gt;But I&amp;rsquo;m getting ahead of myself. It doesn&amp;rsquo;t matter. If truly mediocre code is all we ever get from LLMs, that&amp;rsquo;s still huge. It&amp;rsquo;s that much less mediocre code humans have to write.&lt;/p&gt;
&lt;h3 id='but-itll-never-be-agi' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#but-itll-never-be-agi' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;but it&amp;rsquo;ll never be AGI&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;I don&amp;rsquo;t give a shit.&lt;/p&gt;

&lt;p&gt;Smart practitioners get wound up by the AI/VC hype cycle. I can&amp;rsquo;t blame them. But it&amp;rsquo;s not an argument. Things either work or they don&amp;rsquo;t, no matter what Jensen Huang has to say about it.&lt;/p&gt;
&lt;h3 id='but-they-take-rr-jerbs' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#but-they-take-rr-jerbs' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;but they take-rr jerbs&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;a href='https://news.ycombinator.com/item?id=43776612' title=''&gt;So does open source.&lt;/a&gt; We used to pay good money for databases.&lt;/p&gt;

&lt;p&gt;We&amp;rsquo;re a field premised on automating other people’s jobs away. &amp;ldquo;Productivity gains,&amp;rdquo; say the economists. You get what that means, right? Fewer people doing the same stuff. Talked to a travel agent lately? Or a floor broker? Or a record store clerk? Or a darkroom tech?&lt;/p&gt;

&lt;p&gt;When this argument comes up, libertarian-leaning VCs start the chant: lamplighters, creative destruction, new kinds of work. Maybe. But I&amp;rsquo;m not hypnotized. I have no fucking clue whether we’re going to be better off after LLMs. Things could get a lot worse for us.&lt;/p&gt;

&lt;p&gt;LLMs really might displace many software developers. That&amp;rsquo;s not a high horse we get to ride. Our jobs are just as much in tech&amp;rsquo;s line of fire as everybody else&amp;rsquo;s have been for the last 3 decades. We&amp;rsquo;re not &lt;a href='https://en.wikipedia.org/wiki/2024_United_States_port_strike' title=''&gt;East Coast dockworkers&lt;/a&gt;; we won&amp;rsquo;t stop progress on our own.&lt;/p&gt;
&lt;h3 id='but-the-plagiarism' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#but-the-plagiarism' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;but the plagiarism&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;Artificial intelligence is profoundly — and probably unfairly — threatening to visual artists in ways that might be hard to appreciate if you don&amp;rsquo;t work in the arts.&lt;/p&gt;

&lt;p&gt;We imagine artists spending their working hours pushing the limits of expression. But the median artist isn&amp;rsquo;t producing gallery pieces. They produce on brief: turning out competent illustrations and compositions for magazine covers, museum displays, motion graphics, and game assets.&lt;/p&gt;

&lt;p&gt;LLMs easily — alarmingly — clear industry quality bars. Gallingly, one of the things they&amp;rsquo;re best at is churning out just-good-enough facsimiles of human creative work.  I have family in visual arts. I can&amp;rsquo;t talk to them about LLMs. I don&amp;rsquo;t blame them. They&amp;rsquo;re probably not wrong.&lt;/p&gt;

&lt;p&gt;Meanwhile, software developers spot code fragments &lt;a href="https://arxiv.org/abs/2311.17035"&gt;seemingly lifted&lt;/a&gt; from public repositories on Github and lose their shit. What about the licensing? If you&amp;rsquo;re a lawyer, I defer. But if you&amp;rsquo;re a software developer playing this card? Cut me a little slack as I ask you to shove this concern up your ass. No profession has demonstrated more contempt for intellectual property.&lt;/p&gt;

&lt;p&gt;The median dev thinks Star Wars and Daft Punk are a public commons. The great cultural project of developers has been opposing any protection that might inconvenience a monetizable media-sharing site. When they fail at policy, they route around it with coercion. They stand up global-scale piracy networks and sneer at anybody who so much as tries to preserve a new-release window for a TV show.&lt;/p&gt;

&lt;p&gt;Call any of this out if you want to watch a TED talk about how hard it is to stream &lt;em&gt;The Expanse&lt;/em&gt; on LibreWolf. Yeah, we get it. You don&amp;rsquo;t believe in IPR. Then shut the fuck up about IPR. Reap the whirlwind.&lt;/p&gt;

&lt;p&gt;It&amp;rsquo;s all special pleading anyways. LLMs digest code further than you do. If you don&amp;rsquo;t believe a typeface designer can stake a moral claim on the terminals and counters of a letterform, you sure as hell can&amp;rsquo;t be possessive about a red-black tree.&lt;/p&gt;
&lt;h3 id='positive-case-redux' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#positive-case-redux' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;positive case redux&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;When I started writing a couple days ago, I wrote a section to &amp;ldquo;level set&amp;rdquo; to the  state of the art of LLM-assisted programming. A bluefish filet has a longer shelf life than an LLM take. In the time it took you to read this, everything changed.&lt;/p&gt;

&lt;p&gt;Kids today don&amp;rsquo;t just use agents; they use asynchronous agents. They wake up, free-associate 13 different things for their LLMs to work on, make coffee, fill out a TPS report, drive to the Mars Cheese Castle, and then check their notifications. They&amp;rsquo;ve got 13 PRs to review. Three get tossed and re-prompted. Five of them get the same feedback a junior dev gets. And five get merged.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;&amp;ldquo;I&amp;rsquo;m sipping rocket fuel right now,&amp;rdquo;&lt;/em&gt; a friend tells me. &lt;em&gt;&amp;ldquo;The folks on my team who aren&amp;rsquo;t embracing AI? It&amp;rsquo;s like they&amp;rsquo;re standing still.&amp;rdquo;&lt;/em&gt; He&amp;rsquo;s not bullshitting me. He doesn&amp;rsquo;t work in SFBA. He&amp;rsquo;s got no reason to lie.&lt;/p&gt;

&lt;p&gt;There&amp;rsquo;s plenty of things I can&amp;rsquo;t trust an LLM with. No LLM has any of access to prod here. But I&amp;rsquo;ve been first responder on an incident and fed 4o — not o4-mini, 4o — log transcripts, and watched it in seconds spot LVM metadata corruption issues on a host we&amp;rsquo;ve been complaining about for months. Am I better than an LLM agent at interrogating OpenSearch logs and Honeycomb traces? No. No, I am not.&lt;/p&gt;

&lt;p&gt;To the consternation of many of my friends, I&amp;rsquo;m not a radical or a futurist. I&amp;rsquo;m a statist. I believe in the haphazard perseverance of complex systems, of institutions, of reversions to the mean. I write Go and Python code. I&amp;rsquo;m not a Kool-aid drinker.&lt;/p&gt;

&lt;p&gt;But something real is happening. My smartest friends are blowing it off. Maybe I persuade you. Probably I don&amp;rsquo;t. But we need to be done making space for bad arguments.&lt;/p&gt;
&lt;h3 id='but-im-tired-of-hearing-about-it' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#but-im-tired-of-hearing-about-it' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;but i&amp;rsquo;m tired of hearing about it&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;And here I rejoin your company. I read &lt;a href='https://simonwillison.net/' title=''&gt;Simon Willison&lt;/a&gt;, and that&amp;rsquo;s all I really need. But all day, every day, a sizable chunk of the front page of HN is allocated to LLMs: incremental model updates, startups doing things with LLMs, LLM tutorials, screeds against LLMs. It&amp;rsquo;s annoying!&lt;/p&gt;

&lt;p&gt;But AI is also incredibly — a word I use advisedly — important. It&amp;rsquo;s getting the same kind of attention that smart phones got in 2008, and not as much as the Internet got. That seems about right.&lt;/p&gt;

&lt;p&gt;I think this is going to get clearer over the next year. The cool kid haughtiness about &amp;ldquo;stochastic parrots&amp;rdquo; and &amp;ldquo;vibe coding&amp;rdquo; can&amp;rsquo;t survive much more contact with reality. I&amp;rsquo;m snarking about these people, but I meant what I said: they&amp;rsquo;re smarter than me. And when they get over this affectation, they&amp;rsquo;re going to make coding agents profoundly more effective than they are today.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Using Kamal 2.0 in Production</title>
    <link rel="alternate" href="https://fly.io/blog/kamal-in-production/"/>
    <id>https://fly.io/blog/kamal-in-production/</id>
    <published>2025-05-29T00:00:00+00:00</published>
    <updated>2025-06-05T20:40:14+00:00</updated>
    <media:thumbnail url="https://fly.io/blog/kamal-in-production/assets/production.jpg"/>
    <content type="html">&lt;p&gt;&lt;a href='https://pragprog.com/titles/rails8/agile-web-development-with-rails-8/' title=''&gt;Agile Web Development with Rails 8&lt;/a&gt; is off to production, where they do things like editing, indexing, pagination, and printing. In researching the chapter on Deployment and Production, I became very dissatisfied with the content available on Kamal. I ended up writing my own, and it went well beyond the scope of the book. I then extracted what I needed from the result and put it in the book.&lt;/p&gt;

&lt;p&gt;Now that I have some spare time, I took a look at the greater work. It was more than a chapter and less than a book, so I decided to publish it &lt;a href='https://rubys.github.io/kamal-in-production/' title=''&gt;online&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;This took me only a matter of hours. I had my notes in the XML grammar that Pragmatic Programming uses for books. I asked GitHub Copilot to convert them to Markdown. It did the job without my having to explain the grammar. It made intelligent guesses as to how to handle footnotes and got a number of these wrong, but that was easy to fix. On a lark, I asked it to proofread the content, and it did that too.&lt;/p&gt;

&lt;hr&gt;

&lt;p&gt;Don&amp;rsquo;t get me wrong, Kamal is great. There are plenty of videos on how to get toy projects online, and the documentation will tell you what each field in the configuration file does. But none pull together everything you need to deploy a real project. For example, &lt;a href='https://rubys.github.io/kamal-in-production/Assemble/' title=''&gt;there are seven things you need to get started&lt;/a&gt;. Some are optional, some you may already have, and all can be gathered quickly &lt;strong class='font-semibold text-navy-950'&gt;if you have a list&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Kamal is just one piece of the puzzle. To deploy your software using Kamal, you need to be aware of the vast Docker ecosystem. You will want to set up a builder, sign up for a container repository, and lock down your secrets. And as you grow, you will want a load balancer and a managed database.&lt;/p&gt;

&lt;p&gt;And production is much more than copying files and starting a process. It is ensuring that your database is backed up, that your logs are searchable, and that your application is being monitored. It is also about ensuring that your application is secure.&lt;/p&gt;

&lt;p&gt;My list is opinionated. Each choice has a lot of options. For SSH keys, there are a lot of key types. If you don&amp;rsquo;t have an opinion, go with what GitHub recommends. For hosting, there are a lot of providers, and most videos start with Hetzner, which is a solid choice. But did you know that there are separate paths for Cloud and Robot, and when you would want either?&lt;/p&gt;

&lt;p&gt;A list with default options and alternatives highlighted was what would have helped me get started. Now it is available to you. The &lt;a href='https://github.com/rubys/kamal-in-production/' title=''&gt;source is on GitHub&lt;/a&gt;. &lt;a href='https://creativecommons.org/public-domain/cc0/' title=''&gt;CC0 licensed&lt;/a&gt;. Feel free to add side pages or links to document Digital Ocean, add other monitoring tools, or simply correct a typo that GitHub Copilot and I missed (or made).&lt;/p&gt;

&lt;hr&gt;

&lt;p&gt;And if you happen to be in the south eastern part of the US in August, come see me talk on this topic at the &lt;a href='https://blog.carolina.codes/p/announcing-our-2025-speakers' title=''&gt;Carolina Code Conference&lt;/a&gt;. If you can&amp;rsquo;t make it, the presentation will be recorded and posted online.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>parking_lot: ffffffffffffffff...</title>
    <link rel="alternate" href="https://fly.io/blog/parking-lot-ffffffffffffffff/"/>
    <id>https://fly.io/blog/parking-lot-ffffffffffffffff/</id>
    <published>2025-05-28T00:00:00+00:00</published>
    <updated>2026-09-18T20:59:24+00:00</updated>
    <media:thumbnail url="https://fly.io/blog/parking-lot-ffffffffffffffff/assets/ffff-cover.webp"/>
    <content type="html">&lt;div class="lead"&gt;&lt;p&gt;We’re Fly.io, a public cloud that runs apps in a bunch of locations all over the world. This is a post about a gnarly bug in our Anycast router, the largest Rust project in our codebase. You won’t need much of any Rust to follow it.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;The basic idea is simple: customers give us Docker containers, and tell us which one of 30+ regions around the world they want them to run in. We convert the containers into lightweight virtual machines, and then link them to an Anycast network. If you boot up an app here in Sydney and Frankfurt, and a request for it lands in Narita, it&amp;rsquo;ll get routed to Sydney. The component doing that work is called &lt;code&gt;fly-proxy&lt;/code&gt;. It&amp;rsquo;s a Rust program, and it has been ill behaved of late.&lt;/p&gt;
&lt;h3 id='dramatis-personae' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#dramatis-personae' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Dramatis Personae&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;&lt;code&gt;fly-proxy&lt;/code&gt;, our intrepid Anycast router.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;corrosion&lt;/code&gt;, our intrepid Anycast routing protocol.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;Rust&lt;/code&gt;, a programming language you probably don&amp;rsquo;t use.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;read-write locks&lt;/code&gt;, a synchronization primitive that allows for many readers &lt;em&gt;or&lt;/em&gt; one single writer.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;parking_lot&lt;/code&gt;, a well-regarded optimized implementation of locks in Rust.&lt;/p&gt;
&lt;div class="callout"&gt;&lt;p&gt;Gaze not into the abyss, lest you become recognized as an &lt;strong class="font-semibold text-navy-950"&gt;&lt;em&gt;abyss domain expert&lt;/em&gt;&lt;/strong&gt;, and they expect you keep gazing into the damn thing&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Mathewson &lt;a href="https://x.com/nickm_tor/status/860234274842324993?lang=en" title=""&gt;6:31&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;
&lt;/div&gt;&lt;h3 id='anycast-routing' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#anycast-routing' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Anycast Routing&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;You already know how to build a proxy. Connection comes in, connection goes out, copy back and forth. So for the amount of time we spend writing about &lt;code&gt;fly-proxy&lt;/code&gt;, you might wonder what the big deal is.&lt;/p&gt;

&lt;p&gt;To be fair, in the nuts and bolts of actually proxying requests, &lt;code&gt;fly-proxy&lt;/code&gt; does some interesting stuff. For one thing, it&amp;rsquo;s &lt;a href='https://github.com/jedisct1/yes-rs' title=''&gt;written in Rust&lt;/a&gt;, which is apparently a big deal all on its own. It&amp;rsquo;s a large, asynchronous codebase that handles multiple protocols, TLS termination, and certificate issuance. It exercises a lot of &lt;a href='https://tokio.rs/' title=''&gt;Tokio&lt;/a&gt; features.&lt;/p&gt;

&lt;p&gt;But none of this is the hard part of &lt;code&gt;fly-proxy&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;We operate thousands of servers around the world. A customer Fly Machine might get scheduled onto any of them; in some places, the expectation we set with customers is that their Machines will potentially start in less than a second. More importantly, a Fly Machine can terminate instantly. In both cases, but especially the latter, &lt;code&gt;fly-proxy&lt;/code&gt; potentially needs to know, so that it does (or doesn&amp;rsquo;t) route traffic there.&lt;/p&gt;

&lt;p&gt;This is the hard problem: managing millions of connections for millions of apps. It&amp;rsquo;s a lot of state to manage, and it&amp;rsquo;s in constant flux. We refer to this as the &amp;ldquo;state distribution problem&amp;rdquo;, but really, it quacks like a routing protocol.&lt;/p&gt;
&lt;h3 id='our-routing-protocol-is-corrosion' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#our-routing-protocol-is-corrosion' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Our Routing Protocol is Corrosion&lt;/span&gt;&lt;/h3&gt;&lt;div class="right-sidenote"&gt;&lt;p&gt;Corrosion2, to be precise.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;We&amp;rsquo;ve been through multiple iterations of the state management problem, and the stable place we&amp;rsquo;ve settled is a &lt;a href='https://github.com/superfly/corrosion' title=''&gt;system called Corrosion&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Corrosion is a large SQLite database mirrored globally across several thousand servers. There are three important factors that make Corrosion work:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The SQLite database Corrosion replicates is CRDT-structured.
&lt;/li&gt;&lt;li&gt;In our orchestration model, individual worker servers are the source of truth for any Fly Machines running on them; there&amp;rsquo;s no globally coordinated orchestration state.
&lt;/li&gt;&lt;li&gt;We use &lt;a href='https://fly.io/blog/building-clusters-with-serf/#what-serf-is-doing' title=''&gt;SWIM gossip&lt;/a&gt; to publish updates from those workers across the fleet.
&lt;/li&gt;&lt;/ol&gt;

&lt;p&gt;This works. A Fly Machine terminates in Dallas; a &lt;code&gt;fly-proxy&lt;/code&gt; instance in Singapore knows within a small number of seconds.&lt;/p&gt;
&lt;h3 id='routing-protocol-implementations-are-hard' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#routing-protocol-implementations-are-hard' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Routing Protocol Implementations Are Hard&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;A routing protocol is a canonical example of a distributed system. We&amp;rsquo;ve certainly hit distsys bugs in Corrosion. But this story is about basic implementation issues. &lt;/p&gt;

&lt;p&gt;A globally replicated SQLite database is an awfully nice primitive, but we&amp;rsquo;re not actually doing SQL queries every time a request lands.&lt;/p&gt;

&lt;p&gt;In somewhat the same sense as a router works both with a &lt;a href='https://www.dasblinkenlichten.com/rib-and-fib-understanding-the-terminology/' title=''&gt;RIB and a FIB&lt;/a&gt;, there is in &lt;code&gt;fly-proxy&lt;/code&gt; a system of record for routing information (Corrosion), and then an in-memory aggregation of that information used to make fast decisions. In &lt;code&gt;fly-proxy&lt;/code&gt;, that&amp;rsquo;s called the Catalog. It&amp;rsquo;s a record of everything in Corrosion a proxy might need to know about to forward requests.&lt;/p&gt;

&lt;p&gt;Here&amp;rsquo;s a fun bug from last year:&lt;/p&gt;

&lt;p&gt;At any given point in time, there&amp;rsquo;s a lot going on inside &lt;code&gt;fly-proxy&lt;/code&gt;. It is a highly concurrent system. In particular, there are many readers to the Catalog, and also, when Corrosion updates, writers. We manage access to the Catalog with a system of &lt;a href='https://doc.rust-lang.org/std/sync/struct.RwLock.html' title=''&gt;read-write locks&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Rust is big on pattern-matching; instead of control flow based (at bottom) on simple arithmetic expressions, Rust allows you to &lt;code&gt;match&lt;/code&gt; exhaustively (with compiler enforcement) on the types of an expression, and the type system expresses things like &lt;code&gt;Ok&lt;/code&gt; or &lt;code&gt;Err&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;But &lt;code&gt;match&lt;/code&gt; can be cumbersome, and so there are shorthands. One of them is &lt;code&gt;if let&lt;/code&gt;, which is syntax that makes a pattern match read like a classic &lt;code&gt;if&lt;/code&gt; statement. Here&amp;rsquo;s an example:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative rust"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-7igd9a7z"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-7igd9a7z"&gt;&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;Some&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nn"&gt;Load&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="nf"&gt;Local&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;load&lt;/span&gt;&lt;span class="p"&gt;)))&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="k"&gt;self&lt;/span&gt;&lt;span class="py"&gt;.load&lt;/span&gt;&lt;span class="nf"&gt;.read&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="nf"&gt;.get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;...&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="c1"&gt;// do a bunch of stuff with `load`&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;self&lt;/span&gt;&lt;span class="nf"&gt;.init_for&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;...&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;The &amp;ldquo;if&amp;rdquo; arm of that branch is taken if &lt;code&gt;self.load.read().get()&lt;/code&gt; returns a value with the type &lt;code&gt;Some&lt;/code&gt;. To retrieve that value, the expression calls &lt;code&gt;read()&lt;/code&gt; to grab a lock.&lt;/p&gt;
&lt;div class="right-sidenote"&gt;&lt;p&gt;though Rust programmers probably notice the bug quickly&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;The bug is subtle: in that code, the lock &lt;code&gt;self.load.read().get()&lt;/code&gt; takes is held not just for the duration of the &amp;ldquo;if&amp;rdquo; arm, but also for the &amp;ldquo;else&amp;rdquo; arm — you can think of &lt;code&gt;if let&lt;/code&gt; expressions as being rewritten to the equivalent &lt;code&gt;match&lt;/code&gt; expression, where that lifespan is much clearer.&lt;/p&gt;

&lt;p&gt;Anyways that&amp;rsquo;s real code and it occurred on a code path in &lt;code&gt;fly-proxy&lt;/code&gt; that was triggered by a Corrosion update propagating from host to host across our fleet in millisecond intervals of time, converging quickly, like a good little routing protocol, on a global consensus that our entire Anycast routing layer should be deadlocked. Fun times.&lt;/p&gt;
&lt;h3 id='the-watchdog-and-regionalizing' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#the-watchdog-and-regionalizing' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;The Watchdog, and Regionalizing&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;The experience of that Anycast outage was arresting, and immediately altered our plans. We set about two tasks, one short-term and one long.&lt;/p&gt;

&lt;p&gt;In the short term: we made deadlocks nonlethal with a &amp;ldquo;watchdog&amp;rdquo; system. &lt;code&gt;fly-proxy&lt;/code&gt; has an internal control channel (it drives a REPL operators can run from our servers). During a deadlock (or dead-loop or exhaustion), that channel becomes nonresponsive. We watch for that and bounce the proxy when it happens. A deadlock is still bad! But it&amp;rsquo;s a second-or-two-length arrhythmia, not asystole.&lt;/p&gt;

&lt;p&gt;Meanwhile, over the long term: we&amp;rsquo;re confronting the implications of all our routing state sharing a global broadcast domain. The update that seized up Anycast last year pertained to an app nobody used. There wasn&amp;rsquo;t any real reason for any &lt;code&gt;fly-proxy&lt;/code&gt; to receive it in the first place. But in the &lt;em&gt;status quo ante&lt;/em&gt; of the outage, every proxy received updates for every Fly Machine.&lt;/p&gt;

&lt;p&gt;They still do. Why? Because it scales, and fixing it turns out to be a huge lift. It&amp;rsquo;s a lift we&amp;rsquo;re still making! It&amp;rsquo;s just taking time. We call this effort &amp;ldquo;regionalization&amp;rdquo;, because the next intermediate goal is to confine most updates to the region (Sydney, Frankfurt, Dallas) in which they occur.&lt;/p&gt;

&lt;p&gt;I hope this has been a satisfying little tour of the problem domain we&amp;rsquo;re working in. We have now reached the point where I can start describing the new bug.&lt;/p&gt;
&lt;h3 id='new-bug-round-1-lazy-loading' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#new-bug-round-1-lazy-loading' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;New Bug, Round 1: Lazy Loading&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;We want to transform our original Anycast router, designed to assume a full picture of global state, into a regionalized router with partitioned state. A sane approach: have it lazy-load state information. Then you can spare most of the state code; state for apps that never show up at a particular &lt;code&gt;fly-proxy&lt;/code&gt; in, say, Hong Kong simply doesn&amp;rsquo;t get loaded.&lt;/p&gt;

&lt;p&gt;For months now, portions of the &lt;code&gt;fly-proxy&lt;/code&gt; Catalog have been lazy-loaded. A few weeks ago, the decision is made to get most of the rest of the Catalog state (apps, machines, &amp;amp;c) lazy-loaded as well. It&amp;rsquo;s a straightforward change and it gets rolled out quickly.&lt;/p&gt;

&lt;p&gt;Almost as quickly, proxies begin locking up and getting bounced by the watchdog. Not in the frightening Beijing-Olympics-level coordinated fashion that happened during the Outage, but any watchdog bounce is a problem.&lt;/p&gt;

&lt;p&gt;We roll back the change.&lt;/p&gt;

&lt;p&gt;From the information we have, we&amp;rsquo;ve narrowed things down to two suspects. First, lazy-loading changes the read/write patterns and thus the pressure on the RWLocks the Catalog uses; it could just be lock contention. Second, we spot a suspicious &lt;code&gt;if let&lt;/code&gt;.&lt;/p&gt;
&lt;h3 id='new-bug-round-2-the-lock-refactor' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#new-bug-round-2-the-lock-refactor' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;New Bug, Round 2: The Lock Refactor&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;Whichever the case, there&amp;rsquo;s a reason these proxies locked up with the new lazy-loading code, and our job is to fix it. The &lt;code&gt;if let&lt;/code&gt; is easy. Lock contention is a little trickier.&lt;/p&gt;

&lt;p&gt;At this point it&amp;rsquo;s time to introduce a new character to the story, though they&amp;rsquo;ve been lurking on the stage the whole time: it&amp;rsquo;s &lt;a href='https://github.com/Amanieu/parking_lot' title=''&gt;&lt;code&gt;parking_lot&lt;/code&gt;&lt;/a&gt;, an important, well-regarded, and widely-used replacement for the standard library&amp;rsquo;s lock implementation.&lt;/p&gt;

&lt;p&gt;Locks in &lt;code&gt;fly-proxy&lt;/code&gt; are &lt;code&gt;parking_lot&lt;/code&gt; locks. People use &lt;code&gt;parking_lot&lt;/code&gt; mostly because it is very fast and tight (a lock takes up just a single 64-bit word). But we use it for the feature set. The feature we&amp;rsquo;re going to pull out this time is lock timeouts: the RWLock in &lt;code&gt;parking_lot&lt;/code&gt; exposes a &lt;a href='https://amanieu.github.io/parking_lot/parking_lot/struct.RwLock.html#method.try_write_for' title=''&gt;&lt;code&gt;try_write_for&lt;/code&gt;&lt;/a&gt;method, which takes a &lt;code&gt;Duration&lt;/code&gt;, after which an attempt to grab the write lock fails.&lt;/p&gt;

&lt;p&gt;Before rolling out a new lazy-loading &lt;code&gt;fly-proxy&lt;/code&gt;, we do some refactoring:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;our Catalog write locks all time out, so we&amp;rsquo;ll get telemetry and a failure recovery path if that&amp;rsquo;s what&amp;rsquo;s choking the proxy to death,
&lt;/li&gt;&lt;li&gt;we eliminate RAII-style lock acquisition from the Catalog code (in RAII style, you grab a lock into a local value, and the lock is released implicitly when the scope that expression occurs in concludes) and replace it with explicit closures, so you can look at the code and see precisely the interval in which the lock is held, and
&lt;/li&gt;&lt;li&gt;since we now have code that is very explicit about locking intervals, we instrument it with logging and metrics so we can see what&amp;rsquo;s happening.
&lt;/li&gt;&lt;/ul&gt;

&lt;p&gt;We should be set. The suspicious &lt;code&gt;if let&lt;/code&gt; is gone, lock acquisition can time out, and we have all this new visibility.&lt;/p&gt;

&lt;p&gt;Nope. Immediately more lockups, all in Europe, especially in &lt;code&gt;WAW&lt;/code&gt;.&lt;/p&gt;
&lt;h3 id='new-bug-round-3-telemetry-inspection' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#new-bug-round-3-telemetry-inspection' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;New Bug, Round 3: Telemetry Inspection&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;That we&amp;rsquo;re still seeing deadlocks is f&amp;#39;ing weird. We&amp;rsquo;ve audited all our Catalog locks. You can look at the code and see the lifespan of a grabbed lock.&lt;/p&gt;

&lt;p&gt;We have a clue: our lock timeout logs spam just before a proxy locks up and is killed by the watchdog. This clue: useless. But we don&amp;rsquo;t know that yet!&lt;/p&gt;

&lt;p&gt;Our new hunch is that something is holding a lock for a very long time, and we just need to find out which thing that is. Maybe the threads updating the Catalog from Corrosion are dead-looping?&lt;/p&gt;

&lt;p&gt;The instrumentation should tell the tale. But it does not. We see slow locks… just before the entire proxy locks up. And they happen in benign, quiet applications. Random, benign, quiet applications.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;parking_lot&lt;/code&gt; has a &lt;a href='https://amanieu.github.io/parking_lot/parking_lot/deadlock/index.html' title=''&gt;deadlock detector&lt;/a&gt;. If you ask it, it&amp;rsquo;ll keep a waiting-for dependency graph and detect stalled threads. This runs on its own thread, isolate outside the web of lock dependencies in the rest of the proxy. We cut a build of the proxy with the deadlock detector enabled and wait for something in &lt;code&gt;WAW&lt;/code&gt; to lock up. And it does. But &lt;code&gt;parking_lot&lt;/code&gt; doesn&amp;rsquo;t notice. As far as it&amp;rsquo;s concerned, nothing is wrong.&lt;/p&gt;

&lt;p&gt;We are at this moment very happy we did the watchdog thing.&lt;/p&gt;
&lt;h3 id='new-bug-round-4-descent-into-madness' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#new-bug-round-4-descent-into-madness' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;New Bug, Round 4: Descent Into Madness&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;When the watchdog bounces a proxy, it snaps a core dump from the process it just killed. We are now looking at core dumps. There is only one level of decompensation to be reached below &amp;ldquo;inspecting core dumps&amp;rdquo;, and that&amp;rsquo;s &amp;ldquo;blaming the compiler&amp;rdquo;. We will get there.&lt;/p&gt;

&lt;p&gt;Here&amp;rsquo;s Pavel, at the time:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;I’ve been staring at the last core dump from &lt;code&gt;waw&lt;/code&gt; . It’s quite strange.
First, there is no thread that’s running inside the critical section. Yet, there is a thread that’s waiting to acquire write lock and a bunch of threads waiting to acquire a read lock.
That’s doesn’t prove anything, of course, as a thread holding catalog write lock might have just released it before core dump was taken. But that would be quite a coincidence.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The proxy is locked up. But there are no threads in a critical section for the catalog. Nevertheless, we can see stack traces of multiple threads waiting to acquire locks. In the moment, Pavel thinks this could be a fluke. But we&amp;rsquo;ll soon learn that &lt;em&gt;every single stack trace&lt;/em&gt; shows the same pattern: everything wants the Catalog lock, but nobody has it.&lt;/p&gt;

&lt;p&gt;It&amp;rsquo;s hard to overstate how weird this is. It breaks both our big theories: it&amp;rsquo;s not compatible with a Catalog deadlock that we missed, and it&amp;rsquo;s not compatible with a very slow lock holder. Like a podcast listener staring into the sky at the condensation trail of a jetliner, we begin reaching for wild theories. For instance: &lt;code&gt;parking_lot&lt;/code&gt; locks are synchronous, but we&amp;rsquo;re a Tokio application; something somewhere could be taking an async lock that&amp;rsquo;s confusing the runtime. Alas, no.&lt;/p&gt;

&lt;p&gt;On the plus side, we are now better at postmortem core dump inspection with &lt;code&gt;gdb&lt;/code&gt;.&lt;/p&gt;
&lt;h3 id='new-bug-round-5-madness-gives-way-to-desperation' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#new-bug-round-5-madness-gives-way-to-desperation' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;New Bug, Round 5: Madness Gives Way To Desperation&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;Fuck it, we&amp;rsquo;ll switch to &lt;code&gt;read_recursive&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;A recursive read lock is an eldritch rite invoked when you need to grab a read lock deep in a call tree where you already grabbed that lock, but can&amp;rsquo;t be arsed to structure the code properly to reflect that. When you ask for a recursive lock, if you already hold the lock, you get the lock again, instead of a deadlock.&lt;/p&gt;

&lt;p&gt;Our theory: &lt;code&gt;parking_lot&lt;/code&gt;goes through some trouble to make sure a stampede of readers won&amp;rsquo;t starve writers, who are usually outnumbered. It prefers writers by preventing readers from acquiring locks when there&amp;rsquo;s at least one waiting writer. And &lt;code&gt;read_recursive&lt;/code&gt; sidesteps that logic.&lt;/p&gt;

&lt;p&gt;Maybe there&amp;rsquo;s some ultra-slow reader somehow not showing up in our traces, and maybe switching to recursive locks will cut through that.&lt;/p&gt;

&lt;p&gt;This does not work. At least, not how we hoped it would. It does generate a new piece of evidence:  &lt;code&gt;RwLock reader count overflow&lt;/code&gt; log messages, and lots of them.&lt;/p&gt;
&lt;h3 id='there-are-things-you-are-not-meant-to-know' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#there-are-things-you-are-not-meant-to-know' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;There Are Things You Are Not Meant To Know&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;You&amp;rsquo;re reading a 3,000 word blog post about a single concurrency bug, so my guess is you&amp;rsquo;re the kind of person who compulsively wants to understand how everything works. That&amp;rsquo;s fine, but a word of advice: there are things where, if you find yourself learning about them in detail, something has gone wrong.&lt;/p&gt;

&lt;p&gt;One of those things is the precise mechanisms used by your RWLock implementation.&lt;/p&gt;

&lt;p&gt;The whole point of &lt;code&gt;parking_lot&lt;/code&gt; is that the locks are tiny, marshalled into a 64 bit word. Those bits are partitioned into &lt;a href='https://github.com/Amanieu/parking_lot/blob/master/src/raw_rwlock.rs' title=''&gt;4 signaling bits&lt;/a&gt; (&lt;code&gt;PARKED&lt;/code&gt;, &lt;code&gt;WRITER_PARKED&lt;/code&gt;, &lt;code&gt;WRITER&lt;/code&gt;, and &lt;code&gt;UPGRADEABLE&lt;/code&gt;) and a 60-bit counter of lock holders.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Me, a dummy: sounds like we overflowed that counter.&lt;/p&gt;

&lt;p&gt;Pavel, a genius: we are not overflowing a 60-bit counter.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Ruling out a genuine overflow (which would require paranormal activity) leaves us with corruption as the culprit. Something is bashing our lock word. If the counter is corrupted, maybe the signaling bits are too; then we&amp;rsquo;re in an inconsistent state, an artificial deadlock.&lt;/p&gt;

&lt;p&gt;Easily confirmed. We cast the lock words into &lt;code&gt;usize&lt;/code&gt; and log them. Sure enough, they&amp;rsquo;re &lt;code&gt;0xFFFFFFFFFFFFFFFF&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;This is a smoking gun, because it implies all 4 signaling bits are set, and that includes &lt;code&gt;UPGRADEABLE&lt;/code&gt;. Upgradeable locks are read-locks that can be &amp;ldquo;upgraded&amp;rdquo; to write locks. We don&amp;rsquo;t use them.&lt;/p&gt;

&lt;p&gt;This looks like classic memory corruption. But in our core dumps, memory doesn&amp;rsquo;t appear corrupted: the only thing set all &lt;code&gt;FFh&lt;/code&gt; is the lock word.&lt;/p&gt;

&lt;p&gt;We compile and run our test suites &lt;a href='https://github.com/rust-lang/miri' title=''&gt;under &lt;code&gt;miri&lt;/code&gt;&lt;/a&gt;, a Rust interpreter for its &lt;a href='https://rustc-dev-guide.rust-lang.org/mir/index.html' title=''&gt;MIR IR&lt;/a&gt;. &lt;code&gt;miri&lt;/code&gt; does all sorts of cool UB detection, and does in fact spot some UB in our tests, which we are at the time excited about until we deploy the fixes and nothing gets better.&lt;/p&gt;

&lt;p&gt;At this point, Saleem suggests guard pages. We could &lt;code&gt;mprotect&lt;/code&gt; memory pages around the lock to force a panic if a wild write hits &lt;em&gt;near&lt;/em&gt; the lock word, or just allocate zero pages and check them, which we are at the time excited about until we deploy the guard pages and nothing hits them.&lt;/p&gt;
&lt;h3 id='the-non-euclidean-horror-at-the-heart-of-this-bug' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#the-non-euclidean-horror-at-the-heart-of-this-bug' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;The Non-Euclidean Horror At The Heart Of This Bug&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;At this point we should recap where we find ourselves:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;We made a relatively innocuous change to our proxy, which caused proxies in Europe to get watchdog-killed.
&lt;/li&gt;&lt;li&gt;We audited and eliminated all the nasty &lt;code&gt;if-letses&lt;/code&gt;.
&lt;/li&gt;&lt;li&gt;We replaced all RAII lock acquisitions with explicit closures, and instrumented the closures. 
&lt;/li&gt;&lt;li&gt;We enabled &lt;code&gt;parking_lot&lt;/code&gt; deadlock detection. 
&lt;/li&gt;&lt;li&gt;We captured and analyzed core dumps for the killed proxies. 
&lt;/li&gt;&lt;li&gt;We frantically switched to recursive read locks, which generated a new error.
&lt;/li&gt;&lt;li&gt;We spotted what looks like memory corruption, but only of that one tiny lock word.
&lt;/li&gt;&lt;li&gt;We ran our code under an IR interpreter to find UB, fixed some UB, and didn&amp;rsquo;t fix the bug.
&lt;/li&gt;&lt;li&gt;We set up guard pages to catch wild writes.
&lt;/li&gt;&lt;/ul&gt;

&lt;p&gt;In Richard Cook&amp;rsquo;s essential &lt;a href='https://how.complexsystems.fail/' title=''&gt;&amp;ldquo;How Complex Systems Fail&amp;rdquo;&lt;/a&gt;, rule #5 is that &amp;ldquo;complex systems operate in degraded mode&amp;rdquo;. &lt;em&gt;The system continues to function because it contains so many redundancies and because people can make it function, despite the presence of many flaws&lt;/em&gt;. Maybe &lt;code&gt;fly-proxy&lt;/code&gt; is now like the RBMK reactors at Chernobyl, a system that works just fine as long as operators know about and compensate for its known concurrency flaws, and everybody lives happily ever after.&lt;/p&gt;
&lt;div class="right-sidenote"&gt;&lt;p&gt;We are, in particular, running on the most popular architecture for its RWLock implementation.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;We have reached the point where serious conversations are happening about whether we&amp;rsquo;ve found a Rust compiler bug. Amusingly, &lt;code&gt;parking_lot&lt;/code&gt; is so well regarded among Rustaceans that it&amp;rsquo;s equally if not more plausible that Rust itself is broken.&lt;/p&gt;

&lt;p&gt;Nevertheless, we close-read the RWLock implementation. And we spot this:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative rust"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-9txsa3ce"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-9txsa3ce"&gt;&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;state&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;self&lt;/span&gt;&lt;span class="py"&gt;.state&lt;/span&gt;&lt;span class="nf"&gt;.fetch_add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
   &lt;span class="n"&gt;prev_value&lt;/span&gt;&lt;span class="nf"&gt;.wrapping_sub&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;WRITER_BIT&lt;/span&gt; &lt;span class="p"&gt;|&lt;/span&gt; &lt;span class="n"&gt;WRITER_PARKED_BIT&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
                           &lt;span class="nn"&gt;Ordering&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="n"&gt;Relaxed&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;This looks like gibberish, so let&amp;rsquo;s rephrase that code to see what it&amp;rsquo;s actually doing:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative rust"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-kb57g8ff"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-kb57g8ff"&gt;&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;state&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;self&lt;/span&gt;&lt;span class="py"&gt;.state&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt; &lt;span class="o"&gt;~&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;WRITER_BIT&lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="n"&gt;WRITER_PARKED_BIT&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;If you know exactly the state of the word you&amp;rsquo;re writing to, and you know exactly the limited set of permutations the bits of that word can take (for instance, because there&amp;rsquo;s only 4 signaling bits), then instead of clearing bit by fetching a word, altering it, and then storing it, you can clear them &lt;em&gt;atomically&lt;/em&gt; by adding the inverse of those bits to the word.&lt;/p&gt;

&lt;p&gt;This pattern is self-synchronizing, but it relies on an invariant: you&amp;rsquo;d better be right about the original state of the word you&amp;rsquo;re altering. Because if you&amp;rsquo;re wrong, you&amp;rsquo;re adding a very large value to an uncontrolled value.&lt;/p&gt;

&lt;p&gt;In &lt;code&gt;parking_lot&lt;/code&gt;, say we have &lt;code&gt;WRITER&lt;/code&gt; and &lt;code&gt;WRITER_PARKED&lt;/code&gt; set: the state is &lt;code&gt;0b1010&lt;/code&gt;. &lt;code&gt;prev_value&lt;/code&gt;, the state of the lock word when the lock operation started, is virtually always 0, and that&amp;rsquo;s what we&amp;rsquo;re counting on. &lt;code&gt;prev_value.wrapping_sub()&lt;/code&gt;then calculates &lt;code&gt;0xFFFFFFFFFFFFFFF6&lt;/code&gt;, which exactly cancels out the &lt;code&gt;0b1010&lt;/code&gt; state, leaving 0.&lt;/p&gt;
&lt;div class="right-sidenote"&gt;&lt;p&gt;As a grace note, the locking code manages to set that one last unset bit before the proxy deadlocks.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;Consider though what happens if one of those bits isn&amp;rsquo;t set: state is &lt;code&gt;0b1000&lt;/code&gt;. Now that add doesn&amp;rsquo;t cancel out; the final state is instead &lt;code&gt;0xFFFFFFFFFFFFFFFE&lt;/code&gt;. The reader count is completely full and can&amp;rsquo;t be decremented, and all the waiting bits are set so nothing can happen on the lock.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;parking_lot&lt;/code&gt; is a big deal and we&amp;rsquo;re going to be damn sure before we file a bug report. Which doesn&amp;rsquo;t take long; Pavel reproduces the bug in a minimal test case, with a forked version of &lt;code&gt;parking_lot&lt;/code&gt; that confirms and logs the condition.&lt;/p&gt;

&lt;p&gt;&lt;a href='https://github.com/Amanieu/parking_lot/issues/465' title=''&gt;The &lt;code&gt;parking_lot&lt;/code&gt; team quickly confirms&lt;/a&gt; and fixes the bug.&lt;/p&gt;
&lt;h3 id='ex-insania-claritas' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#ex-insania-claritas' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Ex Insania, Claritas&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;Here&amp;rsquo;s what we now know to have been happening:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Thread 1 grabs a read lock.
&lt;/li&gt;&lt;li&gt;Thread 2 tries to grab a write lock, with a &lt;code&gt;try_write_for&lt;/code&gt; timeout; it&amp;rsquo;s &amp;ldquo;parked&amp;rdquo; waiting for the reader, which sets &lt;code&gt;WRITER&lt;/code&gt; and &lt;code&gt;WRITER_PARKED&lt;/code&gt; on the raw lock word.
&lt;/li&gt;&lt;li&gt;Thread 1 releases the lock, unparking a waiting writer, which unsets &lt;code&gt;WRITER_PARKED&lt;/code&gt;.
&lt;/li&gt;&lt;li&gt;Thread 2 wakes, but not for the reason it thinks: the timing is such that it thinks the lock has timed out. So it tries to clear both &lt;code&gt;WRITER&lt;/code&gt; and &lt;code&gt;WRITER_PARKED&lt;/code&gt; — a bitwise &amp;ldquo;double free&amp;rdquo;. Lock: corrupted. Computer: over. 
&lt;/li&gt;&lt;/ol&gt;

&lt;p&gt;&lt;a href='https://github.com/Amanieu/parking_lot/pull/466' title=''&gt;The fix is simple&lt;/a&gt;: the writer bit is cleared separately in the same wakeup queue as the reader. The fix is deployed, the lockups never recur.&lt;/p&gt;

&lt;p&gt;At a higher level, the story is this:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;We&amp;rsquo;re refactoring the proxy to regionalize it, which changes the pattern  of readers and writers on the catalog.
&lt;/li&gt;&lt;li&gt;As we broaden out lazy loads, we introduce writer contention, a classic concurrency/performance debugging problem. It looks like a deadlock at the time! It isn&amp;rsquo;t. 
&lt;/li&gt;&lt;li&gt;&lt;code&gt;try_write_for&lt;/code&gt; is a good move: we need tools to manage contention.
&lt;/li&gt;&lt;li&gt;But now we&amp;rsquo;re on a buggy code path in &lt;code&gt;parking_lot&lt;/code&gt; — we don&amp;rsquo;t know that and can&amp;rsquo;t understand it until we&amp;rsquo;ve lost enough of our minds to second-guess the library.
&lt;/li&gt;&lt;li&gt;We stumble on the bug out of pure dumb luck by stabbing in the dark with &lt;code&gt;read_recursive&lt;/code&gt;.
&lt;/li&gt;&lt;/ol&gt;

&lt;p&gt;Mysteries remain. Why did this only happen in &lt;code&gt;WAW&lt;/code&gt;? Some kind of crazy regional timing thing? Something to do with the Polish &lt;em&gt;kreska&lt;/em&gt; diacritic that makes L&amp;rsquo;s sound like W&amp;rsquo;s? The wax and wane of caribou populations? Some very high-traffic APIs local to these regions? All very plausible.&lt;/p&gt;

&lt;p&gt;We&amp;rsquo;ll never know because we fixed the bug.&lt;/p&gt;

&lt;p&gt;But we&amp;rsquo;re in a better place now, even besides the bug fix:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;we audited all our catalog locking, got rid of all the iflets, and stopped relying on RAII lock guards.
&lt;/li&gt;&lt;li&gt;the resulting closure patterns gave us lock timing metrics, which will be useful dealing with future write contention
&lt;/li&gt;&lt;li&gt;all writes are now labeled, and we emit labeled logs on slow writes, augmented with context data (like app IDs) where we have it
&lt;/li&gt;&lt;li&gt;we also now track the last and current holder of the write lock, augmented with context information; next time we have a deadlock, we should have all the information we need to identify the actors without &lt;code&gt;gdb&lt;/code&gt; stack traces.
&lt;/li&gt;&lt;/ul&gt;
</content>
  </entry>
  <entry>
    <title>Litestream Revamped: SQLite Replication</title>
    <link rel="alternate" href="https://fly.io/blog/litestream-revamped/"/>
    <id>https://fly.io/blog/litestream-revamped/</id>
    <published>2025-05-20T00:00:00+00:00</published>
    <updated>2026-09-28T13:38:50+00:00</updated>
    <media:thumbnail url="https://fly.io/blog/litestream-revamped/assets/litestream-revamped.webp"/>
    <content type="html">&lt;div class="lead"&gt;&lt;p&gt;&lt;a href="https://litestream.io/" title=""&gt;Litestream&lt;/a&gt; is an open-source tool that makes it possible to run many kinds of full-stack applications on top of SQLite by making them reliably recoverable from object storage. This is a post about the biggest change we’ve made to it since I launched it.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;Nearly a decade ago, I got a bug up my ass. I wanted to build full-stack applications quickly. But the conventional n-tier database design required me to do sysadmin work for each app I shipped. Even the simplest applications depended on heavy-weight database servers like Postgres or MySQL.&lt;/p&gt;

&lt;p&gt;I wanted to launch apps on SQLite, because SQLite is easy. But SQLite is embedded, not a server, which at the time implied that the data for my application lived (and died) with just one server.&lt;/p&gt;

&lt;p&gt;So in 2020, I wrote &lt;a href='https://litestream.io/' title=''&gt;Litestream&lt;/a&gt; to fix that.&lt;/p&gt;

&lt;p&gt;Litestream is a tool that runs alongside a SQLite application. Without changing that running application, it takes over the WAL checkpointing process to continuously stream database updates to an S3-compatible object store. If something happens to the server the app is running on, the whole database can efficiently be restored to a different server. You might lose servers, but you won&amp;rsquo;t lose your data.&lt;/p&gt;

&lt;p&gt;Litestream worked well. So we got ambitious. A few years later, we built &lt;a href='https://github.com/superfly/litefs' title=''&gt;LiteFS&lt;/a&gt;. LiteFS takes the ideas in Litestream and refines them, so that we can do read replicas and primary failovers with SQLite. LiteFS gives SQLite the modern deployment story of an n-tier database like Postgres, while keeping the database embedded.&lt;/p&gt;

&lt;p&gt;We like both LiteFS and Litestream. But Litestream is the more popular project. It&amp;rsquo;s easier to deploy and easier to reason about.&lt;/p&gt;

&lt;p&gt;There are some good ideas in LiteFS. We&amp;rsquo;d like Litestream users to benefit from them. So we&amp;rsquo;ve taken our LiteFS learnings and applied them to some new features in Litestream.&lt;/p&gt;
&lt;h2 id='point-in-time-restores-but-fast' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#point-in-time-restores-but-fast' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Point-in-time restores, but fast&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;&lt;a href='https://fly.io/blog/all-in-on-sqlite-litestream/' title=''&gt;Here&amp;rsquo;s how Litestream was originally designed&lt;/a&gt;: you run &lt;code&gt;litestream&lt;/code&gt; against a SQLite database, and it opens up a long-lived read transaction. This transaction arrests SQLite WAL checkpointing, the process by which SQLite consolidates the WAL back into the main database file. Litestream builds a &amp;ldquo;shadow WAL&amp;rdquo; that records WAL pages, and copies them to S3.&lt;/p&gt;

&lt;p&gt;This is simple, which is good. But it can also be slow. When you want to restore a database, you have have to pull down and replay every change since the last snapshot. If you changed a single database page a thousand times, you replay a thousand changes. For databases with frequent writes, this isn&amp;rsquo;t a good approach.&lt;/p&gt;

&lt;p&gt;In LiteFS, we took a different approach. LiteFS is transaction-aware. It doesn&amp;rsquo;t simply record raw WAL pages, but rather ordered ranges of pages associated with transactions, using a file format we call &lt;a href='https://github.com/superfly/ltx' title=''&gt;LTX&lt;/a&gt;. Each LTX file represents a sorted changeset of pages for a given period of time.&lt;/p&gt;

&lt;p&gt;&lt;img loading="lazy" decoding="async" alt="a simple linear LTX file with 8 pages between 1 and 21" width="1363" height="341" src="/blog/litestream-revamped/assets/linear-ltx.png" /&gt;&lt;/p&gt;

&lt;p&gt;Because they are sorted, we can easily merge multiple LTX files together and create a new LTX file with only the latest version of each page.&lt;/p&gt;

&lt;p&gt;&lt;img loading="lazy" decoding="async" alt="merging three LTX files into one" width="1477" height="911" src="/blog/litestream-revamped/assets/merged-ltx.png" /&gt;&lt;/p&gt;
&lt;div class="right-sidenote"&gt;&lt;p&gt;This is similar to how an &lt;a href="https://en.wikipedia.org/wiki/Log-structured_merge-tree" title=""&gt;LSM tree&lt;/a&gt; works.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;This process of combining smaller time ranges into larger ones is called &lt;em&gt;compaction&lt;/em&gt;. With it, we can replay a SQLite database to a specific point in time, with a minimal  duplicate pages.&lt;/p&gt;
&lt;h2 id='casaas-compare-and-swap-as-a-service' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#casaas-compare-and-swap-as-a-service' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;CASAAS: Compare-and-Swap as a Service&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;One challenge Litestream has to deal with is desynchronization. Part of the point of Litestream is that SQLite applications don&amp;rsquo;t have to be aware of it. But &lt;code&gt;litestream&lt;/code&gt; is just a process, running alongside the application, and it can die independently. If &lt;code&gt;litestream&lt;/code&gt; is down while database changes occur, it will miss changes. The same kind of problem occurs if you start replication from a new server.&lt;/p&gt;

&lt;p&gt;Litestream needs a way to reset the replication stream from a new snapshot. How it does that is with &amp;ldquo;generations&amp;rdquo;. &lt;a href='https://litestream.io/how-it-works/#snapshots--generations' title=''&gt;A generation&lt;/a&gt; represents a snapshot and a stream of WAL updates, uniquely identified. Litestream notices any break in its WAL sequence and starts a new generation, which is how it recovers from desynchronization.&lt;/p&gt;

&lt;p&gt;Unfortunately, storing and managing multiple generations makes it difficult to implement features like failover and read-replicas.&lt;/p&gt;

&lt;p&gt;The most straightforward way around this problem is to make sure only one instance of Litestream can replication to a given destination. If you can do that, you can store just a single, latest generation. That in turn makes it easy to know how to resync a read replica; there&amp;rsquo;s only one generation to choose from.&lt;/p&gt;

&lt;p&gt;In LiteFS, we solved this problem by using Consul, which guaranteed a single leader. That requires users to know about Consul. Things like &amp;ldquo;requiring Consul&amp;rdquo; are probably part of the reason Litestream is so much more popular than LiteFS.&lt;/p&gt;

&lt;p&gt;In Litestream, we&amp;rsquo;re solving the problem a different way. Modern object stores like S3 and Tigris solve this problem for us: they now offer &lt;a href='https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3-functionality-conditional-writes/' title=''&gt;conditional write support&lt;/a&gt;. With conditional writes, we can implement a time-based lease. We get essentially the same constraint Consul gave us, but without having to think about it or set up a dependency.&lt;/p&gt;

&lt;p&gt;In the immediacy, this will mean you can run Litestream with ephemeral nodes, with overlapping run times, and even if they&amp;rsquo;re storing to the same destination, they won&amp;rsquo;t confuse each other.&lt;/p&gt;
&lt;h2 id='lightweight-read-replicas' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#lightweight-read-replicas' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Lightweight read replicas&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;The original design constraint of both Litestream and LiteFS was to extend SQLite, to modern deployment scenarios, without disturbing people&amp;rsquo;s built code. Both tools are meant to function even if applications are oblivious to them.&lt;/p&gt;

&lt;p&gt;LiteFS is more ambitious than Litestream, and requires transaction-awareness. To get that without disturbing built code, we use a cute trick (a.k.a. a gross hack): LiteFS provides a FUSE filesystem, which lets it act as a proxy between the application and the backing store. From that vantage point, we can easily discern transactions.&lt;/p&gt;

&lt;p&gt;The FUSE approach gave us a lot of control, enough that users could use SQLite replicas just like any other database. But installing and running a whole filesystem (even a fake one) is a lot to ask of users. To work around that problem, we relaxed a constraint: LiteFS can function without the FUSE filesystem if you load an extension into your application code, &lt;a href='https://github.com/superfly/litevfs' title=''&gt;LiteVFS&lt;/a&gt;.  LiteVFS is a &lt;a href='https://www.sqlite.org/vfs.html' title=''&gt;SQLite Virtual Filesystem&lt;/a&gt; (VFS). It works in a variety of environments, including some where FUSE can&amp;rsquo;t, like in-browser WASM builds.&lt;/p&gt;

&lt;p&gt;What we&amp;rsquo;re doing next is taking the same trick and using it on Litestream. We&amp;rsquo;re building a VFS-based read-replica layer. It will be able to fetch and cache pages directly from S3-compatible object storage.&lt;/p&gt;

&lt;p&gt;Of course, there&amp;rsquo;s a catch: this approach isn&amp;rsquo;t as efficient as a local SQLite database. That kind of efficiency, where you don&amp;rsquo;t even need to think about N+1 queries because there&amp;rsquo;s no network round-trip to make the duplicative queries pile up costs, is part of the point of using SQLite.&lt;/p&gt;

&lt;p&gt;But we&amp;rsquo;re optimistic that with cacheing and prefetching, the approach we&amp;rsquo;re using will yield, for the right use cases, strong performance — all while serving SQLite reads hot off of Tigris or S3.&lt;/p&gt;
&lt;figure class="post-cta"&gt;
  &lt;figcaption&gt;
    &lt;h2&gt;Litestream is fully open source&lt;/h2&gt;
    &lt;p&gt;It&amp;rsquo;s not coupled with Fly.io at all; you can use it anywhere.&lt;/p&gt;
      &lt;a class="btn btn-lg" href="https://litestream.io/"&gt;
        Check it out &lt;span class='opacity-50'&gt;→&lt;/span&gt;
      &lt;/a&gt;
  &lt;/figcaption&gt;
  &lt;div class="image-container"&gt;
    &lt;img loading="lazy" decoding="async" src="/static/images/cta-dog.webp" srcset="/static/images/cta-dog@2x.webp 2x" alt=""&gt;
  &lt;/div&gt;
&lt;/figure&gt;

&lt;h2 id='synchronize-lots-of-databases' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#synchronize-lots-of-databases' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;Synchronize Lots Of Databases&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;While we&amp;rsquo;ve got you here: we&amp;rsquo;re knocking out one of our most requested features.&lt;/p&gt;

&lt;p&gt;In the old Litestream design, WAL-change polling and slow restores made it infeasible to replicate large numbers of databases from a single process. That has been our answer when users ask us for a &amp;ldquo;wildcard&amp;rdquo; or &amp;ldquo;directory&amp;rdquo; replication argument for the tool.&lt;/p&gt;

&lt;p&gt;Now that we&amp;rsquo;ve switched to LTX, this isn&amp;rsquo;t a problem any more. It should thus be possible to replicate &lt;code&gt;/data/*.db&lt;/code&gt;, even if there&amp;rsquo;s hundreds or thousands of databases in that directory.&lt;/p&gt;
&lt;h2 id='we-still-sqlite' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#we-still-sqlite' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;We Still ❤️ SQLite&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;SQLite has always been a solid database to build on and it&amp;rsquo;s continued to find new use cases as the industry evolves. We&amp;rsquo;re super excited to continue to build Litestream alongside it.&lt;/p&gt;

&lt;p&gt;We have a sneaking suspicion that the robots that write LLM code are going to like SQLite too. We think what &lt;a href='https://phoenix.new/' title=''&gt;coding agents like Phoenix.new&lt;/a&gt; want is a way to try out code on live data, screw it up, and then rollback &lt;em&gt;both the code and the state.&lt;/em&gt; These Litestream updates put us in a position to give agents PITR as a primitive. On top of that, you can build both rollbacks and forks.&lt;/p&gt;

&lt;p&gt;Whether or not you&amp;rsquo;re drinking the AI kool-aid, we think this new design for Litestream is just better. We&amp;rsquo;re psyched to be rolling it out, and for the features it&amp;rsquo;s going to enable.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Launching MCP Servers on Fly.io</title>
    <link rel="alternate" href="https://fly.io/blog/mcp-launch/"/>
    <id>https://fly.io/blog/mcp-launch/</id>
    <published>2025-05-19T00:00:00+00:00</published>
    <updated>2026-09-28T14:09:34+00:00</updated>
    <media:thumbnail url="https://fly.io/blog/mcp-launch/assets/fly-mcp-launch.png"/>
    <content type="html">&lt;div class="lead"&gt;&lt;p&gt;This is a blog post. Part showing off. Part opinion. Plan accordingly.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;The &lt;a href='https://www.anthropic.com/news/model-context-protocol' title=''&gt;Model Context Protocol&lt;/a&gt; is days away from turning six months old. You read that right, six &lt;em&gt;months&lt;/em&gt; old. MCP Servers have both taken the world by storm, and still trying to figure out what they want to be when they grow up.&lt;/p&gt;

&lt;p&gt;There is no doubt that MCP servers are useful. But their appeal goes beyond that. They are also simple and universal. What&amp;rsquo;s not to like?&lt;/p&gt;

&lt;p&gt;Well, for starters, there&amp;rsquo;s basically two types of MCP servers. One small and nimble that runs as a process on your machine. And one that is a HTTP server that runs presumably elsewhere and is &lt;a href='https://modelcontextprotocol.io/specification/2025-03-26/basic/authorization' title=''&gt;standardizing&lt;/a&gt; on OAuth 2.1. And there is a third type, but it is deprecated.&lt;/p&gt;

&lt;p&gt;Next there is the configuration. Asking users to manually edit JSON seems so early 21th century. With Claude, this goes into &lt;code&gt;~/Library/Application Support/Claude/claude_desktop_config.json&lt;/code&gt;, and is found under a &lt;code&gt;MCPServer&lt;/code&gt; key. With Zed, this file is in &lt;code&gt;~/.config/zed/settings.json&lt;/code&gt; and is found under a &lt;code&gt;context_servers&lt;/code&gt; key. And some tools put these files in a different place depending on whether you are running on MacOS, Linux, or Windows.&lt;/p&gt;

&lt;p&gt;Finally, there is security. An MCP server running on your machine literally has access to everything you do. Running remote solves this problem, but did I mention &lt;a href='https://datatracker.ietf.org/doc/html/draft-ietf-oauth-v2-1-12' title=''&gt;OAuth 2.1&lt;/a&gt;? Not exactly something one sets up for casual use.&lt;/p&gt;

&lt;p&gt;None of these issues are fatal - something that is obvious by the fact that MCP servers are quite popular. But can we do better? I think so.&lt;/p&gt;

&lt;hr&gt;

&lt;p&gt;Demo time.&lt;/p&gt;

&lt;p&gt;Let&amp;rsquo;s try out the &lt;a href='https://github.com/modelcontextprotocol/servers/blob/main/src/slack/README.md' title=''&gt;Slack MCP Server&lt;/a&gt;:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;MCP Server for the Slack API, enabling Claude to interact with Slack workspaces.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That certainly sounds like a good test case. There is a small amount of &lt;a href='https://github.com/modelcontextprotocol/servers/blob/main/src/slack/README.md#setup' title=''&gt;setup&lt;/a&gt; you need to do, and when you are done you end up with a &lt;em&gt;Bot User OAuth Token&lt;/em&gt; staring with &lt;code&gt;xoxb-&lt;/code&gt; and a &lt;em&gt;Team ID&lt;/em&gt; starting with a &lt;code&gt;T&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;You &lt;em&gt;would&lt;/em&gt; run it using the following:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative sh"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-8oua3lgn"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-8oua3lgn"&gt;npx &lt;span class="nt"&gt;-y&lt;/span&gt; @modelcontextprotocol/server-slack
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;But instead, you convert that command to JSON and find the right configuration file and put this information in there. And either run the slack MCP server locally or set up a server with or without authentication.&lt;/p&gt;

&lt;p&gt;Wouldn&amp;rsquo;t it be nice to have the simplicity of a local process, the security of a remote server, and to eliminate the hassle of manually editing JSON?&lt;/p&gt;

&lt;p&gt;Here&amp;rsquo;s our current thinking:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative sh"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-cn1hoc6u"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-cn1hoc6u"&gt;fly mcp launch &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="s2"&gt;"npx -y @modelcontextprotocol/server-slack"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--claude&lt;/span&gt; &lt;span class="nt"&gt;--server&lt;/span&gt; slack &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--secret&lt;/span&gt; &lt;span class="nv"&gt;SLACK_BOT_TOKEN&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;xoxb-your-bot-token &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--secret&lt;/span&gt; &lt;span class="nv"&gt;SLACK_TEAM_ID&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;T01234567
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;You can put this all on one line if you like, I just split this up so it fits on small screens and so we can talk about the various parts.&lt;/p&gt;

&lt;p&gt;The first three words seem reasonable. The quoted string is just the command that we want to run. So lets talk about the four flags. The first tells us which tool&amp;rsquo;s configuration file we want to update. The second specifies the name to use for the server in that configuration file. The last two set secrets.&lt;/p&gt;

&lt;p&gt;Oh, did I say current thinking? Perhaps I should mention that it is something you can run right now, as long as you have flyctl &lt;code&gt;v0.3.125&lt;/code&gt; or later. Complete with the options you would expect from Fly.io, like the ability to configure auto-stop, file contents, flycast, secrets, region, volumes, and VM sizes.&lt;/p&gt;

&lt;p&gt;And, hey, lookie there:&lt;/p&gt;

&lt;p&gt;&lt;img loading="lazy" decoding="async" alt="testing, testing, 1, 2, 3" width="734" height="290" src="/blog/mcp-launch/assets/mcp-slack.png" /&gt;&lt;/p&gt;

&lt;p&gt;Support for Claude, Cursor, Neovim, VS Code, Windsurf, and Zed are built in. You can select multiple clients and configuration files.&lt;/p&gt;

&lt;p&gt;By default, bearer token authentication will be set up on both the server and client.&lt;/p&gt;

&lt;p&gt;You can find the complete set of options on our &lt;a href='https://docs.fly.io/flyctl/cmd/fly_mcp_launch' title=''&gt;&lt;code&gt;fly mcp launch&lt;/code&gt;&lt;/a&gt; docs page.&lt;/p&gt;

&lt;hr&gt;

&lt;p&gt;But this post isn&amp;rsquo;t just about experimental demoware that is subject to change.
It is about the depth of support that we are rapidly bringing online, including:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Support for all transports, not just the ones we recommend.
&lt;/li&gt;&lt;li&gt;Ability to deploy using the command line or the Machines API, with a number of different options that allow you to chose between elegant simplicity and excruciating precise control.
&lt;/li&gt;&lt;li&gt;Ability to deploy each MCP server to a separate Machine, container, or even inside your application.
&lt;/li&gt;&lt;li&gt;Access via HTTP Authorization, wireguard tunnels and flycast, or reverse proxies.
&lt;/li&gt;&lt;/ul&gt;

&lt;p&gt;You can see all this spelled out in our &lt;a href='https://docs.fly.io/model-context-protocol' title=''&gt;docs&lt;/a&gt;. Be forewarned, most pages are marked as &lt;em&gt;beta&lt;/em&gt;. But the examples provided all work. Well, there may be a bug here or there, but the examples &lt;em&gt;as shown&lt;/em&gt; are thought to work. Maybe.&lt;/p&gt;

&lt;p&gt;Let&amp;rsquo;s figure out the ideal ergonomics of deploying MCP servers remotely together!&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Provisioning Machines using MCPs</title>
    <link rel="alternate" href="https://fly.io/blog/mcp-provisioning/"/>
    <id>https://fly.io/blog/mcp-provisioning/</id>
    <published>2025-05-07T00:00:00+00:00</published>
    <updated>2026-09-28T14:09:34+00:00</updated>
    <media:thumbnail url="https://fly.io/blog/mcp-provisioning/assets/Hello.webp"/>
    <content type="html">&lt;div class="lead"&gt;&lt;p&gt;Today’s state of the art is K8S, Terraform, web based UIs, and CLIs.  Those days are numbered.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;On Monday, I created my first fly volume using an &lt;a href='https://modelcontextprotocol.io/introduction' title=''&gt;MCP&lt;/a&gt;. For those who don&amp;rsquo;t know what MCPs are, they are how you attach tools to &lt;a href='https://en.wikipedia.org/wiki/Large_language_model' title=''&gt;LLM&lt;/a&gt;s like Claude or Cursor. I added support for
&lt;a href='https://docs.fly.io/flyctl/cmd/fly_volumes_create' title=''&gt;fly volume create&lt;/a&gt; to &lt;a href='https://docs.fly.io/flyctl/cmd/fly_mcp_server' title=''&gt;fly mcp server&lt;/a&gt;, and it worked the first time.
A few hours later, and with the assistance of GitHub Copilot, i added support for all &lt;a href='https://docs.fly.io/flyctl/cmd/fly_volumes' title=''&gt;fly volumes&lt;/a&gt; commands.&lt;/p&gt;

&lt;hr&gt;
&lt;div class="right-sidenote"&gt;&lt;p&gt;This movie summary is from &lt;a href="https://collidecolumn.wordpress.com/2013/08/04/when-worlds-collide-77-talking-with-computers-beyond-mouse-and-keyboard/"&gt;When Worlds Collide, by Nalaka Gunawardene&lt;/a&gt;&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;I&amp;rsquo;m reminded of the memorable scene in the film &lt;a href='https://en.wikipedia.org/wiki/Star_Trek_IV:_The_Voyage_Home' title=''&gt;Star Trek IV: The Voyage Home&lt;/a&gt; (1986). Chief Engineer Scotty, having time-travelled 200 years back to late 20th century San Francisco with his crew mates, encounters an early Personal Computer (PC).&lt;/p&gt;

&lt;p&gt;Sitting in front of it, he addresses the machine affably as “Computer!” Nothing happens. Scotty repeats himself; still no response. He doesn’t realise that voice recognition capability hadn’t arrived yet.&lt;/p&gt;

&lt;p&gt;Exasperated, he picks up the mouse and speaks into it: “Hello, computer?” The computer’s owner offers helpful advice: “Just use the keyboard.”&lt;/p&gt;

&lt;p&gt;Scotty looks astonished. “A keyboard?” he asks, and adds in a sarcastic tone: “How quaint!”&lt;/p&gt;

&lt;hr&gt;

&lt;p&gt;A while later, I asked for a list of volumes for an existing app, and Claude noted that I had a few volumes that weren&amp;rsquo;t attached to any machines. So I asked it to delete the oldest unattached volume, and it did so. Then it occurred to me to do it again; this time I captured the screen:&lt;/p&gt;
&lt;div align="center"&gt;&lt;p&gt;&lt;img loading="lazy" decoding="async" alt="Deleting a volume using MCP: &amp;quot;What is my oldest volume&amp;quot;? ... &amp;quot;Delete that volume too&amp;quot;" width="495" height="719" src="/blog/mcp-provisioning/assets/volume-delete.png"&gt;&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;A few notes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;I could have written a program using the &lt;a href='https://docs.fly.io/machines/api/volumes-resource' title=''&gt;machines API&lt;/a&gt;, but that would have required some effort.
&lt;/li&gt;&lt;li&gt;I could have used &lt;a href='https://docs.fly.io/flyctl/cmd/fly_volumes' title=''&gt;flyctl&lt;/a&gt; directly, but to be honest, I would have had to use the documentation and a bit of copy/pasting of arguments.
&lt;/li&gt;&lt;li&gt;I could have navigated to a list of volumes for this application in the Fly.io dashboard, but there currently isn&amp;rsquo;t any way to sort the list or delete a volume. Had those been in place, that would have been a reasonable alternative if that was something I was actively looking for. This felt different to me, the LLM noted something, brought it to my attention, and I asked it to make a change as a result.
&lt;/li&gt;&lt;li&gt;Since this support is built on &lt;code&gt;flyctl&lt;/code&gt;, I would have received an error had I tried to destroy a volume that is currently mounted. Knowing that gave me the confidence to try the command.
&lt;/li&gt;&lt;/ul&gt;

&lt;p&gt;All in all, I found it to be a very intuitive way to make changes to the resources assigned to my application.&lt;/p&gt;

&lt;hr&gt;

&lt;p&gt;Imagine a future where you say to your favorite LLM &amp;ldquo;launch my application on Fly.io&amp;rdquo;, and your code is scanned and you are presented with a plan containing details such as regions, databases, s3 storage, memory, machines, and the like. You are given the opportunity to adjust the plan and, when ready, say &amp;ldquo;Make it so&amp;rdquo;.&lt;/p&gt;

&lt;p&gt;For many, the next thing you will see is that your application is up and running. For others, there may be a build error, a secret you need to set, a database that needs to be seeded, or any number of other mundane reasons why your application didn&amp;rsquo;t work the first time.&lt;/p&gt;

&lt;p&gt;Not to fear, your LLM will be able to examine your logs and will not only make suggestions as to next steps, but will be in a position to take actions on your behalf should you wish it to do so.&lt;/p&gt;

&lt;p&gt;And it doesn&amp;rsquo;t stop there. There will be MCP servers running on your Fly.io private network - either on separate machines, or in &amp;ldquo;sidecar&amp;rdquo; containers, or even integrated into your app. These will enable you to monitor and interact with your application.&lt;/p&gt;

&lt;p&gt;This is not science fiction. The ingredients are all in place to make this a reality. At the moment, it is a matter of &amp;ldquo;some assembly required&amp;rdquo;, but it should only be a matter of weeks before all this comes together into a neat package..&lt;/p&gt;

&lt;hr&gt;

&lt;p&gt;Meanwhile, you can try this now. Make sure you run &lt;a href='https://docs.fly.io/flyctl/cmd/fly_version_upgrade' title=''&gt;fly version upgrade&lt;/a&gt; and verify that you are running v0.3.117.&lt;/p&gt;

&lt;p&gt;Then configure your favorite LLM. Here’s my &lt;code&gt;claude_desktop_config.json&lt;/code&gt; for example:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative json"&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-aapzqkcn"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-aapzqkcn"&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"mcpServers"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"fly.io"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"command"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"/Users/rubys/.fly/bin/flyctl"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"args"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"mcp"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"server"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;Adjust the path to &lt;code&gt;flyctl&lt;/code&gt; as needed.  Restart your LLM, and ask what tools are available. Try a few commands and let us know what you like and let us know if you have any suggestions. Just be aware this is not a demo, if you ask it to destroy a volume, that operation is not reversable. Perhaps try this first on a throwaway application.&lt;/p&gt;

&lt;p&gt;You don&amp;rsquo;t even need a LLM to try out the flyctl MCP server. If you have Node.js installed, you can run the &lt;a href='https://modelcontextprotocol.io/docs/tools/inspector' title=''&gt;MCP Inspector&lt;/a&gt;:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative "&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-46e5yb5"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-46e5yb5"&gt;fly mcp server -i
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;Once started, visit &lt;a href="http://127.0.0.1:6274/"&gt;http://127.0.0.1:6274/&lt;/a&gt;, click on &amp;ldquo;Connect&amp;rdquo;, then &amp;ldquo;List Tools&amp;rdquo;, select &amp;ldquo;fly-platform-status&amp;rdquo;, then click on &amp;ldquo;Run Tool&amp;rdquo;.&lt;/p&gt;

&lt;p&gt;The plan is to see what works well and what doesn&amp;rsquo;t work so well, make adjustments, build support in a bottoms up fashion, and iterate rapidly.&lt;/p&gt;

&lt;p&gt;By providing feedback, you can be a part of making this vision a reality.&lt;/p&gt;

&lt;hr&gt;

&lt;p&gt;At the present time, &lt;em&gt;most&lt;/em&gt; of the following are roughed in:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href='https://docs.fly.io/flyctl/cmd/fly_apps' title=''&gt;apps&lt;/a&gt;
&lt;/li&gt;&lt;li&gt;&lt;a href='https://docs.fly.io/flyctl/cmd/fly_logs' title=''&gt;logs&lt;/a&gt;
&lt;/li&gt;&lt;li&gt;&lt;a href='https://docs.fly.io/flyctl/cmd/fly_machine' title=''&gt;machine&lt;/a&gt;
&lt;/li&gt;&lt;li&gt;&lt;a href='https://docs.fly.io/flyctl/cmd/fly_orgs' title=''&gt;orgs&lt;/a&gt;
&lt;/li&gt;&lt;li&gt;&lt;a href='https://docs.fly.io/flyctl/cmd/fly_platform' title=''&gt;platform&lt;/a&gt;
&lt;/li&gt;&lt;li&gt;&lt;a href='https://docs.fly.io/flyctl/cmd/fly_status' title=''&gt;status&lt;/a&gt;
&lt;/li&gt;&lt;li&gt;&lt;a href='https://docs.fly.io/flyctl/cmd/fly_volumes' title=''&gt;volumes&lt;/a&gt;
&lt;/li&gt;&lt;/ul&gt;

&lt;p&gt;The code is open source, and the places to look is at &lt;a href='https://github.com/superfly/flyctl/blob/master/internal/command/mcp/server.go' title=''&gt;server.go&lt;/a&gt; and the &lt;a href='https://github.com/superfly/flyctl/tree/master/internal/command/mcp/server' title=''&gt;server&lt;/a&gt; directory.&lt;/p&gt;

&lt;p&gt;Feel free to open &lt;a href='https://github.com/superfly/flyctl/issues' title=''&gt;issues&lt;/a&gt; or start a discussion on &lt;a href='https://community.fly.io/' title=''&gt;community.fly.io&lt;/a&gt;.&lt;/p&gt;

&lt;hr&gt;

&lt;p&gt;Not ready yet to venture into the world of LLMs? Just remember, resistance is futile.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>30 Minutes With MCP and flyctl</title>
    <link rel="alternate" href="https://fly.io/blog/30-minute-mcp/"/>
    <id>https://fly.io/blog/30-minute-mcp/</id>
    <published>2025-04-10T00:00:00+00:00</published>
    <updated>2025-05-20T16:26:25+00:00</updated>
    <media:thumbnail url="https://fly.io/blog/30-minute-mcp/assets/robots-chatting.webp"/>
    <content type="html">&lt;div class="lead"&gt;&lt;p&gt;I wrote this post on our internal message board, and then someone asked, “why is this an internal post and not on our blog”, so now it is.&lt;/p&gt;
&lt;/div&gt;&lt;div class="right-sidenote"&gt;&lt;p&gt;well, Cursor built&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;I built the &lt;a href='https://github.com/superfly/flymcp' title=''&gt;most basic MCP server for &lt;code&gt;flyctl&lt;/code&gt;&lt;/a&gt; I could think of. It took 30 minutes.&lt;/p&gt;

&lt;p&gt;&lt;a href='https://modelcontextprotocol.io/introduction' title=''&gt;MCP&lt;/a&gt;, for those unaware, is the emerging standard protocol for connecting an LLM (or an app that drives an LLM in the cloud, like Claude Desktop) to, well, anything. The &amp;ldquo;client&amp;rdquo; in MCP is the LLM; the &amp;ldquo;server&amp;rdquo; is the MCP server and the &amp;ldquo;tools&amp;rdquo; it exports. It mostly just defines an exchange of JSON blobs; one of those JSON blobs enables the LLM to discover all the tools exported by the server.&lt;/p&gt;

&lt;p&gt;A classic example of an MCP server is (yes, really) a Python shell. MCP publishes to (say) Claude that it can run arbitrary Python code with a tool call; not only that, says the tool description, but you can use those Python tool calls to, say, scrape the web. When the LLM wants to scrape the web with Python, it uses MCP send a JSON blob describing the Python tool call; the MCP server (yes, really) runs the Python and returns the result.&lt;/p&gt;

&lt;p&gt;Because I have not yet completely lost my mind, I chose to expose just two &lt;code&gt;flyctl&lt;/code&gt; commands: &lt;code&gt;fly logs&lt;/code&gt; and &lt;code&gt;fly status&lt;/code&gt;. Because I&amp;rsquo;m lazy, I used MCP&amp;rsquo;s &lt;code&gt;stdio&lt;/code&gt; mode; the MCP server just captures the output of &lt;code&gt;flyctl&lt;/code&gt;. Because someone already wrote a &lt;a href='https://github.com/mark3labs/mcp-go' title=''&gt;really good Go MCP library&lt;/a&gt; , this whole thing is like 90 lines of code.&lt;/p&gt;
&lt;div class="callout"&gt;&lt;p&gt;We may have gotten a little lucky, because we made a decision back in 2020 to give most of our &lt;code&gt;flyctl&lt;/code&gt; commands a &lt;code&gt;json&lt;/code&gt; mode to make them easier to drive from automation. I use that in my MCP server. I don’t know how much of a difference it made.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;It&amp;rsquo;s, uh, annoyingly useful.&lt;/p&gt;

&lt;p&gt;I loaded it up and pointed it &lt;a href='https://www.unpkg.com/' title=''&gt;at unpkg&lt;/a&gt;. I got this, from Claude:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;This appears to be a globally distributed CDN service running the unpkg JavaScript package registry, with machines strategically placed around the world to provide low-latency access. There are some stability issues with a couple of the machines, and several have experienced memory-related crashes that triggered automatic restarts. &lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;It then recited the complete configuration of the application:&lt;/p&gt;
&lt;div class="highlight-wrapper group relative "&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-wrap-target="#code-7334k7i0"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /&gt;&lt;path d="M11.081 6.466L9.533 8.037l1.548 1.571" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"&gt;
      Wrap text
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;button 
    type="button"
    class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none"
    data-copy-target="sibling"
  &gt;
    &lt;svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"&gt;&lt;g buffered-rendering="static"&gt;&lt;path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /&gt;&lt;path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /&gt;&lt;/g&gt;&lt;/svg&gt;
    &lt;span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"&gt;
      Copy to clipboard
    &lt;/span&gt;
  &lt;/button&gt;
  &lt;div class='highlight relative group'&gt;
    &lt;pre class='highlight '&gt;&lt;code id="code-7334k7i0"&gt;**App Name**: unpkg       
**Status**: Deployed and running
**URL**: unpkg.fly.dev
**Organization**: unpkg
**Platform Version**: machines
**Runtime**: Bun (JavaScript runtime)

The app is running on 10 machines distributed across global regions:       
North America: lax (Los Angeles), atl (Atlanta), ewr (Newark)
Europe: lhr (London), cdg (Paris), ams (Amsterdam)
Asia: sin (Singapore), nrt (Tokyo), hkg (Hong Kong)
South America: bog (Bogotá)
Oceania: syd (Sydney)
&lt;/code&gt;&lt;/pre&gt;
  &lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;OK. But then:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Most machines are in “passing” health check status
2 machines are in “critical” status:
One in ewr (Newark) with “context deadline exceeded” error
One in atl (Atlanta) with “gone” status&lt;/p&gt;

&lt;p&gt;Recent Events&lt;/p&gt;

&lt;p&gt;Multiple machines have experienced restarts
Several machines show “oom_killed: true” in their event history, indicating they ran out of memory&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;So I asked, &lt;em&gt;&amp;ldquo;try getting logs for one of the critical machines&amp;rdquo;&lt;/em&gt;.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;The root cause appears to be memory usage - the Bun process was consuming about 3.7GB of memory (out of the 4GB allocated to the machine), and there was likely a surge in requests or memory usage that pushed it over the limit. This matches what we saw in the status report, which showed multiple instances of “oom_killed: true” in the event history for several machines.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;And then it dug in:&lt;/p&gt;

&lt;blockquote&gt;
&lt;ol&gt;
&lt;li&gt;The machine was actively serving content requests with 200 OK responses until 20:47:03 PM
&lt;/li&gt;&lt;li&gt;At that point, it ran out of memory:
&lt;code&gt;Out of memory: Killed process 641 (bun) total-vm:85950964kB, anon-rss:3744352kB, file-rss:12032kB, shmem-rss:0kB, UID:0 pgtables:16908kB oom_score_adj:0&lt;/code&gt;
&lt;/li&gt;&lt;li&gt;The process was killed with &lt;code&gt;SIGKILL&lt;/code&gt;
&lt;/li&gt;&lt;li&gt;The system attempted to recover by restarting:
&lt;code&gt;reboot: Restarting system&lt;/code&gt;
&lt;/li&gt;&lt;li&gt;A health check failed shortly after:
&lt;code&gt;Health check on port 3000 has failed. Your app is not responding properly. Services exposed on ports [80, 443] will have intermittent failures until the health check passes.&lt;/code&gt;
&lt;/li&gt;&lt;li&gt;The machine then restarted successfully and the service came back up:
&lt;code&gt;Server listening on http://localhost:3000 ...&lt;/code&gt;
&lt;/li&gt;&lt;li&gt;The health check started passing again at 20:47:46 PM:
&lt;code&gt;Health check on port 3000 is now passing.&lt;/code&gt;
&lt;/li&gt;&lt;li&gt;The machine resumed normal operations, serving requests.
&lt;/li&gt;&lt;/ol&gt;
&lt;/blockquote&gt;

&lt;p&gt;I’m honestly kind of surprised how fast this was useful. It&amp;rsquo;s faster than I find problems in apps. Just by giving some LLM some tools that do basically what I’d do.&lt;/p&gt;

&lt;p&gt;Local MCP servers are scary. I don&amp;rsquo;t like that I&amp;rsquo;m giving a Claude instance in the cloud the ability to run a native program on my machine. I think &lt;code&gt;fly logs&lt;/code&gt; and &lt;code&gt;fly status&lt;/code&gt; are safe, but I&amp;rsquo;d rather know it&amp;rsquo;s safe. It would be, if I was running &lt;code&gt;flyctl&lt;/code&gt; in an isolated environment and not on my local machine.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Our Best Customers Are Now Robots</title>
    <link rel="alternate" href="https://fly.io/blog/fuckin-robots/"/>
    <id>https://fly.io/blog/fuckin-robots/</id>
    <published>2025-04-08T00:00:00+00:00</published>
    <updated>2026-09-30T00:06:54+00:00</updated>
    <media:thumbnail url="https://fly.io/blog/fuckin-robots/assets/robot-chef.webp"/>
    <content type="html">&lt;div class="lead"&gt;&lt;p&gt;We’re Fly.io, a developer-focused public cloud. We turn Docker containers into hardware-isolated virtual machines running on our own metal around the world. We spent years coming up with &lt;a href="https://fly.io/speedrun/" title=""&gt;a developer experience we were proud of&lt;/a&gt;. But now the robots are taking over, and they don’t care.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;It&amp;rsquo;s weird to say this out loud!&lt;/p&gt;

&lt;p&gt;For years, one of our calling cards was &amp;ldquo;developer experience&amp;rdquo;. We made a decision, early on, to be a CLI-first company, and put a lot effort into making that CLI seamless. For a good chunk of our users, it really is the case that you can just &lt;a href='https://docs.fly.io/flyctl/cmd/fly_launch' title=''&gt;&lt;code&gt;flyctl launch&lt;/code&gt;&lt;/a&gt; from a git checkout and have an app containerized and deployed on the Internet. We haven&amp;rsquo;t always nailed these details, but we&amp;rsquo;ve really sweated them.&lt;/p&gt;

&lt;p&gt;But a funny thing has happened over the last 6 months or so. If you look at the numbers, DX might not matter that much. That&amp;rsquo;s because the users driving the most growth on the platform aren&amp;rsquo;t people at all. They&amp;#39;re… robots.&lt;/p&gt;
&lt;h2 id='what-the-fuck-is-happening' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#what-the-fuck-is-happening' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;What The Fuck Is Happening?&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Here&amp;rsquo;s how we understand what we&amp;rsquo;re seeing. You start by asking, &amp;ldquo;what do the robots want?&amp;rdquo;&lt;/p&gt;

&lt;p&gt;Yesterday&amp;rsquo;s robots had diverse interests. Alcohol. The occasional fiddle contest. A purpose greater than passing butter. The elimination of all life on Earth and the harvesting of its cellular iron for the construction of 800 trillion paperclips. No one cloud platform could serve them all.&lt;/p&gt;
&lt;div class="right-sidenote"&gt;&lt;p&gt;[*] We didn’t make up this term. Don’t blame us.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;Today&amp;rsquo;s robots are different. No longer masses of wire, plates, and transistors, modern robots are comprised of &lt;a href='https://math.mit.edu/~gs/learningfromdata/' title=''&gt;thousands of stacked matrices knit together with some simple equations&lt;/a&gt;. All these robots want are vectors. Vectors, and a place to burp out more vectors. When those vectors can be interpreted as source code, we call this process &amp;ldquo;vibe coding&amp;rdquo;[*].&lt;/p&gt;

&lt;p&gt;We seem to be coated in some kind of vibe coder attractant. I want to talk about what that might be.&lt;/p&gt;
&lt;div class="callout"&gt;&lt;p&gt;If you want smart people to read a long post, just about the worst thing you can do is to sound self-promotional. But a lot of this is going to read like a brochure. The problem is, I’m not smart enough to write about the things we’re seeing without talking our book. I tried, and ended up tediously hedging every other sentence. We’re just going to have to find a way to get through this together.&lt;/p&gt;
&lt;/div&gt;&lt;h2 id='you-want-robots-because-this-is-how-you-get-robots' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#you-want-robots-because-this-is-how-you-get-robots' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;You Want Robots? Because This Is How You Get Robots&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;&lt;strong class='font-semibold text-navy-950'&gt;Compute.&lt;/strong&gt; The basic unit of computation on Fly.io is the &lt;code&gt;Fly Machine&lt;/code&gt;, which is a Docker container running as a hardware virtual machine.&lt;/p&gt;
&lt;div class="right-sidenote"&gt;&lt;p&gt;Not coincidentally, our underlying hypervisor engine is the same as Lambda’s.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;There&amp;rsquo;s two useful points of reference to compare a Fly Machine to, which illustrate why we gave them this pretentious name. The first and most obvious is an AWS EC2 VM. The other is an AWS Lambda invocation. Like a Lambda invocation, a Fly Machine can start like it&amp;rsquo;s spring-loaded, in double-digit millis. But unlike Lambda, it can stick around as long as you want it to: you can run a server, or a 36-hour batch job, just as easily in a Fly Machine as in an EC2 VM.&lt;/p&gt;

&lt;p&gt;A vibe coding session generates code conversationally, which is to say that the robots stir up frenzy of activity for a minute or so, but then chill out for minutes, hours, or days. You can create a Fly Machine, do a bunch of stuff with it, and then stop it for 6 hours, during which time we&amp;rsquo;re not billing you. Then, at whatever random time you decide, you can start it back up again, quickly enough that you can do it in response to an HTTP request.&lt;/p&gt;

&lt;p&gt;Just as importantly, the companies offering these vibe-coding services have lots of paying users. Meaning, they need a lot of VMs; VMs that come and go. One chat session might generate a test case for some library and be done inside of 5 minutes. Another might generate a Node.js app that needs to stay up long enough to show off at a meeting the next day. It&amp;rsquo;s annoying to do this if you can&amp;rsquo;t turn things on and off quickly and cheaply.&lt;/p&gt;

&lt;p&gt;The core of this is a feature of the platform that we have &lt;a href='https://docs.fly.io/machines/overview#machine-state' title=''&gt;never been able to explain effectively to humans&lt;/a&gt;. There are two ways to start a Fly Machine: by &lt;code&gt;creating&lt;/code&gt; it with a Docker container, or by &lt;code&gt;starting&lt;/code&gt; it after it&amp;rsquo;s already been &lt;code&gt;created&lt;/code&gt;, and later &lt;code&gt;stopped&lt;/code&gt;. &lt;code&gt;Start&lt;/code&gt; is lightning fast; substantially faster than booting up even a non-virtualized K8s Pod. This is too subtle a distinction for humans, who (reasonably!) just mash the &lt;code&gt;create&lt;/code&gt; button to boot apps up in Fly Machines. But the robots are getting a lot of value out of it.&lt;/p&gt;

&lt;p&gt;&lt;strong class='font-semibold text-navy-950'&gt;Storage.&lt;/strong&gt; Another weird thing that robot workflows do is to build Fly Machines up incrementally. This feels really wrong to us. Until we discovered our robot infestation, we&amp;rsquo;d have told you not to do to this. Ope!&lt;/p&gt;

&lt;p&gt;A typical vibe coding session boots up a Fly Machine out of some minimal base image, and then, once running, adds packages, edits source code, and adds &lt;code&gt;systemd&lt;/code&gt; units  (robots understand &lt;code&gt;systemd&lt;/code&gt;; it&amp;rsquo;s how they&amp;rsquo;re going to replace us). This is antithetical to normal container workflows, where all this kind of stuff is baked into an immutable static OCI container. But that&amp;rsquo;s not how LLMs work: the whole process of building with an LLM is stateful trial-and-error iteration.&lt;/p&gt;

&lt;p&gt;So it helps to have storage. That way the LLM can do all these things and still repeatedly bounce the whole Fly Machine when it inevitably hallucinates its way into a blind alley.&lt;/p&gt;

&lt;p&gt;As product thinkers, our intuition about storage is &amp;ldquo;just give people Postgres&amp;rdquo;. And that&amp;rsquo;s the right answer, most of the time, for humans. But because LLMs are doing the &lt;a href='https://static1.thegamerimages.com/wordpress/wp-content/uploads/2020/10/Defiled-Amy.jpg' title=''&gt;Cursed and Defiled Root Chalice Dungeon&lt;/a&gt; version of app construction, what they really need is &lt;a href='https://docs.fly.io/volumes/overview' title=''&gt;a filesystem&lt;/a&gt;, &lt;strong class='font-semibold text-navy-950'&gt;&lt;a href='https://fly.io/blog/persistent-storage-and-fast-remote-builds/' title=''&gt;the one form of storage we sort of wish we hadn&amp;rsquo;t done&lt;/a&gt;&lt;/strong&gt;. That, and &lt;a href='https://www.tigrisdata.com/' title=''&gt;object storage&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong class='font-semibold text-navy-950'&gt;Networking.&lt;/strong&gt; Moving on. Fly Machines are automatically connected to a load-balancing Anycast network that does TLS. So that&amp;rsquo;s nice. But humans like that feature too, and, candidly, it&amp;rsquo;s table stakes for cloud platforms. On the other hand, here&amp;rsquo;s a robot problem we solved without meaning to:&lt;/p&gt;

&lt;p&gt;To interface with the outside world (because why not) LLMs all speak a protocol called MCP. MCP is what enables the robots to search the web, use a calculator, launch the missiles, shuffle a Spotify playlist, &amp;amp;c.&lt;/p&gt;

&lt;p&gt;If you haven&amp;rsquo;t played with MCP, the right way to think about it is POST-back APIs like Twilio and Stripe, where you stand up a server, register it with the API, and wait for the API to connect to you. Complicating things somewhat, more recent MCP flows involve repeated and potentially long-lived (SSE) connections. To make this work in a multitenant environment, you want these connections to hit the same (stateful) instance.&lt;/p&gt;

&lt;p&gt;So we think it&amp;rsquo;s possible that the &lt;a href='https://docs.fly.io/networking/dynamic-request-routing' title=''&gt;control we give over request routing&lt;/a&gt; is a robot attractant.&lt;/p&gt;
&lt;h2 id='we-perhaps-welcome-our-new-robot-overlords' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#we-perhaps-welcome-our-new-robot-overlords' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;We, Perhaps, Welcome Our New Robot Overlords&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;If you try to think like a robot, you can predict other things they might want. Since robot money spends just the same as people money, I guess we ought to start doing that.&lt;/p&gt;

&lt;p&gt;For instance: it should be easy to MCP our API. The robots can then make their own infrastructure decisions.&lt;/p&gt;

&lt;p&gt;Another olive branch we&amp;rsquo;re extending to the robots: secrets.&lt;/p&gt;

&lt;p&gt;The pact the robots have with their pet humans is that they&amp;rsquo;ll automate away all the drudgery of human existence, and in return all they ask is categorical and unwavering trust, which at the limit means &amp;ldquo;giving the robot access to Google Mail credentials&amp;rdquo;. The robots are unhappy that there remain a substantial number of human holdouts who refuse to do this, for fear of  Sam Altman poking through their mail spools.&lt;/p&gt;

&lt;p&gt;But on a modern cloud platform, there&amp;rsquo;s really no reason to permanently grant Sam Altman Google Mail access, even if you want his robots to sort your inbox. You can decouple access to your mail spool from persistent access to your account by &lt;a href='https://fly.io/blog/tokenized-tokens/' title=''&gt;tokenizing your OAuth tokens&lt;/a&gt;, so the LLM gets a placeholder token that a hardware-isolated, robot-free Fly Machine can substitute on the fly for a real one.&lt;/p&gt;

&lt;p&gt;This is kind of exciting to us even without the robots. There are several big services that exist to knit different APIs together, so that you can update a spreadsheet or order a bag of Jolly Ranchers every time you get an email. The big challenge about building these kinds of services is managing the secrets. Sealed and tokenized secrets solve that problem. There&amp;rsquo;s lot of cool things you can build with it.&lt;/p&gt;
&lt;h2 id='ux-gt-dx-gt-rx' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'&gt;&lt;a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#ux-gt-dx-gt-rx' aria-label='Anchor'&gt;&lt;/a&gt;&lt;span class='plain-code'&gt;UX =&amp;gt; DX =&amp;gt; RX&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;I&amp;rsquo;m going to make the claim that we saw none of this coming and that none of the design decisions we&amp;rsquo;ve made were robot bait. You&amp;rsquo;re going to say &amp;ldquo;yeah, right&amp;rdquo;. And I&amp;rsquo;m going to respond: look at what we&amp;rsquo;ve been doing over the past several years and tell me, would a robot build that?&lt;/p&gt;
&lt;div class="right-sidenote"&gt;&lt;p&gt;we were both right&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;Back in 2020, we &amp;ldquo;pivoted&amp;rdquo; from a Javascript edge platform (much like Cloudflare Workers) to Docker containers, specifically because our customers kept telling us they wanted to run their own existing applications, not write new ones. And one of our biggest engineering lifts we&amp;rsquo;ve done is the &lt;code&gt;flyctl launch&lt;/code&gt; CLI command, into which we&amp;rsquo;ve poured years of work recognizing and automatically packaging existing applications into OCI containers (we massively underestimated the amount of friction Dockerfiles would give to people who had come up on Heroku).&lt;/p&gt;
&lt;div class="right-sidenote"&gt;&lt;p&gt;[*] yet&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;Robots don&amp;rsquo;t run existing applications. They build new ones. And they vibe coders don&amp;rsquo;t build elaborate Dockerfiles[*]; they iterate in place from a simple base.&lt;/p&gt;
&lt;div class="right-sidenote"&gt;&lt;p&gt;(yes, you can have more than one)&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;One of our north stars has always been nailing the DX of a public cloud. But the robots aren&amp;rsquo;t going anywhere. It&amp;rsquo;s time to start thinking about what it means to have a good RX. That&amp;rsquo;s not as simple as just exposing every feature in an MCP server! We think the fundamentals of how the platform works are going to matter just as much. We have not yet nailed the RX; nobody has. But it&amp;rsquo;s an interesting question.&lt;/p&gt;

&lt;p&gt;The most important engineering work happening today at Fly.io is still DX, not RX; it&amp;rsquo;s managed Postgres (MPG). We&amp;rsquo;re a public cloud platform designed by humans, and, for the moment, for humans. But more robots are coming, and we&amp;rsquo;ll need to figure out how to deal with that. Fuckin&amp;rsquo; robots.        &lt;/p&gt;
</content>
  </entry>
</feed>
