Hello everyone! We’re back with another quarterly update for you on how things are going at SourceHut. It’s been a busy few months, and we’re happy to share what we’re working on with you all.
Drew’s updates
As readers of my personal blog might have noticed when I posted my LLM rant in mid-March, my assessment in the Q1 update that the LLM scrapers were “more or less under control” was a bit optimistic. Since then, I think that we have actually gotten it under control after deploying the nuclear option (Anubis). A few weeks after this blog post, I moved us from Anubis to go-away, which is more configurable and allows us to reduce the user impact of Anubis (e.g. by offering challenges that don’t require JavaScript, or support text-mode browsers better). We have rolled this out on several services now, and unfortunately I think they’re going to remain necessary for a while yet – presumably until the bubble pops, I guess.
Thankfully, the success of Anubis and go-away in mitigating the LLM bots problem has finally allowed me to spend most of my time on our real priorities, which is why I’m getting ready to ship a huge batch of billing upgrades next week. I’ve been hard at work (1) upgrading our Stripe integration to the latest Stripe APIs, and (2) moving most of it into meta.sr.ht’s GraphQL API. You can expect to see major user-facing improvements in the billing UX from next week onwards.
It’s kind of difficult to overstate how much work has gone into the billing system upgrades. The billing system was very simple and unsophisticated prior to all of this work, and the new system is much more capable, flexible, and maintainable – but getting there is a lot of work. Most of it is not obvious to users, since we have to move slowly and carefully to avoid recklessly breaking our revenue sources (or nasty bugs like overcharging users), but this batch of changes will bring the first big group of obvious user-facing improvements. I also encourage you to play with the GraphQL API a bit when this ships next week – the new billing system is designed with an API-first mindset and so a lot of its knobs and dials are available to you to play with.
After this big patch series lands in production, I have some more internal (non-user-facing) work to do to move our payment processing into the EU, but it should be relatively soon that we’re able to add the single longest-awaited feature of the billing upgrades: support for more payment methods, like iDEAL and SEPA transfers. This effort saw its first signs of life recently – last week I collected a payment via iDEAL in my development environment!
Billing (and bot control) have been my two major focus areas this quarter, but I have done a little bit here and there, so I’ll briefly summarize those other accomplishments. A small improvement to the GraphQL APIs includes the addition of better standardized errors, so that you can distinguish error conditions on the client side – for example, readily distinguishing “not found” from “access denied” in your GraphQL client.
In more colorful news, I have installed a server in my bike shed at home. This may seem odd to include here, but I thought I’d mention it because I have used it to replicate an infrastructure environment comparable to SourceHut production, with ceph, libvirt, and k8s all making an appearance. I’ve moved many of the services I rely on privately in my personal infrastructure constellation from The Cloud (VPS providers) to the shed, such as NextCloud, Immich, and Miniflux. I’m using this setup mainly to learn more about ceph and k8s in an environment that has realistic production constraints (I actually need this stuff to work!) but which I can more confidently experiment on without breaking anything for SourceHut users. I hope to apply this experience to help Conrad with his efforts to roll out upgrades to the SourceHut fleet – which I’ll let him share with you now.
Conrad’s updates
This time around I’ll start with the rather tangible items: first, I am happy to announce that the pages.sr.ht API has gained basic support for ACLs. It’s not really well documented yet - and only works for custom domains - but it is possible to allow other users to publish your site. We are using this ourselves to collectively publish docs.sourcehut.org. Pending some better documentation, we hope that this feature may make it easier for teams using SourceHut until there is proper groups support.
Sort of descending into the infrastructure realm, but still tangible: we made the repository that hosts our Kubernetes infrastructure public. After the DDoS, we became a bit cautious about publishing details about our infrastructure. However, we think it’s time to be more transparent about our operations again. This repository is only the stuff that runs in Kubernetes, which isn’t that much yet. But it will become much more, and we will evaluate publishing other bits and pieces over time.
Speaking of Kubernetes, this is may be a good time to lay out the current plans. Much preparation work has happened, and there are currently two distinct milestones we hope to reach soonish.
One is to route all incoming sr.ht traffic through the Kubernetes ingress controller. In the background, much or even most of the traffic might still be handled by the service VMs. But it would provide much better visibility into sr.ht as a whole, and would give us easy knobs to turn when experimenting with running services in Kubernetes. To achieve this milestone we need to change all service’s DNS records to point to the ingress nodes, but that means they’ll also have to handle incoming SSH connections. Hence, the main thing missing is the unified SSH dispatch handler that is currently in review. For sr.ht, we will probably end up employing an SSH ingress solution that leverages the power of CephFS, but we’ll take care to keep things working for smaller instances (or those simply not using Ceph).
The second milestone would be running one actual service completely in Kubernetes. I was initially entertaining the idea that it could be meta, but for various reasons it will probably be something else. Regardless of the service, there are a few things missing. First, the service’s Redict instance will have to be migrated from the on-VM instance to the new network-wide sentinel-based cluster. This way, VMs and pods can share a cache, which is important for some services. Then, of course, there need to be containers. Those are conceptually straightforward, but some minor things will still need adjustment (like the currently chosen-at-random Prometheus port for observing the work queue flushing). Also, a first attempt to add Containerfiles sparked some ideas about our repository layouts, so we may take the occasion to clean those up a bit.
Most of my other work is also still in-flight. We’ve added another server to our Ceph cluster, but the big Ceph upgrade is still to come. And I am evaluating Go-based database migration frameworks, so we can throw out even more Python code (alembic).
Everyone else
I’d like to set out a spot here in our quarterly updates going forward to talk about what all of you have been doing for SourceHut! We’re a free software project after all, and we regularly accept contributions from our users.
First, I’d like to thank our tireless builds.sr.ht image maintainers – most of our builds.sr.ht images are maintained by members of their respective communities (i.e. Fedora images are maintained by Fedora users, not SourceHut). Haowen Liu, Maxwell G, Hugo Osvaldo Barrera, CismonX, and Cayetano Santos all shipped updates to, respectively, Ubuntu, Fedora, OpenBSD, FreeBSD, and Guix this quarter, with Cayetano stepping up as the new Guix maintainer – and we’re seeing off Unwox with a big thanks for all of their work on Guix support to date.
In addition to his work on builds.sr.ht, Hugo Osvaldo Barrera added a long-awaited feature allowing users to rename more resources on SourceHut and set up the appropriate redirects, such as projects on the project hub. Long time contributor наб also made an appearance in this quarter to land a bunch of small improvements, such as improving to diff rendering on git.sr.ht and lists.sr.ht, and building a more sophisticated git-over-HTTP implementation – nice to have you back, наб! Eli Schwartz also returned to revisit his favorite subject, namely making SourceHut more deterministic in support of downstream distributions that download and checksum resources from SourceHut.
I’d also like to thank Simon Martin for making their first contributions to SourceHut this quarter, landing numerous small improvements to lists.sr.ht, such as access control improvements which allow users to update the status of their own patchsets. Be sure to read Simon’s blog post about the experience!
There were many other small contributions from our community this month – thank you to everyone who helped to make SourceHut better!
Thanks for reading our update – we welcome you to discuss it on sr.ht-discuss if you have comments or feedback. See you again in Q3!