Hello everyone! It’s time for another quarterly update on what we’re up to at SourceHut. There’s a lot of great stuff going on since you last heard from us! Let’s get started.
Drew’s update
We finally rolled out the billing overhaul! God, that was so much work. Since the announcement in July we have successfully accepted a couple hundred payments in Euro – thank you to everyone for your support! The rollout was nearly flawless, with just a few things to sort out here and there and just few embarrassing problems to deal with (a fun one: a handful of users got invoices with a “SAMPLE” watermark).
There are still some knock-on things to deal with for billing, especially when our Q3 tax bill is due, but most of it is not user-impacting. We’ll probably roll out a couple of new payment currencies and move all users to our European entity by the end of the year (which will, unfortunately, come with an annoying price increase for some users who will start being charged VAT, you’ll get an email before that happens if you’re affected). Other than that, the billing plans are now pleasantly unremarkable.
Now that I’m free of the huge pile of billing work, I’ve been spending most of my time aggressively paying back tech debt, as promised in previous updates. The biggest change in this respect is that we’re getting serious about removing the legacy REST API in favor of GraphQL. All of the API documentation has been removed, the meta.sr.ht functionality for registering new legacy OAuth clients and personal access tokens has been removed, and I have done an analysis of contemporary legacy API usage and removed all of the API features which are not still in active use. There are about 30 users who are still relying on the legacy API – I have emailed them to inform them of the looming removal of the legacy API and offered to assist in migrating them to GraphQL. Please check your inbox if you’re one of them!
Unfortunately one of those contemporary users of the legacy API is ourselves. I have been working to reduce internal consumers of the legacy API over the past few months, upgrading bits of the project hub, the build submitter for hg.sr.ht, rewriting how man.sr.ht communicates with git.sr.ht, and so on. It’s a work in progress but I’m making good strides towards refactoring all of the remaining code – at this point we’re down to a few legacy webhooks, and those are a bit more complicated to replace.
I’ve also been looking into some more experimental changes. A big one that I’ve been researching is using ariadne-codegen to generate GraphQL clients for our Python frontends to use to talk to their own backends and the APIs of other services with which they integrate. A long term goal is to remove our frontend’s access to the database and get rid of SQLAlchemy, and having our frontends communicate with GraphQL for their work – providing a single source of truth for business logic and database access. Making this work well requires us to have a robust approach to GraphQL clients in the frontend code – and Ariadne seems to be a promising option for that. I’m getting ready to roll it out for meta.sr.ht’s frontend for a start later this week.
I’ve been up to a few other things – I’ve replaced our dependency on minio’s Go SDK to talk to radosgw with the AWS SDK instead, for all of our services that speak to S3. That includes git.sr.ht and builds.sr.ht artifacts, as well as pages.sr.ht’s basic functionality. Since we moved from minio to radosgw, it had seemed a bit odd for us to continue to depend on the minio client SDK as well. I also replaced our dependency on the now-deprecated go-yaml/yaml with goccy/go-yaml, which had some knock-on effects that were resolved with ~meroje’s help.
There’s a bunch of little things all over the place that I’ve improved in my quest to pay back our tech debt – more than I can elaborate on any further without boring you. I plan to continue my work in the coming months, but I will also be working on some new features for you. Some small, nice-to-have features that I have planned include support for multiple registered email addresses on your account, as well as webauthn (FIDO/U2F) support for 2FA. I also intend on overhauling the project hub’s profile page, to act as a sole unified page for your profile across all services – which should reduce some confusion and give you a nice page to show off your work.
Conrad’s update
For me as well, finally, some of the long-term work is coming to fruition! The first one is the site-wide rollout of sourcehut-ssh, our new unified SSH ingress. This is of course one of those under-the-hood changes, but it will make life a lot easier going forward for both admins and developers. It will also be the foundation for our planned SSH ingress handling in Kubernetes.
Another one, hinted at in the last update, is sourcehut-migrate, our anticipated Go replacement for Alembic. It has already been rolled out for pages.sr.ht (which was not actually using Alembic before), but with paste.sr.ht the first switch is in the pipeline as well.
A more user-facing highlight are some of the improvements to the GraphQL
playground. My part in that was actually quite a while ago, adding support for
variables and making some minor UI improvements. Conveniently, that prompted
Drew to add a field providing the currently composed query as a curl request,
providing a simple entrypoint for scripting. He also replaced the unannotated
schema definition on that page with an integration of docs.sourcehut.org.
We hope that all this will help people adopt and experiment with our GraphQL
API.
Our monitoring infrastructure also got some upgrades: metrics.srht.network is now on Prometheus 3 and we added a ntfy.sh sink to alertmanager. The latter is not public (for now) and the alerts will require some tuning to the new use cases that this opens up, but it definitely already improves incident response.
However, my personal highlight for this quarter - and this may sound odd at first - was performing a full Alpine Linux upgrade and subsequent reboot on one of our two Ceph cluster nodes, without any interruption whatsoever. You see, Ceph is not really designed to be operated at this small scale. But we cannot afford a whole server zoo just for storage. Ever since its inception, SourceHut has been operating with a single storage server (initially not Ceph). The anticipated redundancy has always been disk-level, not server-level (we have off-site backups for that). Not too long ago, we added a second storage server, and with this we still don’t have full server-level redundancy. But what we are able to do now is shuffle data around so that we can temporarily take down one server in a very controlled fashion. I feel like it took me a Ph.D. in Cephology to achieve this, so I am now busy writing a lot of documentation. But since this is something that every Ceph expert will (somewhat rightfully) tell you you shouldn’t be doing, I am also very happy right now that I learned the Ceph internals well enough to get it to work at all. Because offsite backups are cool, but upgrades without downtime are even cooler… ;)
Everyone else
SourceHut is 100% free and open source software, and the community is invited to participate in its development. Let’s take a moment to acknowledge the work of the volunteers who use and depend on SourceHut and sent along patches to improve it over the past few months.
First of all, as always our build image maintainers have been hard at work. Taavi Väänänen has been keeping an eye on the Debian builds, fixing some issues with recent upstream Debian changes and shipping a Trixie image. ~arusekk was kind enough to send along an Alpine 3.22 patch as well. As Drew mentioned earlier, ~meroje also sent us a small improvement for our updated YAML parser, which makes round-trips a bit prettier when your build manifests are edited by e.g. git.sr.ht prior to submission.
Thomas Chauchefoin has once again generously sent us a number of security bug reports, most of which had patches attached. Thanks Thomas! If you’re running your own SourceHut instance, make sure you’re subscribed to the sr.ht-admins mailing list to get notified about these issues.
Simon Martin has also continued his work on various SourceHut improvements – chronicled in weekly updates on his blog. Simon fixed a few small bugs and added minor features all over the place, such as showing the branch name on man.sr.ht footers to help contributors figure out where to find the code for your wikis, or fixing race conditions that prevent lists.sr.ht from threading messages properly. Notably Simon also added some heuristics for auto-detecting new patch versions and superseding the old versions automatically – very convenient! In related lists.sr.ht news, Robin Jarry also stopped by this quarter to continue some maintenance work on the lists.sr.ht-ingress daemon.
Big thanks to everyone to worked to make SourceHut better this quarter!
Thanks for reading our update – we welcome you to discuss it on sr.ht-discuss if you have comments or feedback. See you again in Q4!