Hi everyone! We hope you’re reading today’s update under as many blankets as possible, ideally in front of a warm fireplace, as the cold of winter starts to set in. You may have read our last update in the third quarter – it’s time again to take a look at what’s new and what’s next.
Drew’s update
I’ve been knee deep in refactoring over the past few months, and it has been very rewarding. The pile of tech debt has been steadily shrinking! Two of the goals I mentioned in my last status update were met: first, I rolled out ariadne-codegen to all of our frontends to handle their GraphQL requests to each other (and to their own backends) and I have found that it improves the frontend code substantially. And, the one we’ve been waiting years for: the legacy REST API has finally been removed (almost) entirely 🎉
There is still one internal user of the legacy API, which is a legacy webhook subscription used by all services to receive account updates from meta.sr.ht. Completely paying back all of the tech debt incurred by supporting the legacy API is blocked on the removal of this last piece of the puzzle. Conrad and I are debating a few ways to replace it – do we add a user details cache, do we have services fetch user data from meta every time they need it, or do we upgrade it to a GraphQL webhook, or do some other thing? Ideally we like to reduce the number of sources of truth so that we don’t have to worry about them disagreeing with one another – the solution we come up with in the end is something that will have to wait until the next quarter’s update to become clear.
Speaking of the next quarter, I am planning on working on more user-facing improvements. In particular, I have plans to introduce first-class support for commit trailers into the git.sr.ht API as well as into the lists.sr.ht’s API for patchsets. Based on this work I will be updating the project hub to more tightly integrate the various components of SourceHut projects together, for instance by adding ticket references when a patch mentioning them is being discussed on the mailing list. The main motivator for working on this, however, is to add a commit trailer that lets you identify the dependencies of a patch which are not yet merged upstream, and have the CI pull those patches down when testing your work.
The other user-facing changes I mentioned in the last update, such as an improved profile page on the project hub, or FIDO/U2F support for two-factor auth, are still planned for the near term and I hope to find time for them in the coming months. Multiple email addresses may end up being on hold until we figure out what to do with the internal legacy user profile update thing.
Another large change I’ve made is to our process for handling tickets related to SourceHut itself. Each of the old bug trackers, one per service, has been closed to submissions and we’ve declared a sort of bankruptcy on all of the tickets that were there – we were not keeping up with it and many of them were duplciates, obsolete, or irrelevant. Instead we now have a single “sourcehut” bug tracker. I retrieved most of the tickets worth saving from the old bug trackers, and organized them onto this new one, which we are now attempting to better maintain so that you can get a view of our priorities. If we missed any tickets you cared about, please let us know on the sr.ht-discuss mailing list.
One last treat for you before I let Conrad take the mic: just this week I put out an overhaul for our install docs, for those of you who want to run your own SourceHut instance. I won’t pretend that SourceHut is any easier to install than it has ever been, but these docs should give experienced sysadmins a better starting point for their deployments. Enjoy!
Conrad’s update
Welcome back to the department of under-the-hood improvements! I certainly have some treats for you this time around.
As hinted at last time, we’ve made great progress with our Ceph cluster. The Alpine upgrade in the previous quarter unlocked an upgrade of Ceph itself, which I performed this quarter. We are now on Ceph 18 (“Reef”). There was another round of Alpine upgrades, too, so the path to Ceph 19 (“Squid”) is clear. Though I am not sure it will still happen this year. We are prudent with the Ceph version we are running and not in a rush…
The absolute highlight for me this quarter, however, was finally dropping
Alembic from our stack! It has been completely replaced by
sourcehut-migrate, and we are quite happy with the result. The whole
operation went smoothly, but there is one little regression: the
xxx.sr.ht-initdb scripts got dropped without proper replacements. But fear
not: I am almost done implementing an alternative solution in
sourcehut-migrate!
As is customary, I will sprinkle in some user-facing changes: we enabled GraphQL introspection on all services. This facilitates the usage of various GraphQL tools to work with our API, without having to feed them with our schemas first.
Another one that only few users will care about, but those probably all the stronger: lists.sr.ht finally supports emails in non-UTF8 encodings. I don’t think this was ever a conscious decision, it was just a few bits and pieces missing, but now it should be fixed. While we’d certainly still appreciate everyone switching to UTF8, we did not mean to lock anyone out.
Incidentally, I think the most prominent cause of encoding issues was in fact folks importing old(ish) mailing list archives from elsewhere. So while this will hopefully improve the process, there is still plenty of other reasons why an import might fail. But I have good news about that, too: lists.sr.ht will now send you an email if an import fails, providing you with actionable details.
Going back under the hood, I played a bit with golangci-lint. While we had
tons of rather uninteresting warnings, it also did uncover a few very valid
issues. So I bit the bullet and cleaned up the entire Go codebase to the point
that we now run golangci-lint in CI, so that hopefully no new issues will
creep in.
We also moved one more of our ancillary services to Kubernetes: there is now mirror.srht.network, which will soon replace mirror.sr.ht. Right now we only use it for infrastructure-related packages, but with the next Alpine upgrade we will likely switch completely, allowing us to shut down yet another VM.
Last but not least, if you’re still reading, your attention span is certainly long enough, so why not take a look at our new build worker RFC. I cannot take credit for all the ideas put forth in there, but as I sort of started driving this I’ll take the opportunity to solicit feedback from you, the community. Because the TL;DR is that we want to allow you, the community, to run your own build workers! Wouldn’t that be a nice user-facing feature for me to write about?
Everyone else
SourceHut is 100% free and open source software, and the community is invited to participate in its development. Let’s take a moment to acknowledge the work of the volunteers who use and depend on SourceHut and sent along patches to improve it over the past few months.
Simon Martin has been hard at work on dozens of bug fixes and little improvements, and also found the time to develop some nice features, too: thanks to his efforts you’ll now find a little checkbox on your lists.sr.ht dashboard which will cause the mailing list software to copy you on your own emails, if you prefer for it to. We’re not quite done reviewing his work on this next one, but soon we’ll be rolling out another feature he developed which allows you to narrow the conditions under which build jobs are submitted, such as omitting feature branches from CI in your git repos. Simon also fixed a long-standing problem with the project hub recently, de-duplicating events so that an email sent to one mailing list shared among many projects does not fill their event logs up with dozens of events. Thanks, Simon!
It’s been a relatively slow quarter for distro releases, so the workload of our volunteer build image maintainers has been pretty light. But I’d like to thank Maxell G for handling the roll-out of Fedora 44 as the new Rawhide release, and the EOL of Fedora 40, and to welcome Alex Kramer as the new maintainer of our Arch Linux images.
There were dozens of other small patches in the past couple of months, fixing little bugs or making small improvements all over SourceHut, from many different contributors. Thank you to everyone who played a part in making SourceHut better!
Thanks for reading our update – we welcome you to discuss it on sr.ht-discuss if you have comments or feedback. See you again in Q4!