Hello everyone! It’s time for another quarterly update, keeping you up to date on what we’re cooking up here at SourceHut.
Drew’s update
This past quarter I found myself mostly focused on “invisible” labor for SourceHut, which will make for a boring update from me this time. Most of my time was spent preparing a grant proposal, jointly with some other open source forges and related partners, to apply for funding from the EU. We’ll learn how that went sometime next quarter!
Otherwise I’ve been focused on greasing the wheels and keeping the lights on – doing code reviews, fixing little bugs here and there, handling user support, mitigating rolling DDoS attacks (Conrad will elaborate on these in a moment), dealing with the finances (it’s tax season), and enjoying some rest after dealing with all of the above.
In the coming quarter, I plan to write our annual financial report, and to invest more time in user-visible improvements. There’s a lot of work going into our GraphQL APIs now (led by Simon Martin!) which I want to build on. With this momentum I also plan to look into anonymous API access and more standardized and uniform GraphQL API designs, such as support for the connections specification for resource enumeration.
We’ll leverage these API improvements to facilitate some long-awaited features, such as linking resource pages (e.g. git repos) back to the projects they belong to on the project hub. I also plan on doing some more work on the billing system, to finalize the migration to the EU, so if this works out all customers will be moved into the EU billing system soon enough.
Conrad’s update
While I did get good deal done this quarter, some of that work was certainly of the kind I wish I wouldn’t have to do in the first place. Let’s start with the elephant in the room: the DDoS. We still remain cautious about sharing too many details, but we wanted to at least offer a little glimpse into what we were facing. The below graph was provided by our network provider. For scale, note that the baseline traffic you can make out is not just ours - it’s us plus other customers. The visible spikes, however, were unfortunately directed at us alone…
The graph is from some time ago. A few more waves came in after that. We are still on alert and of course discussing what if any mitigations we can put in place for such events in the future.
There is a small silver lining to this. The DDoS came in several waves of different traffic patterns, but it was mostly aimed at network resource exhaustion. This “helped” us identify several places where internal network traffic (such as inter-service requests) was still routed over public (that is, saturated) interfaces. Those were all fixed and we were happy to see that afterwards those few requests that made it to our servers could successfully be handled.
Hot on the heels of the DDoS we were targeted by another huge wave of spam sign-ups. These are accounts that get created solely for link farming. They basically get an advertisement with one or more links in their bio and never get used again. This time around, there seems to have been a serious campaign going on, creating over 300 accounts in a single month. We’ve seen such campaigns before, but we were mostly able to stall them by blocking the email domains they were using, which often seemed to be from obscure, hijacked relays or such. Unfortunately, by now, the main offender for fake accounts has become: Gmail… sad trombone
So we had to resort to other means, and I added a keyword capability to our abuse detection system. All profile updates are now checked against certain keywords, and if there are a certain number of matching keywords the account is suspended right away. We will be very careful with the keywords we add to this to avoid false positives. The kind of crap we are dealing with is fortunately pretty easy to detect with 100% accuracy.
Let’s talk about more interesting stuff. My favorite this quarter is of course that I managed just right on time to finish git.sr.ht deploy keys! In the “Access” tab of your repository settings, you can now add SSH keys which will be able to access only this very repository, either read-write or read-only. This is intended for keys used for example in CI or similar automation.
This work was preceded by a clean-up of the meta.sr.ht SSH key handling, with the user-visible side effect that finally SHA256 fingerprints are used everywhere as opposed to the legacy MD5 fingerprints.
Besides the few fixes here and there I also floated a first patch to replace the builds.sr.ht shell (currently Python) with a Go implementation. It might need a few fix-ups, but it already went through an RFC phase, so I think it’s fair to mention this now and call it a day.
Everyone else
SourceHut is 100% free and open source software, and the community is invited to participate in its development. Let’s take a moment to acknowledge the work of the volunteers who use and depend on SourceHut and sent along patches to improve it over the past few months.
Simon Martin has been back at it again this quarter, writing many patches to improve the project hub. Thanks to Simon, the project hub now has a writable GraphQL API, allowing you to manage projects and project resources via the API. He has a few more patches queued up to improve the API further and reduce our Python footprint there. Simon also added some improvements for lists.sr.ht’s patch review view, associating new patchset revisions with their previous versions and adding a UI for navigating between different versions of a patch. Thanks for these and many other patches, Simon!
Other community-led improvements include CismonX’s improvemnts to PGP keys, allowing one to update an existing PGP key, for example, to bump its expiry date or update subkeys, and some other smaller improvements as well. Our volunteer build image maintainers have also been quietly keeping your build images up to date this month – CismonX was back to update FreeBSD to 14.4 and drop the EOL 13.x branch, and some small improvements for Debian were put forth by Michael Forney and Andrew Oberstar. Haowen Liu began the work to ship Ubuntu 25.10 and 26.04, which still has some growing pains for us to sort out.
Big thanks to everyone who contributed to SourceHut this quarter!