Zurück

We have finally reached a somewhat stable level of operation. In the past hours, we managed to partially restore service, but mostly managed to do so only for a subset of our users (due to DNS propagation delays and IPv6 vs IPv4 connectivity differences).

Currentely, Codeberg is available for all the situations we monitor for, and we expect that most connectivity issues will be resolved soon after all DNS caches clear.

We appreciate all the love and support we receive from you, thank you so much.

However, the sad story is, that this day was a massive disruption for most people who develop software on a serious level on Codeberg, from large Free/libre software projects to companies and freelancers, and we are sorry about this.

We acknowledge if this makes you want to move elsewhere, but we're of course happy about everyone who can stay .

Bad news: DDoS has followed to the new location.
Good news: There, we have at least basic DDoS protection.
Bad news: The server is still unreachable.

We have received first numbers. The DDoS is apparently about 11Gbit/s over UDP traffic currently.

We were still struggling with email delivery from Forgejo. It looks like some queues are corrupted and restoring them is very hard. Most queued messages are spam or registration emails with already expired tokens. Finally, we made the decision to reset the queue and will do that in a few minutes.

We are using the opportunity to switch the queues to #redict / #redis, which was a planned project anyway (a requirement for clustering our Forgejo to multiple instances).

We have just fixed a connection issue resulting in degraded SSH access via IPv4. All services should be fully available now.

We overlooked a mistake in the firewall configuration of one host, because DNS propagation delay has resulted in our tests running against another server.