Storage

Together with the database topic, this topic is surely among the most important ones. Both because storage will become the primary cost source at some point and also needs to deliver high performance, as the repository data is queried a lot.

Object Storage

CodeFloe uses Scaleway Object Storage as its S3-compatible object-storage provider. Live data and backups are kept apart:

• Live data is stored in Scaleway’s fr-par (Paris) region: Forgejo’s packages, attachments, CI artifacts and logs, the forum’s uploads, and the container registry.
• Every backup is stored in a separate Scaleway project in the nl-ams (Amsterdam) region.

Each service reaches object storage with its own credential, and each credential is limited to a single project. A leaked application credential therefore cannot delete the backups, and the backup credential cannot reach the live data.

Git repository data remains on CodeFloe’s own servers and is copied to object storage only as part of the backup system. Every backup is encrypted before it is uploaded, so Scaleway does not receive its plaintext contents: restic file backups, pgBackRest database backups, and Kubernetes volume backups alike.

Object storage keeps less latency-sensitive data off the primary repository disks and reduces storage costs without changing how users access it.

High Availability Storage

As the repository files are stored on disk, disk speed matters. Also, a HA-based architecture is required to prevent hardware failures and allow for seamless node updates. This is why we are opting for a Ceph cluster across the three nodes, which we are currently working on. Right now, the Git service is running on a single node with frequent backups.

With all the HA ideas in mind, there is still a lot of work to be done on the Forgejo side. Having a HA-storage setup and a HA-database is only half the battle. Forgejo is not yet HA-ready, i.e. custom adaptions are required to make the individual components (queue, cache, cron) work reliably in HA mode.