overleaf/object-persistor

Object-persistor is a tool built by Overleaf that helps applications store and retrieve files across different cloud storage services. Instead of locking your app into a single storage provider, it lets you work with Amazon S3, Google Cloud Storage, or even a plain local filesystem. The standout feature is its fallback system: if a file isn't found in your primary storage location, the tool automatically checks a secondary one. At a high level, you configure the module by telling it which storage backend to use and providing the necessary credentials. Once set up, you can perform everyday file operations like uploading, downloading, copying, and deleting. When you upload a file, it goes only to the primary backend. When you retrieve or check for a file, the system looks in the primary first, then falls back to the secondary if needed. If you delete a file, it gets removed from both backends to keep everything in sync. This tool is designed for teams building applications that need reliable file storage without being tightly coupled to one provider. For example, a company migrating files from Google Cloud Storage to Amazon S3 could use the fallback feature to serve files from either system during the transition. It is also useful for maintaining a backup storage location, ensuring files remain accessible even if one backend experiences issues. One notable tradeoff involves the fallback mechanism and direct download links. The tool can generate signed URLs that let users download files directly from the storage provider, bypassing your server. However, this feature doesn't work with the secondary fallback system because there's no way to verify in advance which backend actually holds the file. The README also details a specific workaround for Google Cloud Storage involving a two-bucket system to handle delayed deletions, which GCS's own native rules don't fully support.