Rivet's Exploration of SQLite On The Server at Hyper-Scale

Rivet, an open-source, serverless platform, explores the misunderstood potential of SQLite-on-the-server, especially in large-scale deployments. While SQLite is typically associated with small projects due to its simplicity and minimal cost, its capabilities at scale are often overlooked.

Micro-Scale Deployment:

• SQLite is favored in small applications for its low infrastructure costs, seamless development, minimal management, and proven reliability.
• Tools like LiteFS, Litestream, rqlite, Dqlite, and Bedrock enhance SQLite by providing replication and high availability.

Current High-Scale Challenges:

• Traditional solutions using sharded databases like Cassandra, ScyllaDB, and DynamoDB are common but present challenges:
  • Rigid schema requirements limit flexibility.
  • Schema changes are complex.
  • Cross-partition operations struggle with enforcing ACID properties.
  • Data inconsistency is problematic due to interrupted transactions.

SQLite at Scale:

• Cloudflare Durable Objects and Turso showcase SQLite's scalability. They provide:
  • Dynamic scaling and infinite, low-cost databases.
  • Global distribution for better performance.
  • Built-in replication and durability.

Advantages of SQLite-Per-Partition:

• Facilitates local ACID transactions and efficient I/O.
• Supports complex SQL queries without cross-partition challenges.
• Benefits from SQLite's rich ecosystem of extensions.
• Offers flexible SQL migrations and lazy schema updates.

Limitations:

• Currently lacks an open-source, self-hosted option.
• Does not support built-in cross-database querying without additional tools.
• Limited database tooling compared to other databases.
• Lacks standardized communication protocols.

Despite its current challenges, SQLite-on-the-server provides a powerful alternative for hyper-scale applications, offering flexibility and operational advantages over traditional partitioned databases. Rivet plans to release open-source tools to enhance its capabilities.