High-performance persistent storage designed for AI systems
Support for multiple storage backends including Memory, Lance, and custom implementations. Choose the best storage for your needs.
Built on Lance columnar format for high-performance random access, vector search, and zero-copy versioning.
Seamless integration with Pulsing Actor framework, providing persistence for distributed queues.
Support for dynamic schema evolution without downtime, adapting to changing requirements.
Built-in Prometheus metrics export for real-time monitoring of storage performance and health.
pip install persisting
# Or install with Pulsing
pip install persisting[pulsing]from persisting import Queue
queue = Queue("my_topic", storage_path="./data")await queue.put({"id": "1", "value": 42})
await queue.flush()
records = await queue.get(limit=100)Efficiently store and retrieve KV Cache from LLM inference, supporting cross-session reuse to reduce computation costs.
Persistent storage for RL and training trajectories, supporting offline analysis and experience replay.
Efficiently save and load model parameter checkpoints with support for incremental updates and versioning.
Seamless integration with AI ecosystem
Persisting is a persistent storage solution tailored for AI systems. Built on the Lance columnar format, it provides high-performance data read/write capabilities while deeply integrating with the Pulsing Actor framework to provide reliable persistence for distributed queues. Whether it's KV Cache, Trajectories, or model parameters, Persisting handles them efficiently.
Learn More βPersisting provides a flexible storage backend architecture. Choose Memory backend (for testing), Lance backend (for production), or implement custom backends based on your needs.
All backends follow the unified StorageBackend protocol, ensuring API consistency.
Switch backends with a single parameter change, no business logic modifications required.
from persisting import Queue
# Create queues with different storage paths
queue = Queue("my_topic", storage_path="./data")
# Write and read
await queue.put({"id": "1", "value": 42})
await queue.flush()
records = await queue.get(limit=100)Persisting is open source and community-driven. Get involved!