Once the ledger size grows beyond the terabyte magnitude, how will Cardano address this issue? Storing and querying anything that large seems impractical to me.
1
-
1I have no experience on Cardano development, but a couple of years ago I worked on a now defunct cryptocurrency called Unit-e, and we had a proposal that could help mitigating the problem of the ever growing ledger. A sort of cryptographic snapshots: github.com/dtr-org/uips/blob/master/UIP-0011.md The key is in Elliptic Curve Multiset Hashes .– castarcoMay 26, 2021 at 10:40
Add a comment
|
3 Answers
The answer to this question is not clear yet, as more research is needed into this problem.
For now, the effort is to prevent the blockchain growing into large sizes with using layer one two solutions such as Hydra and possibly with side-chains, where you won't have just one blockchain but many, so the data might be distributed this way.
Shredding will be used too to provide a light-weight view of the blockchain, but many services such as Blockfrost will still have to run full archived nodes to access all the data on the blockchain.
answered May 26, 2021 at 8:25
-
Exactly! Most serious applications like real estate history needs to query the entire thing for a full audit. Getting only recent parts of the history via shredding would not be an option in such cases. I suppose a protocol that splits the data across many nodes might work, but you'd still need to put in some redundancy for security measures, so a level of chain duplication will be required even if it's not the full chain being copied everywhere.– Mr. DMay 26, 2021 at 11:04
Querying large data sizes doesn't have to be an issue necessarily, because you can create multiple indices for data access, which speeds up the search process by a large factor.
For example think about Google that queries essentially the complete web.
It is expected that the transaction volume grows linearly over time while storage capacity grows exponentially. So the storage problem will likely solve itself over large time frames.
Also see this question:
Are there plans to keep the blockchain/node size small enough to maintain decentralisation?
-
The blockchain isn't indexed like a database. To benefit from indices, you'd need to ETL the chain information into a queryable database, right? I'm not sure what you mean otherwise. Even then, you'd essentially be making write operations much slower by making the reads faster. And even then, querying any database with billions/trillions of rows would be insanely slow.– Mr. DMay 26, 2021 at 10:33
-
@Mr.D That's quite hard to answer. I think you can build indices and query data even if the blockchain itself has no API for that. I also think that indexing does not have to be made in sync with write operations. Theoretically, you can create a new index every day for older entries. Maybe you can even outsource indexing. But you are right that this wouldn't be an easy task. May 26, 2021 at 10:43
In the near term, the solution is to keep metadata off chain as much as possible. Proofs/Hashes of the metadata can be used to maintain a reference to the data and guarantee it hasn't changed.
This of course relies on the assumption that said metadata will be available as long as it is needed.
If you look on the IOG GitHub, they have a "testnet" repo for the metadata registry and documentation scattered about in the other repos about this. When the metadata registry format is finalised, documentation should be clearly available. Since these registries are just git repositories, it is trivial to spin one up or mirror one. Additionally, since git is already a decentralised protocol, it will fit in quite well with the existing ecosystem.
With this format it seems the goal will be to have multiple community registries and these community registries will likely have multiple independent mirrors (due to the nature of git). This federated system will allow the community to discard unused data as necessary (and said data can be kept by anyone who needs it).
Other responses have already addressed parts of the long term goal however I would recommend looking through the IOG research catalogue. I believe there are a few papers in there addressing the topic.
Additionally, like other comments have mentioned, side-chains and state channels will allow history to be compressed as much as possible.
