Chain Indexers with Stream Endpoints

Question

I have been taking a look at different chain indexers (or APIs to indexers) to see what's available today and how they differ - see resource list below.

Question

I still wanted to raise the question, what would be a good option for an event-based backend (not necessarily written in Haskell) that shall process on-chain events as they happen. In particular, we want to observe certain addresses and aggregate on-chain data using an CQRS / event sourcing architecture.

My current take

I wouldn't want to poll data via REST which makes Kupo and Blockfrost not an ideal candidate.

Oura looks like a great candidate but from what I see, it's not really indexing and much rather filtering on event types (like blocks, txs, assets etc.) and would still require some more processing on the backend to actually get the data I am interested in. What I like about it is that it support Node-to-Client (N2C) communication.

Scrolls as of now looks like a great candidate as it offers opinionated read-optimized views. I would imagine a Redis centered event store architecture combined with Redis Streams. The only thing I'd love support for is N2C which looks like is not yet implemented with version 0.4.2 mainly for improved performance.

Ogmios looks also very attractive as it would easily allow for a Node.js backend that could connect via websockets to directly read from the node. It's not an indexer I believe, but more an interface to the node. Hence, it requires additional filtering/ indexing in the backend similar to Oura.

Carp by dcSpark looks interesting too. It indexes and stores everything to a Postgres db, allows for N2C. Might very well be one of the strongest candidates and most likely complete in terms of development.

Lastly, Marconi looks like the best alternative to Scrolls & Carp unless I am forgetting about some other available resource. It's Haskell based and unfortunately, as of now has no event streams implemented. It will offer it in the near future.

Summary

In short, it looks like the available options are still very early in development but very promising. Nevertheless, I would appreciate feedback, things I've missed or other advice from the community!

Thanks!

Resources

• Kupo
• Oura
• Scrolls
• Marconi
• Blockfrost
• Ogmios
• Carp

Answer 1

Hi there (Kupo's maintainer here)!

Given the use case you're describing, Scrolls seems the most suited solution at present -- especially because this is exactly the direction Scrolls is moving into.

We've been discussing that on several occasions with Santiago (Scrolls' maintainer), and we try to not overlap too much between Scrolls and Kupo; both solutions emerged roughly around the same time and we're now trying to make them focus on a different set of use cases.

About the N2C in Scrolls, keep in mind that the project is open source and I am sure that you'd find either the resources to implement it or, if kindly ask, you may have maintainers or community-members consider prioritizing it.

Regarding Oura & Ogmios, you're right that they are more fundamental building blocks, not indexers, but they would allow you to build your own indexer should Kupo or Scrolls not meet your needs in the end.

Carp is a good alternative to db-sync, probably overkill for small projects.

Marconi, as far as I can tell, is Kupo with less features and less documentation. I wasn't aware that this project even existed (and... now a bit baffled to be honest).

Answer 2

Agree with all that's been said already, just important (unfortunate) gap to highlight (since we're talking about comparison), most of the solutions are not able to fetch certain offline or historical information from ledger-state, tho that is a limitation that comes from consensus/ledger teams instead - and a very non-trivial piece of work to keep up with.

So while things like historical reward history, epoch nonce, etc may not useful to all (especially in DeFi world), other solutions (for instance Carp) unfortunately do not exactly qualify as alternative/replacement to dbsync yet - tho hoping things improve on that front all around.