Performance Issues

by BlakeKeiller, Monday, October 19, 2020, 08:27 (412 days ago)
edited by BlakeKeiller, Monday, October 19, 2020, 08:32

My proofgold nodes have gotten slower and consumed more memory as time goes on. I've tried to track down the main bottlenecks. It looks like one of the main bottlenecks is computing and saving new "ctree elements". A ctree is a representation of (part of) a ledger tree containing all assets. It's theoretically a binary tree of depth 162. It's very sparse, of course, and the representation tries to avoid storing more than it needs to. A "ctree element" is a ctree of depth (at most) 9 giving a part of a full ctree, leaving deeper ctrees abbreviated by hashes.

When processing a new block (technically, the delta part of the block), the ctree given in the block is extracted (this is easy), the ctree is expanded to consist only of elements using the elements in the local database (this turns out to take a long time), the expanded ctree is transformed by the transactions in the block. Then this transformed ctree is split into elements and the elements are saved into the database. I suspect it's also the case that the expanded ctree is sometimes using a lot of RAM, causing some of the memory issues.

I'm experimenting with an alternative of saving "atoms" instead of "elements" -- where an "atom" is only one level deep instead of nine. After some debugging it turns out I still need to do some expansions to include leaves that are abbreviated by lower atoms, so it doesn't totally avoid needing to expand the ctree, but it definitely requires less expansion. It isn't clear yet if this helps.

One thing that I think could help a lot would be if some of the code (hashing? serialization? database fetches? database writes? expanding ctrees?) could be written in a lower level language that are called from ocaml via an FFI. This goes against my inclination to keep things simple, but it's clear to me that alternatives to make things faster and less memory heavy need to be considered.

Edit to add: Ctrees and ctree elements are described in Chapter 11 of the Qeditas Technical Documentation:

http://qeditas.org/docs/QeditasTechDoc.pdf

Some details have changed since then, but it's mostly still accurate.

Incidentally, if someone wants to do a low level implementation of some of the operations, the Qeditas Technical Documentation is probably essential.