Nyzo techRelease notesNyzo 565: historical block manager

Nyzo 565: historical block manager

Nyzo version 565 (commit on GitHub) adds the HistoricalBlockManager to allow the client to efficiently access blocks behind the BlockManager's retention edge.

This version affects the client primarily. Some small changes affect code used by other run modes.

In BlockFileConsolidator, a condition was added to allow consolidation to occur even if the retention edge is not set. This condition ensures that files are eventually consolidated even when a Nyzo instance has frequent discontinuities in its tracking. This change affects all run modes that use the BlockFileConsolidator.

RN_565 code 0

Later in BlockFileConsolidator, a consolidated file's corresponding offset file is deleted just before the consolidated file is written. If the consolidated file is new, this has no effect. If the consolidated file is replacing another consolidated file, this will remove the offset file that contains offsets for the previous consolidated file. As the comment explains, the HistoricalBlockManager will create a new offset file if one is needed.

RN_565 code 1

HistoricalBlockManager calculates offsets to allow efficient reading of blocks directly from consolidated block files. The startManagerKey is used in the preferences file to indicate that the HistoricalBlockManager should be started. The alive field is used as a mutex for the thread.

RN_565 code 2

The HistoricalBlockManager.start() method starts a thread for generating offset files. The comment explains time considerations.

RN_565 code 3

The default setting is not starting HistoricalBlockManager. If you want to use the HistoricalBlockManager on your client installation, add the following line to /var/lib/nyzo/production/preferences.

start_historical_block_manager=1

HistoricalBlockManager.buildOffsetFile() creates an offset file for no more than one consolidated file. The first part of the method looks for a consolidated file for which an offset file does not exist. If a file is found, the second part of the method generates the offsets and writes the file.

RN_565 code 4

HistoricalBlockManager.blockOffsetsForConsolidatedFile() determines the start and end byte offsets for each block contained in a consolidated file. The result always contains 1000 offset pairs. If a block is not present in the file, its offsets are negative.

RN_565 code 5

HistoricalBlockManager.blockForHeight() provides a block on demand from the file system. It first looks to individual files, and then it looks to consolidated files for which an offset is available. No blocks are cached in memory, and no blocks are read other than the block requested.

RN_565 code 6

HistoricalBlockManager.offsetFileForHeight() provides a File object representing the location of the offset file for a specified height. This is derived from the path of the consolidated file, appending "_offsets" to the file extension.

RN_565 code 7

In Verifier, checking of the WebListener.startWebListenerKey preference was eliminated. This has been moved inside the WebListener class to follow the pattern established by BlockFileConsolidator and followed by HistoricalBlockManager.

RN_565 code 8

In Client.main(), checking of the WebListener.startWebListenerKey preference was eliminated. Also, the BlockFileConsolidator and HistoricalBlockManager are now started. Termination was moved outside the startedDataManager if/else block to allow the client to terminate properly when the command loop exits normally.

RN_565 code 9

At the end of Client.runCommandLoop(), an incorrect comment was corrected.

RN_565 code 10

In CommandEndpointWeb, the value of actionValueRun was simplified from "run command" to "run" to improve readability when used in a query string.

RN_565 code 11

To allow commands to be executed easily with pre-determined parameter values (e.g., http://client.nyzo.co/transactionSearch?timestamp=1564666847588&action=run), the CommandEndpointWeb.getResponse() method now checks if command parameter values are provided in the query string. If such values are provided, the form is processed.

RN_565 code 12

In CommandEndpointWeb.processForm(), query parameters are now used in the appropriate context.

RN_565 code 13

In TransactionSearchCommand, HistoricalBlockManager is now used as a supplemental block source. Note that HistoricalBlockManager is not wired into BlockManager, nor should it be. Over time, BlockManager should be expected to become more efficient and more focused on providing streamlined, lightweight handling of the blocks necessary for blockchain processing, and HistoricalBlockManager should be expected to provide a more comprehensive view of the full blockchain history.

RN_565 code 14

In Sentinel, the WebListener preference is no longer checked. As noted above, this preference is now checked inside the WebListener class.

RN_565 code 15

In WebListener, an unused contentType constant was removed. As noted multiple times above, this class now checks the start_web_listener preference internally. The WebListener is active by default for the Micropay and documentation-server run modes, and it is inactive by default for all other run modes.

RN_565 code 16