From 4f32508bc458bed47acb8fe4f7975b2f7e0c6ebd Mon Sep 17 00:00:00 2001 From: tjayrush Date: Thu, 16 Jun 2022 06:40:19 -0400 Subject: [PATCH] Bumps version to 0.37.0-beta - Centre release --- content/api/openapi.yaml | 25 ++++++++++++++++++++----- content/docs/chifra/accounts.md | 28 ++++++++++++++++------------ content/docs/chifra/admin.md | 20 ++++++++++---------- content/docs/chifra/chaindata.md | 6 +++--- 4 files changed, 49 insertions(+), 30 deletions(-) diff --git a/content/api/openapi.yaml b/content/api/openapi.yaml index 08965cc9d..4b8e72011 100644 --- a/content/api/openapi.yaml +++ b/content/api/openapi.yaml @@ -7,7 +7,7 @@ info: license: name: GPL 3.0 url: http://www.gnu.org/licenses/ - version: 0.36.0-beta + version: 0.37.0-beta description: > A REST layer over the TrueBlocks application. With `chifra serve`, you can @@ -446,7 +446,7 @@ paths: parameters: - name: addrs description: one or more addresses (0x...) to process - required: true + required: false style: form in: query explode: true @@ -463,6 +463,22 @@ paths: explode: true schema: type: boolean + - name: watch + description: continually scan for new blocks and extract data for monitored addresses + required: false + style: form + in: query + explode: true + schema: + type: boolean + - name: sleep + description: seconds to sleep between monitor passes + required: false + style: form + in: query + explode: true + schema: + type: number - name: delete description: delete the item, but do not remove it required: false @@ -1466,9 +1482,8 @@ paths: items: type: string enum: - - indexer - - monitors - - both + - run + - stop - name: sleep description: seconds to sleep between scraper passes required: false diff --git a/content/docs/chifra/accounts.md b/content/docs/chifra/accounts.md index 79652121e..4d7620941 100644 --- a/content/docs/chifra/accounts.md +++ b/content/docs/chifra/accounts.md @@ -2,7 +2,7 @@ title: "Accounts" description: "" lead: "" -date: 2022-05-29T07:16:47 +date: 2022-06-10T21:58:15 lastmod: - :git - lastmod @@ -111,11 +111,13 @@ Notes: ## chifra monitors -A TrueBlocks monitor is simply a file on your computer that represents the transactional history of a given Ethereum address. Monitors do not exist until you indicate your interest in a certain address. (See `chifra list`.) +A TrueBlocks monitor tool has two purposes. The first is to `--watch` a set of addresses. This function is in its early stages and will be better explained elsewhere. Please see an example of what one may do with `chifra monitors --watch` [here](https://tokenomics.io/). -You may use the `--delete` command to delete (or undelete if already deleted) an address. The monitor is not removed from your computer if you delete it. It is just marked as deleted making it invisible to the TrueBlocks explorer. +A "monitor" is simply a file on a hard drive that represents the transactional history of a given Ethereum address. Monitors are very small, being only the `` pair representing each appearance of an address. Monitor files are only created when a user expresses interest in a particular address. In this way, TrueBlock is able to continue to work on small desktop or even laptop computers. (See `chifra list`.) -Use the `--remove` command to permanently remove a monitor from your computer. This is an irreversible operation. +You may use the `--delete` command to delete (or `--undelete` if already deleted) an address. The monitor is not removed from your computer if you delete it. It is just marked as being deleted making it invisible to the TrueBlocks explorer. + +Use the `--remove` command to permanently remove a monitor from your computer. This is an irreversible operation and requires the monitor to have been previously deleted. ```[plaintext] Purpose: @@ -125,16 +127,18 @@ Usage: chifra monitors [flags]
[address...] Arguments: - addrs - one or more addresses (0x...) to process (required) + addrs - one or more addresses (0x...) to process Flags: - --clean clean (i.e. remove duplicate appearances) from monitors - --delete delete a monitor, but do not remove it - --undelete undelete a previously deleted monitor - --remove remove a previously deleted monitor - -x, --fmt string export format, one of [none|json*|txt|csv|api] - -v, --verbose enable verbose (increase detail with --log_level) - -h, --help display this help screen + --clean clean (i.e. remove duplicate appearances) from monitors + --delete delete a monitor, but do not remove it + --undelete undelete a previously deleted monitor + --remove remove a previously deleted monitor + --watch continually scan for new blocks and extract data for monitored addresses + -E, --sleep float seconds to sleep between monitor passes (default 14) + -x, --fmt string export format, one of [none|json*|txt|csv|api] + -v, --verbose enable verbose (increase detail with --log_level) + -h, --help display this help screen Notes: - An address must start with '0x' and be forty-two characters long. diff --git a/content/docs/chifra/admin.md b/content/docs/chifra/admin.md index 79f5cd564..60301c69c 100644 --- a/content/docs/chifra/admin.md +++ b/content/docs/chifra/admin.md @@ -2,7 +2,7 @@ title: "Admin" description: "" lead: "" -date: 2022-06-02T07:06:13 +date: 2022-06-10T16:43:36 lastmod: - :git - lastmod @@ -55,9 +55,9 @@ Notes: The `chifra scrape` application creates TrueBlocks' index of address appearances -- the fundamental data structure of the entire system. It also, optionally, pins the index to IPFS. -`chifra scrape` is a long running process, therefore we advise you run it as a service or in terminal multiplexer such as `tmux`. It is possible to start and stop `chifra scrape` as needed, but doing so means the scraper needs to catch up to the front of the chain, a process that may take some time depending on how frequently the scraper is run. See below for a more in depth explanation of how the scraping process works and prerequisites for it proper operation. +`chifra scrape` is a long running process, therefore we advise you run it as a service or in terminal multiplexer such as `tmux`. You may start and stop `chifra scrape` as needed, but doing so means the scraper will have to catch up to the front of the chain the next time it runs, a process that may take several hours depending on how long ago it was last run. See below for a more in depth explanation of how the scraping process works and prerequisites for it proper operation. -The scraper can scrape either the index only, previously created monitors only, both, or neither. If you specify `none`, timestamps will be scraped but nothing else. If you're scraping monitors, you may tell the system to cache traces and transactions. This will speed up access, but take addition hard drive space. You may also adjust the speed of operation on different machines with the `--sleep` and `--block_cnt` options. Finally, you may choose to optionally `--pin` each new chunk to IPFS. +You may adjust the speed of the index creation with the `--sleep` and `--block_cnt` options. On some machines, or when running against some EVM node software, you may overburden the hardware. Slowing things down will ensure proper operation. Finally, you may optionally `--pin` each new chunk to IPFS which naturally shards the database among all users. ```[plaintext] Purpose: @@ -68,7 +68,7 @@ Usage: Arguments: modes - which scraper(s) to control (required) - One or more of [ indexer | monitors | both ] + One or more of [ run | stop ] Flags: -s, --sleep float seconds to sleep between scraper passes (default 14) @@ -81,19 +81,19 @@ Flags: ### explainer -Each time `chifra scrape` runs, it begins at the last block it completed (plus one) and descends as deeply as it can into the block's data. (This is why we need a `--tracing` node.) As address appearances are encountered, the system adds the appearance to a binary index. Periodically (at the end of the block containing the 2,000,000th appearance), the system consolidates a **chunk**. +Each time `chifra scrape` runs, it begins at the last block it completed processing (plus one). With each pass, it descends as deeply as it can into the block's data. (This is why the indexer requires a `--tracing` node.) As addresses appear in the blocks, the system adds the appearance to a binary index. Periodically (at the end of the block containing the 2,000,000th appearance), the system consolidates a **chunk**. -A **chunk** is a portion of the index containing approximately 2,000,000 records. As part of the consolidation, the scraper creates a Bloom filter representing the chunk. The Bloom filters are an order of magnitude or more smaller than the chunks. The system then pushes both the chunk and the Bloom filter to IPFS. In this way, TrueBlocks creates an immutable, uncapturable index of appearances that can be used not only by TrueBlocks, but any member of the community who needs it. (Hint: we all need it.) +A **chunk** is a portion of the index containing approximately 2,000,000 records (although, this number is adjustable for different chains). As part of the consolidation, the scraper creates a Bloom filter representing the set membership in the chunk. The Bloom filters are an order of magnitude (or more) smaller than the chunks. The system then pushes both the chunk and the Bloom filter to IPFS. In this way, TrueBlocks creates an immutable, uncapturable index of appearances that can be used not only by TrueBlocks, but any member of the community who needs it. (Hint: We all, every one of us, need it.) -Users of the [TrueBlocks Explorer](https://github.com/TrueBlocks/trueblocks-explorer) (or any other software, for that matter) subsequently downloads the Bloom filters, queries them to determine which **chunks** need to be downloaded to the user's machine and thereby build a historical list of transactions for a given address. This is accomplished while imposing a minimum amount of data on the end user's machine. +Users of the [TrueBlocks Explorer](https://github.com/TrueBlocks/trueblocks-explorer) (or any other software) may subsequently download the Bloom filters, query them to determine which **chunks** need to be downloaded, and thereby build a historical list of transactions for a given address. This is accomplished while imposing a minimum amount of resource requirement on the end user's machine. -In future versions of the software, we will pin these shared chunks and blooms on end user's machines. They need the data for the software to operate and sharing it makes all users better off. A naturally-born network effect. +In future versions of the software, we will pin these downloaded chunks and blooms on end user's machines. The user needs the data for the software to operate and sharing it makes all everyone better off. A naturally-occuring network effect. ### prerequisites -`chifra scrape` does not work without an RPC endpoint to an Ethereum node. The software works without an `--archive` node, but it works significantly better with one. An additional requirement for the software to work properly is an RPC that provides OpenEthereum's `trace_` routines. We've tested with OpenEthereum and Erigon. +`chifra scrape` does not work without an RPC endpoint to an Ethereum node. The software *works* without an `archive` node, but it works significantly better with one. An additional requirement is an RPC that provides OpenEthereum's `trace_` routines. We suggest strongly that you use Erigon for many reasons. -Please see [this article](.) for more information about running the scraper and building and sharing the index of appearances. +Please [see this article](https://trueblocks.io/blog/a-long-winded-explanation-of-trueblocks/) for more information about running the scraper and building and sharing the index of appearances. **Source code**: [`internal/scrape`](https://github.com/TrueBlocks/trueblocks-core/tree/master/src/apps/chifra/internal/scrape) diff --git a/content/docs/chifra/chaindata.md b/content/docs/chifra/chaindata.md index a6c5f8462..bcb3c8840 100644 --- a/content/docs/chifra/chaindata.md +++ b/content/docs/chifra/chaindata.md @@ -2,7 +2,7 @@ title: "Chain data" description: "" lead: "" -date: 2022-06-01T15:21:39 +date: 2022-06-10T08:26:35 lastmod: - :git - lastmod @@ -40,11 +40,11 @@ Arguments: Flags: -e, --hashes display only transaction hashes, default is to display full transaction detail - -U, --uncles display uncle blocks (if any) instead of the requested block + -c, --uncles display uncle blocks (if any) instead of the requested block -t, --trace export the traces from the block as opposed to the block data -s, --apps display a list of uniq address appearances in the block -u, --uniq display a list of uniq address appearances per transaction - -c, --count display the number of the lists of appearances for --addrs or --uniq + -U, --count display the number of the lists of appearances for --addrs or --uniq -o, --cache force a write of the block to the cache -x, --fmt string export format, one of [none|json*|txt|csv|api] -v, --verbose enable verbose (increase detail with --log_level)