merge to the latest (#4)

* compact StepIdentifier into single uint64_t

* (mongo) drop block.finalizationPoint unique constraint

* rename OtsTree to BmPrivateKeyTree

* add AggregateBmPrivateKeyTree and split signing and wiping behavior

* update StepIdentifierToBmKeyIdentifier to use Epoch instead of Point

* call canSign before signing messages and support null messages in orchestrator

* add multi-tree support to AggregateBmPrivateKeyTree

* regenerate builders from catbuffer

* (test) move size + alignment tests first

* (zeromq) pack Finalized_Block_Marker message into single part

* add UserConfiguration::VotingKeysDirectory

* (asan) only wipe KeyPair from memory, not entire SignedPrivateKey

* (mongo) improve finalizedBlocks indexes

* add voter eligibility check before sending message

* TrailingVariableDataLayout should be NonCopyable

* raise nemesis finalized block notification on first boot

* (#84) fix state hash calculation during harvesting with maximize-fee option

when account is added by address and public key and then removed by public key, it should be effectively added by address
prior to fix, entire account was being improperly removed

* (#85) validate nemesis block

* fix deadlock in FinalizationContextFactory due to out of order lock acquisition

* (lint) add space between closing paren and opening brace at end of line

* add Version to FinalizationMessage and include HashesCount in signed data payload

* restructure KeyPair and PrivateKey and add SecureByteArray

* refactor basic Sign Verify tests into traits

* add VotingKeyPair, VotingPrivateKey and VotingSignature and use as appropriate

* add and use SecureByteArray::FromBufferSecure

* update computed BlockChainConfiguration properties to provide reasonable values when MaxRollbackBlocks is zero

* spool high value accounts to disk at each recalculation interval to enable deep rollbacks

* make namespace state consistent in presence of delayed prune

- do not allow renewal of inactive namespace

* add LockInfoHistory and serializers

* replace LockInfo with LockInfoHistory in caches

* bump version to 10.0.1, update changelog

* allow PacketIo to be used cross module

* fix BalanceValidator when DynamicFeeMultiplier is zero

* fix StorageImportanceCalculatorFactory to properly restore activity buckets

* specify range when requesting messages to reduce network traffic

* optimize CompareChains to do binary search

* update AggregateProofStorage to not delegate when attempt is made to save older proof

* change time-based pruning to be finalization aware

* don't punish remote if local node syncs and improves its score during compare chains

* add some additional logging

* bump version to 10.0.2, update changelog

* add CatapultCacheDelta::Disposition and only write out importance files when cache is attached

* (mongo) drop phantom message.type

* add TransactionPlugin::embeddedCount

* (mongo) store totalTransactionsCount in block metadata

* change maxTransactionsPerBlock to apply against embedded transactions in addition to top-level transactions

* set and check directory permissions

* strip RPATHs from built modules by default

* bump version to 10.0.3, update changelog

* (#90) add finalization support for catastrophic recovery

* (#82) resolve confirmed transaction addresses

* (#93) finalization and partial connections need unique service ids

* (tool) network - print node identity host

* (docs) add supported compiler versions in conan instructions

* (docs) fix format and links in README files

- Fix broken links in RUNPEERLIN.md
- Fix format, whitespace and typos:
    Rationale: by using actual markdown lists (1.) instead
    of text lists (1\.) the renderer has the chance to display
    the list items in any way it wants (indented or not).

    Also, the file is easier to read as plain text too.

* (#96) harvesting needs to count top-level as well as embedded transactions

* (#97) check node version when adding node to container

* (#73) (docs) add Voting section to RUNNETWORKLIN.md

* (#98) add NetworkIdentifier::Private and NetworkIdentifier::Private_Test

* (#55) add NodeConfiguration::ListenInterface

* (tool) add --inclusion-filter to testvectors

* (#99) allow non-voting nodes to pull finalization proofs more aggressively

* avoid pulling duplicate finalization proofs

* (#100) store server version in NodeMetadata::Version

* (lint) prefer struct initialization to struct assignment

* (test) remove tests dependent on 'localhost' resolution

* (pylint) fix warning

* bypass version check for Dynamic_Incoming nodes

* flatten three level voting tree into two levels

* use smaller Ed25519 sizes with voting

* delete resolved todo

* (mongo) log collection on bulk_write error

* update RepairSpooling (and tests) to explicitly handle 'finalization' queue

* (mongo) upsert finalized blocks and fix indexes

* make ListenInterface mandatory and test IPv6 support

* encode supported protocols in NodeMetadata

* move IpProtocol from extensions to ionet

* update PacketSocket Connect to select 'best' protocol endpoint

* use sentinel host '_local_' for local node representation and filtering

* add zeromq IPv6 support with custom listenInterface

* (#101) server can crash when processing harvesters.dat file during deep rollbacks

* (#104) size validation needs to check header size before accessing derived header fields

* (#105) prune delta cache before raising state change notifications

* (mongo) rename meta.active to meta.latest

* replace LocalFinalizedHeightSupplier with LocalFinalizedHeightHashPairSupplier

* differentiate between score and score delta by adding ChainScore::Delta

* prefer finalized block in fork resolution

* fix license link

* Add some harvesting configuration docs

* pass PrevoteDescriptorConsumer to CreateFinalizationMessageFactory to allow prevote chain saving

* FinalizationPatchingSubscriber and scaffolding

* use alignDown(h - multiple , multiple) for alignment, when preparing prevote

* add backup FilePrevoteChainStorage

* derive FilePrevoteChainStorage from PrevoteChainStorage interface

* integrate FilePrevoteChainStorage into services

* fix FinalizationMessageFactory when proof storage is ahead of block storage

* bump version add changelog

* (#108) need to differentiate between local and network finalized height hash pairs

node (especially non-voting) needs to be able to rollback network finalized *height*
when corresponding block isn't in its local chain but it should never rollback
finalized block in its local chain

* add ReadOnlyAccountStateCache::highValueAccountStatistics

* add block footers to model

* update BlockMapper to handle ImportanceBlockFooter

* update nemgen to set ImportanceBlockFooter

* add ImportanceBlockValidator

* add BlockTypeValidator

* enable new validators and fix int tests

* (config) require VotingSetGrouping to be multiple of ImportanceGrouping

* add PreviousImportanceBlockHash validation to consumers

* add Importance_Block generation to Harvester and HarvesterUtFacade

* (ubsan) fix transaction plugin tests

* (asan) fixes introduced by #103 resolution

* (build) fixes for gcc 10.2.0

* (build) fixes for clang 11.0.1

* (docs) rewrite non-conan build instructions

* (build) bump (ex boost) dependency versions

* (build) update boost to version 74

* boost => std filesystem

* replace boost::thread_group with custom ThreadGroup

* reduce memory allocation in TreeNode processing

* update copyrights

* friendlier process name

* (asan) fix violation in finalization tests by setting block type in SaveBlocks

* (tsan) various fixes

1. SpinReaderWriterLockTests: store level two threads in different container

2. suppress false positive in boost's 1.74 executor

* (#113) add meta.version to all mongo state documents

* (#114) only serialize non-empty AccountRestrictions

* replace EncodeKey with function that doesn't require allocations

* add BranchTreeNode::hasLinkedNode to reduce allocations in PatriciaTree::saveAll

* add CalculateBlockTypeFromHeight to model and refactor

* limit Node strings (Endpoint.Host, Metadata.Name) to 255 characters

* (#115) move state versions from meta to primary documents

* (#116) fix race condition between harvester and dispatcher service shutdown

* use pool allocator for openssl hash digest contexts

* (log) add finalization trace logging

* add NodeConfiguration::CacheDatabase to allow finer tuning of rocks database

* update changelog with additional 0.10.0.4 changes

* (ubsan) fix implicit conversion in EntityHasherTests

* default initialize StorageConfiguration::CacheDatabaseConfig to fix nemgen

* use checked_cast in GetPackedSize as precaution

* (build) require boost date_time component

* (docs) update BUILD-manual instructions

- thanks to @ryo-opening-line and @rg911 for identifying problems

* (docs) verified in the ubuntu environment and updated manual build procedures

* (build) require exact find_package versions

* (build) fix conan build on windows

* fix calculation of VotingEligibleAccountsCount to only include currently eligible accounts

* (#118) guarantee HighValueAccountCommitObserver is executed AFTER all other state-changing observers

* (#119) write importance files to importance/wip and add commit step

* (#120) prune newer account histories during rollback

* (log) write server version to log file

* (#124) update conan build to work after boost recipe change

* update recovery tool to handle 'importance' and 'importance/wip'

* update AccountImportanceSnapshots and AccountActivityBuckets to support pushing empty values

* update HighValueAccounts to include removedAddresses

* (breaking) insert gaps into AccountState stacks to fix deep rollback bug

* always track removed accounts with historical information

* add removedAddresses pruning

* (build) update boost to version 75

* (ubsan) MacOS vptr warning for PacketIoPicker cross module calls

* (ubsan) implicit conversion warning in MetadataValueTests

* (#135) (mongo) cap size of transactionStatuses collection

* add diagnostic counters tracking total mem used by dispatchers

* add memory limits on dispatcher queues

* (#126) fix infinite loop in CompareChains when remote returns less than configured hashes

* change PT and UT limits from count to size

* add MinDeadline filter to PT and UT requests

* (#137) fix potential deadlock when harvesting importance block

- deadlock between HarvestingUtFacade and BlockChainSyncConsumer

* (mongo) update AccountStateMapper to map all importance information or none

* (log) reduce log bloat by logging failed transactions at trace level

* (log) change WorkingBuffer log to trace

* (docs) Update build docs

- Clarify a bit manual build instructions

- Fix links to BUILD docs

- Add installation instructions

- Fix BUILD-conan formatting

* (build) conan - add `boost:without_json = True`

* (tsan) refactoring to avoid false positives in ConsumerInputTests

* (test) fix race condition in PtDispatcherServiceTests

- PtDispatcherServiceTests.Dispatcher_UpdaterForwardsToConsumerWhenValidationSucceeded
  can fail when cosignatures fill up pt cache before all transactions are added

* drop premature Remote_Chain_Too_Many_Blocks validation

* Update BUILD-conan.md docs

- Clarify Windows prerequisites
- Add libstdc++11 note for Linux
- Clarify runtime dependencies location on each OS
- Add optional installation and verification steps
- Formatting
- Add note regarding PYTHON_EXECUTABLE

* Update runtime docs

- Clarify concepts
- Update property names
- Add verification step

* only propagate valid PTs and UTs

* punish stateful transaction failures like any other transaction failures

* pull finalization proofs more aggressively when unfinalizedBlocksDuration is 0

* add configuration to ban nodes that send a lot of bad transactions

* (tsan) fix race condition in PtDispatcherServiceTests

* [cosmetic] help comment no longer needed

* Update node property docs

- Version and role properties needed clarification

* [cosmetic] change private_test network id from 0x98 -> 0xA8, group test ids together

* bump version to 10.0.5, update changelog

* [cosmetic] make shutdown log important

* (#142) remote harvesting account should not require NodeLink

* (tool) fix health tool REST querying and add --preferRestApi

* bypass MaxTransactions validation for nemesis

* (mongo) add maxDropBatchSize

* (mongo) add writeTimeout

* (recovery) allow recovery to reseed databases from blocks

* (travis) add python and shell linters

* (tool) redo address tool to support input files and output formats

* (tool) add addressgen with support for vanity address generation

* (docs) Moved code from documentation to script

* OptimizeForPointLookup accepts mb not bytes

* add MaxTimeBehindPullTransactionsStart to delay transaction pulls until a node is almost synced

* ignore pushed transactions until node is almost synced

* (asan) fix ConsumerInputTests

* (tsan) fix CanCreateWriterWithCustomOptions

* (test) make resource templates consistent with ./seed/private-test

* (#148) prevent resolution statements from being created during undo

* allow remote links in linker tool

* [cosmetic] log service id and ban reason in hex

* set install prefix in configure script

* update changelog, bump version

* (#153) make --input parameter optional in catapult.tools.addressgen

* initial version of FileDatabase

* update FileBlockStorage and FileProofStorage to use FileDatabase

* add seedDirectory => dataDirectory import to LocalNode

* (#142) check linked public key (remote) instead of main public key

* (tool) minor tool refactoring

- drop articles in options descriptions

- refactor resources option handling

* (log) throttle 'message with out of bounds round' log in FinalizationMessageProcessingService

* FileDatabase: rename exists => contains

* (#155) recovery process needs to drop orphaned documents associated with previous block

* local preparations for importer tool

* add importer tool initial version

* add generation of finalization notifications

* (tool) initial version of verify tool

* (#151) fix overflow in CalculateTransactionFee

* configurable number of parallel tasks in deps compilation

* (#158) prevent pruning of primary signing account

* (#159) open FileLock in RW instead of RO mode to fix on NFS

* bump version to 10.0.7, update changelog

* Update README to match 0.10.0.7 changes

Basically, do not manually copy things from seed to data.
Also, simplified instructions by removing intermediate directory
in seed/network-test/00000.

* (#162) fix deadlock between timesync and node selection

problem:
  timesync SelectNodes acquires CatapultCache lock followed by NodeContainer lock
  peer SelectNodes acquires NodeContainer lock followed by CatapultCache lock

solution:
 always acquire NodeContainer lock before CatapultCache lock

* (tool) addressgen - fix matching logic when no substrings match

* (sdk) add BIP32 derivation

* (sdk) add BIP39 support

* bump version to 10.0.8, update changelog

* (ubsan) implicit conversion in Bip39

* (tool) minor addressgen refactoring

* (tool) fix addresssgen when no search pattern is specified

* (test) extend test vectors to check Verify and Bip39

* (pylint) fix errors after travis update

* (#165) send full block header (including footer) over zmq

* bump version

* (docs) Note on backing up the voting key

* (#168) server should make files writeable when copying seed

* (#170) add Failure_Link_Start_Epoch_Invalid when voting link is attempted with zero StartEpoch

Co-authored-by: Jaguar0625 <[email protected]>
Co-authored-by: gimer <[email protected]>
Co-authored-by: Xavi Artigas <[email protected]>
Co-authored-by: Ryo Sasaki <[email protected]>
Co-authored-by: gimre <[email protected]>
Co-authored-by: ocp9r <[email protected]>
Co-authored-by: Marcos Mayorga <[email protected]>

Author: 8 people

.travis.yml

@@ -0,0 +1,18 @@
+language: python
+sudo: false
+
+addons:
+  apt:
+    packages:
+    - shellcheck
+
+install:
+ - pip3 install ply pycodestyle pylint pylint-quotes
+
+script:
+  - mkdir logs
+  - python3 scripts/lint/checkProjectStructure.py --text --depCheckDir src --depCheckDir extensions --depCheckDir plugins --destDir logs
+  - pushd scripts
+  - pylint --rcfile .pylintrc --load-plugins pylint_quotes --output-format=parseable ./*/*.py
+  - pycodestyle --config=.pycodestyle ./*/*.py
+  - shellcheck --format=gcc ./*/*.sh