Commit Graph

107 Commits

Author SHA1 Message Date
NoodleDoodleNoodleDoodleNoodleDoodleNoo e5d2680094 ** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY)
Bockchain:
1. Optim: Multi-thread long-hash computation when encountering groups of blocks.
2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible.
3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible.
4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks.
5. Optim: Multi-thread signature computation whenever possible.
6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD)
7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???).
8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads).

Berkeley-DB:
1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc).
2. Fix: Unable to pop blocks on reorganize due to transaction errors.
3. Patch: Large number of transaction aborts when running multi-threaded bulk queries.
4. Patch: Insufficient locks error when running full sync.
5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation.
6. Optim: Add bulk queries to get output global indices.
7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3)
8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key
9. Optim: Added thread-safe buffers used when multi-threading bulk queries.
10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details)
11. Mod: Added checkpoint thread and auto-remove-logs option.
12. *Now usable on 32-bit systems like RPI2.

LMDB:
1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect)
2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3)
3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key
4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details)
5. Mod: Auto resize to +1GB instead of multiplier x1.5

ETC:
1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete.
2. Fix: 32-bit saturation bug when computing next difficulty on large blocks.

[PENDING ISSUES]
1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization.
   This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD.
2. Berkeley db, possible bug "unable to allocate memory". TBD.

[NEW OPTIONS] (*Currently all enabled for testing purposes)
1. --fast-block-sync arg=[0:1] (default: 1)
	a. 0 = Compute long hash per block (may take a while depending on CPU)
	b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence)
2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000)
	a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions.
	b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache.
	Fast    - Write meta-data but defer data flush.
	Fastest - Defer meta-data and data flush.
	Sync    - Flush data after nblocks_per_sync and wait.
	Async   - Flush data after nblocks_per_sync but do not wait for the operation to finish.
3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower)
        Max number of threads to use when computing long-hash in groups.
4. --show-time-stats arg=[0:1] (default: 1)
	Show benchmark related time stats.
5. --db-auto-remove-logs arg=[0:1] (default: 1)
	For berkeley-db only. Auto remove logs if enabled.

**Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version.
	At the moment, you need a full resync to use this optimized version.

[PERFORMANCE COMPARISON]
**Some figures are approximations only.
Using a baseline machine of an i7-2600K+SSD+(with full pow computation):
1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain.
2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain.
3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain.

Averate procesing times (with full pow computation):
lmdb-optimized:
1. tx_ave = 2.5 ms / tx
2. block_ave = 5.87 ms / block
memory-official-repo:
1. tx_ave = 8.85 ms / tx
2. block_ave = 19.68 ms / block
lmdb-official-repo (0f4a036437)
1. tx_ave = 47.8 ms / tx
2. block_ave = 64.2 ms / block

**Note: The following data denotes processing times only (does not include p2p download time)
lmdb-optimized processing times (with full pow computation):
1. Desktop,  Quad-core / 8-threads 2600k  (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000).
2. Laptop,   Dual-core / 4-threads U4200  (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000).
3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000).

lmdb-optimized processing times (with per-block-checkpoint)
1. Desktop,  Quad-core / 8-threads 2600k  (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000).

berkeley-db optimized processing times (with full pow computation)
1. Desktop, Quad-core / 8-threads 2600k  (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000).
2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).

berkeley-db optimized processing times (with per-block-checkpoint)
1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-15 23:20:16 -07:00
moneromooo-monero 8623492150
Interpret x86_64 as x86-64 for architecture 2015-07-09 14:56:20 +01:00
Riccardo Spagni ce974949e2
fixed static assert test 2015-05-31 15:39:56 +02:00
Riccardo Spagni 5001489353
explicitly include libc++ with clang 2015-05-26 11:07:58 +02:00
warptangent 5c025f99a8
Check for Clang before adding compiler flag
See eb565a1ce2
2015-05-13 09:20:02 -07:00
Sergey Kazenyuk 867e98d739 Merge branch 'clang_fixes' into clang_fixes-master 2015-05-06 19:10:51 +03:00
Sergey Kazenyuk eb565a1ce2 Suppress 'register storage class is deprecated' warning in boost dependency library 2015-04-20 20:02:11 +03:00
Thomas Winget a8bc7182ea
Merge BlockchainDB into upstream 2015-04-07 17:56:18 -04:00
Thomas Winget 7292dea192
Merges PR #37
New/updated Makefile targets
2015-04-07 16:13:42 -04:00
Thomas Winget 9519526224
Only compile BerkeleyDB as an option in non-static 2015-04-07 15:02:20 -04:00
Riccardo Spagni 4222b372e4
add arm6 target and flags 2015-04-06 14:00:09 +02:00
rfree2monero c511abf005 remerged; commands JSON. logging upgrade. doxygen 2015-04-01 19:00:45 +02:00
rfree2monero 3cbdf198f1 Merge remote-tracking branch 'monero-official/master' into network-1.6-work1 2015-04-01 18:24:45 +02:00
Thomas Winget 874f48bc82
update berkeleydb branch to blockchain branch 2015-03-25 07:30:23 -04:00
Thomas Winget 5c0bc0050c
Merge upstream updates into blockchain branch 2015-03-25 05:56:36 -04:00
Thomas Winget 51e3579a80
Fixed bug in static linking boost on MINGW
There was a workaround for linking to boost at all on MINGW, but
unfortunately this workaround would not correctly link to boost
statically.  This workaround for that workaround works around the issue
that that workaround had.
2015-03-22 06:25:53 -04:00
Thomas Winget 8e3347f310
Pull blockchain changes into berkeleydb branch 2015-03-17 19:52:53 -04:00
Thomas Winget 6e9e8ab620
Move db_drivers/ to external/
Also change LMDB Cmake variables to CACHE rather than upgrading them
through several parent scopes.
2015-03-17 18:55:24 -04:00
Riccardo Spagni 56a1bf527b
fixed BUILD_64 option 2015-03-17 18:26:58 -04:00
Riccardo Spagni c97a685a37
fixed arch_width option 2015-03-17 18:26:48 -04:00
Riccardo Spagni 3a3c07c330
fixed msys2 / mingw folders based on architecture, added license to unbound CMakeList as that is not part of standard Unbound 2015-03-17 18:26:38 -04:00
Riccardo Spagni 3b3da86483
moved 32-bit/64-bit detection into main CMakeLists 2015-03-17 18:26:29 -04:00
Thomas Winget 11e815761f
Revert "Moved db_drivers/ into external/ for consistency"
This reverts commit b21335642e.
2015-03-17 18:03:11 -04:00
Thomas Winget cade0da8f1
CMake wiring, minor cleanup, minor test addition
Make Cmake things aware of BerkeleyDB and BlockchainBDB

Make the BlockchainDB unit tests aware of BlockchainBDB
2015-03-16 04:17:53 -04:00
Thomas Winget b21335642e
Moved db_drivers/ into external/ for consistency 2015-03-09 15:48:46 -04:00
rfree2monero 5ce4256e3d 2014 network limit 1.1 +utils +toc -doc -drmonero
Update of the PR with network limits

works very well for all speeds
(but remember that low download speed can stop upload
because we then slow down downloading of blockchain
requests too)

more debug options

fixed pedantic warnings in our code
should work again on Mac OS X and FreeBSD
fixed warning about size_t
tested on Debian, Ubuntu, Windows(testing now)

TCP options and ToS (QoS) flag
FIXED peer number limit
FIXED some spikes in ingress/download
FIXED problems when other up and down limit
2015-02-20 22:13:00 +01:00
rfree2monero eabb519605 2014 network limit 1.0a +utils +toc -doc -drmonero
commands and options for network limiting
works very well e.g. for 50 KiB/sec up and down
ToS (QoS) flag
peer number limit
TODO some spikes in ingress/download
TODO problems when other up and down limit
added "otshell utils" - simple logging (with colors, text files channels)
2015-02-20 22:13:00 +01:00
warptangent 84fe5fbd65
Add compile-time support for both db implementations: in-memory and LMDB
Usage:

default is lmdb for blockchain branch:
$ make release

same as:
$ DATABASE=lmdb make release

for original in-memory implementation:
$ DATABASE=memory make release
2015-02-02 11:53:09 -08:00
warptangent 79ea1f6ff0
Merge upstream
Merge branch 'monero-project/master' into blockchain
2015-01-30 16:19:22 -08:00
Riccardo Spagni 2b3b70a261
remove aes flag, detect no sse2 2015-01-27 09:05:47 +02:00
Riccardo Spagni bb85fea35a
fixed AES support flag 2015-01-27 07:45:21 +02:00
Riccardo Spagni 615079e2f4
allow for explicitly disabling AES-NI 2015-01-26 23:19:53 +02:00
Thomas Winget 34883599bc
Add in-source lmdb to build process
Working on Linux, needs verified for other platforms but should be fine.
2015-01-18 18:30:31 -05:00
moneromooo-monero 0886183568 build: add liblmdb to the cmake autodetection system
update for rebase (warptangent 2015-01-04)
  src/cryptonote_core/CMakeLists.txt (edit)
  - replace LMDB_LIBRARIES with LMDB_LIBRARY set from autodetection
2015-01-04 19:39:40 -08:00
Thomas Winget 1240cf805b
BlockchainDB unit tests, lmdb linker flag
Some BlockchainDB unit testing fleshed out (and working), rudimentary
linker flag for lmdb in CMakeLists, but should probably be done
"correctly" at some point (find it on whatever system you're building on
and all that jazz).

update for rebase (warptangent 2015-01-04)
  fix conflicts with upstream CMakeLists.txt files

  tests/CMakeLists.txt (remove edits from original commit)
2015-01-04 19:00:31 -08:00
Riccardo Spagni f4b69d553a
year updated in license 2015-01-02 18:52:46 +02:00
Riccardo Spagni affde29754
moved rapidjson to external folder, fixed CMake 2014-12-01 21:15:50 +02:00
Riccardo Spagni 4cd15cdde5
Merge pull request #1
f1eaf88 Prints seed after wallet upgrade. Removed iostream include. (Oran Juice)
70971be Doxygen comments (Oran Juice)
031ca23 Rewrites to old wallet file correctly (Oran Juice)
1f833dc Doxygen comments in (Oran Juice)
0bd88ff Writes seed language while generating wallet. Wallet open fix. (Oran Juice)
09a659e Stores seed language in wallet file. added rapidjson. Yet to test backward compatibility (Oran Juice)
2014-12-01 20:15:06 +02:00
Riccardo Spagni 685dd4a5d3
don't build tests unless we're building all-* or *-test builds 2014-12-01 20:00:22 +02:00
Ben Boeckel ce71c01f11 cmake: work around a bug with implicit link directories
Unfortunately, this is necessary because CMake doesn't detect whether
mingw libraries are static or shared and doesn't put a -static flag around
the -lfoo argument which then makes the shared library be linked to.
2014-11-18 17:04:47 -05:00
Ben Boeckel 18c56abe4b msys: don't use LTO
This causes duplicate symbol errors on msys' mingw install.
2014-11-17 18:19:54 -05:00
Ben Boeckel 5680c9cac7 msys: factor out -Werror on msys
There are Windows-related warnings turned into errors here.
2014-11-17 18:19:54 -05:00
Ben Boeckel 475154255a msys: look in msys' directory for files
This tells find_path and find_library to look under this directory first
which is what we want on msys2.
2014-11-17 18:19:53 -05:00
Ben Boeckel 7d708e4223 cmake: support 2.8.7
Older versions of CMake support LINK_{PUBLIC,PRIVATE} while newer
versions prefer PUBLIC and PRIVATE instead, but still support the LINK_
prefix.
2014-10-24 15:29:51 -04:00
Ben Boeckel 9689df925c cmake: clean up EXTRA_LIBRARIES
pthread is handled by CMAKE_THREAD_LIBS_INIT while libdl isn't needed.
2014-10-23 16:42:35 -04:00
Ben Boeckel 3b7bdcb790 cmake: set the project name 2014-10-23 16:42:35 -04:00
Ben Boeckel 0f0efc4af2 cmake: prepend to CMAKE_MODULE_PATH 2014-10-23 16:42:35 -04:00
Ben Boeckel abbd5c0bdb unbound: import cmake build system 2014-10-23 16:42:35 -04:00
Ben Boeckel 475fe209a8 cmake: minor cleanups (indentation and typos) 2014-10-23 16:42:35 -04:00
Ben Boeckel 799e8b241f cmake: use option() for STATIC 2014-10-23 16:42:35 -04:00