Commit Graph

415 Commits

Author SHA1 Message Date
xiphon fc99f177f3 lmdb: fix gcc 7.3.0 'implicit-fallthrough' warning 2018-11-19 22:36:10 +00:00
Martijn Otto bd98e99c80
Removed a lot of unnecessary includes 2018-11-15 17:29:34 +01:00
Riccardo Spagni 0cc3fc3756
Merge pull request #4769
5808530f blockchain: remove unused output_scan_worker parameter (moneromooo-monero)
1426209a blockchain: don't run threads if we have just one function to run (moneromooo-monero)
6f7a5fd4 db_lmdb: slight speedup getting array data from the blockchain (moneromooo-monero)
99fbe100 db_lmdb: save some string copies for readonly db keys/values (moneromooo-monero)
bf31447e tx_pool: speed up take_tx for transactions from blocks (moneromooo-monero)
4f005a77 tx_pool: remove unnecessary get_transaction_hash (moneromooo-monero)
593ef598 perf_timer: call reserve on new timer array (moneromooo-monero)
6ecc99ad core: avoid unnecessary tx/blob conversions (moneromooo-monero)
00cc1a16 unit_tests: notify test special case for the usual weirdo (moneromooo-monero)
2018-11-14 21:35:03 +02:00
moneromooo-monero dffec2587f
db_lmdb: error out if the db needs migration in read only mode 2018-11-09 09:39:22 +00:00
moneromooo-monero 6f7a5fd4f7
db_lmdb: slight speedup getting array data from the blockchain 2018-11-04 13:23:00 +00:00
moneromooo-monero 97cad7e979
db_lmdb: log which output was not found in get_output_key 2018-11-02 11:49:39 +00:00
Riccardo Spagni 9a4b15c631
Merge pull request #4664
61304151 db_lmdb: use MDB_MULTIPLE_NEXT where possible for some speedup (moneromooo-monero)
2018-11-01 22:34:20 +02:00
Riccardo Spagni 4f7cdd90fd
Merge pull request #4639
ec301390 Update db_lmdb.cpp (Cactii1)
2018-11-01 22:24:11 +02:00
moneromooo-monero 99fbe1008b
db_lmdb: save some string copies for readonly db keys/values 2018-11-01 17:26:09 +00:00
moneromooo-monero 6ecc99ad1f
core: avoid unnecessary tx/blob conversions 2018-11-01 15:51:19 +00:00
moneromooo-monero 61304151b4
db_lmdb: use MDB_MULTIPLE_NEXT where possible for some speedup 2018-10-19 16:47:29 +00:00
Cactii1 ec3013903b
Update db_lmdb.cpp 2018-10-17 21:36:10 +02:00
moneromooo-monero 4eca42b241
blockchain_db: initialize m_hardfork in ctor just in case
Coverity 136568
2018-10-16 11:58:46 +00:00
Howard Chu 7c790f11f2
Fix rtxn usage in BlockchainLMDB::get_estimated_batch_size
Should only stop the rtxn if we actually started it
Fixes Coverity 184960
2018-10-02 16:14:20 +01:00
Riccardo Spagni effcbf2060
Merge pull request #4459
bcf3f6af fuzz_tests: catch unhandled exceptions (moneromooo-monero)
3ebd05d4 miner: restore stream flags after changing them (moneromooo-monero)
a093092e levin_protocol_handler_async: do not propagate exception through dtor (moneromooo-monero)
1eebb82b net_helper: do not propagate exceptions through dtor (moneromooo-monero)
fb6a3630 miner: do not propagate exceptions through dtor (moneromooo-monero)
2e2139ff epee: do not propagate exception through dtor (moneromooo-monero)
0749a8bd db_lmdb: do not propagate exceptions in dtor (moneromooo-monero)
1b0afeeb wallet_rpc_server: exit cleanly on unhandled exceptions (moneromooo-monero)
418a9936 unit_tests: catch unhandled exceptions (moneromooo-monero)
ea7f9543 threadpool: do not propagate exceptions through the dtor (moneromooo-monero)
6e855422 gen_multisig: nice exit on unhandled exception (moneromooo-monero)
53df2deb db_lmdb: catch error in mdb_stat calls during migration (moneromooo-monero)
e67016dd blockchain_blackball: catch failure to commit db transaction (moneromooo-monero)
661439f4 mlog: don't remove old logs if we failed to rename the current file (moneromooo-monero)
5fdcda50 easylogging++: test for NULL before dereference (moneromooo-monero)
7ece1550 performance_test: fix bad last argument calling add_arg (moneromooo-monero)
a085da32 unit_tests: add check for page size > 0 before dividing (moneromooo-monero)
d8b1ec8b unit_tests: use std::shared_ptr to shut coverity up about leaks (moneromooo-monero)
02563bf4 simplewallet: top level exception catcher to print nicer messages (moneromooo-monero)
c57a65b2 blockchain_blackball: fix shift range for 32 bit archs (moneromooo-monero)
2018-09-29 22:20:38 +02:00
Riccardo Spagni 09fb2bb5f8
Merge pull request #4445
fe125647 Fixup RENAME_DB() macro (Howard Chu)
2018-09-29 22:20:12 +02:00
moneromooo-monero befdcbf4be
db_lmdb: do not use base for cumulative distribution
it's confusing and needlessly complicated
2018-09-12 19:20:13 +00:00
moneromooo-monero 5ffb2ff9b7
v8: per byte fee, pad bulletproofs, fixed 11 ring size 2018-09-11 13:38:07 +00:00
luigi1111 ab85b924c9
Merge pull request #4293
9d65399 is_hdd update (p8p)
2018-09-10 15:07:02 -05:00
luigi1111 46bcd91db4
Merge pull request #4256
d6fc870 db_lmdb: resize blockchain database when 90% filled (moneromooo-monero)
2018-09-10 15:04:50 -05:00
luigi1111 77ed11e627
Merge pull request #4270
29dea03 epee: resize vectors where possible in serialization (moneromooo-monero)
76affd9 epee: some speedup in parsing (moneromooo-monero)
dc6c069 db_lmdb: speedup the get_output_distribution common case (moneromooo-monero)
76ac5a8 wallet2: ask for a binary output distribution, for speed (moneromooo-monero)
2018-09-04 13:19:58 -05:00
p8p 9d6539923e
is_hdd update 2018-08-25 04:31:22 -07:00
luigi1111 4a4ea78ecd
Merge pull request #4204
b278b83 core: sync database based on bytes added, not blocks added (moneromooo-monero)
2018-08-22 20:43:51 -05:00
luigi1111 7e4507a048
Merge pull request #4200
43f7110 blockchain_db: remove unused get_output_key variant (moneromooo-monero)
2018-08-22 20:40:37 -05:00
moneromooo-monero dc6c0696fd
db_lmdb: speedup the get_output_distribution common case 2018-08-19 15:58:21 +00:00
luigi1111 2530dc6710
Merge pull request #4161
be02eb9 db_lmdb: demote the 'batch already enabled' log, it's harmless (moneromooo-monero)
2018-08-15 17:27:48 -05:00
moneromooo-monero d6fc870d04
db_lmdb: resize blockchain database when 90% filled
instead of a random ratio from 60% to 90%.
2018-08-14 08:55:49 +00:00
moneromooo-monero b278b83860
core: sync database based on bytes added, not blocks added
Blocks have a very wide range, whereas actual size is the relevant
quantity to consider when syncing
2018-08-12 16:13:46 +00:00
moneromooo-monero 43f71100bc
blockchain_db: remove unused get_output_key variant
It was actually incorrect, as it would not return commitment
2018-07-30 21:57:47 +00:00
luigi1111 0dddfeacc9
Merge pull request #4129
ee31383 db_lmdb: don't sync a read only DB (moneromooo-monero)
2018-07-27 14:30:57 -05:00
moneromooo-monero be02eb9389
db_lmdb: demote the "batch already enabled" log, it's harmless 2018-07-20 23:41:40 +01:00
luigi1111 3e026ff6ed
Merge pull request #4013
e5592c4 rpc: add blockchain disk size to getinfo (moneromooo-monero)
2018-07-19 13:40:42 -05:00
luigi1111 21afa71ba6
Merge pull request #3981
45e419b db: store cumulative rct output distribution in the db for speed (moneromooo-monero)
2018-07-19 13:39:24 -05:00
luigi1111 025187e6c9
Merge pull request #3854
149da42 db_lmdb: enable batch transactions by default (stoffu)
34cb6b4 add --regtest and --fixed-difficulty for regression testing (vicsn)
9e1403e update get_info RPC and bump RPC version (vicsn)
207b66e first new functional tests (vicsn)
2018-07-19 13:35:25 -05:00
moneromooo-monero 45e419bd5c
db: store cumulative rct output distribution in the db for speed
This gets rid of the temporary precalc cache.

Also make the RPC able to send data back in binary or JSON,
since there can be a lot of data

This bumps the LMDB database format to v3, with migration.
2018-07-13 11:37:04 +01:00
moneromooo-monero ee31383a52
db_lmdb: don't sync a read only DB
This would only throw
2018-07-11 18:14:22 +01:00
victorsintnicolaas 34cb6b4b70 add --regtest and --fixed-difficulty for regression testing
on_generateblocks RPC call combines functionality from the on_getblocktemplate and on_submitblock RPC calls to allow rapid block creation. Difficulty is set permanently to 1 for regtest.
Makes use of FAKECHAIN network type, but takes hard fork heights from mainchain
Default reserve_size in generate_blocks RPC call is now 1. If it is 0, the following error occurs 'Failed to calculate offset for'.
Queries hard fork heights info of other network types
2018-06-29 10:06:11 +02:00
luigi1111 cbfa0a2a7e
Merge pull request #3987
2d10830 db_ldmb: print db version when complaining about incompatibility (moneromooo-monero)
2018-06-27 16:23:17 -05:00
luigi1111 c3ec5373b3
Merge pull request #3936
d81e042 tx_pool: initialize bitflags padding since it gets written to storage (moneromooo-monero)
2018-06-25 14:53:27 -05:00
moneromooo-monero e5592c4bab
rpc: add blockchain disk size to getinfo
This should help new nodes predict how much disk space will be
needed for a full sync
2018-06-20 23:48:37 +01:00
stoffu 149da420e9 db_lmdb: enable batch transactions by default 2018-06-14 20:56:11 +02:00
moneromooo-monero 2d10830cfb
db_ldmb: print db version when complaining about incompatibility 2018-06-11 17:57:02 +01:00
moneromooo-monero 284fe6438d
db_lmdb: warn about slowness when running off a spinning disk 2018-06-08 20:59:02 +01:00
moneromooo-monero d81e042306
tx_pool: initialize bitflags padding since it gets written to storage
Avoids valgrind reporting uninitialized data usage
2018-06-05 20:26:44 +01:00
luigi1111 a7ea14dc6a
Merge pull request #3876
740da1b core: fix automatic safe db sync mode switching (moneromooo-monero)
e942d34 protocol: do not switch to unsafe sync mode for just a few blocks (moneromooo-monero)
2018-05-30 15:46:13 -05:00
moneromooo-monero 740da1ba95
core: fix automatic safe db sync mode switching 2018-05-28 12:14:04 +01:00
moneromooo-monero b9389e582e
db_lmdb: save pruned and prunable tx data separately
This bumps DB version to 2, migration code will run for v1 DBs
2018-05-23 22:48:12 +01:00
moneromooo-monero 5f8ffca34d
speed up get_output_distribution (and precalc common case) 2018-05-21 19:51:58 +01:00
moneromooo-monero ffeeefde60
speedup get_output_histogram for all amounts when min_count > 0
This skips the vast majority of "dust" output amounts with just
one instance on the chain. Clocks in at 0.15% of the original
time on testnet.
2018-04-23 10:54:28 +01:00
moneromooo-monero 41f727ce42
add RPC to get a histogram of outputs of a given amount 2018-03-16 10:32:26 +00:00
Riccardo Spagni 0e5c006207
Merge pull request #3307
223d7d0c db_lmdb: fix free space reporting (moneromooo-monero)
2018-03-05 19:11:52 +02:00
Edward Betts fbcc91c2a4 Correct spelling mistakes. 2018-03-05 17:00:40 +00:00
moneromooo-monero 223d7d0c7a
db_lmdb: fix free space reporting
reported by Brad Richards
2018-02-23 09:06:10 +00:00
moneromooo-monero 6514f0ed7b
db_lmdb: fix return code mixup in for_all_* 2018-02-19 14:55:58 +00:00
Riccardo Spagni f4a6bc79d9
Merge pull request #3226
e4646379 keccak: fix mdlen bounds sanity checking (moneromooo-monero)
2e3e90ac pass large parameters by const ref, not value (moneromooo-monero)
61defd89 blockchain: sanity check number of precomputed hash of hash blocks (moneromooo-monero)
9af6b2d1 ringct: fix infinite loop in unused h2b function (moneromooo-monero)
8cea8d0c simplewallet: double check a new multisig wallet is multisig (moneromooo-monero)
9b98a6ac threadpool: catch exceptions in dtor, to avoid terminate (moneromooo-monero)
24803ed9 blockchain_export: fix buffer overflow in exporter (moneromooo-monero)
f3f7da62 perf_timer: rewrite to make it clear there is no division by zero (moneromooo-monero)
c6ea3df0 performance_tests: remove add_arg call stray extra param (moneromooo-monero)
fa6b4566 fuzz_tests: fix an uninitialized var in setup (moneromooo-monero)
03887f11 keccak: fix sanity check bounds test (moneromooo-monero)
ad11db91 blockchain_db: initialize m_open in base class ctor (moneromooo-monero)
bece67f9 miner: restore std::cout precision after modification (moneromooo-monero)
1aabd14c db_lmdb: check hard fork info drop succeeded (moneromooo-monero)
2018-02-16 14:26:58 +01:00
Riccardo Spagni 260d666cf4
Merge pull request #3231
84a8b2da Don't create readtxn until after txn_safe gate check (Howard Chu)
2018-02-16 14:24:01 +01:00
moneromooo-monero ad11db9144
blockchain_db: initialize m_open in base class ctor
It's cleaner this way, since it's a base class field

Coverity 136568
2018-02-02 15:21:39 +00:00
moneromooo-monero 1aabd14c21
db_lmdb: check hard fork info drop succeeded
Coverity 136364
2018-02-02 15:08:22 +00:00
Howard Chu 84a8b2da14
Don't create readtxn until after txn_safe gate check 2018-01-31 18:18:01 +00:00
xmr-eric 18216f19dd Update 2018 copyright 2018-01-26 10:03:20 -05:00
Riccardo Spagni 0a110aefbb
Merge pull request #2971
ae860230 Fix exceptions not finding txpool txes when relaying (moneromooo-monero)
2018-01-10 11:45:15 +01:00
Howard Chu 294adc8341
Additional fix for core_tests
Reset thread-local info if it doesn't match the current env.
Only happens when a process opens/closes env multiple times in the
same process, doesn't affect monerod.
2017-12-28 20:24:08 +00:00
Howard Chu d52b732efb
Fix stale readcursor flags
Reset thread-specific flags when a write txn is started.
Also remove some redundant start-readtxn code.
2017-12-28 00:59:04 +00:00
moneromooo-monero ae8602303a
Fix exceptions not finding txpool txes when relaying 2017-12-20 13:02:53 +00:00
moneromooo-monero a4240d9ffc
catch const exceptions 2017-12-18 15:15:43 +00:00
moneromooo-monero d753d716a6
fix a few leaks by throwing objects, not newed pointers to objects 2017-12-18 15:15:13 +00:00
moneromooo-monero 09ce03d612
move includes around to lessen overall load 2017-12-16 22:46:38 +00:00
flozilla f9fad18608
blockchain_db: sanity check on tx/hash vector sizes
It could trip on a corrupt/crafted file if the user has disabled
input verification.
2017-11-15 10:02:23 +00:00
Riccardo Spagni 5de492be40
Merge pull request #2629
383ff4f6 remove "using namespace std" from headers (moneromooo-monero)
2017-11-14 21:29:49 +02:00
moneromooo-monero 383ff4f689
remove "using namespace std" from headers
It's nasty, and actually breaks on Solaris, where if.h fails to
build due to:

  struct map *if_memmap;
2017-11-14 16:56:10 +00:00
Riccardo Spagni 0d0ce8c21e
Merge pull request #2742
7c7d3672 Increase LMDB maxreaders if large number of threads in use (Howard Chu)
6738753b Use max_concurrency as-is (Howard Chu)
2017-11-14 15:30:03 +02:00
Riccardo Spagni fd0740e5e7
Merge pull request #2615
10013e94 Protect node privacy by proper filtering in restricted-mode RPC answers (binaryFate)
2017-11-14 14:53:10 +02:00
binaryFate 10013e9434 Protect node privacy by proper filtering in restricted-mode RPC answers
This patch allows to filter out sensitive information for queries that rely on the pool state, when running in restricted mode.
This filtering is only applied to data sent back to RPC queries. Results of inline commands typed locally in the daemon are not affected.
In practice, when running with `--restricted-rpc`:
* get_transaction_pool will list relayed transactions with the fields "last relayed time" and "received time" set to zero.
* get_transaction_pool will not list transaction that have do_not_relay set to true, and will not list key images that are used only for such transactions
* get_transaction_pool_hashes.bin will not list such transaction
* get_transaction_pool_stats will not count such transactions in any of the aggregated values that are computed

The implementation does not make filtering the default, so developers should be mindful of this if they add new RPC functionality.
Fixes #2590.
2017-11-08 13:07:07 +01:00
moneromooo-monero ccf53a566c
track double spending in the txpool
Transactions in the txpool are marked when another transaction
is seen double spending one or more of its inputs.
This is then exposed wherever appropriate.

Note that being marked with this "double spend seen" flag does
NOT mean this transaction IS a double spend and will never be
mined: it just means that the network has seen at least another
transaction spending at least one of the same inputs, so care
should be taken to wait for a few confirmations before acting
upon that transaction (ie, mostly of use for merchants wanting
to accept unconfirmed transactions).
2017-11-06 00:05:44 +00:00
Howard Chu 7c7d36725a
Increase LMDB maxreaders if large number of threads in use 2017-11-02 16:33:48 +00:00
ston1th f946bb7be6 fixed an LMDB issue on OpenBSD (#2699) which leads to a crash in monerod 2017-10-26 17:16:06 +02:00
Riccardo Spagni b293ad9a1a
Merge pull request #2640
22b51e06 db_lmdb: include chain height when failing to find an output key (moneromooo-monero)
5db433b3 blockchain: avoid exceptions in output verification (moneromooo-monero)
2017-10-15 18:59:45 +02:00
moneromooo-monero 22b51e0672
db_lmdb: include chain height when failing to find an output key
To help debugging logs.
2017-10-12 13:28:28 +01:00
moneromooo-monero bc1b93333e
frob level 1 logs a bit for consistency
Level 1 logs map to INFO, so setting log level to 1 should
show these. Demote some stuff to DEBUG to avoid spam, though.
2017-10-07 20:03:42 +01:00
Riccardo Spagni e5a2cdbfad
Merge pull request #2457
d2d8a41c Use actual batch size for resize estimates (Howard Chu)
2017-09-25 16:57:07 +02:00
moneromooo-monero 9aeae7a224
db_lmdb: fix use of uninitialized key in for_blocks_range 2017-09-19 14:11:24 +01:00
Howard Chu d2d8a41c47
Use actual batch size for resize estimates
And optimize import startup:

Remember start_height position during initial count_blocks pass
to avoid having to reread entire file again to arrive at start_height
2017-09-17 22:08:40 +01:00
Riccardo Spagni 1e57e48342
Merge pull request #2372
c867357a cryptonote_protocol: error handling on cleanup_handle_incoming_blocks (moneromooo-monero)
ce901fcb Fix blockchain_import wedge on exception in cleanup_handle_incoming_blocks (moneromooo-monero)
84fa015e core: guard against exceptions in handle_incoming_{block,tx} (moneromooo-monero)
2017-09-02 11:30:57 +02:00
moneromooo-monero ce901fcb31
Fix blockchain_import wedge on exception in cleanup_handle_incoming_blocks 2017-08-29 15:43:32 +01:00
moneromooo-monero a95e460c71
move db specific options to BlockchainDB
Avoids common depending on blockchain_db, which can cause
link errors.
2017-08-29 11:43:34 +01:00
Riccardo Spagni 36b978f50e
Merge pull request #2314
c22d22e2 Cleanup test impact of adding safesyncmode() method (Howard Chu)
9a859844 Toggle SAFE syncmode on and off automatically (Howard Chu)
2017-08-26 23:43:22 +02:00
Howard Chu 9a859844f4
Toggle SAFE syncmode on and off automatically
If monerod is started with default sync mode, set it to SAFE after
synchronization completes. Set it back to FAST if synchronization
restarts (e.g. because another peer has a longer blockchain).

If monerod is started with an explicit sync mode, none of this
automation takes effect.
2017-08-20 16:30:28 +01:00
Howard Chu 80344740bd
More DB support cleanup
Hide DB types from db_types.h - no reason to recompile dependencies
when DB types change.

Also remove lingering in-memory DB references, they've been
obsolete since 9e82b694da
2017-08-20 13:57:36 +01:00
Howard Chu 4c7f8ac04f
DB cleanup
Hide LMDB-specific stuff behind blockchain_db.h. Nobody besides blockchain_db.cpp
should ever be including DB-specific headers any more.
2017-08-19 18:11:38 +01:00
moneromooo-monero c6e200a8ab
core: add --db-salvage command line flag
Use to load the database when the primary meta page is corrupted
2017-08-12 19:04:44 +01:00
moneromooo-monero 214fd81e93
some include cleanup 2017-07-31 16:36:52 +01:00
moneromooo-monero 235df7f484
blockchain_db: add a txpool tx getter which returns existence
Avoids exception spam for the "nope, not found" case
2017-06-11 15:36:48 +01:00
Howard Chu d17c0fc2d0
Don't copy blockchain for coinbase_tx_sum
Changed Blockchain::for_all_blocks() to for_blocks_range()
Operate on blockchain in-place instead of building a copy first.
2017-06-01 14:14:24 +01:00
Riccardo Spagni a4c13ea092
Merge pull request #2039
4b932ff3 changed crypto to cncrypto so it generated libcncrypto (Gentian)
2017-05-30 21:24:11 +02:00
moneromooo-monero b52abd1370
Move txpool to the database
Integration could go further (ie, return_tx_to_pool calls should
not be needed anymore, possibly other things).

poolstate.bin is now obsolete.
2017-05-25 22:23:37 +01:00
Gentian 4b932ff314 changed crypto to cncrypto so it generated libcncrypto
fix a cmakelist
2017-05-23 07:45:40 -04:00
Howard Chu d3aaf74080
Partial fix #1876, check thread in block_txn_start()
Don't allow use of existing batch txn if it's from the wrong thread
2017-03-19 12:27:03 +00:00
kenshi84 7d07c64fe5
fix dependency: put HardFork back to cryptonote_basic, made some BlockchainDB functions virtual again to avoid missing symbols error 2017-03-10 11:22:39 +09:00
Howard Chu eb62dcc871
Remove redundant num_txs() method
Cleanup of bf1348b7e2
2017-03-03 16:52:45 +00:00
moneromooo-monero 12adb4a3f3
core: move hardfork back to cryptonote_core
should fix a cross dependency betewen cryptonote_basic and
blockchain_db
2017-02-25 16:41:35 +00:00
Riccardo Spagni c3599fa7b9
update copyright year, fix occasional lack of newline at line end 2017-02-21 19:38:18 +02:00
Riccardo Spagni 49efd3add9
Merge pull request #1727
0288310e blockchain_db: add "raw" blobdata getters for block and transaction (moneromooo-monero)
2017-02-21 11:27:15 +02:00
Riccardo Spagni 59d96eedd8
Merge pull request #1724
cca95c1c blockchain_db: do not throw on expected partial results getting keys (moneromooo-monero)
2017-02-21 11:23:03 +02:00
Howard Chu 2e9136767e
Handle map resizes from other processes 2017-02-18 22:22:24 +00:00
Howard Chu bf1348b7e2
Can't cache num_txs or num_outputs either
Same reason as 3ff54bdd7a
2017-02-18 21:01:24 +00:00
Howard Chu dc53e9eef2
Add a few read txns to streamline
Slight perf gain, but mainly to reduce spam at loglevel 3
2017-02-18 21:01:24 +00:00
moneromooo-monero 0288310e3b
blockchain_db: add "raw" blobdata getters for block and transaction
This speeds up operations such as serving blocks to syncing peers
2017-02-13 21:11:37 +00:00
moneromooo-monero cca95c1c7a
blockchain_db: do not throw on expected partial results getting keys
When scanning for outputs used in a set of incoming blocks,
we expect that some of the inputs in their transactions will
not be found in the blockchain, as they could be in previous
blocks in that set. Those outputs will be scanned there at
a later point. In this case, we add a flag to control wehther
an output not being found is expected or not.
2017-02-13 19:05:30 +00:00
moneromooo-monero f2986ccfc1
db_lmdb: fix bad height saved in tx data
The recent change to not keep separate track of the blockchain
height caused the reported height to jump early in the lmdb
transaction (when the block data is added to the blocks table),
rather than at the end, after everything succeeded. Since the
block data is added before the transaction data, this caused
the transaction data to be saved with a height one more than
its expected value.

Fix this by saving the block data last. This should have no
side effects.
2017-02-11 10:16:18 +00:00
kenshi84 8027ce0c75 extract some basic code from libcryptonote_core into libcryptonote_basic 2017-02-08 22:45:15 +09:00
Miguel Herranz 95f3e193f0 Update and use blockchain data files defines 2017-01-28 15:55:14 +01:00
Howard Chu 843769f88f
Tweak some msg loglevels 2017-01-16 20:42:59 +00:00
Howard Chu 990e08f090
Fix PR#1506, off by one in chain height 2017-01-16 10:00:06 +00:00
moneromooo-monero 5833d66f65
Change logging to easylogging++
This replaces the epee and data_loggers logging systems with
a single one, and also adds filename:line and explicit severity
levels. Categories may be defined, and logging severity set
by category (or set of categories). epee style 0-4 log level
maps to a sensible severity configuration. Log files now also
rotate when reaching 100 MB.

To select which logs to output, use the MONERO_LOGS environment
variable, with a comma separated list of categories (globs are
supported), with their requested severity level after a colon.
If a log matches more than one such setting, the last one in
the configuration string applies. A few examples:

This one is (mostly) silent, only outputting fatal errors:

MONERO_LOGS=*:FATAL

This one is very verbose:

MONERO_LOGS=*:TRACE

This one is totally silent (logwise):

MONERO_LOGS=""

This one outputs all errors and warnings, except for the
"verify" category, which prints just fatal errors (the verify
category is used for logs about incoming transactions and
blocks, and it is expected that some/many will fail to verify,
hence we don't want the spam):

MONERO_LOGS=*:WARNING,verify:FATAL

Log levels are, in decreasing order of priority:
FATAL, ERROR, WARNING, INFO, DEBUG, TRACE

Subcategories may be added using prefixes and globs. This
example will output net.p2p logs at the TRACE level, but all
other net* logs only at INFO:

MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE

Logs which are intended for the user (which Monero was using
a lot through epee, but really isn't a nice way to go things)
should use the "global" category. There are a few helper macros
for using this category, eg: MGINFO("this shows up by default")
or MGINFO_RED("this is red"), to try to keep a similar look
and feel for now.

Existing epee log macros still exist, and map to the new log
levels, but since they're used as a "user facing" UI element
as much as a logging system, they often don't map well to log
severities (ie, a log level 0 log may be an error, or may be
something we want the user to see, such as an important info).
In those cases, I tried to use the new macros. In other cases,
I left the existing macros in. When modifying logs, it is
probably best to switch to the new macros with explicit levels.

The --log-level options and set_log commands now also accept
category settings, in addition to the epee style log levels.
2017-01-16 00:25:46 +00:00
Riccardo Spagni 65e33b1bc5
Merge pull request #1506
3ff54bdd Check for correct thread before ending batch transaction (Howard Chu)
eaf8470b Must wait for previous batch to finish before starting new one (Howard Chu)
c903c554 Don't cache block height, always get from DB (Howard Chu)
eb1fb601 Tweak default db-sync-mode to fast:async:1 (Howard Chu)
0693cff9 Use batch transactions when syncing (Howard Chu)
2017-01-15 14:43:12 -05:00
Howard Chu 3ff54bdd7a
Check for correct thread before ending batch transaction 2017-01-15 15:51:57 +00:00
Howard Chu c903c5541e
Don't cache block height, always get from DB 2017-01-14 22:43:06 +00:00
Howard Chu 0693cff925
Use batch transactions when syncing
Faster throughput while avoiding corruption. I.e., makes
running with --db-sync-mode safe more tolerable.
2017-01-14 22:43:06 +00:00
Riccardo Spagni 006bb0e957
Merge pull request #1538
aff28178 Remove db-auto-remove-logs (Miguel Herranz)
1229c685 Remove berkeley from db_type initialization (Miguel Herranz)
e3090558 Show available types for db-type command (Miguel Herranz)
046ab33d Remove berkeley from blockchain_db_types (Miguel Herranz)
2017-01-13 14:33:59 -05:00
Miguel Herranz 046ab33d1b Remove berkeley from blockchain_db_types 2017-01-10 11:17:48 +01:00
moneromooo-monero 0478ac6848
blockchain: allow marking "tx not found" without an exception
This is a normal occurence in many cases, and there is no need
to spam the log with those when it is.
2017-01-07 20:52:17 +00:00
moneromooo-monero 0e18f46522
db_lmdb: add info in an error message when we can't get an output
Will be useful to debug
2016-12-01 14:31:40 +00:00
moneromooo-monero c96f9b0255
db_lmdb: guard against going out of sync on unexpected db results
m_num_outputs keeps track of the number of outputs, which should
be the same as the size of both the output_txs and output_amounts
databases. If one goes out of sync, we need to throw to abort
whatever it is we were doing.
2016-12-01 14:29:35 +00:00
moneromooo-monero bef51e677e
db_lmdb: minor pedantic tweaks
Add consts in a few places where it makes sense, avoid unnecessary
memory reallocation where we know the full size needed at the outset,
simplify and avoid memory copy.
2016-12-01 14:28:09 +00:00
moneromooo-monero 3465c4ebc7
db_lmdb: set same packing format for output_data_t and pre_rct_output_data_t
For safety, though it seems to have been the case already.

Also add a comment about the necessary layout identity.
2016-12-01 14:26:18 +00:00
moneromooo-monero 88faec75fe
wallet: select part of the fake outs from recent outputs
25% of the outputs are selected from the last 5 days (if possible),
in order to avoid the common case of sending recently received
outputs again. 25% and 5 days are subject to review later, since
it's just a wallet level change.
2016-10-15 18:17:16 +01:00
Riccardo Spagni 4cb1348548
Merge pull request #1139
01ec195 Update CMakeLists.txt (codehalo)
446ebbc Update CMakeLists.txt (codehalo)
bd773e7 Update CMakeLists.txt (codehalo)
3627cea Cleanup. Dropped "bit" from bitmonero. (Randi Joseph)
2a51396 Dropped "bit" from bitmonero. (Randi Joseph)
78b13d6 Cleanup. Dropped "bit" from bitmonero. (Randi Joseph)
1e6aedb Cleanup. Dropped "bit" from bitmonero. (Randi Joseph)
9e54616 Dropped "bit" from bitmonero. (Randi Joseph)
2016-10-04 12:11:03 +02:00
Riccardo Spagni 995dde4f95
Merge pull request #1128
d2e11f3 db_lmdb: do not error if dropping a non existent hard fork table (moneromooo-monero)
2016-09-29 22:16:15 +02:00
Randi Joseph 9e54616924 Dropped "bit" from bitmonero. 2016-09-26 17:22:30 -04:00
moneromooo-monero d2e11f374a
db_lmdb: do not error if dropping a non existent hard fork table 2016-09-24 16:15:50 +01:00
Myagui 12427dd8d3 Fix minor typo in "need resize" message.
Message observed while synchronizing a node from scratch.
"LMDB memory map needs resized"
Proposing a change to:
"LMDB memory map needs to be resized"
2016-09-21 20:40:07 +01:00
redfish e1c7af35d4 cmake: transitive deps and remove deprecated LINK_*
Keep the immediate direct deps at the library that depends on them,
declare deps as PUBLIC so that targets that link against that library
get the library's deps as transitive deps.

Break dep cycle between blockchain_db <-> crytonote_core.
No code refactoring, just hide cycle from cmake so that
it doesn't complain (cycles are allowed only between
static libs, not shared libs).

This is in preparation for supproting BUILD_SHARED_LIBS cmake
built-in option for building internal libs as shared.
2016-09-18 02:56:26 -04:00
moneromooo-monero 6cf8ca2a7f
core: faster find_blockchain_supplement
Since this queries block heights for blocks that may or may not
exist, queries for non existing blocks would throw an exception,
and that would slow down the loop a lot. 7 seconds to go through
a 30 hash list.

Fix this by adding an optional return block height to block_exists
and using this instead. Actual errors will still throw an
exception.

This also cuts down on log exception spam.
2016-08-31 10:03:32 +01:00
moneromooo-monero 1cdba42a77
db_lmdb: fix reorg with "halfway rct" coinbase outputs 2016-08-29 22:44:17 +01:00
moneromooo-monero c3b3260ae5
New "Halfway RingCT" outputs for coinbase transactions
When RingCT is enabled, outputs from coinbase transactions
are created as a single output, and stored as RingCT output,
with a fake mask. Their amount is not hidden on the blockchain
itself, but they are then able to be used as fake inputs in
a RingCT ring. Since the output amounts are hidden, their
"dustiness" is not an obstacle anymore to mixing, and this
makes the coinbase transactions a lot smaller, as well as
helping the TXO set to grow more slowly.

Also add a new "Null" type of rct signature, which decreases
the size required when no signatures are to be stored, as
in a coinbase tx.
2016-08-28 21:30:26 +01:00
moneromooo-monero 59a66e209a
move the rct commitments to the output_amounts database
Since these are needed at the same time as the output pubkeys,
this is a whole lot faster, and takes less space. Only outputs
of 0 amount store the commitment. When reading other outputs,
a fake commitment is regenerated on the fly. This avoids having
to rewrite the database to add space for fake commitments for
existing outputs.

This code relies on two things:

- LMDB must support fixed size records per key, rather than
per database (ie, all records on key 0 are the same size, all
records for non 0 keys are same size, but records from key 0
and non 0 keys do have different sizes).

- the commitment must be directly after the rest of the data
in outkey and output_data_t.
2016-08-28 21:29:02 +01:00
moneromooo-monero dc4aad7eb5
add rct to the protocol
It is not yet constrained to a fork, so don't use on the real network
or you'll be orphaned or rejected.
2016-08-28 21:28:37 +01:00
moneromooo-monero 211d1db762
db_lmdb: update reset for recent db changes
- we need to drop the new m_tx_indices database
- we reset the version to current version

This fixes the core tests failing to initialize.
2016-08-28 21:28:35 +01:00
moneromooo-monero eb56d0f994
blockchain_db: add functions for adding/removing/getting rct commitments 2016-08-28 21:28:11 +01:00
moneromooo-monero f88029e72a
db_lmdb: do not try to modify the database in read only mode 2016-08-26 19:33:25 +01:00
Riccardo Spagni 0faf572db8
Merge pull request #948
11dc091 Fake outs set is now decided by the wallet (moneromooo-monero)
1593553 new unlocked parameter to output_histogram (moneromooo-monero)
2016-08-11 22:43:14 +02:00
moneromooo-monero 240864f529
db_lmdb: errors dropping hf starting height db on reset are not fatal
This db is now dropped unconditionally, so may or may not be there
in the first place.
2016-08-10 10:50:17 +01:00
moneromooo-monero 1593553e03
new unlocked parameter to output_histogram
This constrains the number of instances of any amount
to the unlocked ones (as defined by the default unlock time
setting: outputs with non default unlock time are not
considered, so may be counted as unlocked even if they are
not actually unlocked).
2016-08-01 22:16:00 +01:00
moneromooo-monero 121165f222
db_lmdb: add some missing api call checks 2016-07-26 22:39:51 +01:00
moneromooo-monero d7b681cd65
remove hf_starting_height db
It's not really needed, it used to be an optimization for when
that code was not using the db and needed to recalculate things
fast on startup.
2016-07-13 21:38:34 +01:00
Riccardo Spagni 30dce45643 Merge branch 'performance' of https://github.com/LMDB/bitmonero 2016-04-29 21:41:43 +02:00
moneromooo-monero 513a658c87
add a --max-concurrency flag
It sets the max number of threads to use for a parallel job.
This is different that the number of total threads, since monero
binaries typically start a lot of them.
2016-04-28 20:33:59 +01:00
Howard Chu 2b0fa05f0d Another take on migration
Delete old indices and recreate them, rather than updating them
Maybe not quite as slow as before.
2016-04-24 17:46:50 +01:00
Howard Chu c14f9efd52 Migration
Migrate from DB version 0 to version 1 on startup
2016-04-08 03:11:05 +01:00
Howard Chu d7ea7d9a23 Merge branch 'performance' into master 2016-04-05 21:13:16 +01:00