Commit Graph

100 Commits

Author SHA1 Message Date
Howard Chu 118dd69dd5 Use DUPFIXED for block_info and output_txs
Saves another ~150MB or so on the full blockchain
2016-04-05 20:55:16 +01:00
Howard Chu 6225716f3c More outputs consolidation
Also bumped DB VERSION to 1
Another significant speedup and space savings:
Get rid of global_output_indices, remove indirection from output to keys

This is the change warptangent described on irc but never got to finish.
2016-04-05 20:55:12 +01:00
Howard Chu 7c5abdc3a3 Use DUPFIXED for output_keys
Saves another 90MB on 200000 block import.
Had to bring back compare_uint64 for this, but it's safe since
this table is always 64-bit aligned.
2016-04-05 20:54:42 +01:00
Howard Chu 8e9d8e3364 Use DUPFIXED for tx_indices
Small space savings, no measurable speedup
2016-04-05 20:54:39 +01:00
Howard Chu a12f9365f8 Use DUPFIXED for block_heights
Only a small savings...
2016-04-05 20:54:07 +01:00
Howard Chu 38c2277d6f Use DUPFIXED for spent_keys 2016-04-05 20:54:07 +01:00
warptangent bf769c32ba Add back changes from revert.
m_tx_outputs doesn't need to be changed, as it's no longer dup list.
2016-04-05 20:54:07 +01:00
warptangent 46b991b362 Use MDB_APPEND mode with two tx subdbs
This is possible on those using a tx index as a key.
2016-04-05 20:54:07 +01:00
warptangent 9aadedb1d0 Schema update: tx_indices - consolidate the tx subdbs from 5 to 3 2016-04-05 20:54:06 +01:00
warptangent a2f518aa01 Schema update: tx_indices - yet less indirection 2016-04-05 20:54:06 +01:00
warptangent 8d12a8df2c Schema update: tx_indices - improve further with less indirection 2016-04-05 20:54:06 +01:00
warptangent 7c013f66e9 Add batch warning for further review 2016-04-05 20:54:06 +01:00
warptangent ae0854a431 Schema update: tx_indices 2016-04-05 20:54:06 +01:00
Howard Chu 8d252a4214 Consolidated block info 2016-04-05 20:53:59 +01:00
warptangent 132c666f67 Update schema for "tx_outputs" to use array containing amount output indices
This speeds up wallet refresh by directly retrieving a tx's amount output indices.

It removes the indirection and walking the amount output duplicate list
for every amount in each requested tx.

"tx_outputs" is used by:
Amount output indices are needed for wallet refresh.
Global output indices are needed for removing a tx.

Both amount output indices and global output indices are now stored in
an array of 64-bit unsigned ints:

tx_outputs[<tx_hash>] -> [ <a1_oi, a1_gi, a2_oi, a2_gi, ...> ]

Previously it was:
tx_outputs[<tx_hash>] -> duplicate list of <a1_gi, a2_gi, a3_gi, ...>

The amount output list had to be walked for every amount in order to
find each amount's output index, by comparing the amount's global output
index with each one in the duplicate list until a match was found.

See also d045dfa7ce
2016-04-05 20:30:50 +01:00
Riccardo Spagni 32dc08d19a
Merge pull request #700
bdec7cb BlockchainLMDB: Use DB error helper consistently (warptangent)
c5932eb BlockchainLMDB: Add DB error to exception (warptangent)
a49c355 Blockchain: Omit verbose time stats messages by default (warptangent)
2016-03-05 23:10:38 +02:00
Riccardo Spagni c3af15702f
Merge pull request #698
ee7a8b8 Get rid of lmdb_cur (Howard Chu)
2016-03-05 23:06:26 +02:00
warptangent bdec7cbfb3
BlockchainLMDB: Use DB error helper consistently 2016-03-03 21:27:13 -08:00
warptangent c5932eb5ea BlockchainLMDB: Add DB error to exception 2016-03-03 20:15:37 -08:00
Howard Chu ee7a8b87f9 Get rid of lmdb_cur
We don't need it now with per-txn cursors.
2016-03-03 19:06:27 +00:00
Howard Chu f5affbef26 More for df239428c0
Let ARMv7 work again
2016-03-03 00:13:13 +00:00
warptangent 2c823c1589
Make partial revert consistent
If user-defined comparator is used, subdb shouldn't be opened with
MDB_INTEGERKEY.

TODO: Again, this will be added back with future schema updates.
2016-03-01 02:47:16 -08:00
warptangent df239428c0
Revert part of 7db89ed2ee
For now, so existing databases work.

TODO: add these back with future schema updates.
2016-02-29 18:03:41 -08:00
warptangent c9c4060dc6
BlockchainLMDB: Add height and db error to exception 2016-02-29 17:52:54 -08:00
Howard Chu 7db89ed2ee ARMv7: fix unaligned accesses
And cleanup some key comparators
2016-02-25 13:57:00 +00:00
Howard Chu 8cc7a36f0b read txn/cursor stuff
Could wrap more later.
2016-02-23 20:47:15 +00:00
Howard Chu 02abe3590d Use MDB_PREV_MULTIPLE
in get_global_output_indices
2016-02-17 16:14:50 +00:00
Howard Chu 7a4755d3a4 Fixup after lmdb master resync 2016-02-17 16:14:38 +00:00
Howard Chu 3b13a7473e Shutup about VERSION 0 2016-02-17 05:06:19 +00:00
Howard Chu 1537477c9f Use cursor in get_output_key 2016-02-17 04:05:29 +00:00
Howard Chu f2faf8cdd9 Use MDB_APPEND mode where possible
When keys are contiguous and monotonically increasing, this gets
denser page utilization (doesn't leave padding in page splits).
Can't be used for keys that are inserted in random order (e.g. hashes)
In total this only saves around 1.5% of space compared to original
DB code. The previous patch accounted for 0.8% savings on its own;
the blocks tables just aren't that big.
2016-02-17 04:05:29 +00:00
Howard Chu 090b548c3b Use cursors in write txns
Saves a bit of seek overhead. LMDB frees them automatically
in txn_(commit|abort) so they need no cleanup.
2016-02-17 04:05:29 +00:00
Howard Chu ed08d2152e Keep a running blocksize count
Used in batch size estimation, avoids rereading already processed
blocks during import
2016-02-17 04:05:28 +00:00
Howard Chu 0fc93345ea Win32 import batchsize tweaks
Reduce frequency of resizes: bump minimum increase from 128MB to 512MB
Use a bigger safety margin at small batch sizes
2016-02-17 04:05:28 +00:00
warptangent 1995923559
BlockchainLMDB: Deal with DB exceptions at block level with particularity
Add another DB error exception type to distinguish failed txn setup from
general use of txn.

This keeps the error handling flow the same as before the block-level
txn setup changes that moved control up a layer to BlockchainDB.
2016-02-13 05:12:39 -08:00
warptangent c16cc204d3
BlockchainLMDB: Add sanity check for inconsistent state
This hasn't been known to occur in block-level txn abort, but throw
exception if it does.
2016-02-13 04:17:37 -08:00
warptangent 9118d0a44c
BlockchainLMDB: Call destructor on allocated txn if setup fails 2016-02-13 04:17:37 -08:00
warptangent f5581c3536
BlockchainLMDB: Replace remaining txn pointer NULLs with nullptr
For consistency.
2016-02-13 04:17:36 -08:00
warptangent 3800875406
Make HardFork object available to BlockchainDB and derived DB implementations
This will later allow the HardFork object's DB update functions to be
called when the DB transaction that persists across block add/remove is
open.
2016-02-08 09:28:17 -08:00
warptangent fd46c96dce
BlockchainDB/LMDB: Refactor block-scope DB txn handling for add block
Move block-scope txn start and stop from BlockchainLMDB to BlockchainDB.
2016-02-08 09:28:15 -08:00
warptangent f3a6000094
BlockchainDB/LMDB/BDB: Extract DB txn functions for block add/remove 2016-02-08 09:28:14 -08:00
warptangent f47d5b0fe3
BlockchainLMDB: Allow two HardFork functions to update DB during block add
Note that this doesn't yet cause them to be called during block add.
2016-02-08 09:28:12 -08:00
warptangent 15ee0bef4b
BlockchainLMDB: extract txn macros used during block add/remove 2016-02-08 09:28:10 -08:00
warptangent f7e337e625
fake_core: Check if hard fork subdbs need reset at start 2016-02-08 08:50:48 -08:00
warptangent c657e772c4
blockchain_import: Add --drop-hard-fork command 2016-02-08 08:50:47 -08:00
warptangent 24b66243c2
db_lmdb: Continue when tx has no outputs 2016-01-19 16:55:31 -08:00
Howard Chu 30f92f5630 Fix hf when import with verify off
Delete the hf tables, so the next open will rescan and regenerate
2016-01-15 17:26:19 +00:00
Riccardo Spagni 810a11267c
fixed copyrights with bad year references 2015-12-31 08:37:27 +02:00
Riccardo Spagni fd36eea6dd
Merge pull request #572
b39aae7 Tweak 45800a25e9 (hyc)
4a5a5ff blockchain: always stop the ioservice before returning (moneromooo-monero)
78b65cf db_lmdb: safety close db at exit (moneromooo-monero)
45800a2 db_lmdb: fix a strdup/delete[] mistmatch (moneromooo-monero)
2015-12-30 09:38:41 +02:00
hyc b39aae7aa5 Tweak 45800a25e9
trivial cleanup
2015-12-29 00:11:35 +00:00