** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY)

Bockchain:
1. Optim: Multi-thread long-hash computation when encountering groups of blocks.
2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible.
3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible.
4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks.
5. Optim: Multi-thread signature computation whenever possible.
6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD)
7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???).
8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads).

Berkeley-DB:
1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc).
2. Fix: Unable to pop blocks on reorganize due to transaction errors.
3. Patch: Large number of transaction aborts when running multi-threaded bulk queries.
4. Patch: Insufficient locks error when running full sync.
5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation.
6. Optim: Add bulk queries to get output global indices.
7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3)
8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key
9. Optim: Added thread-safe buffers used when multi-threading bulk queries.
10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details)
11. Mod: Added checkpoint thread and auto-remove-logs option.
12. *Now usable on 32-bit systems like RPI2.

LMDB:
1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect)
2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3)
3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key
4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details)
5. Mod: Auto resize to +1GB instead of multiplier x1.5

ETC:
1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete.
2. Fix: 32-bit saturation bug when computing next difficulty on large blocks.

[PENDING ISSUES]
1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization.
   This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD.
2. Berkeley db, possible bug "unable to allocate memory". TBD.

[NEW OPTIONS] (*Currently all enabled for testing purposes)
1. --fast-block-sync arg=[0:1] (default: 1)
	a. 0 = Compute long hash per block (may take a while depending on CPU)
	b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence)
2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000)
	a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions.
	b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache.
	Fast    - Write meta-data but defer data flush.
	Fastest - Defer meta-data and data flush.
	Sync    - Flush data after nblocks_per_sync and wait.
	Async   - Flush data after nblocks_per_sync but do not wait for the operation to finish.
3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower)
        Max number of threads to use when computing long-hash in groups.
4. --show-time-stats arg=[0:1] (default: 1)
	Show benchmark related time stats.
5. --db-auto-remove-logs arg=[0:1] (default: 1)
	For berkeley-db only. Auto remove logs if enabled.

**Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version.
	At the moment, you need a full resync to use this optimized version.

[PERFORMANCE COMPARISON]
**Some figures are approximations only.
Using a baseline machine of an i7-2600K+SSD+(with full pow computation):
1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain.
2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain.
3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain.

Averate procesing times (with full pow computation):
lmdb-optimized:
1. tx_ave = 2.5 ms / tx
2. block_ave = 5.87 ms / block
memory-official-repo:
1. tx_ave = 8.85 ms / tx
2. block_ave = 19.68 ms / block
lmdb-official-repo (0f4a036437)
1. tx_ave = 47.8 ms / tx
2. block_ave = 64.2 ms / block

**Note: The following data denotes processing times only (does not include p2p download time)
lmdb-optimized processing times (with full pow computation):
1. Desktop,  Quad-core / 8-threads 2600k  (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000).
2. Laptop,   Dual-core / 4-threads U4200  (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000).
3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000).

lmdb-optimized processing times (with per-block-checkpoint)
1. Desktop,  Quad-core / 8-threads 2600k  (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000).

berkeley-db optimized processing times (with full pow computation)
1. Desktop, Quad-core / 8-threads 2600k  (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000).
2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).

berkeley-db optimized processing times (with per-block-checkpoint)
1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
This commit is contained in:
NoodleDoodleNoodleDoodleNoodleDoodleNoo 2015-07-10 13:09:32 -07:00
parent 1f83444d3d
commit e5d2680094
33 changed files with 4052 additions and 2373 deletions

View File

@ -45,6 +45,35 @@ function (die msg)
message(FATAL_ERROR "${BoldRed}${msg}${ColourReset}")
endfunction ()
if (NOT ${ARCH} STREQUAL "")
string(SUBSTRING ${ARCH} 0 5 ARM_TEST)
string(TOLOWER ${ARM_TEST} ARM_TEST)
if (${ARM_TEST} STREQUAL "armv6")
set(ARM6 1)
else()
set(ARM6 0)
endif()
if (${ARM_TEST} STREQUAL "armv7")
set(ARM7 1)
else()
set(ARM7 0)
endif()
endif()
if(WIN32 OR ARM7 OR ARM6)
set(CMAKE_C_FLAGS_RELEASE "-O2 -DNDEBUG")
set(CMAKE_CXX_FLAGS_RELEASE "-O2 -DNDEBUG")
endif()
# set this to 0 if per-block checkpoint needs to be disabled
set(PER_BLOCK_CHECKPOINT 1)
if(PER_BLOCK_CHECKPOINT)
add_definitions("-DPER_BLOCK_CHECKPOINT")
endif()
list(INSERT CMAKE_MODULE_PATH 0
"${CMAKE_SOURCE_DIR}/cmake")
@ -156,14 +185,42 @@ if (DEFINED ENV{DATABASE})
else()
message(STATUS "Could not find DATABASE in env (not required unless you want to change database type from default: ${DATABASE})")
endif()
set(BERKELEY_DB 0)
if (DATABASE STREQUAL "lmdb")
set(BLOCKCHAIN_DB DB_LMDB)
# temporarily allow mingw to compile with berkeley_db,
# regardless if building static or not
if(NOT STATIC OR MINGW)
find_package(BerkeleyDB)
if(NOT BERKELEY_DB_LIBRARIES)
message(STATUS "BerkeleyDB not found and has been disabled.")
else()
message(STATUS "Found BerkeleyDB include (db.h) in ${BERKELEY_DB_INCLUDE_DIR}")
if(BERKELEY_DB_LIBRARIES)
message(STATUS "Found BerkeleyDB shared library")
set(BDB_STATIC false CACHE BOOL "BDB Static flag")
set(BDB_INCLUDE ${BERKELEY_DB_INCLUDE_DIR} CACHE STRING "BDB include path")
set(BDB_LIBRARY ${BERKELEY_DB_LIBRARIES} CACHE STRING "BDB library name")
set(BDB_LIBRARY_DIRS "" CACHE STRING "BDB Library dirs")
set(BERKELEY_DB 1)
else()
message(STATUS "Found BerkeleyDB includes, but could not find BerkeleyDB library. Please make sure you have installed libdb and libdb-dev or the equivalent")
endif()
endif()
endif()
elseif (DATABASE STREQUAL "memory")
set(BLOCKCHAIN_DB DB_MEMORY)
else()
die("Invalid database type: ${DATABASE}")
endif()
if(BERKELEY_DB)
add_definitions("-DBERKELEY_DB")
endif()
add_definitions("-DBLOCKCHAIN_DB=${BLOCKCHAIN_DB}")
if (UNIX AND NOT APPLE)
@ -192,7 +249,7 @@ include_directories(external/rapidjson)
include_directories(${LMDB_INCLUDE})
# Final setup for Berkeley DB
if (NOT STATIC)
if (BERKELEY_DB)
include_directories(${BDB_INCLUDE})
endif()
@ -208,7 +265,8 @@ if(MSVC)
include_directories(SYSTEM src/platform/msc)
else()
set(ARCH native CACHE STRING "CPU to build for: -march value or default")
if(ARCH STREQUAL "default")
# -march=armv7-a conflicts with -mcpu=cortex-a7
if(ARCH STREQUAL "default" OR ARM7)
set(ARCH_FLAG "")
else()
if(ARCH STREQUAL "x86_64")
@ -258,14 +316,18 @@ else()
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++11 -D_GNU_SOURCE ${MINGW_FLAG} ${WARNINGS} ${CXX_WARNINGS} ${ARCH_FLAG} -maes")
endif()
string(SUBSTRING ${ARCH} 0 3 ARM_TEST)
string(TOLOWER ${ARM_TEST} ARM_TEST)
if(${ARM_TEST} STREQUAL "arm")
message(STATUS "Setting ARM C and C++ flags")
if(ARM6)
message(STATUS "Setting ARM6 C and C++ flags")
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -mfpu=vfp -mfloat-abi=hard")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -mfpu=vfp -mfloat-abi=hard")
endif()
if(ARM7)
message(STATUS "Setting ARM7 C and C++ flags")
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -O2 -mcpu=cortex-a7 -mfloat-abi=hard -mfpu=vfpv4 -funsafe-math-optimizations -mtune=cortex-a7")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -O2 -mcpu=cortex-a7 -mfloat-abi=hard -mfpu=vfpv4 -funsafe-math-optimizations -mtune=cortex-a7")
endif()
if(APPLE)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -DGTEST_HAS_TR1_TUPLE=0")
endif()

View File

@ -58,6 +58,14 @@ release-all:
mkdir -p build/release
cd build/release && cmake -D BUILD_TESTS=ON -D CMAKE_BUILD_TYPE=release ../.. && $(MAKE)
release-arm6:
mkdir -p build/release
cd build/release && cmake -D BUILD_TESTS=OFF -D ARCH="armv6zk" -D BUILD_64=OFF -D NO_AES=ON -D CMAKE_BUILD_TYPE=release ../.. && $(MAKE)
release-arm7:
mkdir -p build/release
cd build/release && cmake -D BUILD_TESTS=OFF -D ARCH="armv7-a" -D BUILD_64=OFF -D NO_AES=ON -D CMAKE_BUILD_TYPE=release ../.. && $(MAKE)
release-static: release-static-64
release-static-64:
@ -68,10 +76,6 @@ release-static-32:
mkdir -p build/release
cd build/release && cmake -D STATIC=ON -D ARCH="i686" -D BUILD_64=OFF -D CMAKE_BUILD_TYPE=release ../.. && $(MAKE)
release-static-arm6:
mkdir -p build/release
cd build/release && cmake -D STATIC=ON -D ARCH="armv6zk" -D BUILD_64=OFF -D NO_AES=ON -D CMAKE_BUILD_TYPE=release ../.. && $(MAKE)
release-static-win64:
mkdir -p build/release
cd build/release && cmake -G "MSYS Makefiles" -D STATIC=ON -D ARCH="x86-64" -D BUILD_64=ON -D CMAKE_BUILD_TYPE=Release -D CMAKE_TOOLCHAIN_FILE=../../cmake/64-bit-toolchain.cmake -D MSYS2_FOLDER=c:/msys64 ../.. && $(MAKE)

View File

@ -31,24 +31,4 @@
message(STATUS "Using ${ARCH_WIDTH}-bit LMDB from source tree")
add_subdirectory(liblmdb${ARCH_WIDTH})
set(LMDB_INCLUDE "${CMAKE_CURRENT_SOURCE_DIR}/liblmdb${ARCH_WIDTH}" CACHE STRING "LMDB Include path")
set(LMDB_LIBRARY "lmdb" CACHE STRING "LMDB Library name")
if (NOT STATIC)
find_package(BerkeleyDB)
if(NOT BERKELEY_DB_LIBRARIES)
die("BerkeleyDB not found. At this time it should be installed in your system for a non-static build.")
else()
message(STATUS "Found BerkeleyDB include (db.h) in ${BERKELEY_DB_INCLUDE_DIR}")
if(BERKELEY_DB_LIBRARIES)
message(STATUS "Found BerkeleyDB shared library")
set(BDB_STATIC false CACHE BOOL "BDB Static flag")
set(BDB_INCLUDE ${BERKELEY_DB_INCLUDE_DIR} CACHE STRING "BDB include path")
set(BDB_LIBRARY ${BERKELEY_DB_LIBRARIES} CACHE STRING "BDB library name")
set(BDB_LIBRARY_DIRS "" CACHE STRING "BDB Library dirs")
else()
die("Found BerkeleyDB includes, but could not find BerkeleyDB library. Please make sure you have installed libdb and libdb-dev or the equivalent")
endif()
endif()
endif()

View File

@ -103,3 +103,7 @@ add_subdirectory(daemonizer)
add_subdirectory(daemon)
add_subdirectory(blockchain_utilities)
if(PER_BLOCK_CHECKPOINT)
add_subdirectory(blocks)
endif()

View File

@ -31,7 +31,7 @@ set(blockchain_db_sources
lmdb/db_lmdb.cpp
)
if (NOT STATIC)
if (BERKELEY_DB)
set(blockchain_db_sources
${blockchain_db_sources}
berkeleydb/db_bdb.cpp
@ -46,7 +46,7 @@ set(blockchain_db_private_headers
lmdb/db_lmdb.h
)
if (NOT STATIC)
if (BERKELEY_DB)
set(blockchain_db_private_headers
${blockchain_db_private_headers}
berkeleydb/db_bdb.h

File diff suppressed because it is too large Load Diff

View File

@ -30,6 +30,11 @@
#include "blockchain_db/blockchain_db.h"
#include "cryptonote_protocol/blobdatatype.h" // for type blobdata
#include <unordered_map>
// ND: Enables multi-threaded bulk reads for when getting indices.
// TODO: Disabled for now, as it doesn't seem to provide noticeable improvements (??. Reason: TBD.
// #define BDB_BULK_CAN_THREAD
namespace cryptonote
{
@ -83,10 +88,145 @@ struct bdb_txn_safe
{
return &m_txn;
}
private:
DbTxn* m_txn;
};
// ND: Class to handle buffer management when doing bulk queries
// (DB_MULTIPLE). Allocates buffers then handles thread queuing
// so a fixed set of buffers can be used (instead of allocating
// every time a bulk query is needed).
template <typename T>
class bdb_safe_buffer
{
// limit the number of buffers to 8
const size_t MaxAllowedBuffers = 8;
public:
bdb_safe_buffer(size_t num_buffers, size_t count)
{
if(num_buffers > MaxAllowedBuffers)
num_buffers = MaxAllowedBuffers;
set_count(num_buffers);
for (size_t i = 0; i < num_buffers; i++)
m_buffers.push_back((T) malloc(sizeof(T) * count));
m_buffer_count = count;
}
~bdb_safe_buffer()
{
for (size_t i = 0; i < m_buffers.size(); i++)
{
if (m_buffers[i])
{
free(m_buffers[i]);
m_buffers[i] = nullptr;
}
}
m_buffers.resize(0);
}
T acquire_buffer()
{
std::unique_lock<std::mutex> lock(m_lock);
m_cv.wait(lock, [&]{ return m_count > 0; });
--m_count;
size_t index = -1;
for (size_t i = 0; i < m_open_slot.size(); i++)
{
if (m_open_slot[i])
{
m_open_slot[i] = false;
index = i;
break;
}
}
assert(index >= 0);
T buffer = m_buffers[index];
m_buffer_map.emplace(buffer, index);
return buffer;
}
void release_buffer(T buffer)
{
std::unique_lock<std::mutex> lock(m_lock);
assert(buffer != nullptr);
auto it = m_buffer_map.find(buffer);
if (it != m_buffer_map.end())
{
auto index = it->second;
assert(index < m_open_slot.size());
assert(m_open_slot[index] == false);
assert(m_count < m_open_slot.size());
++m_count;
m_open_slot[index] = true;
m_buffer_map.erase(it);
m_cv.notify_one();
}
}
size_t get_buffer_size() const
{
return m_buffer_count * sizeof(T);
}
size_t get_buffer_count() const
{
return m_buffer_count;
}
typedef T type;
private:
void set_count(size_t count)
{
assert(count > 0);
m_open_slot.resize(count, true);
m_count = count;
}
std::vector<T> m_buffers;
std::unordered_map<T, size_t> m_buffer_map;
std::condition_variable m_cv;
std::vector<bool> m_open_slot;
size_t m_count;
std::mutex m_lock;
size_t m_buffer_count;
};
template <typename T>
class bdb_safe_buffer_autolock
{
public:
bdb_safe_buffer_autolock(T &safe_buffer, typename T::type &buffer) :
m_safe_buffer(safe_buffer), m_buffer(nullptr)
{
m_buffer = m_safe_buffer.acquire_buffer();
buffer = m_buffer;
}
~bdb_safe_buffer_autolock()
{
if (m_buffer != nullptr)
{
m_safe_buffer.release_buffer(m_buffer);
m_buffer = nullptr;
}
}
private:
T &m_safe_buffer;
typename T::type m_buffer;
};
class BlockchainBDB : public BlockchainDB
{
public:
@ -159,8 +299,9 @@ public:
virtual uint64_t get_num_outputs(const uint64_t& amount) const;
virtual crypto::public_key get_output_key(const uint64_t& amount, const uint64_t& index) const;
virtual output_data_t get_output_key(const uint64_t& amount, const uint64_t& index);
virtual output_data_t get_output_key(const uint64_t& global_index) const;
virtual void get_output_key(const uint64_t &amount, const std::vector<uint64_t> &offsets, std::vector<output_data_t> &outputs);
virtual tx_out get_output(const crypto::hash& h, const uint64_t& index) const;
/**
@ -175,9 +316,11 @@ public:
tx_out get_output(const uint64_t& index) const;
virtual tx_out_index get_output_tx_and_index_from_global(const uint64_t& index) const;
virtual void get_output_tx_and_index_from_global(const std::vector<uint64_t> &global_indices,
std::vector<tx_out_index> &tx_out_indices) const;
virtual tx_out_index get_output_tx_and_index(const uint64_t& amount, const uint64_t& index) const;
virtual void get_output_tx_and_index(const uint64_t& amount, std::vector<uint64_t> &offsets, std::vector<tx_out_index> &indices) const;
virtual tx_out_index get_output_tx_and_index(const uint64_t& amount, const uint64_t& index);
virtual void get_output_tx_and_index(const uint64_t& amount, const std::vector<uint64_t> &offsets, std::vector<tx_out_index> &indices);
virtual std::vector<uint64_t> get_tx_output_indices(const crypto::hash& h) const;
virtual std::vector<uint64_t> get_tx_amount_output_indices(const crypto::hash& h) const;
@ -198,7 +341,12 @@ public:
virtual void batch_abort();
virtual void pop_block(block& blk, std::vector<transaction>& txs);
virtual bool has_bulk_indices() const { return true; }
#if defined(BDB_BULK_CAN_THREAD)
virtual bool can_thread_bulk_indices() const { return true; }
#else
virtual bool can_thread_bulk_indices() const { return false; }
#endif
private:
virtual void add_block( const block& blk
@ -214,7 +362,7 @@ private:
virtual void remove_transaction_data(const crypto::hash& tx_hash, const transaction& tx);
virtual void add_output(const crypto::hash& tx_hash, const tx_out& tx_output, const uint64_t& local_index);
virtual void add_output(const crypto::hash& tx_hash, const tx_out& tx_output, const uint64_t& local_index, const uint64_t unlock_time);
virtual void remove_output(const tx_out& tx_output);
@ -227,6 +375,7 @@ private:
virtual void remove_spent_key(const crypto::key_image& k_image);
void get_output_global_indices(const uint64_t& amount, const std::vector<uint64_t> &offsets, std::vector<uint64_t> &global_indices);
/**
* @brief convert a tx output to a blob for storage
*
@ -253,10 +402,13 @@ private:
*
* @return the global index of the desired output
*/
uint64_t get_output_global_index(const uint64_t& amount, const uint64_t& index) const;
uint64_t get_output_global_index(const uint64_t& amount, const uint64_t& index);
void checkpoint_worker() const;
void check_open() const;
void *m_buffer;
bool m_run_checkpoint;
std::unique_ptr<boost::thread> m_checkpoint_thread;
typedef bdb_safe_buffer<void *> bdb_safe_buffer_t;
bdb_safe_buffer_t m_buffer;
DbEnv* m_env;

View File

@ -64,7 +64,7 @@ void BlockchainDB::add_transaction(const crypto::hash& blk_hash, const transacti
{
for (uint64_t i = 0; i < tx.vout.size(); ++i)
{
add_output(tx_hash, tx.vout[i], i);
add_output(tx_hash, tx.vout[i], i, tx.unlock_time);
}
for (const txin_v& tx_input : tx.vin)

View File

@ -138,6 +138,15 @@ namespace cryptonote
// typedef for convenience
typedef std::pair<crypto::hash, uint64_t> tx_out_index;
#pragma pack(push, 1)
struct output_data_t
{
crypto::public_key pubkey;
uint64_t unlock_time;
uint64_t height;
};
#pragma pack(pop)
/***********************************
* Exception Definitions
***********************************/
@ -279,7 +288,7 @@ private:
virtual void remove_transaction_data(const crypto::hash& tx_hash, const transaction& tx) = 0;
// tells the subclass to store an output
virtual void add_output(const crypto::hash& tx_hash, const tx_out& tx_output, const uint64_t& local_index) = 0;
virtual void add_output(const crypto::hash& tx_hash, const tx_out& tx_output, const uint64_t& local_index, const uint64_t unlock_time) = 0;
// tells the subclass to remove an output
virtual void remove_output(const tx_out& tx_output) = 0;
@ -313,7 +322,7 @@ protected:
mutable uint64_t time_tx_exists = 0;
uint64_t time_commit1 = 0;
bool m_auto_remove_logs = true;
public:
@ -461,7 +470,8 @@ public:
virtual uint64_t get_num_outputs(const uint64_t& amount) const = 0;
// return public key for output with global output amount <amount> and index <index>
virtual crypto::public_key get_output_key(const uint64_t& amount, const uint64_t& index) const = 0;
virtual output_data_t get_output_key(const uint64_t& amount, const uint64_t& index) = 0;
virtual output_data_t get_output_key(const uint64_t& global_index) const = 0;
// returns the output indexed by <index> in the transaction with hash <h>
virtual tx_out get_output(const crypto::hash& h, const uint64_t& index) const = 0;
@ -471,9 +481,11 @@ public:
// returns the transaction-local reference for the output with <amount> at <index>
// return type is pair of tx hash and index
virtual tx_out_index get_output_tx_and_index(const uint64_t& amount, const uint64_t& index) const = 0;
virtual void get_output_tx_and_index(const uint64_t& amount, std::vector<uint64_t> &offsets, std::vector<tx_out_index> &indices) const = 0;
virtual bool has_bulk_indices() const = 0;
virtual tx_out_index get_output_tx_and_index(const uint64_t& amount, const uint64_t& index) = 0;
virtual void get_output_tx_and_index(const uint64_t& amount, const std::vector<uint64_t> &offsets, std::vector<tx_out_index> &indices) = 0;
virtual void get_output_key(const uint64_t &amount, const std::vector<uint64_t> &offsets, std::vector<output_data_t> &outputs) = 0;
virtual bool can_thread_bulk_indices() const = 0;
// return a vector of indices corresponding to the global output index for
// each output in the transaction with hash <h>
@ -485,7 +497,10 @@ public:
// returns true if key image <img> is present in spent key images storage
virtual bool has_key_image(const crypto::key_image& img) const = 0;
void set_auto_remove_logs(bool auto_remove) { m_auto_remove_logs = auto_remove; }
bool m_open;
mutable epee::critical_section m_synchronization_lock;
}; // class BlockchainDB

View File

@ -65,10 +65,19 @@ struct lmdb_cur
done = false;
}
~lmdb_cur() { close(); }
~lmdb_cur()
{
close();
}
operator MDB_cursor*() { return m_cur; }
operator MDB_cursor**() { return &m_cur; }
operator MDB_cursor*()
{
return m_cur;
}
operator MDB_cursor**()
{
return &m_cur;
}
void close()
{
@ -87,7 +96,8 @@ private:
template<typename T>
struct MDB_val_copy: public MDB_val
{
MDB_val_copy(const T &t): t_copy(t)
MDB_val_copy(const T &t) :
t_copy(t)
{
mv_size = sizeof (T);
mv_data = &t_copy;
@ -99,7 +109,8 @@ private:
template<>
struct MDB_val_copy<cryptonote::blobdata>: public MDB_val
{
MDB_val_copy(const cryptonote::blobdata &bd): data(new char[bd.size()])
MDB_val_copy(const cryptonote::blobdata &bd) :
data(new char[bd.size()])
{
memcpy(data.get(), bd.data(), bd.size());
mv_size = bd.size();
@ -109,7 +120,8 @@ private:
std::unique_ptr<char[]> data;
};
auto compare_uint64 = [](const MDB_val *a, const MDB_val *b) {
auto compare_uint64 = [](const MDB_val *a, const MDB_val *b)
{
const uint64_t va = *(const uint64_t*)a->mv_data;
const uint64_t vb = *(const uint64_t*)b->mv_data;
if (va < vb) return -1;
@ -117,6 +129,20 @@ auto compare_uint64 = [](const MDB_val *a, const MDB_val *b) {
else return 1;
};
int compare_hash32(const MDB_val *a, const MDB_val *b)
{
uint32_t *va = (uint32_t*) a->mv_data;
uint32_t *vb = (uint32_t*) b->mv_data;
for (int n = 7; n >= 0; n--)
{
if (va[n] == vb[n])
continue;
return va[n] < vb[n] ? -1 : 1;
}
return 0;
}
const char* const LMDB_BLOCKS = "blocks";
const char* const LMDB_BLOCK_TIMESTAMPS = "block_timestamps";
const char* const LMDB_BLOCK_HEIGHTS = "block_heights";
@ -235,6 +261,26 @@ void mdb_txn_safe::allow_new_txns()
void BlockchainLMDB::do_resize(uint64_t increase_size)
{
CRITICAL_REGION_LOCAL(m_synchronization_lock);
const uint64_t add_size = 1LL << 30;
// check disk capacity
try
{
boost::filesystem::path path(m_folder);
boost::filesystem::space_info si = boost::filesystem::space(path);
if(si.available < add_size)
{
LOG_PRINT_RED_L0("!! WARNING: Insufficient free space to extend database !!: " << si.available / 1LL << 20L);
return;
}
}
catch(...)
{
// print something but proceed.
LOG_PRINT_YELLOW("Unable to query free disk space.", LOG_LEVEL_0);
}
MDB_envinfo mei;
mdb_env_info(m_env, &mei);
@ -250,6 +296,9 @@ void BlockchainLMDB::do_resize(uint64_t increase_size)
if (increase_size > 0)
new_mapsize = mei.me_mapsize + increase_size;
// add 1Gb per resize, instead of doing a percentage increase
// uint64_t new_mapsize = (double) mei.me_mapsize + add_size;
new_mapsize += (new_mapsize % mst.ms_psize);
mdb_txn_safe::prevent_new_txns();
@ -270,9 +319,7 @@ void BlockchainLMDB::do_resize(uint64_t increase_size)
mdb_env_set_mapsize(m_env, new_mapsize);
LOG_PRINT_L0("LMDB Mapsize increased."
<< " Old: " << mei.me_mapsize / (1024 * 1024) << "MiB"
<< ", New: " << new_mapsize / (1024 * 1024) << "MiB");
LOG_PRINT_GREEN("LMDB Mapsize increased." << " Old: " << mei.me_mapsize / (1024 * 1024) << "MiB" << ", New: " << new_mapsize / (1024 * 1024) << "MiB", LOG_LEVEL_0);
mdb_txn_safe::allow_new_txns();
}
@ -280,6 +327,7 @@ void BlockchainLMDB::do_resize(uint64_t increase_size)
// threshold_size is used for batch transactions
bool BlockchainLMDB::need_resize(uint64_t threshold_size) const
{
#if defined(ENABLE_AUTO_RESIZE)
MDB_envinfo mei;
mdb_env_info(m_env, &mei);
@ -311,12 +359,19 @@ bool BlockchainLMDB::need_resize(uint64_t threshold_size) const
return false;
}
if ((double)size_used / mei.me_mapsize > RESIZE_PERCENT)
std::mt19937 engine(std::random_device{}());
std::uniform_real_distribution<double> fdis(0.6, 0.9);
double resize_percent = fdis(engine);
if ((double)size_used / mei.me_mapsize > resize_percent)
{
LOG_PRINT_L1("Threshold met (percent-based)");
return true;
}
return false;
#else
return false;
#endif
}
void BlockchainLMDB::check_and_resize_for_batch(uint64_t batch_num_blocks)
@ -389,12 +444,8 @@ uint64_t BlockchainLMDB::get_estimated_batch_size(uint64_t batch_num_blocks) con
return threshold_size;
}
void BlockchainLMDB::add_block( const block& blk
, const size_t& block_size
, const difficulty_type& cumulative_difficulty
, const uint64_t& coins_generated
, const crypto::hash& blk_hash
)
void BlockchainLMDB::add_block(const block& blk, const size_t& block_size, const difficulty_type& cumulative_difficulty, const uint64_t& coins_generated,
const crypto::hash& blk_hash)
{
LOG_PRINT_L3("BlockchainLMDB::" << __func__);
check_open();
@ -545,7 +596,7 @@ void BlockchainLMDB::remove_transaction_data(const crypto::hash& tx_hash, const
}
void BlockchainLMDB::add_output(const crypto::hash& tx_hash, const tx_out& tx_output, const uint64_t& local_index)
void BlockchainLMDB::add_output(const crypto::hash& tx_hash, const tx_out& tx_output, const uint64_t& local_index, const uint64_t unlock_time)
{
LOG_PRINT_L3("BlockchainLMDB::" << __func__);
check_open();
@ -574,10 +625,15 @@ void BlockchainLMDB::add_output(const crypto::hash& tx_hash, const tx_out& tx_ou
if (tx_output.target.type() == typeid(txout_to_key))
{
MDB_val_copy<crypto::public_key> val_pubkey(boost::get<txout_to_key>(tx_output.target).key);
result = mdb_put(*m_write_txn, m_output_keys, &k, &val_pubkey, 0);
if (result)
throw0(DB_ERROR(std::string("Failed to add output pubkey to db transaction: ").append(mdb_strerror(result)).c_str()));
output_data_t od;
od.pubkey = boost::get < txout_to_key > (tx_output.target).key;
od.unlock_time = unlock_time;
od.height = m_height;
MDB_val_copy<output_data_t> data(od);
//MDB_val_copy<crypto::public_key> val_pubkey(boost::get<txout_to_key>(tx_output.target).key);
if (mdb_put(*m_write_txn, m_output_keys, &k, &data, 0))
throw0(DB_ERROR("Failed to add output pubkey to db transaction"));
}
@ -808,47 +864,17 @@ tx_out BlockchainLMDB::output_from_blob(const blobdata& blob) const
return o;
}
uint64_t BlockchainLMDB::get_output_global_index(const uint64_t& amount, const uint64_t& index) const
uint64_t BlockchainLMDB::get_output_global_index(const uint64_t& amount, const uint64_t& index)
{
LOG_PRINT_L3("BlockchainLMDB::" << __func__);
check_open();
mdb_txn_safe txn;
if (mdb_txn_begin(m_env, NULL, MDB_RDONLY, txn))
throw0(DB_ERROR("Failed to create a transaction for the db"));
lmdb_cur cur(txn, m_output_amounts);
MDB_val_copy<uint64_t> k(amount);
MDB_val v;
auto result = mdb_cursor_get(cur, &k, &v, MDB_SET);
if (result == MDB_NOTFOUND)
std::vector <uint64_t> offsets;
std::vector <uint64_t> global_indices;
offsets.push_back(index);
get_output_global_indices(amount, offsets, global_indices);
if (!global_indices.size())
throw1(OUTPUT_DNE("Attempting to get an output index by amount and amount index, but amount not found"));
else if (result)
throw0(DB_ERROR("DB error attempting to get an output"));
size_t num_elems = 0;
mdb_cursor_count(cur, &num_elems);
if (num_elems <= index)
throw1(OUTPUT_DNE("Attempting to get an output index by amount and amount index, but output not found"));
mdb_cursor_get(cur, &k, &v, MDB_FIRST_DUP);
for (uint64_t i = 0; i < index; ++i)
{
mdb_cursor_get(cur, &k, &v, MDB_NEXT_DUP);
}
mdb_cursor_get(cur, &k, &v, MDB_GET_CURRENT);
uint64_t glob_index = *(const uint64_t*)v.mv_data;
cur.close();
txn.commit();
return glob_index;
return global_indices[0];
}
void BlockchainLMDB::check_open() const
@ -903,8 +929,7 @@ void BlockchainLMDB::open(const std::string& filename, const int mdb_flags)
// check for existing LMDB files in base directory
boost::filesystem::path old_files = direc.parent_path();
if (boost::filesystem::exists(old_files / "data.mdb") ||
boost::filesystem::exists(old_files / "lock.mdb"))
if (boost::filesystem::exists(old_files / "data.mdb") || boost::filesystem::exists(old_files / "lock.mdb"))
{
LOG_PRINT_L0("Found existing LMDB files in " << old_files.string());
LOG_PRINT_L0("Move data.mdb and/or lock.mdb to " << filename << ", or delete them, and then restart");
@ -970,7 +995,7 @@ void BlockchainLMDB::open(const std::string& filename, const int mdb_flags)
lmdb_db_open(txn, LMDB_OUTPUT_TXS, MDB_INTEGERKEY | MDB_CREATE, m_output_txs, "Failed to open db handle for m_output_txs");
lmdb_db_open(txn, LMDB_OUTPUT_INDICES, MDB_INTEGERKEY | MDB_CREATE, m_output_indices, "Failed to open db handle for m_output_indices");
lmdb_db_open(txn, LMDB_OUTPUT_AMOUNTS, MDB_INTEGERKEY | MDB_DUPSORT | MDB_CREATE, m_output_amounts, "Failed to open db handle for m_output_amounts");
lmdb_db_open(txn, LMDB_OUTPUT_AMOUNTS, MDB_INTEGERKEY | MDB_DUPSORT | MDB_DUPFIXED | MDB_CREATE, m_output_amounts, "Failed to open db handle for m_output_amounts");
lmdb_db_open(txn, LMDB_OUTPUT_KEYS, MDB_INTEGERKEY | MDB_CREATE, m_output_keys, "Failed to open db handle for m_output_keys");
/*************** not used, but kept for posterity
@ -982,6 +1007,11 @@ void BlockchainLMDB::open(const std::string& filename, const int mdb_flags)
mdb_set_dupsort(txn, m_output_amounts, compare_uint64);
mdb_set_dupsort(txn, m_tx_outputs, compare_uint64);
mdb_set_compare(txn, m_spent_keys, compare_hash32);
mdb_set_compare(txn, m_block_heights, compare_hash32);
mdb_set_compare(txn, m_txs, compare_hash32);
mdb_set_compare(txn, m_tx_unlocks, compare_hash32);
mdb_set_compare(txn, m_tx_heights, compare_hash32);
// get and keep current height
MDB_stat db_stats;
@ -995,6 +1025,34 @@ void BlockchainLMDB::open(const std::string& filename, const int mdb_flags)
throw0(DB_ERROR("Failed to query m_output_indices"));
m_num_outputs = db_stats.ms_entries;
// ND: This "new" version of the lmdb database is incompatible with
// the previous version. Ensure that the output_keys database is
// sizeof(output_data_t) in length. Otherwise, inform user and
// terminate.
if(m_height > 0)
{
MDB_val_copy<uint64_t> k(0);
MDB_val v;
auto get_result = mdb_get(txn, m_output_keys, &k, &v);
if(get_result != MDB_SUCCESS)
{
txn.abort();
m_open = false;
return;
}
// LOG_PRINT_L0("Output keys size: " << v.mv_size);
if(v.mv_size != sizeof(output_data_t))
{
txn.abort();
mdb_env_close(m_env);
m_open = false;
LOG_PRINT_RED_L0("Existing lmdb database is incompatible with this version.");
LOG_PRINT_RED_L0("Please delete the existing database and resync.");
return;
}
}
// commit the transaction
txn.commit();
@ -1604,26 +1662,33 @@ uint64_t BlockchainLMDB::get_num_outputs(const uint64_t& amount) const
return num_elems;
}
crypto::public_key BlockchainLMDB::get_output_key(const uint64_t& amount, const uint64_t& index) const
output_data_t BlockchainLMDB::get_output_key(const uint64_t &global_index) const
{
LOG_PRINT_L3("BlockchainLMDB::" << __func__);
check_open();
uint64_t glob_index = get_output_global_index(amount, index);
mdb_txn_safe txn;
if (mdb_txn_begin(m_env, NULL, MDB_RDONLY, txn))
throw0(DB_ERROR("Failed to create a transaction for the db"));
MDB_val_copy<uint64_t> k(glob_index);
MDB_val_copy<uint64_t> k(global_index);
MDB_val v;
auto get_result = mdb_get(txn, m_output_keys, &k, &v);
if (get_result == MDB_NOTFOUND)
throw0(DB_ERROR("Attempting to get output pubkey by global index, but key does not exist"));
else if (get_result)
throw0(DB_ERROR("Error attempting to retrieve an output pubkey from the db"));
txn.commit();
return *(output_data_t *) v.mv_data;
}
return *(crypto::public_key*)v.mv_data;
output_data_t BlockchainLMDB::get_output_key(const uint64_t& amount, const uint64_t& index)
{
LOG_PRINT_L3("BlockchainLMDB::" << __func__);
check_open();
uint64_t glob_index = get_output_global_index(amount, index);
return get_output_key(glob_index);
}
tx_out BlockchainLMDB::get_output(const crypto::hash& h, const uint64_t& index) const
@ -1731,53 +1796,17 @@ tx_out_index BlockchainLMDB::get_output_tx_and_index_from_global(const uint64_t&
return tx_out_index(tx_hash, *(const uint64_t *)v.mv_data);
}
tx_out_index BlockchainLMDB::get_output_tx_and_index(const uint64_t& amount, const uint64_t& index) const
tx_out_index BlockchainLMDB::get_output_tx_and_index(const uint64_t& amount, const uint64_t& index)
{
LOG_PRINT_L3("BlockchainLMDB::" << __func__);
check_open();
mdb_txn_safe txn;
mdb_txn_safe* txn_ptr = &txn;
if (m_batch_active)
txn_ptr = m_write_txn;
else
{
if (mdb_txn_begin(m_env, NULL, MDB_RDONLY, txn))
throw0(DB_ERROR("Failed to create a transaction for the db"));
}
lmdb_cur cur(*txn_ptr, m_output_amounts);
MDB_val_copy<uint64_t> k(amount);
MDB_val v;
auto result = mdb_cursor_get(cur, &k, &v, MDB_SET);
if (result == MDB_NOTFOUND)
std::vector < uint64_t > offsets;
std::vector<tx_out_index> indices;
offsets.push_back(index);
get_output_tx_and_index(amount, offsets, indices);
if (!indices.size())
throw1(OUTPUT_DNE("Attempting to get an output index by amount and amount index, but amount not found"));
else if (result)
throw0(DB_ERROR("DB error attempting to get an output"));
size_t num_elems = 0;
mdb_cursor_count(cur, &num_elems);
if (num_elems <= index)
throw1(OUTPUT_DNE("Attempting to get an output index by amount and amount index, but output not found"));
mdb_cursor_get(cur, &k, &v, MDB_FIRST_DUP);
for (uint64_t i = 0; i < index; ++i)
{
mdb_cursor_get(cur, &k, &v, MDB_NEXT_DUP);
}
mdb_cursor_get(cur, &k, &v, MDB_GET_CURRENT);
uint64_t glob_index = *(const uint64_t*)v.mv_data;
cur.close();
if (! m_batch_active)
txn.commit();
return get_output_tx_and_index_from_global(glob_index);
return indices[0];
}
std::vector<uint64_t> BlockchainLMDB::get_tx_output_indices(const crypto::hash& h) const
@ -2016,12 +2045,8 @@ void BlockchainLMDB::set_batch_transactions(bool batch_transactions)
LOG_PRINT_L3("batch transactions " << (m_batch_transactions ? "enabled" : "disabled"));
}
uint64_t BlockchainLMDB::add_block( const block& blk
, const size_t& block_size
, const difficulty_type& cumulative_difficulty
, const uint64_t& coins_generated
, const std::vector<transaction>& txs
)
uint64_t BlockchainLMDB::add_block(const block& blk, const size_t& block_size, const difficulty_type& cumulative_difficulty, const uint64_t& coins_generated,
const std::vector<transaction>& txs)
{
LOG_PRINT_L3("BlockchainLMDB::" << __func__);
check_open();
@ -2103,4 +2128,213 @@ void BlockchainLMDB::pop_block(block& blk, std::vector<transaction>& txs)
--m_height;
}
void BlockchainLMDB::get_output_tx_and_index_from_global(const std::vector<uint64_t> &global_indices,
std::vector<tx_out_index> &tx_out_indices) const
{
LOG_PRINT_L3("BlockchainLMDB::" << __func__);
check_open();
tx_out_indices.clear();
mdb_txn_safe txn;
mdb_txn_safe* txn_ptr = &txn;
if (m_batch_active)
txn_ptr = m_write_txn;
else
{
if (mdb_txn_begin(m_env, NULL, MDB_RDONLY, txn))
throw0(DB_ERROR("Failed to create a transaction for the db"));
}
for (const uint64_t &index : global_indices)
{
MDB_val_copy<uint64_t> k(index);
MDB_val v;
auto get_result = mdb_get(*txn_ptr, m_output_txs, &k, &v);
if (get_result == MDB_NOTFOUND)
throw1(OUTPUT_DNE("output with given index not in db"));
else if (get_result)
throw0(DB_ERROR("DB error attempting to fetch output tx hash"));
crypto::hash tx_hash = *(crypto::hash*) v.mv_data;
get_result = mdb_get(*txn_ptr, m_output_indices, &k, &v);
if (get_result == MDB_NOTFOUND)
throw1(OUTPUT_DNE("output with given index not in db"));
else if (get_result)
throw0(DB_ERROR("DB error attempting to fetch output tx index"));
auto result = tx_out_index(tx_hash, *(const uint64_t *) v.mv_data);
tx_out_indices.push_back(result);
}
if (!m_batch_active)
txn.commit();
}
void BlockchainLMDB::get_output_global_indices(const uint64_t& amount, const std::vector<uint64_t> &offsets,
std::vector<uint64_t> &global_indices)
{
LOG_PRINT_L3("BlockchainLMDB::" << __func__);
TIME_MEASURE_START(txx);
check_open();
global_indices.clear();
uint64_t max = 0;
for (const uint64_t &index : offsets)
{
if (index > max)
max = index;
}
mdb_txn_safe txn;
mdb_txn_safe* txn_ptr = &txn;
if(m_batch_active)
txn_ptr = m_write_txn;
else
{
if (mdb_txn_begin(m_env, NULL, MDB_RDONLY, txn))
throw0(DB_ERROR("Failed to create a transaction for the db"));
}
lmdb_cur cur(*txn_ptr, m_output_amounts);
MDB_val_copy<uint64_t> k(amount);
MDB_val v;
auto result = mdb_cursor_get(cur, &k, &v, MDB_SET);
if (result == MDB_NOTFOUND)
throw1(OUTPUT_DNE("Attempting to get an output index by amount and amount index, but amount not found"));
else if (result)
throw0(DB_ERROR("DB error attempting to get an output"));
size_t num_elems = 0;
mdb_cursor_count(cur, &num_elems);
if (max <= 1 && num_elems <= max)
throw1(OUTPUT_DNE("Attempting to get an output index by amount and amount index, but output not found"));
uint64_t t_dbmul = 0;
uint64_t t_dbscan = 0;
if (max <= 1)
{
for (const uint64_t& index : offsets)
{
mdb_cursor_get(cur, &k, &v, MDB_FIRST_DUP);
for (uint64_t i = 0; i < index; ++i)
{
mdb_cursor_get(cur, &k, &v, MDB_NEXT_DUP);
}
mdb_cursor_get(cur, &k, &v, MDB_GET_CURRENT);
uint64_t glob_index = *(const uint64_t*) v.mv_data;
LOG_PRINT_L3("Amount: " << amount << " M0->v: " << glob_index);
global_indices.push_back(glob_index);
}
}
else
{
uint32_t curcount = 0;
uint32_t blockstart = 0;
for (const uint64_t& index : offsets)
{
if (index >= num_elems)
{
LOG_PRINT_L1("Index: " << index << " Elems: " << num_elems << " partial results found for get_output_tx_and_index");
break;
}
while (index >= curcount)
{
TIME_MEASURE_START(db1);
if (mdb_cursor_get(cur, &k, &v, curcount == 0 ? MDB_GET_MULTIPLE : MDB_NEXT_MULTIPLE) != 0)
{
// allow partial results
result = false;
break;
}
int count = v.mv_size / sizeof(uint64_t);
blockstart = curcount;
curcount += count;
TIME_MEASURE_FINISH(db1);
t_dbmul += db1;
}
LOG_PRINT_L3("Records returned: " << curcount << " Index: " << index);
TIME_MEASURE_START(db2);
uint64_t actual_index = index - blockstart;
uint64_t glob_index = ((const uint64_t*) v.mv_data)[actual_index];
LOG_PRINT_L3("Amount: " << amount << " M1->v: " << glob_index);
global_indices.push_back(glob_index);
TIME_MEASURE_FINISH(db2);
t_dbscan += db2;
}
}
cur.close();
if(!m_batch_active)
txn.commit();
TIME_MEASURE_FINISH(txx);
LOG_PRINT_L3("txx: " << txx << " db1: " << t_dbmul << " db2: " << t_dbscan);
}
void BlockchainLMDB::get_output_key(const uint64_t &amount, const std::vector<uint64_t> &offsets, std::vector<output_data_t> &outputs)
{
LOG_PRINT_L3("BlockchainLMDB::" << __func__);
TIME_MEASURE_START(db3);
check_open();
outputs.clear();
std::vector <uint64_t> global_indices;
get_output_global_indices(amount, offsets, global_indices);
if (global_indices.size() > 0)
{
mdb_txn_safe txn;
if (mdb_txn_begin(m_env, NULL, MDB_RDONLY, txn))
throw0(DB_ERROR("Failed to create a transaction for the db"));
for (const uint64_t &index : global_indices)
{
MDB_val_copy<uint64_t> k(index);
MDB_val v;
auto get_result = mdb_get(txn, m_output_keys, &k, &v);
if (get_result != 0)
throw0(DB_ERROR("Attempting to get output pubkey by global index, but key does not exist"));
else if (get_result)
throw0(DB_ERROR("Error attempting to retrieve an output pubkey from the db"));
output_data_t data = *(output_data_t *) v.mv_data;
outputs.push_back(data);
}
txn.commit();
}
TIME_MEASURE_FINISH(db3);
LOG_PRINT_L3("db3: " << db3);
}
void BlockchainLMDB::get_output_tx_and_index(const uint64_t& amount, const std::vector<uint64_t> &offsets, std::vector<tx_out_index> &indices)
{
LOG_PRINT_L3("BlockchainLMDB::" << __func__);
check_open();
indices.clear();
std::vector <uint64_t> global_indices;
get_output_global_indices(amount, offsets, global_indices);
TIME_MEASURE_START(db3);
if(global_indices.size() > 0)
{
get_output_tx_and_index_from_global(global_indices, indices);
}
TIME_MEASURE_FINISH(db3);
LOG_PRINT_L3("db3: " << db3);
}
} // namespace cryptonote

View File

@ -33,6 +33,8 @@
#include <lmdb.h>
#define ENABLE_AUTO_RESIZE
namespace cryptonote
{
@ -159,7 +161,9 @@ public:
virtual uint64_t get_num_outputs(const uint64_t& amount) const;
virtual crypto::public_key get_output_key(const uint64_t& amount, const uint64_t& index) const;
virtual output_data_t get_output_key(const uint64_t& amount, const uint64_t& index);
virtual output_data_t get_output_key(const uint64_t& global_index) const;
virtual void get_output_key(const uint64_t &amount, const std::vector<uint64_t> &offsets, std::vector<output_data_t> &outputs);
virtual tx_out get_output(const crypto::hash& h, const uint64_t& index) const;
@ -175,12 +179,12 @@ public:
tx_out get_output(const uint64_t& index) const;
virtual tx_out_index get_output_tx_and_index_from_global(const uint64_t& index) const;
virtual void get_output_tx_and_index_from_global(const std::vector<uint64_t> &global_indices,
std::vector<tx_out_index> &tx_out_indices) const;
virtual tx_out_index get_output_tx_and_index(const uint64_t& amount, const uint64_t& index) const;
virtual void get_output_tx_and_index(const uint64_t& amount, std::vector<uint64_t> &offsets, std::vector<tx_out_index> &indices) const
{
// do nothing
};
virtual tx_out_index get_output_tx_and_index(const uint64_t& amount, const uint64_t& index);
virtual void get_output_tx_and_index(const uint64_t& amount, const std::vector<uint64_t> &offsets, std::vector<tx_out_index> &indices);
virtual void get_output_global_indices(const uint64_t& amount, const std::vector<uint64_t> &offsets, std::vector<uint64_t> &indices);
virtual std::vector<uint64_t> get_tx_output_indices(const crypto::hash& h) const;
virtual std::vector<uint64_t> get_tx_amount_output_indices(const crypto::hash& h) const;
@ -202,7 +206,7 @@ public:
virtual void pop_block(block& blk, std::vector<transaction>& txs);
virtual bool has_bulk_indices() const { return false; }
virtual bool can_thread_bulk_indices() const { return true; }
private:
void do_resize(uint64_t size_increase=0);
@ -223,7 +227,7 @@ private:
virtual void remove_transaction_data(const crypto::hash& tx_hash, const transaction& tx);
virtual void add_output(const crypto::hash& tx_hash, const tx_out& tx_output, const uint64_t& local_index);
virtual void add_output(const crypto::hash& tx_hash, const tx_out& tx_output, const uint64_t& local_index, const uint64_t unlock_time);
virtual void remove_output(const tx_out& tx_output);
@ -262,7 +266,7 @@ private:
*
* @return the global index of the desired output
*/
uint64_t get_output_global_index(const uint64_t& amount, const uint64_t& index) const;
uint64_t get_output_global_index(const uint64_t& amount, const uint64_t& index);
void check_open() const;
@ -299,9 +303,18 @@ private:
bool m_batch_transactions; // support for batch transactions
bool m_batch_active; // whether batch transaction is in progress
constexpr static uint64_t DEFAULT_MAPSIZE = 1 << 30;
#if defined(__arm__)
// force a value so it can compile with 32-bit ARM
constexpr static uint64_t DEFAULT_MAPSIZE = 1LL << 31;
#else
#if defined(ENABLE_AUTO_RESIZE)
constexpr static uint64_t DEFAULT_MAPSIZE = 1LL << 30;
#else
constexpr static uint64_t DEFAULT_MAPSIZE = 1LL << 33;
#endif
#endif
constexpr static float RESIZE_PERCENT = 0.8f;
constexpr static float RESIZE_FACTOR = 1.5f;
};
} // namespace cryptonote

37
src/blocks/CMakeLists.txt Normal file
View File

@ -0,0 +1,37 @@
# Copyright (c) 2014-2015, The Monero Project
#
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without modification, are
# permitted provided that the following conditions are met:
#
# 1. Redistributions of source code must retain the above copyright notice, this list of
# conditions and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright notice, this list
# of conditions and the following disclaimer in the documentation and/or other
# materials provided with the distribution.
#
# 3. Neither the name of the copyright holder nor the names of its contributors may be
# used to endorse or promote products derived from this software without specific
# prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
# THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
# STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
# THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
if(APPLE)
add_library(blocks STATIC blockexports.c)
set_target_properties(blocks PROPERTIES LINKER_LANGUAGE C)
else()
add_custom_command(OUTPUT blocks.o COMMAND cd ${CMAKE_CURRENT_SOURCE_DIR} && ld -r -b binary -o ${CMAKE_CURRENT_BINARY_DIR}/blocks.o blocks.dat)
add_library(blocks STATIC blocks.o blockexports.c)
set_target_properties(blocks PROPERTIES LINKER_LANGUAGE C)
endif()

48
src/blocks/blockexports.c Normal file
View File

@ -0,0 +1,48 @@
#include <stddef.h>
#if defined(__APPLE__)
#include <mach-o/getsect.h>
#if !defined(__LP64__)
extern const struct mach_header _mh_execute_header;
#else
extern const struct mach_header_64 _mh_execute_header;
#endif
const unsigned char *get_blocks_dat_start()
{
size_t size;
return getsectiondata(&_mh_execute_header, "__DATA", "__blocks_dat", &size);
}
size_t get_blocks_dat_size()
{
size_t size;
getsectiondata(&_mh_execute_header, "__DATA", "__blocks_dat", &size);
return size;
}
#else
#if defined(_WIN32) && !defined(_WIN64)
#define _binary_blocks_start binary_blocks_dat_start
#define _binary_blocks_end binary_blocks_dat_end
#else
#define _binary_blocks_start _binary_blocks_dat_start
#define _binary_blocks_end _binary_blocks_dat_end
#endif
extern const unsigned char _binary_blocks_start[];
extern const unsigned char _binary_blocks_end[];
const unsigned char *get_blocks_dat_start(void)
{
return _binary_blocks_start;
}
size_t get_blocks_dat_size(void)
{
return (size_t) (_binary_blocks_end - _binary_blocks_start);
}
#endif

BIN
src/blocks/blocks.dat Normal file

Binary file not shown.

16
src/blocks/blocks.h Normal file
View File

@ -0,0 +1,16 @@
#ifndef SRC_BLOCKS_BLOCKS_H_
#define SRC_BLOCKS_BLOCKS_H_
#ifdef __cplusplus
extern "C" {
#endif
const unsigned char *get_blocks_dat_start();
size_t get_blocks_dat_size();
#ifdef __cplusplus
}
#endif
#endif /* SRC_BLOCKS_BLOCKS_H_ */

View File

@ -140,7 +140,15 @@ extern "C"
d_4(uint32_t, t_dec(f,n), sb_data, u0, u1, u2, u3);
void aesb_single_round(const uint8_t *in, uint8_t *out, uint8_t *expandedKey)
#if !defined(STATIC)
#define STATIC
#endif
#if !defined(INLINE)
#define INLINE
#endif
STATIC INLINE void aesb_single_round(const uint8_t *in, uint8_t *out, uint8_t *expandedKey)
{
uint32_t b0[4], b1[4];
const uint32_t *kp = (uint32_t *) expandedKey;
@ -151,7 +159,7 @@ void aesb_single_round(const uint8_t *in, uint8_t *out, uint8_t *expandedKey)
state_out(out, b1);
}
void aesb_pseudo_round(const uint8_t *in, uint8_t *out, uint8_t *expandedKey)
STATIC INLINE void aesb_pseudo_round(const uint8_t *in, uint8_t *out, uint8_t *expandedKey)
{
uint32_t b0[4], b1[4];
const uint32_t *kp = (uint32_t *) expandedKey;

View File

@ -624,6 +624,196 @@ void cn_slow_hash(const void *data, size_t length, char *hash)
extra_hashes[state.hs.b[0] & 3](&state, 200, hash);
}
#elif defined(__arm__)
// ND: Some minor optimizations for ARM7 (raspberrry pi 2), effect seems to be ~40-50% faster.
// Needs more work.
void slow_hash_allocate_state(void)
{
// Do nothing, this is just to maintain compatibility with the upgraded slow-hash.c
return;
}
void slow_hash_free_state(void)
{
// As above
return;
}
static void (*const extra_hashes[4])(const void *, size_t, char *) = {
hash_extra_blake, hash_extra_groestl, hash_extra_jh, hash_extra_skein
};
#define MEMORY (1 << 21) /* 2 MiB */
#define ITER (1 << 20)
#define AES_BLOCK_SIZE 16
#define AES_KEY_SIZE 32 /*16*/
#define INIT_SIZE_BLK 8
#define INIT_SIZE_BYTE (INIT_SIZE_BLK * AES_BLOCK_SIZE)
#if defined(__GNUC__)
#define RDATA_ALIGN16 __attribute__ ((aligned(16)))
#define STATIC static
#define INLINE inline
#else
#define RDATA_ALIGN16
#define STATIC static
#define INLINE
#endif
#define U64(x) ((uint64_t *) (x))
#include "aesb.c"
STATIC INLINE void ___mul128(uint32_t *a, uint32_t *b, uint32_t *h, uint32_t *l)
{
// ND: 64x64 multiplication for ARM7
__asm__ __volatile__
(
// lo hi
"umull %[r0], %[r1], %[b], %[d]\n\t" // bd [r0 = bd.lo]
"umull %[r2], %[r3], %[b], %[c]\n\t" // bc
"umull %[b], %[c], %[a], %[c]\n\t" // ac
"adds %[r1], %[r1], %[r2]\n\t" // r1 = bd.hi + bc.lo
"adcs %[r2], %[r3], %[b]\n\t" // r2 = ac.lo + bc.hi + carry
"adc %[r3], %[c], #0\n\t" // r3 = ac.hi + carry
"umull %[b], %[a], %[a], %[d]\n\t" // ad
"adds %[r1], %[r1], %[b]\n\t" // r1 = bd.hi + bc.lo + ad.lo
"adcs %[r2], %[r2], %[a]\n\t" // r2 = ac.lo + bc.hi + ad.hi + carry
"adc %[r3], %[r3], #0\n\t" // r3 = ac.hi + carry
: [r0]"=&r"(l[0]), [r1]"=&r"(l[1]), [r2]"=&r"(h[0]), [r3]"=&r"(h[1])
: [a]"r"(a[1]), [b]"r"(a[0]), [c]"r"(b[1]), [d]"r"(b[0])
: "cc"
);
}
STATIC INLINE void mul(const uint8_t* a, const uint8_t* b, uint8_t* res)
{
___mul128((uint32_t *) a, (uint32_t *) b, (uint32_t *) (res + 0), (uint32_t *) (res + 8));
}
STATIC INLINE void sum_half_blocks(uint8_t* a, const uint8_t* b)
{
uint64_t a0, a1, b0, b1;
a0 = U64(a)[0];
a1 = U64(a)[1];
b0 = U64(b)[0];
b1 = U64(b)[1];
a0 += b0;
a1 += b1;
U64(a)[0] = a0;
U64(a)[1] = a1;
}
STATIC INLINE void swap_blocks(uint8_t *a, uint8_t *b)
{
uint64_t t[2];
U64(t)[0] = U64(a)[0];
U64(t)[1] = U64(a)[1];
U64(a)[0] = U64(b)[0];
U64(a)[1] = U64(b)[1];
U64(b)[0] = U64(t)[0];
U64(b)[1] = U64(t)[1];
}
STATIC INLINE void xor_blocks(uint8_t* a, const uint8_t* b)
{
U64(a)[0] ^= U64(b)[0];
U64(a)[1] ^= U64(b)[1];
}
#pragma pack(push, 1)
union cn_slow_hash_state
{
union hash_state hs;
struct
{
uint8_t k[64];
uint8_t init[INIT_SIZE_BYTE];
};
};
#pragma pack(pop)
void cn_slow_hash(const void *data, size_t length, char *hash)
{
uint8_t long_state[MEMORY];
uint8_t text[INIT_SIZE_BYTE];
uint8_t a[AES_BLOCK_SIZE];
uint8_t b[AES_BLOCK_SIZE];
uint8_t d[AES_BLOCK_SIZE];
uint8_t aes_key[AES_KEY_SIZE];
RDATA_ALIGN16 uint8_t expandedKey[256];
union cn_slow_hash_state state;
size_t i, j;
uint8_t *p = NULL;
oaes_ctx *aes_ctx;
static void (*const extra_hashes[4])(const void *, size_t, char *) =
{
hash_extra_blake, hash_extra_groestl, hash_extra_jh, hash_extra_skein
};
hash_process(&state.hs, data, length);
memcpy(text, state.init, INIT_SIZE_BYTE);
aes_ctx = (oaes_ctx *) oaes_alloc();
oaes_key_import_data(aes_ctx, state.hs.b, AES_KEY_SIZE);
// use aligned data
memcpy(expandedKey, aes_ctx->key->exp_data, aes_ctx->key->exp_data_len);
for(i = 0; i < MEMORY / INIT_SIZE_BYTE; i++)
{
for(j = 0; j < INIT_SIZE_BLK; j++)
aesb_pseudo_round(&text[AES_BLOCK_SIZE * j], &text[AES_BLOCK_SIZE * j], expandedKey);
memcpy(&long_state[i * INIT_SIZE_BYTE], text, INIT_SIZE_BYTE);
}
U64(a)[0] = U64(&state.k[0])[0] ^ U64(&state.k[32])[0];
U64(a)[1] = U64(&state.k[0])[1] ^ U64(&state.k[32])[1];
U64(b)[0] = U64(&state.k[16])[0] ^ U64(&state.k[48])[0];
U64(b)[1] = U64(&state.k[16])[1] ^ U64(&state.k[48])[1];
for(i = 0; i < ITER / 2; i++)
{
#define MASK ((uint32_t)(((MEMORY / AES_BLOCK_SIZE) - 1) << 4))
#define state_index(x) ((*(uint32_t *) x) & MASK)
// Iteration 1
p = &long_state[state_index(a)];
aesb_single_round(p, p, a);
xor_blocks(b, p);
swap_blocks(b, p);
swap_blocks(a, b);
// Iteration 2
p = &long_state[state_index(a)];
mul(a, p, d);
sum_half_blocks(b, d);
swap_blocks(b, p);
xor_blocks(b, p);
swap_blocks(a, b);
}
memcpy(text, state.init, INIT_SIZE_BYTE);
oaes_key_import_data(aes_ctx, &state.hs.b[32], AES_KEY_SIZE);
memcpy(expandedKey, aes_ctx->key->exp_data, aes_ctx->key->exp_data_len);
for(i = 0; i < MEMORY / INIT_SIZE_BYTE; i++)
{
for(j = 0; j < INIT_SIZE_BLK; j++)
{
xor_blocks(&text[j * AES_BLOCK_SIZE], &long_state[i * INIT_SIZE_BYTE + j * AES_BLOCK_SIZE]);
aesb_pseudo_round(&text[AES_BLOCK_SIZE * j], &text[AES_BLOCK_SIZE * j], expandedKey);
}
}
oaes_free((OAES_CTX **) &aes_ctx);
memcpy(state.init, text, INIT_SIZE_BYTE);
hash_permutation(&state.hs);
extra_hashes[state.hs.b[0] & 3](&state, 200, hash);
}
#else
// Portable implementation as a fallback

View File

@ -62,6 +62,12 @@ set(cryptonote_core_private_headers
tx_pool.h
verification_context.h)
if(PER_BLOCK_CHECKPOINT)
set(Blocks "blocks")
else()
set(Blocks "")
endif()
bitmonero_private_headers(cryptonote_core
${crypto_private_headers})
bitmonero_add_library(cryptonote_core
@ -78,6 +84,7 @@ target_link_libraries(cryptonote_core
${Boost_PROGRAM_OPTIONS_LIBRARY}
${Boost_SERIALIZATION_LIBRARY}
LINK_PRIVATE
${Blocks}
${Boost_FILESYSTEM_LIBRARY}
${Boost_SYSTEM_LIBRARY}
${Boost_THREAD_LIBRARY}

File diff suppressed because it is too large Load Diff

View File

@ -56,6 +56,13 @@ namespace cryptonote
{
class tx_memory_pool;
enum blockchain_db_sync_mode
{
db_sync,
db_async,
db_nosync
};
/************************************************************************/
/* */
/************************************************************************/
@ -94,6 +101,8 @@ namespace cryptonote
crypto::hash get_block_id_by_height(uint64_t height) const;
bool get_block_by_hash(const crypto::hash &h, block &blk) const;
void get_all_known_block_ids(std::list<crypto::hash> &main, std::list<crypto::hash> &alt, std::list<crypto::hash> &invalid) const;
bool prepare_handle_incoming_blocks(const std::list<block_complete_entry> &blocks);
bool cleanup_handle_incoming_blocks(bool force_sync = false);
template<class archive_t>
void serialize(archive_t & ar, const unsigned int version);
@ -102,16 +111,13 @@ namespace cryptonote
bool have_tx_keyimges_as_spent(const transaction &tx) const;
bool have_tx_keyimg_as_spent(const crypto::key_image &key_im) const;
template<class visitor_t>
bool scan_outputkeys_for_indexes(const txin_to_key& tx_in_to_key, visitor_t& vis, uint64_t* pmax_related_block_height = NULL) const;
uint64_t get_current_blockchain_height() const;
crypto::hash get_tail_id() const;
crypto::hash get_tail_id(uint64_t& height) const;
difficulty_type get_difficulty_for_next_block() const;
difficulty_type get_difficulty_for_next_block();
bool add_new_block(const block& bl_, block_verification_context& bvc);
bool reset_and_set_genesis_block(const block& b);
bool create_block_template(block& b, const account_public_address& miner_address, difficulty_type& di, uint64_t& height, const blobdata& ex_nonce) const;
bool create_block_template(block& b, const account_public_address& miner_address, difficulty_type& di, uint64_t& height, const blobdata& ex_nonce);
bool have_block(const crypto::hash& id) const;
size_t get_total_transactions() const;
bool get_short_chain_history(std::list<crypto::hash>& ids) const;
@ -123,9 +129,8 @@ namespace cryptonote
bool get_random_outs_for_amounts(const COMMAND_RPC_GET_RANDOM_OUTPUTS_FOR_AMOUNTS::request& req, COMMAND_RPC_GET_RANDOM_OUTPUTS_FOR_AMOUNTS::response& res) const;
bool get_tx_outputs_gindexs(const crypto::hash& tx_id, std::vector<uint64_t>& indexs) const;
bool store_blockchain();
bool check_tx_input(const txin_to_key& txin, const crypto::hash& tx_prefix_hash, const std::vector<crypto::signature>& sig, uint64_t* pmax_related_block_height = NULL) const;
bool check_tx_inputs(const transaction& tx, uint64_t* pmax_used_block_height = NULL) const;
bool check_tx_inputs(const transaction& tx, uint64_t& pmax_used_block_height, crypto::hash& max_used_block_id) const;
bool check_tx_inputs(const transaction& tx, uint64_t& pmax_used_block_height, crypto::hash& max_used_block_id, bool kept_by_block = false);
uint64_t get_current_cumulative_blocksize_limit() const;
bool is_storing_blockchain()const{return m_is_blockchain_storing;}
uint64_t block_difficulty(uint64_t i) const;
@ -145,11 +150,23 @@ namespace cryptonote
void set_enforce_dns_checkpoints(bool enforce);
bool update_checkpoints(const std::string& file_path, bool check_dns);
// user options, must be called before calling init()
void set_user_options(uint64_t block_threads, uint64_t blocks_per_sync,
blockchain_db_sync_mode sync_mode, bool fast_sync);
void set_show_time_stats(bool stats) { m_show_time_stats = stats; }
BlockchainDB& get_db()
{
return *m_db;
}
void output_scan_worker(const uint64_t amount,const std::vector<uint64_t> &offsets,
std::vector<output_data_t> &outputs, std::unordered_map<crypto::hash,
cryptonote::transaction> &txs) const;
void block_longhash_worker(const uint64_t height, const std::vector<block> &blocks,
std::unordered_map<crypto::hash, crypto::hash> &map) const;
private:
typedef std::unordered_map<crypto::hash, size_t> blocks_by_id_index;
typedef std::unordered_map<crypto::hash, transaction_chain_entry> transactions_container;
@ -171,6 +188,29 @@ namespace cryptonote
key_images_container m_spent_keys;
size_t m_current_block_cumul_sz_limit;
std::unordered_map<crypto::hash, std::unordered_map<crypto::key_image, std::vector<output_data_t>>> m_scan_table;
std::unordered_map<crypto::hash, std::pair<bool, uint64_t>> m_check_tx_inputs_table;
std::unordered_map<crypto::hash, crypto::hash> m_blocks_longhash_table;
// SHA-3 hashes for each block and for fast pow checking
std::vector<crypto::hash> m_blocks_hash_check;
std::vector<crypto::hash> m_blocks_txs_check;
blockchain_db_sync_mode m_db_sync_mode;
bool m_fast_sync;
bool m_show_time_stats;
uint64_t m_db_blocks_per_sync;
uint64_t m_max_prepare_blocks_threads;
uint64_t m_fake_pow_calc_time;
uint64_t m_fake_scan_time;
uint64_t m_sync_counter;
std::vector<uint64_t> m_timestamps;
std::vector<difficulty_type> m_difficulties;
uint64_t m_timestamps_and_difficulties_height;
boost::asio::io_service m_async_service;
boost::thread_group m_async_pool;
std::unique_ptr<boost::asio::io_service::work> m_async_work_idle;
// all alternative chains
blocks_ext_by_hash m_alternative_chains; // crypto::hash -> block_extended_info
@ -185,6 +225,11 @@ namespace cryptonote
std::atomic<bool> m_is_blockchain_storing;
bool m_enforce_dns_checkpoints;
template<class visitor_t>
inline bool scan_outputkeys_for_indexes(const txin_to_key& tx_in_to_key, visitor_t &vis, const crypto::hash &tx_prefix_hash, uint64_t* pmax_related_block_height = NULL) const;
bool check_tx_input(const txin_to_key& txin, const crypto::hash& tx_prefix_hash, const std::vector<crypto::signature>& sig, std::vector<crypto::public_key> &output_keys, uint64_t* pmax_related_block_height);
bool check_tx_inputs(const transaction& tx, uint64_t* pmax_used_block_height = NULL);
bool switch_to_alternative_blockchain(std::list<blocks_ext_by_hash::iterator>& alt_chain, bool discard_disconnected_chain);
block pop_block_from_blockchain();
bool purge_transaction_from_blockchain(const crypto::hash& tx_id);
@ -213,6 +258,9 @@ namespace cryptonote
bool update_next_cumulative_size_limit();
bool check_for_double_spend(const transaction& tx, key_images_container& keys_this_block) const;
void get_timestamp_and_difficulty(uint64_t &timestamp, difficulty_type &difficulty, const int offset) const;
void check_ring_signature(const crypto::hash &tx_prefix_hash, const crypto::key_image &key_image,
const std::vector<crypto::public_key> &pubkeys, const std::vector<crypto::signature> &sig, uint64_t &result);
};

View File

@ -104,9 +104,6 @@ namespace cryptonote
bool have_tx_keyimg_as_spent(const crypto::key_image &key_im) const;
const transaction *get_tx(const crypto::hash &id) const;
template<class visitor_t>
bool scan_outputkeys_for_indexes(const txin_to_key& tx_in_to_key, visitor_t& vis, uint64_t* pmax_related_block_height = NULL) const;
uint64_t get_current_blockchain_height() const;
crypto::hash get_tail_id() const;
crypto::hash get_tail_id(uint64_t& height) const;
@ -127,9 +124,7 @@ namespace cryptonote
bool get_backward_blocks_sizes(size_t from_height, std::vector<size_t>& sz, size_t count) const;
bool get_tx_outputs_gindexs(const crypto::hash& tx_id, std::vector<uint64_t>& indexs) const;
bool store_blockchain();
bool check_tx_input(const txin_to_key& txin, const crypto::hash& tx_prefix_hash, const std::vector<crypto::signature>& sig, uint64_t* pmax_related_block_height = NULL) const;
bool check_tx_inputs(const transaction& tx, const crypto::hash& tx_prefix_hash, uint64_t* pmax_used_block_height = NULL) const;
bool check_tx_inputs(const transaction& tx, uint64_t* pmax_used_block_height = NULL) const;
bool check_tx_inputs(const transaction& tx, uint64_t& pmax_used_block_height, crypto::hash& max_used_block_id) const;
uint64_t get_current_cumulative_blocksize_limit() const;
bool is_storing_blockchain()const{return m_is_blockchain_storing;}
@ -229,6 +224,13 @@ namespace cryptonote
bool m_enforce_dns_checkpoints;
bool m_testnet;
// made private for consistency with blockchain.h
template<class visitor_t>
bool scan_outputkeys_for_indexes(const txin_to_key& tx_in_to_key, visitor_t& vis, uint64_t* pmax_related_block_height = NULL) const;
bool check_tx_input(const txin_to_key& txin, const crypto::hash& tx_prefix_hash, const std::vector<crypto::signature>& sig, uint64_t* pmax_related_block_height = NULL) const;
bool check_tx_inputs(const transaction& tx, const crypto::hash& tx_prefix_hash, uint64_t* pmax_used_block_height = NULL) const;
bool check_tx_inputs(const transaction& tx, uint64_t* pmax_used_block_height = NULL) const;
bool switch_to_alternative_blockchain(std::list<blocks_ext_by_hash::iterator>& alt_chain, bool discard_disconnected_chain);
bool pop_block_from_blockchain();
bool purge_block_data_from_blockchain(const block& b, size_t processed_tx_count);

View File

@ -99,7 +99,11 @@ namespace cryptonote {
assert(current_block_size < std::numeric_limits<uint32_t>::max());
uint64_t product_hi;
uint64_t product_lo = mul128(base_reward, current_block_size * (2 * median_size - current_block_size), &product_hi);
// BUGFIX: 32-bit saturation bug (e.g. ARM7), the result was being
// treated as 32-bit by default.
uint64_t multiplicand = 2 * median_size - current_block_size;
multiplicand *= current_block_size;
uint64_t product_lo = mul128(base_reward, multiplicand, &product_hi);
uint64_t reward_hi;
uint64_t reward_lo;

View File

@ -46,7 +46,7 @@ using namespace epee;
#include "cryptonote_core/checkpoints_create.h"
#include "blockchain_db/blockchain_db.h"
#include "blockchain_db/lmdb/db_lmdb.h"
#ifndef STATICLIB
#if defined(BERKELEY_DB)
#include "blockchain_db/berkeleydb/db_bdb.h"
#endif
@ -212,18 +212,27 @@ namespace cryptonote
#if BLOCKCHAIN_DB == DB_LMDB
std::string db_type = command_line::get_arg(vm, daemon_args::arg_db_type);
std::string db_sync_mode = command_line::get_arg(vm, daemon_args::arg_db_sync_mode);
bool fast_sync = command_line::get_arg(vm, daemon_args::arg_fast_block_sync) != 0;
uint64_t blocks_threads = command_line::get_arg(vm, daemon_args::arg_prep_blocks_threads);
BlockchainDB* db = nullptr;
uint64_t BDB_FAST_MODE = 0;
uint64_t BDB_FASTEST_MODE = 0;
uint64_t BDB_SAFE_MODE = 0;
if (db_type == "lmdb")
{
db = new BlockchainLMDB();
}
else if (db_type == "berkeley")
{
#ifndef STATICLIB
#if defined(BERKELEY_DB)
db = new BlockchainBDB();
BDB_FAST_MODE = DB_TXN_WRITE_NOSYNC;
BDB_FASTEST_MODE = DB_TXN_NOSYNC;
BDB_SAFE_MODE = DB_TXN_SYNC;
#else
LOG_ERROR("BlockchainBDB not supported on STATIC builds");
LOG_ERROR("BerkeleyDB support disabled.");
return false;
#endif
}
@ -240,9 +249,71 @@ namespace cryptonote
LOG_PRINT_L0("Loading blockchain from folder " << folder.string() << " ...");
const std::string filename = folder.string();
// temporarily default to fastest:async:1000
blockchain_db_sync_mode sync_mode = db_async;
uint64_t blocks_per_sync = 1000;
try
{
db->open(filename);
uint64_t db_flags = 0;
bool islmdb = db_type == "lmdb";
std::vector<std::string> options;
boost::trim(db_sync_mode);
boost::split(options, db_sync_mode, boost::is_any_of(" :"));
for(const auto &option : options)
LOG_PRINT_L0("option: " << option);
// temporarily default to fastest:async:1000
uint64_t DEFAULT_FLAGS = islmdb ? MDB_WRITEMAP | MDB_MAPASYNC | MDB_NORDAHEAD | MDB_NOMETASYNC | MDB_NOSYNC :
BDB_FASTEST_MODE;
if(options.size() == 0)
{
// temporarily default to fastest:async:1000
db_flags = DEFAULT_FLAGS;
}
bool safemode = false;
if(options.size() >= 1)
{
if(options[0] == "safe")
{
safemode = true;
db_flags = islmdb ? MDB_NORDAHEAD : BDB_SAFE_MODE;
sync_mode = db_nosync;
}
else if(options[0] == "fast")
db_flags = islmdb ? MDB_NOMETASYNC | MDB_NOSYNC | MDB_NORDAHEAD : BDB_FAST_MODE;
else if(options[0] == "fastest")
db_flags = islmdb ? MDB_WRITEMAP | MDB_MAPASYNC | MDB_NORDAHEAD | MDB_NOMETASYNC | MDB_NOSYNC : BDB_FASTEST_MODE;
else
db_flags = DEFAULT_FLAGS;
}
if(options.size() >= 2 && !safemode)
{
if(options[1] == "sync")
sync_mode = db_sync;
else if(options[1] == "async")
sync_mode = db_async;
}
if(options.size() >= 3 && !safemode)
{
blocks_per_sync = atoll(options[2].c_str());
if(blocks_per_sync > 5000)
blocks_per_sync = 5000;
if(blocks_per_sync == 0)
blocks_per_sync = 1;
}
bool auto_remove_logs = command_line::get_arg(vm, daemon_args::arg_db_auto_remove_logs) != 0;
db->set_auto_remove_logs(auto_remove_logs);
db->open(filename, db_flags);
if(!db->m_open)
return false;
}
catch (const DB_ERROR& e)
{
@ -250,7 +321,13 @@ namespace cryptonote
return false;
}
m_blockchain_storage.set_user_options(blocks_threads,
blocks_per_sync, sync_mode, fast_sync);
r = m_blockchain_storage.init(db, m_testnet);
bool show_time_stats = command_line::get_arg(vm, daemon_args::arg_show_time_stats) != 0;
m_blockchain_storage.set_show_time_stats(show_time_stats);
#else
r = m_blockchain_storage.init(m_config_folder, m_testnet);
#endif
@ -587,6 +664,25 @@ namespace cryptonote
{
return m_blockchain_storage.add_new_block(b, bvc);
}
//-----------------------------------------------------------------------------------------------
bool core::prepare_handle_incoming_blocks(const std::list<block_complete_entry> &blocks)
{
#if BLOCKCHAIN_DB == DB_LMDB
m_blockchain_storage.prepare_handle_incoming_blocks(blocks);
#endif
return true;
}
//-----------------------------------------------------------------------------------------------
bool core::cleanup_handle_incoming_blocks(bool force_sync)
{
#if BLOCKCHAIN_DB == DB_LMDB
m_blockchain_storage.cleanup_handle_incoming_blocks(force_sync);
#endif
return true;
}
//-----------------------------------------------------------------------------------------------
bool core::handle_incoming_block(const blobdata& block_blob, block_verification_context& bvc, bool update_miner_blocktemplate)
{
@ -678,7 +774,8 @@ namespace cryptonote
return m_blockchain_storage.get_block_id_by_height(height);
}
//-----------------------------------------------------------------------------------------------
bool core::get_block_by_hash(const crypto::hash &h, block &blk) {
bool core::get_block_by_hash(const crypto::hash &h, block &blk)
{
return m_blockchain_storage.get_block_by_hash(h, blk);
}
//-----------------------------------------------------------------------------------------------
@ -723,11 +820,13 @@ namespace cryptonote
return true;
}
//-----------------------------------------------------------------------------------------------
void core::set_target_blockchain_height(uint64_t target_blockchain_height) {
void core::set_target_blockchain_height(uint64_t target_blockchain_height)
{
m_target_blockchain_height = target_blockchain_height;
}
//-----------------------------------------------------------------------------------------------
uint64_t core::get_target_blockchain_height() const {
uint64_t core::get_target_blockchain_height() const
{
return m_target_blockchain_height;
}
//-----------------------------------------------------------------------------------------------

View File

@ -66,6 +66,9 @@ namespace cryptonote
bool on_idle();
bool handle_incoming_tx(const blobdata& tx_blob, tx_verification_context& tvc, bool keeped_by_block);
bool handle_incoming_block(const blobdata& block_blob, block_verification_context& bvc, bool update_miner_blocktemplate = true);
bool prepare_handle_incoming_blocks(const std::list<block_complete_entry> &blocks);
bool cleanup_handle_incoming_blocks(bool force_sync = false);
bool check_incoming_block_size(const blobdata& block_blob);
i_cryptonote_protocol* get_protocol(){return m_pprotocol;}

View File

@ -45,10 +45,7 @@ namespace cryptonote {
using std::uint64_t;
using std::vector;
#if defined(_MSC_VER) || defined(__MINGW32__)
#include <windows.h>
#include <winnt.h>
#if defined(__x86_64__)
static inline void mul(uint64_t a, uint64_t b, uint64_t &low, uint64_t &high) {
low = mul128(a, b, &high);
}

View File

@ -128,7 +128,11 @@ namespace cryptonote
crypto::hash max_used_block_id = null_hash;
uint64_t max_used_block_height = 0;
#if BLOCKCHAIN_DB == DB_LMDB
bool ch_inp_res = m_blockchain.check_tx_inputs(tx, max_used_block_height, max_used_block_id, kept_by_block);
#else
bool ch_inp_res = m_blockchain.check_tx_inputs(tx, max_used_block_height, max_used_block_id);
#endif
CRITICAL_REGION_LOCAL(m_transactions_lock);
if(!ch_inp_res)
{
@ -203,6 +207,9 @@ namespace cryptonote
bool tx_memory_pool::remove_transaction_keyimages(const transaction& tx)
{
CRITICAL_REGION_LOCAL(m_transactions_lock);
// ND: Speedup
// 1. Move transaction hash calcuation outside of loop. ._.
crypto::hash actual_hash = get_transaction_hash(tx);
BOOST_FOREACH(const txin_v& vi, tx.vin)
{
CHECKED_GET_SPECIFIC_VARIANT(vi, const txin_to_key, txin, false);
@ -211,11 +218,11 @@ namespace cryptonote
<< "transaction id = " << get_transaction_hash(tx));
std::unordered_set<crypto::hash>& key_image_set = it->second;
CHECK_AND_ASSERT_MES(key_image_set.size(), false, "empty key_image set, img=" << txin.k_image << ENDL
<< "transaction id = " << get_transaction_hash(tx));
<< "transaction id = " << actual_hash);
auto it_in_set = key_image_set.find(get_transaction_hash(tx));
auto it_in_set = key_image_set.find(actual_hash);
CHECK_AND_ASSERT_MES(it_in_set != key_image_set.end(), false, "transaction id not found in key_image set, img=" << txin.k_image << ENDL
<< "transaction id = " << get_transaction_hash(tx));
<< "transaction id = " << actual_hash);
key_image_set.erase(it_in_set);
if(!key_image_set.size())
{

View File

@ -315,6 +315,9 @@ namespace cryptonote
if(context.m_state != cryptonote_connection_context::state_normal)
return 1;
std::list<block_complete_entry> blocks;
blocks.push_back(arg.b);
m_core.prepare_handle_incoming_blocks(blocks);
for(auto tx_blob_it = arg.b.txs.begin(); tx_blob_it!=arg.b.txs.end();tx_blob_it++)
{
cryptonote::tx_verification_context tvc = AUTO_VAL_INIT(tvc);
@ -323,6 +326,7 @@ namespace cryptonote
{
LOG_PRINT_CCONTEXT_L1("Block verification failed: transaction verification failed, dropping connection");
m_p2p->drop_connection(context);
m_core.cleanup_handle_incoming_blocks();
return 1;
}
}
@ -331,6 +335,7 @@ namespace cryptonote
block_verification_context bvc = boost::value_initialized<block_verification_context>();
m_core.pause_mine();
m_core.handle_incoming_block(arg.b.block, bvc); // got block from handle_notify_new_block
m_core.cleanup_handle_incoming_blocks(true);
m_core.resume_mine();
if(bvc.m_verifivation_failed)
{
@ -536,7 +541,7 @@ namespace cryptonote
if (m_core.get_test_drop_download() && m_core.get_test_drop_download_height()) { // DISCARD BLOCKS for testing
m_core.prepare_handle_incoming_blocks(arg.blocks);
BOOST_FOREACH(const block_complete_entry& block_entry, arg.blocks)
{
// process transactions
@ -550,6 +555,7 @@ namespace cryptonote
LOG_ERROR_CCONTEXT("transaction verification failed on NOTIFY_RESPONSE_GET_OBJECTS, \r\ntx_id = "
<< epee::string_tools::pod_to_hex(get_blob_hash(tx_blob)) << ", dropping connection");
m_p2p->drop_connection(context);
m_core.cleanup_handle_incoming_blocks();
return 1;
}
}
@ -566,12 +572,14 @@ namespace cryptonote
{
LOG_PRINT_CCONTEXT_L1("Block verification failed, dropping connection");
m_p2p->drop_connection(context);
m_core.cleanup_handle_incoming_blocks();
return 1;
}
if(bvc.m_marked_as_orphaned)
{
LOG_PRINT_CCONTEXT_L1("Block received at sync phase was marked as orphaned, dropping connection");
m_p2p->drop_connection(context);
m_core.cleanup_handle_incoming_blocks();
return 1;
}
@ -582,7 +590,7 @@ namespace cryptonote
epee::net_utils::data_logger::get_instance().add_data("block_processing", 1);
} // each download block
m_core.cleanup_handle_incoming_blocks();
} // if not DISCARD BLOCK

View File

@ -26,6 +26,12 @@
# STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
# THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
set(blocksdat "")
if(APPLE AND PER_BLOCK_CHECKPOINT)
add_custom_command(OUTPUT blocksdat.o COMMAND cd ${CMAKE_CURRENT_SOURCE_DIR} && ${CMAKE_C_COMPILER} -o stub.o -c stub.c COMMAND cd ${CMAKE_CURRENT_SOURCE_DIR} && ld -r -sectcreate __DATA __blocks_dat ../blocks/blocks.dat -o ${CMAKE_CURRENT_BINARY_DIR}/blocksdat.o stub.o)
set(blocksdat "blocksdat.o")
endif()
set(daemon_sources
command_parser_executor.cpp
command_server.cpp
@ -69,7 +75,9 @@ bitmonero_private_headers(daemon
bitmonero_add_executable(daemon
${daemon_sources}
${daemon_headers}
${daemon_private_headers})
${daemon_private_headers}
${blocksdat}
)
target_link_libraries(daemon
LINK_PRIVATE
rpc
@ -90,8 +98,7 @@ target_link_libraries(daemon
${CMAKE_THREAD_LIBS_INIT}
${UPNP_LIBRARIES}
${EXTRA_LIBRARIES})
add_dependencies(daemon
version)
add_dependencies(daemon version)
set_property(TARGET daemon
PROPERTY
OUTPUT_NAME "bitmonerod")

View File

@ -75,7 +75,32 @@ namespace daemon_args
, "Specify database type"
, "lmdb"
};
const command_line::arg_descriptor<uint64_t> arg_prep_blocks_threads = {
"prep-blocks-threads"
, "Max number of threads to use when preparing block hashes in groups."
, 4
};
const command_line::arg_descriptor<uint64_t> arg_fast_block_sync = {
"fast-block-sync"
, "Test fast block-sync option using temporarily embedded known block hashes."
, 1
};
const command_line::arg_descriptor<uint64_t> arg_show_time_stats = {
"show-time-stats"
, "Show time-stats when processing blocks/txs and disk synchronization."
, 1
};
const command_line::arg_descriptor<uint64_t> arg_db_auto_remove_logs = {
"db-auto-remove-logs"
, "For BerkeleyDB only. Remove transactions logs automatically."
, 1
};
const command_line::arg_descriptor<std::string> arg_db_sync_mode = {
"db-sync-mode"
, "Specify sync option, using format [safe|fast|fastest]:[sync|async]:[nblocks_per_sync]."
, "fastest:async:1000"
};
;
} // namespace daemon_args
#endif // DAEMON_COMMAND_LINE_ARGS_H

View File

@ -86,6 +86,12 @@ int main(int argc, char const * argv[])
command_line::add_arg(core_settings, daemon_args::arg_testnet_on);
command_line::add_arg(core_settings, daemon_args::arg_dns_checkpoints);
command_line::add_arg(core_settings, daemon_args::arg_db_type);
command_line::add_arg(core_settings, daemon_args::arg_prep_blocks_threads);
command_line::add_arg(core_settings, daemon_args::arg_fast_block_sync);
command_line::add_arg(core_settings, daemon_args::arg_db_sync_mode);
command_line::add_arg(core_settings, daemon_args::arg_show_time_stats);
command_line::add_arg(core_settings, daemon_args::arg_db_auto_remove_logs);
daemonizer::init_options(hidden_options, visible_options);
daemonize::t_executor::init_options(core_settings);

View File

@ -209,11 +209,12 @@ namespace nodetool
bool set_rate_up_limit(const boost::program_options::variables_map& vm, int64_t limit);
bool set_rate_down_limit(const boost::program_options::variables_map& vm, int64_t limit);
bool set_rate_limit(const boost::program_options::variables_map& vm, uint64_t limit);
bool set_rate_limit(const boost::program_options::variables_map& vm, int64_t limit);
void kill() { ///< will be called e.g. from deinit()
_info("Killing the net_node");
is_closing = true;
if(mPeersLoggerThread != nullptr)
mPeersLoggerThread->join(); // make sure the thread finishes
_info("Joined extra background net_node threads");
}

View File

@ -67,6 +67,8 @@ namespace nodetool
{
namespace
{
const int64_t default_limit_up = 2048;
const int64_t default_limit_down = 8192;
const command_line::arg_descriptor<std::string> arg_p2p_bind_ip = {"p2p-bind-ip", "Interface for p2p network protocol", "0.0.0.0"};
const command_line::arg_descriptor<std::string> arg_p2p_bind_port = {
"p2p-bind-port"
@ -93,7 +95,7 @@ namespace nodetool
const command_line::arg_descriptor<int64_t> arg_limit_rate_up = {"limit-rate-up", "set limit-rate-up [kB/s]", -1};
const command_line::arg_descriptor<int64_t> arg_limit_rate_down = {"limit-rate-down", "set limit-rate-down [kB/s]", -1};
const command_line::arg_descriptor<uint64_t> arg_limit_rate = {"limit-rate", "set limit-rate [kB/s]", 128};
const command_line::arg_descriptor<int64_t> arg_limit_rate = {"limit-rate", "set limit-rate [kB/s]", -1};
const command_line::arg_descriptor<bool> arg_save_graph = {"save-graph", "Save data for dr monero", false};
}
@ -1446,7 +1448,7 @@ namespace nodetool
this->islimitup=true;
if (limit==-1) {
limit=128;
limit=default_limit_up;
this->islimitup=false;
}
@ -1461,7 +1463,7 @@ namespace nodetool
{
this->islimitdown=true;
if(limit==-1) {
limit=128;
limit=default_limit_down;
this->islimitdown=false;
}
limit *= 1024;
@ -1471,19 +1473,31 @@ namespace nodetool
}
template<class t_payload_net_handler>
bool node_server<t_payload_net_handler>::set_rate_limit(const boost::program_options::variables_map& vm, uint64_t limit)
bool node_server<t_payload_net_handler>::set_rate_limit(const boost::program_options::variables_map& vm, int64_t limit)
{
int64_t limit_up = 0;
int64_t limit_down = 0;
if(limit == -1)
{
limit_up = default_limit_up * 1024;
limit_down = default_limit_down * 1024;
}
else
{
limit_up = limit * 1024;
limit_down = limit * 1024;
}
limit *= 1024;
if(this->islimitdown==false && this->islimitup==false) {
epee::net_utils::connection<epee::levin::async_protocol_handler<p2p_connection_context> >::set_rate_up_limit( limit );
epee::net_utils::connection<epee::levin::async_protocol_handler<p2p_connection_context> >::set_rate_down_limit( limit );
LOG_PRINT_L0("Set limit to " << limit/1024 << " kB/s");
epee::net_utils::connection<epee::levin::async_protocol_handler<p2p_connection_context> >::set_rate_up_limit(limit_up);
epee::net_utils::connection<epee::levin::async_protocol_handler<p2p_connection_context> >::set_rate_down_limit(limit_down);
}
else if(this->islimitdown==false && this->islimitup==true ) {
epee::net_utils::connection<epee::levin::async_protocol_handler<p2p_connection_context> >::set_rate_down_limit( limit );
epee::net_utils::connection<epee::levin::async_protocol_handler<p2p_connection_context> >::set_rate_down_limit(limit_down);
}
else if(this->islimitdown==true && this->islimitup==false ) {
epee::net_utils::connection<epee::levin::async_protocol_handler<p2p_connection_context> >::set_rate_up_limit( limit );
epee::net_utils::connection<epee::levin::async_protocol_handler<p2p_connection_context> >::set_rate_up_limit(limit_up);
}
return true;

View File

@ -85,5 +85,7 @@ namespace tests
cryptonote::blockchain_storage &get_blockchain_storage() { throw std::runtime_error("Called invalid member function: please never call get_blockchain_storage on the TESTING class proxy_core."); }
bool get_test_drop_download() {return true;}
bool get_test_drop_download_height() {return true;}
bool prepare_handle_incoming_blocks(const std::list<cryptonote::block_complete_entry> &blocks) { return true; }
bool cleanup_handle_incoming_blocks(bool force_sync = false) { return true; }
};
}