When the disk cache is used, the cache we give back to applications is a
dummy. Therefore, try to use the disk cache blob if we detect a useless
application blob.
Signed-off-by: Hans-Kristian Arntzen <post@arntzen-software.no>
With VKD3D_SHADER_CACHE_PATH, we can add automatic serialization of pipeline
blobs to disk, even for games which do not make any use of GetCachedBlob
of ID3D12PipelineLibrary interfaces. Most applications expect drivers to
have some kind of internal caching.
This is implemented as a system where a disk
thread will manage a private ID3D12PipelineLibrary, and new PSOs are
automatically committed to this library. PSO creation will also consult
this internal pipeline library if applications do not provide their own
blob.
The strategy for updating the cache is based on a read-only cache which
is mmaped from disk, with an exclusive write-only portion for new blobs,
which ensures some degree of safety if there are multiple
concurrent processes using the same cache.
The memory layout of the disk cache is optimized to be very efficient
for appending new blobs, just simple fwrites + fflush.
The format is also robust against sliced files, which solves the problem
where applications tear down without destroying the D3D12 device
properly.
This structure is very similar to Fossilize, and in fact the idea is to
move towards actually using the Fossilize format directly later.
This implementation prepares us for this scenario where e.g. Steam could
potentially manage the vkd3d-proton cache.
The main complication in this implementation is that we have to merge
the read-only and write caches.
Signed-off-by: Hans-Kristian Arntzen <post@arntzen-software.no>
For internal pipeline libraries, we want a somewhat different strategy.
- PSOs are keyed by hash instead of user key.
- We want the option to conditionally store SPIR-V and PSO blobs.
For internal caches, there isn't much of a reason to store PSO blobs
since the disk cache is going to be primed anyways.
Signed-off-by: Hans-Kristian Arntzen <post@arntzen-software.no>
Rather than having to take writer lock on serialize calls from the
outside, we should just take locks when accessing the internal hashmaps
instead.
Signed-off-by: Hans-Kristian Arntzen <post@arntzen-software.no>
If outer code has taken a reader lock, we don't need to lock again.
Also allows a reader lock to go GetSerializedSize + Serialize with one
reader lock.
This will be relevant for magic cache implementation.
Signed-off-by: Hans-Kristian Arntzen <post@arntzen-software.no>
In pipeline libraries, the library holds on to private references of the
libraries so that they can be rapidly loaded on-demand.
This behavior is verifed by API tests.
Signed-off-by: Hans-Kristian Arntzen <post@arntzen-software.no>
Elden Ring does not detect the proper error code and create a new
pipeline library. Instead, create a fresh new library, which works
around the issue.
The game has a pattern of LoadPipeline -> if fail -> CreatePSO ->
StorePipeline. Sometimes, in the same process it will LoadLibrary from
its own cache (could explain some stutters),
so it's very useful to have this either way.
Signed-off-by: Hans-Kristian Arntzen <post@arntzen-software.no>
This became basically a rewrite in the end, and it got too awkward to
split these commits in any meaningful way.
The goals here were primarily to:
- Support serializing SPIR-V and load SPIR-V.
To do this robustly requires a lot more validation and checks to make
sure end up compiling the same SPIR-V that we load from cache.
This is critical for performance when games have primed their pipeline
libraries and expect that loading a PSO should be fast. Without this,
we will hit vkd3d-shader for every PSO, causing very long load times.
- Implement the required validation for mismatched PSO descriptions.
- Rewrite the binary layout of the pipeline library for flexibility
concerns and performance.
If the pipeline library is mmap-ed from disk - which appears to be
the intended use - we only need to scan through the TOC to fully parse
the library contents.
From a flexibility concern, a blob needs to support inlined data,
but a library can use referential links. We introduce separate
hashmaps which store deduplicated SPIR-V and pipeline cache blobs,
which significantly drop memory and storage requirements.
For future improvements, it should be fairly easy to add information
which lets us avoid SPIR-V or pipeline cache data altogether if
relevant changes to Vulkan/drivers are made.
Signed-off-by: Hans-Kristian Arntzen <post@arntzen-software.no>
If we're going to create different SPIR-V files from what the
VkPipelineCache represents, it's meaningless to load it.
Signed-off-by: Hans-Kristian Arntzen <post@arntzen-software.no>
Wraps the D3D12 struct with a pipeline library handle.
This is needed if the blob contains references to external data,
which then needs to be resolved.
Signed-off-by: Hans-Kristian Arntzen <post@arntzen-software.no>
As per MSDN, SetName is just a wrapper around SetPrivateData and a specific GUID.
Some apps and tools will use this to retrieve their name back.
So instead, just forward the name to Vulkan in the SetPrivateData call.
Signed-off-by: Joshua Ashton <joshua@froggi.es>
Version string is used in logging for information purposes, but pipelines blobs and libraries use uint64_t–based commit hash. Using fixed–size integer silences warnings about string length and makes storing build info a little more efficient.
The hash is obtained separately from version string and is shifted to the left by 4 bits if the working tree is dirty.
Signed-off-by: Krzysztof Bogacki <krzysztof.bogacki@leancode.pl>
On systems without extended dynamic state, or for certain pipelines,
it is possible for vk_pso_cache to be VK_NULL_HANDLE, so we need to
check for this during serialization.
Signed-off-by: Philip Rebohle <philip.rebohle@tu-dortmund.de>
This is used extensively by Horizon Zero Dawn, and allows us
to skip the compile screen after the initial first run.
Signed-off-by: Philip Rebohle <philip.rebohle@tu-dortmund.de>
Hans-Kristian Arntzen
Joshua Ashton
Krzysztof Bogacki
Philip Rebohle