liblzma (XZ Utils) 5.4.4
|
Handling of .xz Index and related information. More...
Data Structures | |
struct | lzma_index_iter |
Iterator to get information about Blocks and Streams. More... | |
Typedefs | |
typedef struct lzma_index_s | lzma_index |
Opaque data type to hold the Index(es) and other information. | |
Enumerations | |
enum | lzma_index_iter_mode { LZMA_INDEX_ITER_ANY = 0 , LZMA_INDEX_ITER_STREAM = 1 , LZMA_INDEX_ITER_BLOCK = 2 , LZMA_INDEX_ITER_NONEMPTY_BLOCK = 3 } |
Operation mode for lzma_index_iter_next() More... | |
Functions | |
uint64_t | lzma_index_memusage (lzma_vli streams, lzma_vli blocks) lzma_nothrow |
Calculate memory usage of lzma_index. | |
uint64_t | lzma_index_memused (const lzma_index *i) lzma_nothrow |
Calculate the memory usage of an existing lzma_index. | |
lzma_index * | lzma_index_init (const lzma_allocator *allocator) lzma_nothrow |
Allocate and initialize a new lzma_index structure. | |
void | lzma_index_end (lzma_index *i, const lzma_allocator *allocator) lzma_nothrow |
Deallocate lzma_index. | |
lzma_ret | lzma_index_append (lzma_index *i, const lzma_allocator *allocator, lzma_vli unpadded_size, lzma_vli uncompressed_size) lzma_nothrow lzma_attr_warn_unused_result |
Add a new Block to lzma_index. | |
lzma_ret | lzma_index_stream_flags (lzma_index *i, const lzma_stream_flags *stream_flags) lzma_nothrow lzma_attr_warn_unused_result |
Set the Stream Flags. | |
uint32_t | lzma_index_checks (const lzma_index *i) lzma_nothrow lzma_attr_pure |
Get the types of integrity Checks. | |
lzma_ret | lzma_index_stream_padding (lzma_index *i, lzma_vli stream_padding) lzma_nothrow lzma_attr_warn_unused_result |
Set the amount of Stream Padding. | |
lzma_vli | lzma_index_stream_count (const lzma_index *i) lzma_nothrow lzma_attr_pure |
Get the number of Streams. | |
lzma_vli | lzma_index_block_count (const lzma_index *i) lzma_nothrow lzma_attr_pure |
Get the number of Blocks. | |
lzma_vli | lzma_index_size (const lzma_index *i) lzma_nothrow lzma_attr_pure |
Get the size of the Index field as bytes. | |
lzma_vli | lzma_index_stream_size (const lzma_index *i) lzma_nothrow lzma_attr_pure |
Get the total size of the Stream. | |
lzma_vli | lzma_index_total_size (const lzma_index *i) lzma_nothrow lzma_attr_pure |
Get the total size of the Blocks. | |
lzma_vli | lzma_index_file_size (const lzma_index *i) lzma_nothrow lzma_attr_pure |
Get the total size of the file. | |
lzma_vli | lzma_index_uncompressed_size (const lzma_index *i) lzma_nothrow lzma_attr_pure |
Get the uncompressed size of the file. | |
void | lzma_index_iter_init (lzma_index_iter *iter, const lzma_index *i) lzma_nothrow |
Initialize an iterator. | |
void | lzma_index_iter_rewind (lzma_index_iter *iter) lzma_nothrow |
Rewind the iterator. | |
lzma_bool | lzma_index_iter_next (lzma_index_iter *iter, lzma_index_iter_mode mode) lzma_nothrow lzma_attr_warn_unused_result |
Get the next Block or Stream. | |
lzma_bool | lzma_index_iter_locate (lzma_index_iter *iter, lzma_vli target) lzma_nothrow |
Locate a Block. | |
lzma_ret | lzma_index_cat (lzma_index *dest, lzma_index *src, const lzma_allocator *allocator) lzma_nothrow lzma_attr_warn_unused_result |
Concatenate lzma_indexes. | |
lzma_index * | lzma_index_dup (const lzma_index *i, const lzma_allocator *allocator) lzma_nothrow lzma_attr_warn_unused_result |
Duplicate lzma_index. | |
lzma_ret | lzma_index_encoder (lzma_stream *strm, const lzma_index *i) lzma_nothrow lzma_attr_warn_unused_result |
Initialize .xz Index encoder. | |
lzma_ret | lzma_index_decoder (lzma_stream *strm, lzma_index **i, uint64_t memlimit) lzma_nothrow lzma_attr_warn_unused_result |
Initialize .xz Index decoder. | |
lzma_ret | lzma_index_buffer_encode (const lzma_index *i, uint8_t *out, size_t *out_pos, size_t out_size) lzma_nothrow |
Single-call .xz Index encoder. | |
lzma_ret | lzma_index_buffer_decode (lzma_index **i, uint64_t *memlimit, const lzma_allocator *allocator, const uint8_t *in, size_t *in_pos, size_t in_size) lzma_nothrow |
Single-call .xz Index decoder. | |
lzma_ret | lzma_file_info_decoder (lzma_stream *strm, lzma_index **dest_index, uint64_t memlimit, uint64_t file_size) lzma_nothrow |
Initialize a .xz file information decoder. | |
Handling of .xz Index and related information.
typedef struct lzma_index_s lzma_index |
Opaque data type to hold the Index(es) and other information.
lzma_index often holds just one .xz Index and possibly the Stream Flags of the same Stream and size of the Stream Padding field. However, multiple lzma_indexes can be concatenated with lzma_index_cat() and then there may be information about multiple Streams in the same lzma_index.
Notes about thread safety: Only one thread may modify lzma_index at a time. All functions that take non-const pointer to lzma_index modify it. As long as no thread is modifying the lzma_index, getting information from the same lzma_index can be done from multiple threads at the same time with functions that take a const pointer to lzma_index or use lzma_index_iter. The same iterator must be used only by one thread at a time, of course, but there can be as many iterators for the same lzma_index as needed.
enum lzma_index_iter_mode |
Operation mode for lzma_index_iter_next()
Calculate memory usage of lzma_index.
On disk, the size of the Index field depends on both the number of Records stored and the size of the Records (due to variable-length integer encoding). When the Index is kept in lzma_index structure, the memory usage depends only on the number of Records/Blocks stored in the Index(es), and in case of concatenated lzma_indexes, the number of Streams. The size in RAM is almost always significantly bigger than in the encoded form on disk.
This function calculates an approximate amount of memory needed to hold the given number of Streams and Blocks in lzma_index structure. This value may vary between CPU architectures and also between liblzma versions if the internal implementation is modified.
streams | Number of Streams |
blocks | Number of Blocks |
uint64_t lzma_index_memused | ( | const lzma_index * | i | ) |
Calculate the memory usage of an existing lzma_index.
This is a shorthand for lzma_index_memusage(lzma_index_stream_count(i), lzma_index_block_count(i)).
i | Pointer to lzma_index structure |
lzma_index * lzma_index_init | ( | const lzma_allocator * | allocator | ) |
Allocate and initialize a new lzma_index structure.
allocator | lzma_allocator for custom allocator functions. Set to NULL to use malloc() and free(). |
void lzma_index_end | ( | lzma_index * | i, |
const lzma_allocator * | allocator | ||
) |
Deallocate lzma_index.
If i is NULL, this does nothing.
i | Pointer to lzma_index structure to deallocate |
allocator | lzma_allocator for custom allocator functions. Set to NULL to use malloc() and free(). |
lzma_ret lzma_index_append | ( | lzma_index * | i, |
const lzma_allocator * | allocator, | ||
lzma_vli | unpadded_size, | ||
lzma_vli | uncompressed_size | ||
) |
Add a new Block to lzma_index.
i | Pointer to a lzma_index structure |
allocator | lzma_allocator for custom allocator functions. Set to NULL to use malloc() and free(). |
unpadded_size | Unpadded Size of a Block. This can be calculated with lzma_block_unpadded_size() after encoding or decoding the Block. |
uncompressed_size | Uncompressed Size of a Block. This can be taken directly from lzma_block structure after encoding or decoding the Block. |
Appending a new Block does not invalidate iterators. For example, if an iterator was pointing to the end of the lzma_index, after lzma_index_append() it is possible to read the next Block with an existing iterator.
lzma_ret lzma_index_stream_flags | ( | lzma_index * | i, |
const lzma_stream_flags * | stream_flags | ||
) |
Set the Stream Flags.
Set the Stream Flags of the last (and typically the only) Stream in lzma_index. This can be useful when reading information from the lzma_index, because to decode Blocks, knowing the integrity check type is needed.
i | Pointer to lzma_index structure |
stream_flags | Pointer to lzma_stream_flags structure. This is copied into the internal preallocated structure, so the caller doesn't need to keep the flags' data available after calling this function. |
uint32_t lzma_index_checks | ( | const lzma_index * | i | ) |
Get the types of integrity Checks.
If lzma_index_stream_flags() is used to set the Stream Flags for every Stream, lzma_index_checks() can be used to get a bitmask to indicate which Check types have been used. It can be useful e.g. if showing the Check types to the user.
The bitmask is 1 << check_id, e.g. CRC32 is 1 << 1 and SHA-256 is 1 << 10.
i | Pointer to lzma_index structure |
lzma_ret lzma_index_stream_padding | ( | lzma_index * | i, |
lzma_vli | stream_padding | ||
) |
Set the amount of Stream Padding.
Set the amount of Stream Padding of the last (and typically the only) Stream in the lzma_index. This is needed when planning to do random-access reading within multiple concatenated Streams.
By default, the amount of Stream Padding is assumed to be zero bytes.
lzma_vli lzma_index_stream_count | ( | const lzma_index * | i | ) |
Get the number of Streams.
i | Pointer to lzma_index structure |
lzma_vli lzma_index_block_count | ( | const lzma_index * | i | ) |
Get the number of Blocks.
This returns the total number of Blocks in lzma_index. To get number of Blocks in individual Streams, use lzma_index_iter.
i | Pointer to lzma_index structure |
lzma_vli lzma_index_size | ( | const lzma_index * | i | ) |
Get the size of the Index field as bytes.
This is needed to verify the Backward Size field in the Stream Footer.
i | Pointer to lzma_index structure |
lzma_vli lzma_index_stream_size | ( | const lzma_index * | i | ) |
Get the total size of the Stream.
If multiple lzma_indexes have been combined, this works as if the Blocks were in a single Stream. This is useful if you are going to combine Blocks from multiple Streams into a single new Stream.
i | Pointer to lzma_index structure |
lzma_vli lzma_index_total_size | ( | const lzma_index * | i | ) |
Get the total size of the Blocks.
This doesn't include the Stream Header, Stream Footer, Stream Padding, or Index fields.
i | Pointer to lzma_index structure |
lzma_vli lzma_index_file_size | ( | const lzma_index * | i | ) |
Get the total size of the file.
When no lzma_indexes have been combined with lzma_index_cat() and there is no Stream Padding, this function is identical to lzma_index_stream_size(). If multiple lzma_indexes have been combined, this includes also the headers of each separate Stream and the possible Stream Padding fields.
i | Pointer to lzma_index structure |
lzma_vli lzma_index_uncompressed_size | ( | const lzma_index * | i | ) |
Get the uncompressed size of the file.
i | Pointer to lzma_index structure |
void lzma_index_iter_init | ( | lzma_index_iter * | iter, |
const lzma_index * | i | ||
) |
Initialize an iterator.
This function associates the iterator with the given lzma_index, and calls lzma_index_iter_rewind() on the iterator.
This function doesn't allocate any memory, thus there is no lzma_index_iter_end(). The iterator is valid as long as the associated lzma_index is valid, that is, until lzma_index_end() or using it as source in lzma_index_cat(). Specifically, lzma_index doesn't become invalid if new Blocks are added to it with lzma_index_append() or if it is used as the destination in lzma_index_cat().
It is safe to make copies of an initialized lzma_index_iter, for example, to easily restart reading at some particular position.
iter | Pointer to a lzma_index_iter structure |
i | lzma_index to which the iterator will be associated |
void lzma_index_iter_rewind | ( | lzma_index_iter * | iter | ) |
Rewind the iterator.
Rewind the iterator so that next call to lzma_index_iter_next() will return the first Block or Stream.
iter | Pointer to a lzma_index_iter structure |
lzma_bool lzma_index_iter_next | ( | lzma_index_iter * | iter, |
lzma_index_iter_mode | mode | ||
) |
Get the next Block or Stream.
iter | Iterator initialized with lzma_index_iter_init() |
mode | Specify what kind of information the caller wants to get. See lzma_index_iter_mode for details. |
lzma_bool lzma_index_iter_locate | ( | lzma_index_iter * | iter, |
lzma_vli | target | ||
) |
Locate a Block.
If it is possible to seek in the .xz file, it is possible to parse the Index field(s) and use lzma_index_iter_locate() to do random-access reading with granularity of Block size.
If the target is smaller than the uncompressed size of the Stream (can be checked with lzma_index_uncompressed_size()):
If the target is greater than the uncompressed size of the Stream, *iter is not modified.
iter | Iterator that was earlier initialized with lzma_index_iter_init(). |
target | Uncompressed target offset which the caller would like to locate from the Stream |
lzma_ret lzma_index_cat | ( | lzma_index * | dest, |
lzma_index * | src, | ||
const lzma_allocator * | allocator | ||
) |
Concatenate lzma_indexes.
Concatenating lzma_indexes is useful when doing random-access reading in multi-Stream .xz file, or when combining multiple Streams into single Stream.
[out] | dest | lzma_index after which src is appended |
src | lzma_index to be appended after dest. If this function succeeds, the memory allocated for src is freed or moved to be part of dest, and all iterators pointing to src will become invalid. | |
allocator | lzma_allocator for custom allocator functions. Set to NULL to use malloc() and free(). |
lzma_index * lzma_index_dup | ( | const lzma_index * | i, |
const lzma_allocator * | allocator | ||
) |
Duplicate lzma_index.
i | Pointer to lzma_index structure to be duplicated |
allocator | lzma_allocator for custom allocator functions. Set to NULL to use malloc() and free(). |
lzma_ret lzma_index_encoder | ( | lzma_stream * | strm, |
const lzma_index * | i | ||
) |
Initialize .xz Index encoder.
strm | Pointer to properly prepared lzma_stream |
i | Pointer to lzma_index which should be encoded. |
The valid `action' values for lzma_code() are LZMA_RUN and LZMA_FINISH. It is enough to use only one of them (you can choose freely).
lzma_ret lzma_index_decoder | ( | lzma_stream * | strm, |
lzma_index ** | i, | ||
uint64_t | memlimit | ||
) |
Initialize .xz Index decoder.
strm | Pointer to properly prepared lzma_stream | |
[out] | i | The decoded Index will be made available via this pointer. Initially this function will set *i to NULL (the old value is ignored). If decoding succeeds (lzma_code() returns LZMA_STREAM_END), *i will be set to point to a new lzma_index, which the application has to later free with lzma_index_end(). |
memlimit | How much memory the resulting lzma_index is allowed to require. liblzma 5.2.3 and earlier don't allow 0 here and return LZMA_PROG_ERROR; later versions treat 0 as if 1 had been specified. |
Valid `action' arguments to lzma_code() are LZMA_RUN and LZMA_FINISH. There is no need to use LZMA_FINISH, but it's allowed because it may simplify certain types of applications.
lzma_ret lzma_index_buffer_encode | ( | const lzma_index * | i, |
uint8_t * | out, | ||
size_t * | out_pos, | ||
size_t | out_size | ||
) |
Single-call .xz Index encoder.
i | lzma_index to be encoded | |
[out] | out | Beginning of the output buffer |
[out] | out_pos | The next byte will be written to out[*out_pos]. *out_pos is updated only if encoding succeeds. |
out_size | Size of the out buffer; the first byte into which no data is written to is out[out_size]. |
lzma_ret lzma_index_buffer_decode | ( | lzma_index ** | i, |
uint64_t * | memlimit, | ||
const lzma_allocator * | allocator, | ||
const uint8_t * | in, | ||
size_t * | in_pos, | ||
size_t | in_size | ||
) |
Single-call .xz Index decoder.
[out] | i | If decoding succeeds, *i will point to a new lzma_index, which the application has to later free with lzma_index_end(). If an error occurs, *i will be NULL. The old value of *i is always ignored and thus doesn't need to be initialized by the caller. |
[out] | memlimit | Pointer to how much memory the resulting lzma_index is allowed to require. The value pointed by this pointer is modified if and only if LZMA_MEMLIMIT_ERROR is returned. |
allocator | lzma_allocator for custom allocator functions. Set to NULL to use malloc() and free(). | |
in | Beginning of the input buffer | |
in_pos | The next byte will be read from in[*in_pos]. *in_pos is updated only if decoding succeeds. | |
in_size | Size of the input buffer; the first byte that won't be read is in[in_size]. |
lzma_ret lzma_file_info_decoder | ( | lzma_stream * | strm, |
lzma_index ** | dest_index, | ||
uint64_t | memlimit, | ||
uint64_t | file_size | ||
) |
Initialize a .xz file information decoder.
This decoder decodes the Stream Header, Stream Footer, Index, and Stream Padding field(s) from the input .xz file and stores the resulting combined index in *dest_index. This information can be used to get the uncompressed file size with lzma_index_uncompressed_size(*dest_index) or, for example, to implement random access reading by locating the Blocks in the Streams.
To get the required information from the .xz file, lzma_code() may ask the application to seek in the input file by returning LZMA_SEEK_NEEDED and having the target file position specified in lzma_stream.seek_pos. The number of seeks required depends on the input file and how big buffers the application provides. When possible, the decoder will seek backward and forward in the given buffer to avoid useless seek requests. Thus, if the application provides the whole file at once, no external seeking will be required (that is, lzma_code() won't return LZMA_SEEK_NEEDED).
The value in lzma_stream.total_in can be used to estimate how much data liblzma had to read to get the file information. However, due to seeking and the way total_in is updated, the value of total_in will be somewhat inaccurate (a little too big). Thus, total_in is a good estimate but don't expect to see the same exact value for the same file if you change the input buffer size or switch to a different liblzma version.
Valid `action' arguments to lzma_code() are LZMA_RUN and LZMA_FINISH. You only need to use LZMA_RUN; LZMA_FINISH is only supported because it might be convenient for some applications. If you use LZMA_FINISH and if lzma_code() asks the application to seek, remember to reset `action' back to LZMA_RUN unless you hit the end of the file again.
Possible return values from lzma_code():
strm | Pointer to a properly prepared lzma_stream | |
[out] | dest_index | Pointer to a pointer where the decoder will put the decoded lzma_index. The old value of *dest_index is ignored (not freed). |
memlimit | How much memory the resulting lzma_index is allowed to require. Use UINT64_MAX to effectively disable the limiter. | |
file_size | Size of the input .xz file |