Creation and Interpretation#

Tibs and Mutibs can be constructed from a number of different types. The constructors for both types are identical, so I’ll use Tibs in this section, but it all applies equally well to Mutibs.

A wide range of from_ constructor methods are provided:

Some examples:

# Five bits from a binary string
a = Tibs.from_bin('11001')

# Directly from bytes, bytearray or a memoryview. Useful if creating from a file.
b = Tibs.from_bytes(b'some_bytes')

# Create bits from the truthiness of any iterator.
c = Tibs.from_bools([1, 0, 1, 1, 1])

# Optionally seeded random bits. There's also an option to use the OS's secure generator.
d = Tibs.from_random(1000, seed=b'a_seed')

# From a signed integer. The length can be any value up to 128 bits.
e = Tibs.from_i(-384, 20)

# From an unsigned integer. For whole-byte lengths a byte order can be used.
f = Tibs.from_u(3, 32, byte_order=ByteOrder.Little)

# Floating point values need to have a length of 16, 32 or 64.
g = Tibs.from_f(-0.125, 16)

# Hex, binary and octal strings can be parsed.
h = Tibs.from_string('0xff01, 0b101')

# An efficient way to join many other Tibs together.
i = Tibs.from_joined([a, b, c, d, e, f, g, h])

Promotion to Tibs#

The __init__ method can also be called directly, which is often more convenient, if ever so slightly slower. This will look at the type of object it’s been given and try to promote it to a Tibs by delegating to Tibs.from_string() or Tibs.from_bytes() for strings and bytes-like types, or by accepting strict list/tuple bit patterns directly. So for example

s = Tibs('0xabc')     # Same as Tibs.from_string('0xabc')
t = Tibs([1, 0, 1])   # Strict list/tuple bit-pattern shorthand
u = Tibs(b'hello')    # Same as Tibs.from_bytes(b'hello')

These types (string, strict list/tuple bit patterns and bytes/bytearray/memoryview) can also be automatically promoted to Tibs. Most methods that take a bit sequence will also accept another type they can promote in this way. So, for example, if you want to count how many times the bit pattern 101 appears in a random bit sequence you could write:

t = Tibs.from_random(1_000_000)  # A million random bits
c = t.count(Tibs.from_bools([1, 0, 1]))

but it’s more natural to use automatic promotion

c = t.count([1, 0, 1])

The list and tuple shorthand is intentionally strict: only True, False, 0 and 1 are accepted. Use Tibs.from_bools() when you want to convert arbitrary truthy values or an arbitrary iterator to bits. Use Tibs.from_values() for numeric values with a fixed width, for example Tibs.from_values("u8", [1, 2, 3]). File-like objects such as io.BytesIO are not read implicitly; pass the bytes you want explicitly, for example Tibs.from_bytes(stream.getvalue()), Tibs.from_bytes(stream.read()) or Tibs.from_bytes(stream.getbuffer()). Similarly, pass memoryview(arr) to Tibs.from_bytes() for an array.array only when that raw byte representation is intended.

This automatic promotion of these types to Tibs is quite pervasive in the library, and is generally recommended for conciseness and clarity. An exception is when performance is critical and not having the small overhead of examining the type and dispatching to another method is significant — in this rare case using an explicit from_ method for construction is preferred.

Data representations#

When a Tibs has been created there are multiple ways to interpret the data. These methods start with to_.

A subset of these methods return lossless representations of the exact bit sequence, as a string or bytes.

There is also Tibs.to_padded_bytes(), which appends 0 to 7 zero bits on the right before converting to bytes.

These to_ methods accept optional start and end bit positions when you only want to convert part of the data. With no parameters, the properties are provided as a convenient alias. So instead of using t.to_bin() you can use just t.bin when you want the whole value. For Tibs instances these properties are read-only.

Many of these representations need the data to have a length that’s a correct multiple, for example bytes needs the data length to be a multiple of 8:

>>> t = Tibs('0x4145c')
>>> len(t)
20
>>> t.bin
'01000001010001011100'
>>> t.bytes
Traceback (most recent call last):
...
ValueError: Cannot interpret as bytes - length of 20 is not a multiple of 8 bits.

Note

Tibs can be arbitrary sizes, so lengths are always given in bits and not bytes.

To convert to a bytes object we need to change the length, for example by extending it with four 0 bits:

>>> (t + '0x0').bytes
b'AE\xc0'

Here we used the hex string '0x0' where a Tibs was expected, so it was promoted to a 4-bit Tibs before being used to create a 24-bit value that we could interpret as bytes.

This is a common enough operation that the Tibs.to_padded_bytes() method is provided:

>>> t.to_padded_bytes()
b'AE\xc0'

When you have one of these lossless representations, you can always reconstruct the original Tibs - there is a 1:1 relationship. So t == Tibs.from_bin(t.bin) will always be true.

Data interpretations#

There are also a number of data interpretations that complement the data representations:

Unlike the data representations, the interpretations can have a many-to-one relationship. For example there are many ways for a Tibs to be constructed from the unsigned integer 3:

>>> u1 = Tibs.from_u(3, 5)   # binary 00011
>>> u2 = Tibs.from_u(3, 16)  # binary 00000000_00000011
>>> u3 = Tibs.from_u(3, 16, ByteOrder.Little)  # binary 00000011_00000000

These are three different Tibs, but they can all have equal interpretations:

>>> set([u1, u2, u3])
{Tibs('0b00011'), Tibs('0x0003'), Tibs('0x0300')}
>>> set([u1.u, u2.u, u3.le.u])
{3}

For the value stored in u3 a little-endian View was used - we’ll cover that later.

Repeated fixed-width values#

When every item uses the same fixed-width encoding, Dtype strings make the intent explicit and avoid writing a construction loop yourself. The most common dtype forms are unsigned integers such as "u8" or "u12", signed integers such as "i16", floats such as "f32", and string or byte representations such as "hex16" and "bytes32". Use "bool" for a single Python boolean bit, or "bitsN" when each value is itself a fixed-size bit sequence decoded as Tibs.

Use Tibs.from_value() for one value, or Tibs.from_values() for an iterable of values:

>>> Tibs.from_value("u8", 15)
Tibs('0x0f')
>>> samples = Tibs.from_values("u12", [0, 103, 2048, 4095])
>>> samples.hex
'000067800fff'

The matching interpretation methods decode values back from a bit sequence:

>>> samples.to_values("u12")
[0, 103, 2048, 4095]

For whole-byte numeric values, append _le or _be to the dtype string when byte order matters. These suffixes mean little-endian and big-endian byte order respectively:

>>> Tibs.from_values("u16_le", [0x1234, 0xabcd]).hex
'3412cdab'

Encoding as bytes#

You can use Tibs.to_bytes() and Tibs.from_bytes() to store and retrieve whole-byte bit sequences. If it isn’t whole byte then Tibs.to_padded_bytes() will add zero bits to the end to make up to a whole number of bytes, but this then can’t be round-tripped back to a Tibs as the bit length information has been lost.

For an arbitrary-length Tibs that needs to round-trip through bytes by itself, use Tibs.encode() and Tibs.decode():

>>> t = Tibs('0b001110')
>>> encoded = t.encode()
>>> encoded
b'\xce'
>>> Tibs.decode(encoded)
Tibs('0b001110')

The encoded form stores the bit length as well as the data, so it can reconstruct values whose length is not a multiple of eight. Various compression and storage codec options are provided and it is very efficient at storing short sequences and compressing sparse data.

>>> m = Mutibs.from_zeros(1_000_000_000)
>>> m.set([4, 876, 999_999_999])
>>> encoded = m.encode()
>>> encoded
b'\x0e\x0c\xe6\x00\x00\x00 \x00\x00\xd9\xfa\xe6\xb1\xa4\x80'
>>> t = Tibs.decode(encoded)
>>> t.find_all([1])
[4, 876, 999999999]

The detailed byte format is described in Byte encoding format.

Switching between Tibs and Mutibs#

The other to_ methods are for changing from the immutable Tibs to the mutable Mutibs and vice versa.

If you have a Tibs but want to use one of the in-place modifying methods like Mutibs.reverse(), then you can first use Tibs.to_mutibs() to create a mutable copy:

>>> t = Tibs.from_i(-99, 16)
>>> t.bin
'1111111110011101'
>>> m = t.to_mutibs()
>>> m.reverse()
>>> m.bin
'1011100111111111'

In this simple case it’s better to use the Tibs.reversed() method, which creates and returns a new reversed Tibs. There are a number of these reverse / reversed method pairs which either modify in place (for Mutibs only) or return a new instance.

There are also some methods that are only available on Tibs, which take advantage of its immutable nature. For example the Tibs.chunks_iter() method, which returns an iterator over equal sized chunks of the data, is not available for Mutibs as its data could change while the iterator is active. The list-returning chunks method is available on both Tibs and Mutibs, while for the iterator form we can use Mutibs.to_tibs():

>>> m = Mutibs('0xb2')
>>> m *= 3
>>> for c in m.chunks(12):
...     print(c.hex)
b2b
2b2
>>> for c in m.to_tibs().chunks_iter(12):
...     print(c.hex)
b2b
2b2

There is also the Mutibs.as_tibs() method, which moves the data to a Tibs instead of making a copy. This is more efficient if you don’t need to use the Mutibs any more (as it will be empty after the move).