Data.ByteString.Builder.Internal

Copyright	(c) 2010 - 2011 Simon Meier
License	BSD3-style (see LICENSE)
Maintainer	Simon Meier <[email protected]>
Stability	unstable, private
Portability	GHC
Safe Haskell	Unsafe
Language	Haskell98

Buffer management

data Buffer Source

A Buffer together with the BufferRange of free bytes. The filled space starts at offset 0 and ends at the first free byte.

data BufferRange Source

A range of bytes in a buffer represented by the pointer to the first byte of the range and the pointer to the first byte after the range.

newBuffer :: Int -> IO Buffer Source

Allocate a new buffer of the given size.

bufferSize :: Buffer -> Int Source

Combined size of the filled and free space in the buffer.

byteStringFromBuffer :: Buffer -> ByteString Source

Convert the filled part of a Buffer to a strict ByteString.

data ChunkIOStream a Source

A stream of chunks that are constructed in the IO monad.

This datatype serves as the common interface for the buffer-by-buffer execution of a BuildStep by buildStepToCIOS. Typical users of this interface are ciosToLazyByteString or iteratee-style libraries like enumerator.

buildStepToCIOS Source

Convert a BuildStep to a ChunkIOStream stream by executing it on Buffers allocated according to the given AllocationStrategy.

ciosUnitToLazyByteString :: AllocationStrategy -> ByteString -> ChunkIOStream () -> ByteString Source

Convert a ChunkIOStream () to a lazy ByteString using unsafeDupablePerformIO.

ciosToLazyByteString :: AllocationStrategy -> (a -> (b, ByteString)) -> ChunkIOStream a -> (b, ByteString) Source

Convert a ChunkIOStream to a lazy tuple of the result and the written ByteString using unsafeDupablePerformIO.

Build signals and steps

data BuildSignal a Source

BuildSignals abstract signals to the caller of a BuildStep. There are three signals: done, bufferFull, or 'insertChunks signals

type BuildStep a = BufferRange -> IO (BuildSignal a) Source

BuildSteps may be called *multiple times* and they must not rise an async. exception.

finalBuildStep :: BuildStep () Source

The final build step that returns the done signal.

done Source

Signal that the current BuildStep is done and has computed a value.

bufferFull Source

Signal that the current buffer is full.

insertChunk Source

Signal that a ByteString chunk should be inserted directly.

fillWithBuildStep Source

Fill a BufferRange using a BuildStep.

The Builder monoid

data Builder Source

Builders denote sequences of bytes. They are Monoids where mempty is the zero-length sequence and mappend is concatenation, which runs in O(1).

builder Source

Construct a Builder. In contrast to BuildSteps, Builders are referentially transparent.

runBuilder Source

Run a Builder with the finalBuildStep.

runBuilderWith Source

Run a Builder.

Primitive combinators

empty :: Builder Source

The Builder denoting a zero-length sequence of bytes. This function is only exported for use in rewriting rules. Use mempty otherwise.

append :: Builder -> Builder -> Builder Source

Concatenate two Builders. This function is only exported for use in rewriting rules. Use mappend otherwise.

flush :: Builder Source

Flush the current buffer. This introduces a chunk boundary.

ensureFree :: Int -> Builder Source

Ensure that there are at least n free bytes for the following Builder.

byteStringCopy :: ByteString -> Builder Source

Construct a Builder that copies the strict ByteString.

Use this function to create Builders from smallish (<= 4kb) ByteStrings or if you need to guarantee that the ByteString is not shared with the chunks generated by the Builder.

byteStringInsert :: ByteString -> Builder Source

Construct a Builder that always inserts the strict ByteString directly as a chunk.

This implies flushing the output buffer, even if it contains just a single byte. You should therefore use byteStringInsert only for large (> 8kb) ByteStrings. Otherwise, the generated chunks are too fragmented to be processed efficiently afterwards.

byteStringThreshold :: Int -> ByteString -> Builder Source

Construct a Builder that copies the strict ByteStrings, if it is smaller than the treshold, and inserts it directly otherwise.

For example, byteStringThreshold 1024 copies strict ByteStrings whose size is less or equal to 1kb, and inserts them directly otherwise. This implies that the average chunk-size of the generated lazy ByteString may be as low as 513 bytes, as there could always be just a single byte between the directly inserted 1025 byte, strict ByteStrings.

lazyByteStringCopy :: ByteString -> Builder Source

Construct a Builder that copies the lazy ByteString.

lazyByteStringInsert :: ByteString -> Builder Source

Construct a Builder that inserts all chunks of the lazy ByteString directly.

lazyByteStringThreshold :: Int -> ByteString -> Builder Source

Construct a Builder that uses the thresholding strategy of byteStringThreshold for each chunk of the lazy ByteString.

shortByteString :: ShortByteString -> Builder Source

Construct a Builder that copies the ShortByteString.

maximalCopySize :: Int Source

The maximal size of a ByteString that is copied. 2 * smallChunkSize to guarantee that on average a chunk is of smallChunkSize.

byteString :: ByteString -> Builder Source

Create a Builder denoting the same sequence of bytes as a strict ByteString. The Builder inserts large ByteStrings directly, but copies small ones to ensure that the generated chunks are large on average.

lazyByteString :: ByteString -> Builder Source

Create a Builder denoting the same sequence of bytes as a lazy ByteString. The Builder inserts large chunks of the lazy ByteString directly, but copies small ones to ensure that the generated chunks are large on average.

Execution

toLazyByteStringWith Source

Heavy inlining. Execute a Builder with custom execution parameters.

This function is inlined despite its heavy code-size to allow fusing with the allocation strategy. For example, the default Builder execution function toLazyByteString is defined as follows.

{-# NOINLINE toLazyByteString #-}
toLazyByteString =
  toLazyByteStringWith (safeStrategy smallChunkSize defaultChunkSize) L.empty

where L.empty is the zero-length lazy ByteString.

In most cases, the parameters used by toLazyByteString give good performance. A sub-performing case of toLazyByteString is executing short (<128 bytes) Builders. In this case, the allocation overhead for the first 4kb buffer and the trimming cost dominate the cost of executing the Builder. You can avoid this problem using

toLazyByteStringWith (safeStrategy 128 smallChunkSize) L.empty

This reduces the allocation and trimming overhead, as all generated ByteStrings fit into the first buffer and there is no trimming required, if more than 64 bytes and less than 128 bytes are written.

data AllocationStrategy Source

A buffer allocation strategy for executing Builders.

safeStrategy Source

Use this strategy for generating lazy ByteStrings whose chunks are likely to survive one garbage collection. This strategy trims buffers that are filled less than half in order to avoid spilling too much memory.

untrimmedStrategy Source

Use this strategy for generating lazy ByteStrings whose chunks are discarded right after they are generated. For example, if you just generate them to write them to a network socket.

customStrategy Source

Create a custom allocation strategy. See the code for safeStrategy and untrimmedStrategy for examples.

smallChunkSize :: Int Source

The recommended chunk size. Currently set to 4k, less the memory management overhead

defaultChunkSize :: Int Source

The chunk size used for I/O. Currently set to 32k, less the memory management overhead

chunkOverhead :: Int Source

The memory management overhead. Currently this is tuned for GHC only.

The Put monad

data Put a Source

A Put action denotes a computation of a value that writes a stream of bytes as a side-effect. Puts are strict in their side-effect; i.e., the stream of bytes will always be written before the computed value is returned.

Puts are a generalization of Builders. The typical use case is the implementation of an encoding that might fail (e.g., an interface to the zlib compression library or the conversion from Base64 encoded data to 8-bit data). For a Builder, the only way to handle and report such a failure is ignore it or call error. In contrast, Put actions are expressive enough to allow reportng and handling such a failure in a pure fashion.

Put () actions are isomorphic to Builders. The functions putBuilder and fromPut convert between these two types. Where possible, you should use Builders, as sequencing them is slightly cheaper than sequencing Puts because they do not carry around a computed value.

put Source

Construct a Put action. In contrast to BuildSteps, Puts are referentially transparent in the sense that sequencing the same Put multiple times yields every time the same value with the same side-effect.

runPut Source

Run a Put.

Execution

putToLazyByteString Source

Execute a Put and return the computed result and the bytes written during the computation as a lazy ByteString.

This function is strict in the computed result and lazy in the writing of the bytes. For example, given

infinitePut = sequence_ (repeat (putBuilder (word8 1))) >> return 0
 

evaluating the expression

fst $ putToLazyByteString infinitePut
 

does not terminate, while evaluating the expression

L.head $ snd $ putToLazyByteString infinitePut
 

does terminate and yields the value 1 :: Word8.

An illustrative example for these strictness properties is the implementation of Base64 decoding (http://en.wikipedia.org/wiki/Base64).

type DecodingState = ...

decodeBase64 :: ByteString -> DecodingState -> Put (Maybe DecodingState)
decodeBase64 = ...
 

The above function takes a strict ByteString supposed to represent Base64 encoded data and the current decoding state. It writes the decoded bytes as the side-effect of the Put and returns the new decoding state, if the decoding of all data in the ByteString was successful. The checking if the strict ByteString represents Base64 encoded data and the actual decoding are fused. This makes the common case, where all data represents Base64 encoded data, more efficient. It also implies that all data must be decoded before the final decoding state can be returned. Puts are intended for implementing such fused checking and decoding/encoding, which is reflected in their strictness properties.

putToLazyByteStringWith Source

Execute a Put with a buffer-allocation strategy and a continuation. For example, putToLazyByteString is implemented as follows.

putToLazyByteString = putToLazyByteStringWith
    (safeStrategy smallChunkSize defaultChunkSize) (x -> (x, L.empty))
 

hPut :: forall a. Handle -> Put a -> IO a Source

Run a Put action redirecting the produced output to a Handle.

The output is buffered using the Handles associated buffer. If this buffer is too small to execute one step of the Put action, then it is replaced with a large enough buffer.

Conversion to and from Builders

putBuilder :: Builder -> Put () Source

Run a Builder as a side-effect of a Put () action.

fromPut :: Put () -> Builder Source

Convert a Put () action to a Builder.