std.digest

This module describes the digest APIs used in Phobos. All digests follow these APIs. Additionally, this module contains useful helper methods which can be used with every digest type.

Category	Functions
Template API	isDigest DigestType hasPeek hasBlockSize ExampleDigest digest hexDigest makeDigest
OOP API	Digest
Helper functions	toHexString secureEqual
Implementation helpers	digestLength WrapperDigest

APIs

There are two APIs for digests: The template API and the OOP API. The template API uses structs and template helpers like isDigest. The OOP API implements digests as classes inheriting the Digest interface. All digests are named so that the template API struct is called "x" and the OOP API class is called "xDigest". For example we have MD5 <--> MD5Digest, CRC32 <--> CRC32Digest, etc.

The template API is slightly more efficient. It does not have to allocate memory dynamically, all memory is allocated on the stack. The OOP API has to allocate in the finish method if no buffer was provided. If you provide a buffer to the OOP APIs finish function, it doesn't allocate, but the Digest classes still have to be created using new which allocates them using the GC. The OOP API is useful to change the digest function and/or digest backend at 'runtime'. The benefit here is that switching e.g. Phobos MD5Digest and an OpenSSLMD5Digest implementation is ABI compatible. If just one specific digest type and backend is needed, the template API is usually a good fit. In this simplest case, the template API can even be used without templates: Just use the "x" structs directly.

License:

Boost License 1.0.

Authors:

Johannes Pfau

Source

std/digest/package.d

CTFE

Digests do not work in CTFE

TODO

Digesting single bits (as opposed to bytes) is not implemented. This will be done as another template constraint helper (hasBitDigesting!T) and an additional interface (BitDigest)

Examples:

import std.digest.crc;

//Simple example
char[8] hexHash = hexDigest!CRC32("The quick brown fox jumps over the lazy dog");
writeln(hexHash); // "39A34F41"

//Simple example, using the API manually
CRC32 context = makeDigest!CRC32();
context.put(cast(ubyte[])"The quick brown fox jumps over the lazy dog");
ubyte[4] hash = context.finish();
writeln(toHexString(hash)); // "39A34F41"

Examples:

//Generating the hashes of a file, idiomatic D way
import std.digest.crc, std.digest.md, std.digest.sha;
import std.stdio;

// Digests a file and prints the result.
void digestFile(Hash)(string filename)
if (isDigest!Hash)
{
    auto file = File(filename);
    auto result = digest!Hash(file.byChunk(4096 * 1024));
    writefln("%s (%s) = %s", Hash.stringof, filename, toHexString(result));
}

void main(string[] args)
{
    foreach (name; args[1 .. $])
    {
        digestFile!MD5(name);
        digestFile!SHA1(name);
        digestFile!CRC32(name);
    }
}

Examples:

//Generating the hashes of a file using the template API
import std.digest.crc, std.digest.md, std.digest.sha;
import std.stdio;
// Digests a file and prints the result.
void digestFile(Hash)(ref Hash hash, string filename)
if (isDigest!Hash)
{
    File file = File(filename);

    //As digests imlement OutputRange, we could use std.algorithm.copy
    //Let's do it manually for now
    foreach (buffer; file.byChunk(4096 * 1024))
        hash.put(buffer);

    auto result = hash.finish();
    writefln("%s (%s) = %s", Hash.stringof, filename, toHexString(result));
}

void uMain(string[] args)
{
    MD5 md5;
    SHA1 sha1;
    CRC32 crc32;

    md5.start();
    sha1.start();
    crc32.start();

    foreach (arg; args[1 .. $])
    {
        digestFile(md5, arg);
        digestFile(sha1, arg);
        digestFile(crc32, arg);
    }
}

Examples:

import std.digest.crc, std.digest.md, std.digest.sha;
import std.stdio;

// Digests a file and prints the result.
void digestFile(Digest hash, string filename)
{
    File file = File(filename);

    //As digests implement OutputRange, we could use std.algorithm.copy
    //Let's do it manually for now
    foreach (buffer; file.byChunk(4096 * 1024))
      hash.put(buffer);

    ubyte[] result = hash.finish();
    writefln("%s (%s) = %s", typeid(hash).toString(), filename, toHexString(result));
}

void umain(string[] args)
{
    auto md5 = new MD5Digest();
    auto sha1 = new SHA1Digest();
    auto crc32 = new CRC32Digest();

    foreach (arg; args[1 .. $])
    {
      digestFile(md5, arg);
      digestFile(sha1, arg);
      digestFile(crc32, arg);
    }
}

struct ExampleDigest;

This documents the general structure of a Digest in the template API. All digest implementations should implement the following members and therefore pass the isDigest test.

Note

• A digest must be a struct (value type) to pass the isDigest test.
• A digest passing the isDigest test is always an OutputRange

Examples:

//Using the OutputRange feature
import std.algorithm.mutation : copy;
import std.digest.md;
import std.range : repeat;

auto oneMillionRange = repeat!ubyte(cast(ubyte)'a', 1000000);
auto ctx = makeDigest!MD5();
copy(oneMillionRange, &ctx); //Note: You must pass a pointer to copy!
writeln(ctx.finish().toHexString()); // "7707D6AE4E027C70EEA2A935C2296F21"

@trusted void put(scope const(ubyte)[] data...);

Use this to feed the digest with data. Also implements the std.range.primitives.isOutputRange interface for ubyte and const(ubyte)[]. The following usages of put must work for any type which passes isDigest:

Example

ExampleDigest dig;
dig.put(cast(ubyte) 0); //single ubyte
dig.put(cast(ubyte) 0, cast(ubyte) 0); //variadic
ubyte[10] buf;
dig.put(buf); //buffer

@trusted void start();

This function is used to (re)initialize the digest. It must be called before using the digest and it also works as a 'reset' function if the digest has already processed data.

@trusted ubyte[16] finish();

The finish function returns the final hash sum and resets the Digest.

Note

The actual type returned by finish depends on the digest implementation. ubyte[16] is just used as an example. It is guaranteed that the type is a static array of ubytes.

• Use DigestType to obtain the actual return type.
• Use digestLength to obtain the length of the ubyte array.

enum bool isDigest(T);

Use this to check if a type is a digest. See ExampleDigest to see what a type must provide to pass this check.

Note

This is very useful as a template constraint (see examples)

Bugs:

• Does not yet verify that put takes scope parameters.
• Should check that finish() returns a ubyte[num] array

Examples:

import std.digest.crc;
static assert(isDigest!CRC32);

Examples:

import std.digest.crc;
void myFunction(T)()
if (isDigest!T)
{
    T dig;
    dig.start();
    auto result = dig.finish();
}
myFunction!CRC32();

template DigestType(T)

Use this template to get the type which is returned by a digest's finish method.

Examples:

import std.digest.crc;
assert(is(DigestType!(CRC32) == ubyte[4]));

Examples:

import std.digest.crc;
CRC32 dig;
dig.start();
DigestType!CRC32 result = dig.finish();

enum bool hasPeek(T);

Used to check if a digest supports the peek method. Peek has exactly the same function signatures as finish, but it doesn't reset the digest's internal state.

Note

• This is very useful as a template constraint (see examples)
• This also checks if T passes isDigest

Examples:

import std.digest.crc, std.digest.md;
assert(!hasPeek!(MD5));
assert(hasPeek!CRC32);

Examples:

import std.digest.crc;
void myFunction(T)()
if (hasPeek!T)
{
    T dig;
    dig.start();
    auto result = dig.peek();
}
myFunction!CRC32();

template hasBlockSize(T) if (isDigest!T)

Checks whether the digest has a blockSize member, which contains the digest's internal block size in bits. It is primarily used by std.digest.hmac.HMAC.

Examples:

import std.digest.hmac, std.digest.md;
static assert(hasBlockSize!MD5        && MD5.blockSize      == 512);
static assert(hasBlockSize!(HMAC!MD5) && HMAC!MD5.blockSize == 512);

DigestType!Hash digest(Hash, Range)(auto ref Range range)
Constraints: if (!isArray!Range && isDigestibleRange!Range);

This is a convenience function to calculate a hash using the template API. Every digest passing the isDigest test can be used with this function.

Parameters:

Range range

an InputRange with ElementType ubyte, ubyte[] or ubyte[num]

Examples:

import std.digest.md;
import std.range : repeat;
auto testRange = repeat!ubyte(cast(ubyte)'a', 100);
auto md5 = digest!MD5(testRange);

DigestType!Hash digest(Hash, T...)(scope const T data)
Constraints: if (allSatisfy!(isArray, typeof(data)));

This overload of the digest function handles arrays.

Parameters:

T data

one or more arrays of any type

Examples:

import std.digest.crc, std.digest.md, std.digest.sha;
auto md5   = digest!MD5(  "The quick brown fox jumps over the lazy dog");
auto sha1  = digest!SHA1( "The quick brown fox jumps over the lazy dog");
auto crc32 = digest!CRC32("The quick brown fox jumps over the lazy dog");
writeln(toHexString(crc32)); // "39A34F41"

Examples:

import std.digest.crc;
auto crc32 = digest!CRC32("The quick ", "brown ", "fox jumps over the lazy dog");
writeln(toHexString(crc32)); // "39A34F41"

char[digestLength!Hash * 2] hexDigest(Hash, Order order = Order.increasing, Range)(ref Range range)
Constraints: if (!isArray!Range && isDigestibleRange!Range);

This is a convenience function similar to digest, but it returns the string representation of the hash. Every digest passing the isDigest test can be used with this function.

Parameters:

order	the order in which the bytes are processed (see toHexString)
Range range	an InputRange with ElementType ubyte, ubyte[] or ubyte[num]

Examples:

import std.digest.md;
import std.range : repeat;
auto testRange = repeat!ubyte(cast(ubyte)'a', 100);
writeln(hexDigest!MD5(testRange)); // "36A92CC94A9E0FA21F625F8BFB007ADF"

char[digestLength!Hash * 2] hexDigest(Hash, Order order = Order.increasing, T...)(scope const T data)
Constraints: if (allSatisfy!(isArray, typeof(data)));

This overload of the hexDigest function handles arrays.

Parameters:

order	the order in which the bytes are processed (see toHexString)
T data	one or more arrays of any type

Examples:

import std.digest.crc;
// "414FA339"
writeln(hexDigest!(CRC32, Order.decreasing)("The quick brown fox jumps over the lazy dog"));

Examples:

import std.digest.crc;
// "414FA339"
writeln(hexDigest!(CRC32, Order.decreasing)("The quick ", "brown ", "fox jumps over the lazy dog"));

Hash makeDigest(Hash)();

This is a convenience function which returns an initialized digest, so it's not necessary to call start manually.

Examples:

import std.digest.md;
auto md5 = makeDigest!MD5();
md5.put(0);
writeln(toHexString(md5.finish())); // "93B885ADFE0DA089CDF634904FD59F71"

interface Digest;

This describes the OOP API. To understand when to use the template API and when to use the OOP API, see the module documentation at the top of this page.

The Digest interface is the base interface which is implemented by all digests.

Note

A Digest implementation is always an OutputRange

Examples:

//Using the OutputRange feature
import std.algorithm.mutation : copy;
import std.digest.md;
import std.range : repeat;

auto oneMillionRange = repeat!ubyte(cast(ubyte)'a', 1000000);
auto ctx = new MD5Digest();
copy(oneMillionRange, ctx);
writeln(ctx.finish().toHexString()); // "7707D6AE4E027C70EEA2A935C2296F21"

Examples:

import std.digest.crc, std.digest.md, std.digest.sha;
ubyte[] md5   = (new MD5Digest()).digest("The quick brown fox jumps over the lazy dog");
ubyte[] sha1  = (new SHA1Digest()).digest("The quick brown fox jumps over the lazy dog");
ubyte[] crc32 = (new CRC32Digest()).digest("The quick brown fox jumps over the lazy dog");
writeln(crcHexString(crc32)); // "414FA339"

Examples:

import std.digest.crc;
ubyte[] crc32 = (new CRC32Digest()).digest("The quick ", "brown ", "fox jumps over the lazy dog");
writeln(crcHexString(crc32)); // "414FA339"

Examples:

void test(Digest dig)
{
    dig.put(cast(ubyte) 0); //single ubyte
    dig.put(cast(ubyte) 0, cast(ubyte) 0); //variadic
    ubyte[10] buf;
    dig.put(buf); //buffer
}

abstract nothrow @trusted void put(scope const(ubyte)[] data...);

Use this to feed the digest with data. Also implements the std.range.primitives.isOutputRange interface for ubyte and const(ubyte)[].

Example

void test(Digest dig)
{
    dig.put(cast(ubyte) 0); //single ubyte
    dig.put(cast(ubyte) 0, cast(ubyte) 0); //variadic
    ubyte[10] buf;
    dig.put(buf); //buffer
}

abstract nothrow @trusted void reset();

Resets the internal state of the digest.

Note

finish calls this internally, so it's not necessary to call reset manually after a call to finish.

abstract const nothrow @property @trusted size_t length();

This is the length in bytes of the hash value which is returned by finish. It's also the required size of a buffer passed to finish.

abstract nothrow @trusted ubyte[] finish();

abstract nothrow ubyte[] finish(ubyte[] buf);

The finish function returns the hash value. It takes an optional buffer to copy the data into. If a buffer is passed, it must be at least length bytes big.

final nothrow @trusted ubyte[] digest(scope const(void[])[] data...);

This is a convenience function to calculate the hash of a value using the OOP API.

enum Order: bool;

See toHexString

Examples:

import std.digest.crc : CRC32;

auto crc32 = digest!CRC32("The quick ", "brown ", "fox jumps over the lazy dog");
writeln(crc32.toHexString!(Order.decreasing)); // "414FA339"
writeln(crc32.toHexString!(LetterCase.lower, Order.decreasing)); // "414fa339"

increasing

decreasing

char[num * 2] toHexString(Order order = Order.increasing, size_t num, LetterCase letterCase = LetterCase.upper)(const ubyte[num] digest);

char[num * 2] toHexString(LetterCase letterCase, Order order = Order.increasing, size_t num)(in ubyte[num] digest);

string toHexString(Order order = Order.increasing, LetterCase letterCase = LetterCase.upper)(in ubyte[] digest);

string toHexString(LetterCase letterCase, Order order = Order.increasing)(in ubyte[] digest);

Used to convert a hash value (a static or dynamic array of ubytes) to a string. Can be used with the OOP and with the template API.

The additional order parameter can be used to specify the order of the input data. By default the data is processed in increasing order, starting at index 0. To process it in the opposite order, pass Order.decreasing as a parameter.

The additional letterCase parameter can be used to specify the case of the output data. By default the output is in upper case. To change it to the lower case pass LetterCase.lower as a parameter.

Note

The function overloads returning a string allocate their return values using the GC. The versions returning static arrays use pass-by-value for the return value, effectively avoiding dynamic allocation.

Examples:

import std.digest.crc;
//Test with template API:
auto crc32 = digest!CRC32("The quick ", "brown ", "fox jumps over the lazy dog");
//Lower case variant:
writeln(toHexString!(LetterCase.lower)(crc32)); // "39a34f41"
//Usually CRCs are printed in this order, though:
writeln(toHexString!(Order.decreasing)(crc32)); // "414FA339"
writeln(toHexString!(LetterCase.lower, Order.decreasing)(crc32)); // "414fa339"

Examples:

import std.digest.crc;
// With OOP API
auto crc32 = (new CRC32Digest()).digest("The quick ", "brown ", "fox jumps over the lazy dog");
//Usually CRCs are printed in this order, though:
writeln(toHexString!(Order.decreasing)(crc32)); // "414FA339"

class WrapperDigest(T) if (isDigest!T): Digest;

Wraps a template API hash struct into a Digest interface. Modules providing digest implementations will usually provide an alias for this template (e.g. MD5Digest, SHA1Digest, ...).

Examples:

import std.digest.md;
//Simple example
auto hash = new WrapperDigest!MD5();
hash.put(cast(ubyte) 0);
auto result = hash.finish();

Examples:

//using a supplied buffer
import std.digest.md;
ubyte[16] buf;
auto hash = new WrapperDigest!MD5();
hash.put(cast(ubyte) 0);
auto result = hash.finish(buf[]);
//The result is now in result (and in buf). If you pass a buffer which is bigger than
//necessary, result will have the correct length, but buf will still have it's original
//length

this();

Initializes the digest.

nothrow @trusted void put(scope const(ubyte)[] data...);

Use this to feed the digest with data. Also implements the std.range.primitives.isOutputRange interface for ubyte and const(ubyte)[].

nothrow @trusted void reset();

Resets the internal state of the digest.

Note

finish calls this internally, so it's not necessary to call reset manually after a call to finish.

const pure nothrow @property @trusted size_t length();

This is the length in bytes of the hash value which is returned by finish. It's also the required size of a buffer passed to finish.

nothrow ubyte[] finish(ubyte[] buf);

nothrow @trusted ubyte[] finish();

The finish function returns the hash value. It takes an optional buffer to copy the data into. If a buffer is passed, it must have a length at least length bytes.

Example

import std.digest.md;
ubyte[16] buf;
auto hash = new WrapperDigest!MD5();
hash.put(cast(ubyte) 0);
auto result = hash.finish(buf[]);
//The result is now in result (and in buf). If you pass a buffer which is bigger than
//necessary, result will have the correct length, but buf will still have it's original
//length

const @trusted ubyte[] peek(ubyte[] buf);

const @trusted ubyte[] peek();

Works like finish but does not reset the internal state, so it's possible to continue putting data into this WrapperDigest after a call to peek.

These functions are only available if hasPeek!T is true.

bool secureEqual(R1, R2)(R1 r1, R2 r2)
Constraints: if (isInputRange!R1 && isInputRange!R2 && !isInfinite!R1 && !isInfinite!R2 && (isIntegral!(ElementEncodingType!R1) || isSomeChar!(ElementEncodingType!R1)) && !is(CommonType!(ElementEncodingType!R1, ElementEncodingType!R2) == void));

Securely compares two digest representations while protecting against timing attacks. Do not use == to compare digest representations.

The attack happens as follows:

1. An attacker wants to send harmful data to your server, which requires a integrity HMAC SHA1 token signed with a secret.
2. The length of the token is known to be 40 characters long due to its format, so the attacker first sends "0000000000000000000000000000000000000000", then "1000000000000000000000000000000000000000", and so on.
3. The given HMAC token is compared with the expected token using the == string comparison, which returns false as soon as the first wrong element is found. If a wrong element is found, then a rejection is sent back to the sender.
4. Eventually, the attacker is able to determine the first character in the correct token because the sever takes slightly longer to return a rejection. This is due to the comparison moving on to second item in the two arrays, seeing they are different, and then sending the rejection.
5. It may seem like too small of a difference in time for the attacker to notice, but security researchers have shown that differences as small as 20µs can be reliably distinguished even with network inconsistencies.
6. Repeat the process for each character until the attacker has the whole correct token and the server accepts the harmful data. This can be done in a week with the attacker pacing the attack to 10 requests per second with only one client.

This function defends against this attack by always comparing every single item in the array if the two arrays are the same length. Therefore, this function is always Ο(n) for ranges of the same length.

This attack can also be mitigated via rate limiting and banning IPs which have too many rejected requests. However, this does not completely solve the problem, as the attacker could be in control of a bot net. To fully defend against the timing attack, rate limiting, banning IPs, and using this function should be used together.

Parameters:

R1 r1	A digest representation
R2 r2	A digest representation

Returns:

true if both representations are equal, false otherwise

See Also:

The Wikipedia article on timing attacks.

Examples:

import std.digest.hmac : hmac;
import std.digest.sha : SHA1;
import std.string : representation;

// a typical HMAC data integrity verification
auto secret = "A7GZIP6TAQA6OHM7KZ42KB9303CEY0MOV5DD6NTV".representation;
auto data = "data".representation;

auto hex1 = data.hmac!SHA1(secret).toHexString;
auto hex2 = data.hmac!SHA1(secret).toHexString;
auto hex3 = "data1".representation.hmac!SHA1(secret).toHexString;

assert( secureEqual(hex1[], hex2[]));
assert(!secureEqual(hex1[], hex3[]));