How to Calculate Checksums for Large Data or Streams

Ok, so you probably know how to calculate checksums in REBOL. It is easy. For example, a simple checksum is:

sum: checksum data

For a stronger checksum, you can use a secure hashing algorithm, such as SHA1 with code like this:

sum: checksum/secure data

But, what happens if your data is too large to fit in memory, or you are streaming data that needs to be checked on the fly?

The answer is: checksum ports.

A checksum port is like any other REBOL port. Here's a quick example of how to compute an ongoing (streaming) checksum:

sport: open [scheme: 'checksum]

insert sport "this is some text"
insert sport "this is more text"
insert sport "this is the final text"
insert sport ...

update sport
sum: copy sport
close sport

print sum

The update function is required to compute the final checksum value. Without it, the copy function will return none.

The default algorithm for the checksum above is SHA1. The open line above is equivalent to writing:

sport: open [scheme: 'checksum algorithm: 'sha1]

If you want MD5 instead, use this line:

sport: open [scheme: 'checksum algorithm: 'md5]

I hope you find this checksum technique of value for your networking and file operations.

14 Comments

Updated 4-Mar-2024   -   Copyright Carl Sassenrath   -   WWW.REBOL.COM   -   Edit   -   Blogger Source Code