How to Hash Files in Python Using hashlib: A Step-by-Step Guide

File hashing is a crucial technique in modern computing used to verify the integrity and authenticity of files. A hash is a function that takes an input (or 'message') and returns a fixed-size string of characters, typically a hexadecimal number. This unique representation allows us to ensure that files haven't been altered, corrupted, or tampered with over time.

In this article, we’ll explore how to hash files in Python using the hashlib library. We'll walk through examples for both small and large files, and explain why file hashing is essential for data integrity.

What is File Hashing?

Hashing is a process that converts variable-length data (like a file) into a fixed-length value or "digest." For instance, when you hash a file, the result is a string of characters that represents the file’s content. Even a small change in the file content will result in a completely different hash, making it an excellent tool for file comparison and verification.

Common use cases for file hashing include:

• Verifying File Integrity: After downloading a file or transferring it over a network, you can hash the file and compare the result with the original hash to check if the file has been modified.
• Detecting Duplicate Files: Hashing allows you to identify duplicate files by comparing their hash values.
• Password Security: Hashing is widely used in security to store password hashes and prevent plaintext password storage.

Python’s hashlib Library

Python's hashlib module provides a simple and efficient way to work with hash functions like SHA-256, SHA-1, and MD5. We'll use SHA-256 for this tutorial, which is a widely-used and secure hashing algorithm.

Hashing Small Files

For small files, you can simply read the entire file into memory and generate a hash.

Here’s how to hash a small file using SHA-256:

import hashlib
 
# Initialize the SHA-256 hash object
hasher = hashlib.new('sha256')
 
# Open the file and read its contents
with open('myfile', 'r') as f:
    contents = f.read()
    # Update the hash object with the file contents
    hasher.update(contents.encode())  # Encoding the string to bytes
 
# Print the resulting hash
print(hasher.hexdigest())

Explanation:

• We initialize a SHA-256 hash object using hashlib.new('sha256').
• The file 'myfile' is opened and its contents are read into the contents variable.
• hasher.update(contents.encode()) updates the hash with the file contents (note that the contents are encoded to bytes, as hash functions work with bytes, not strings).
• Finally, we print the hexadecimal representation of the hash with hasher.hexdigest().

Important Note:

This method works well for small files that can easily fit into memory. However, when dealing with large files, it’s better to avoid reading the entire file into memory at once.

Hashing Large Files

For larger files, reading the entire file into memory at once may not be feasible. Instead, you can read the file in chunks (buffers) to generate the hash incrementally. This approach ensures that we don’t run out of memory while processing large files.

Here’s how you can hash large files using a buffer:

import hashlib
 
# Set the buffer size (in bytes)
SIZE = 65536  # 64 KB
 
# Initialize the SHA-256 hash object
hasher = hashlib.new('sha256')
 
# Open the large file and read it in chunks
with open('myfile', 'r') as f:
    # Read the first chunk of the file
    buffer = f.read(SIZE)
    
    # Continue reading in chunks until the entire file is processed
    while len(buffer) > 0:
        hasher.update(buffer.encode())  # Encoding to bytes before updating
        buffer = f.read(SIZE)  # Read the next chunk
 
# Print the resulting hash
print(hasher.hexdigest())

Explanation:

• We define a buffer size of 64 KB (SIZE = 65536).
• The file is opened and read in chunks using f.read(SIZE), and the hasher.update() method is called for each chunk.
• Each chunk is encoded to bytes using .encode() before being passed to the update() method.
• The process continues until the entire file is read and hashed.
• Finally, we print the hexadecimal hash of the entire file.

Why Use Buffering for Large Files?

Buffering allows you to read and hash the file incrementally, without loading the entire file into memory. This is especially important when dealing with large files, as it helps conserve system memory and improves performance.

Verifying File Integrity

Once you have the hash of a file, you can use it to verify the file’s integrity. For example, after downloading a file, you can compare its hash with the hash provided by the source (e.g., a website or a file server). If the hashes match, the file is intact and hasn’t been tampered with.

To verify a file:

1. Get the original hash of the file (e.g., from the website).
2. Hash the downloaded file using the same method.
3. Compare the two hashes: If they match, the file is authentic; otherwise, it may have been corrupted or tampered with.

Here’s an example of how you could verify a file's integrity:

def verify_file_integrity(file_path, expected_hash):
    # Calculate the file's hash
    hasher = hashlib.new('sha256')
    with open(file_path, 'r') as f:
        buffer = f.read(SIZE)
        while len(buffer) > 0:
            hasher.update(buffer.encode())
            buffer = f.read(SIZE)
    
    # Compare the calculated hash with the expected hash
    if hasher.hexdigest() == expected_hash:
        print("File is intact.")
    else:
        print("File has been altered or corrupted.")

Other Hashing Algorithms

Python's hashlib supports various hashing algorithms. While SHA-256 is the most commonly used, you can also use other algorithms such as:

• MD5: Older and faster, but not secure for cryptographic purposes.
• SHA-1: More secure than MD5, but still considered weak.
• SHA-512: Similar to SHA-256 but generates a 512-bit hash.

You can switch to a different algorithm by passing its name to hashlib.new(). For example:

hasher = hashlib.new('sha512')

Conclusion

File hashing is a simple but powerful technique to verify the integrity of files and ensure that data has not been tampered with. Whether you are verifying downloaded files, checking for duplicate files, or implementing secure file storage, hashing is a critical tool in data security.

In this guide, we explored how to hash both small and large files in Python using the hashlib library. We also demonstrated how to verify file integrity by comparing hashes, which is a useful practice for ensuring data authenticity and preventing file corruption.

Now that you understand how file hashing works, you can incorporate it into your own Python projects to enhance file security and integrity.