The Big Fuzz Theory: Fuzzing Primer

Introduction

Let’s start with how hacks happen!

In most cases, security breaches in softwares are a result of unexpected scenarios that haven't been anticipated while unit testing and therefore do not have written tests for.

Imagine if I were to suggest that it's possible to handle such a unique edge case and compose a single test that could scrutinize nearly all potential scenarios.

In this article series, we will try answering one question at a time, starting with what is fuzz testing. Let's dive right in. 

What is fuzz testing and how can it be applied to smart contracts in Solidity?

I will start with the first half of the question, and we will then connect the dots with the second part by the end of this article.

Well, for starters, there is a formal definition of fuzzing and fuzz testing available, which goes by the book, but that’s not what you guys are here for, right? 🙂

Let me keep it easy and simple for you. While writing tests, you make sure that code coverage should always be 100%, but even when the coverage is 100%, you can never ensure that no bugs have been left in your smart contract code. That’s where fuzzers come in, fuzzers generate a set of inputs for your contract’s test cases by reaching the boundaries that are missed while writing unit tests.

Formally! Fuzz Testing or Fuzzing is when you supply random data to your system in an attempt to break it.

But it still depends on how good fuzz tests you have written. 

Fuzzers as software are dumb, basically they lack intelligence and the computational boundaries within which they operate. 

In such a context where multiple actions are defined, a fuzzer is liable to choose any of them at random for execution. This raises a  challenge where the fuzzer chooses an action for a particular situation that is inappropriate, e.g., the fuzzer is misled by an inaccurate address for an onlyOwner type function, resulting in the expected reversion.

This issue is considered a low-hanging fruit since we anticipate that the contract should revert in such scenarios, which should be covered within our unit tests. Hence, it would be ideal if our invariant tests could bypass these actions. 

This ultimately would prevent wasting valuable fuzzing calls that can be more effectively utilized on valid edge cases. Therefore, one of the challenges of writing Fuzz Tests is getting the most value out of them.

Fuzzing is just a technique used to improve security. This can uncover vulnerabilities that manual testing might miss since it covers a wider range of potential input scenarios.

Nevertheless, while fuzzing enhances the security of smart contracts, it doesn't always guarantee absolute security. It's just another method of reducing security risks but it does not completely eliminate them. This is because while fuzzing can test many different inputs and scenarios, it may not cover every possible scenario, especially those that involve complex multistep interactions with other contracts.

Introduction of Invariant/Property

Now, let's get an introduction to Invariant, a.k.a. Property. This is where things get little bit complicated (or rather holistic, but not as much as you think).

To put it simply, Invariant is the property that you bet that the system should always hold. During exhaustive testing on this part, you can anticipate that fuzz testing is much more dynamic as compared to unit testing. In unit testing, you supply a single input and get the expected/unexpected results, but in the case of invariant testing, you bet that specific property should be held!

During an invariant test run, the fuzzer will call the test with many randomly generated values, verifying that our assertion holds for each one. This lets us test a specific property of a specific function in a specific contract.

The term "invariant" in the context of DeFi protocols refers to a particular property or rule that must never be violated, no matter what actions are taken within the system. Essentially, these invariants are the core principles or 'laws' that the protocol operates by to ensure the system's stability and fairness.

Lending Markets Invariant

In lending markets like Compound or Aave, there is an important rule that helps ensure the system's overall safety. The rule states that

A user cannot take any action that puts their account, or any other account, into a situation where the value of their borrowed assets exceeds the value of their collateral.

To explain further, when you borrow assets in these markets, you must provide collateral of greater value. This collateral acts as a safety net for the protocol and its lenders. The 'safety threshold' defines the maximum ratio of borrowing to collateral allowed. If the value of the borrowed assets starts approaching this threshold, the account is deemed unsafe. Users are restricted from taking actions that would push accounts into this unsafe state or further worsen an already unsafe state. It's like preventing someone from borrowing more than their house is worth in a mortgage agreement.

Automated Market Makers (AMMs) Invariant

AMMs like Uniswap or Sushiswap, have a core invariant based on a mathematical formula that maintains the relationship between the amounts of two tokens in a liquidity pool.

A widely used invariant is x*y=k, where x and y represent the quantities of two tokens in a pair, and k is a constant value. This equation ensures that the product of the amounts of tokens is always constant. It helps determine the price for each token and maintains a balance of liquidity between them. If more of one token is bought, the price of that token rises to maintain the constant k.

Liquidity Mining/Staking Invariant

Similarly, in liquidity mining or staking protocols like Yearn Finance or Synthetix, a key rule is

A user can only withdraw the same number of staking tokens they initially deposited.

This means if you deposit 10 tokens into the protocol for staking or liquidity mining, you can only withdraw those 10 tokens back. You might earn rewards for participating, but the amount of staking tokens you initially put in remains constant. It's similar to only being able to withdraw the exact amount of money you put into a savings account in a bank, regardless of the interest you've earned.

Invariants are the protocols' backbone, ensuring the systems remain stable and operate as intended.

Basic of Fuzzing

Now let’s move towards an interim question “How fuzzer generates inputs and discovers edge cases?”

To answer this question, we will use a code example for better understanding, but before that, we need to understand one more important thing.

While Invariant testing applies the same idea to the system as a whole, rather than defining properties of specific functions, we define "invariant properties" about a specific contract or system of contracts that should always hold. Invariant tests can be a great tool for shaking out invalid assumptions, providing a holistic approach to testing smart contracts. By examining the entire system, these tests uncover vulnerabilities, complex edge cases, and unexpected interactions. 

Let’s look at this ​​crowdfunding contract.

pragma solidity ^0.8.0;

contract Crowdfunding {
    uint256 public fundingGoal;
    uint256 public deadline;
    mapping(address => uint256) public contributions;

    constructor(uint256 _fundingGoal, uint256 _deadline) {
        fundingGoal = _fundingGoal;
        deadline = _deadline;
    }

    function contribute() public payable {
        require(block.timestamp < deadline, "Deadline has passed");
        contributions[msg.sender] += msg.value;
    }

    function withdraw(uint256 amount) public {
        require(block.timestamp > deadline, "Deadline has not passed");
        require(contributions[msg.sender] > 0, "No contributions");
        require(amount <= address(this).balance);
        
        contributions[msg.sender] = 0;
        payable(msg.sender).transfer(amount);
    }
}

Introducing a Bug:

We will intentionally introduce a bug in the contract to demonstrate the effectiveness of fuzz testing. In the withdraw() function, we remove a vital constraint, that is, to check if the amount from the function argument is equal to or less than the user deposited balance to cover the withdrawal amount before transferring funds to the contributor. This oversight may allow an attacker to drain the contract's balance entirely.

Let’s say we have extracted a property that says, “The withdrawal amount should be less or equal to the deposited amount” (that's really basic, I know, but let's take it for the sake of learning…) Additionally, we could have extracted more properties, but right now I wanna jump to the point I know where this smart contract won't behave as expected.

So, our Property, though in pseudocode, sounds like the statement above, but in code, it will look like the following snippet.

assert( amount <= contributions[msg.sender]);

Think positive while extraction properties

This invariant is very simple to understand at this point. Unlike writing unit tests, where you often think about what could go wrong and you think offensive, this way, the invariants help you focus on how the system functions when everything is going right. In essence, you're studying the system and identifying what changes occur when specific actions are taken. After these actions are performed, you observe the transformations that happen within the system and to its state. 

It is this set of consistent and predictable changes where you think as a system developer and build your invariant around. So, instead of looking for potential threats to the system, 

You're concentrating on the inherent properties that ensure the system's stability and successful operation, which ultimately ensures the security of a system.

TL;DR

Fuzz testing, or fuzzing, is a technique used to improve the security of software, including smart contracts in Solidity. It involves supplying random or unexpected data as inputs to a system in an attempt to break it and uncover vulnerabilities that manual testing might miss. Fuzzers generate a set of inputs for testing scenarios that may have been missed during unit testing, helping to identify bugs and potential security issues.

Invariants, also known as properties, are specific rules or principles that should always hold true within a system. They are essential for ensuring the stability and integrity of protocols like lending markets, automated market makers (AMMs), and liquidity mining/staking systems. Invariant testing involves checking if these properties hold true by using randomly generated values and verifying the assertions for each input.

By using fuzzing and invariant testing together, developers can identify vulnerabilities, complex edge cases, and unexpected interactions within smart contracts. However, while fuzzing improves security, it does not guarantee absolute security. It is just one method to reduce security risks and should be used in conjunction with other security practices and techniques.

Also, read our audit course series "Infiltrating the EVM".