The double rounding problem means that a high-precision reference cannot be used to verify the correctness of lower-precision implementations, even for a single operation. For example, naively using double-precision arithmetic to verify single-precision results is not enough to ensure correctness: double rounding may cause the reference result, rounded to single-precision, to disagree with the correctly-rounded single-precision result. When developing correctly-rounded implementations of floating-point functions, double rounding implies that we cannot simply compute the result at higher precision and then round it to the target precision. In the worst case, a correctly-rounded implementation must be specifically designed for each target precision, adding significant time and complexity to the implementation and verification effort.

Across the literature on floating-point arithmetic, one rounding mode offers a solution to double rounding: round to odd (RTO). This blog post covers the common rounding modes, and the definition and properties of round to odd. Finally, it concludes with the essential property of round to odd: safe re-rounding.

Floating-Point Numbers

To begin, I’ll briefly review floating-point numbers. Floating-point numbers are numbers represented in the form:

where \(s \in \{0, 1\}\) is the sign, \(c \in \mathbb{Z}_{\geq 0}\) is the significand, and \(exp \in \mathbb{Z}\) is the (unnormalized) exponent. The fewest number of digits that can represent \(c\) is called the precision, \(p\), of the number: if \(c > 0\), then \(2^{p - 1} \leq c < 2^p\). Alternatively, we can represent floating-point numbers in normalized form:

where \(1 \leq m < 2\) is the mantissa and \(e \in \mathbb{Z}\) is the (normalized) exponent. The relationship between the two forms is given by the equations:

and

For example, we can represent \(1.25\) using 3 bits of precision by \(101 \cdot 2^{-2}\) and \(1.01 \cdot 2^{0}\) in unnormalized and normalized forms, respectively.

The IEEE 754 standard extends floating-point numbers to include special values: including negative zero, \(-0\); positive infinity, \(+ \infty\); negative infinity, \(- \infty\); and Not a Number, \(\mathrm{NaN}\). Number formats define discrete sets of floating-point numbers that approximate real numbers.

A rounding operation maps real numbers to representable values of a number format according to rounding modes. Rounding modes determine which floating-point value to choose in cases where the real number cannot be represented exactly. There are several rounding modes in use today. For example, the IEEE 754 standard [1] defines five rounding modes: round to nearest, ties to even (RNE); round to nearest, ties away from zero (RNA); round to positive infinity (RTP); round to negative infinity (RTN); and round toward zero (RTZ).

Correctly-Rounded Functions

Rounding extends naturally to real-valued functions through correctly-rounded functions. We say that an implementation \(f^{*}\) of a real-valued function \(f\), is correctly-rounded if its result is the infinitely precise result of \(f\), rounded to the target number format according to a specified rounding mode. We’ll only consider floating-point numbers with a fixed precision of \(p\) digits, so I’ll denote the rounding operation in the style of Boldo and Melquiond [2] as \(\mathrm{rnd}^{p}_{rm}\). Using this notation, the correctly-rounded implementation of \(f\) in precision \(p\) under rounding mode \(rm\) can be expressed as:

Requiring correct rounding has strong arguments [3]: it provides a clear specification for the behavior of \(f^{*}\); it ensures reproducibility of results across different implementations of \(f^{*}\); and it bounds the numerical error introduced by rounding.

We can now formalize the double rounding problem. In general, for precisions \(p_2 < p_1\) and rounding modes \(rm_1, rm_2\):

More succinctly, rounding operations do not compose:

We will see that round to odd provides a solution to this problem.

Rounding Modes

Once we compute the infinitely precise result of \(f(x)\) for a real number \(x\), we need to round the result to get \(f^{*}(x) = \mathrm{rnd}^{p}_{rm}(f(x))\). To illustrate the different rounding modes, we’ll consider three rounding modes: round to nearest, ties to even (RNE); round toward zero (RTZ); and round away from zero (RAZ).

If \(f(x)\) is representable in the target number format, then no more work is needed: \(f^{*}(x) = f(x)\). Otherwise, \(f(x)\) lies between two representable floating-point numbers, which we’ll denote as \(y_1 < f(x) < y_2\). To simplify the discussion, let’s assume that \(y_1\) and \(y_2\) are both positive, and that \(y_1\) has an even significand \(1XXX\ldots0\) and \(y_2\) has an odd significand \(1XXX\ldots1\).

The rules for each rounding mode are as follows:

• RNE: round to the nearest representable number; if \(f(x)\) is exactly halfway between \(y_1\) and \(y_2\), round to the one with an even significand \(c\);
• RTZ: round to the number in the direction of zero;
• RAZ: round to the number in the opposite direction of zero.

Let’s first assume that \(f(x)\) is closer to \(y_1\) than to \(y_2\).

In this case, \(f(x)\) rounds to

• RNE: \(y_1\) since it is the nearest representable number,
• RTZ: \(y_1\) since it is in the direction of zero,
• RAZ: \(y_2\) since it is in the opposite direction.

Now, let’s assume that \(f(x)\) is closer to \(y_2\) than to \(y_1\).

The only difference is that \(f(x)\) rounds to \(y_2\) under RNE, since \(y_2\) is now the nearest representable number.

Finally, let’s consider the case where \(f(x)\) is exactly halfway between \(y_1\) and \(y_2\).

Rounding under RNE will tie-break to the even significand, in this case, \(y_1\). Under RTZ and RAZ, the result is the same as before: RTZ rounds to \(y_1\) and RAZ rounds to \(y_2\).

Other rounding modes like round to nearest, ties away from zero (RNA); round to positive infinity (RTP); and round to negative infinity (RTN) can be analyzed similarly. RNA is similar to RNE, except that it tie-breaks to the representable value away from zero. RTP rounds towards positive infinity: it is the same as RAZ for positive numbers, and RTN is the same as RTZ for positive numbers. RTN is the opposite of RTP.

Round-to-Odd

Round to odd is only a slight adaptation of the rounding modes we’ve seen so far. Like before, if \(f(x)\) is representable in the target number format, then no rounding is required: \(f^{*}(x) = f(x)\). Otherwise, \(f(x)\) lies between two representable floating-point numbers, which we’ll again denote as \(y_1 < f(x) < y_2\). Under RTO, we round to the representable number with an odd significand \(c\). In our example, this means that we always round to \(y_2\).

Round to odd should not be confused with round to nearest, ties to odd (RNO); the rounding mode that uses the opposite tie-breaking rule of round to nearest, ties to even (RNE). RTO is not a nearest rounding mode. For RNE and RNO, the “even” (and “odd”) refers to the tie-breaking rule when \(f(x)\) is exactly halfway between \(y_1\) and \(y_2\). For RTO, the “odd” refers to the rule whenever \(f(x)\) is not representable.

Like RTZ and RAZ, any value between \(y_1\) and \(y_2\) rounds to the same result, in this case, \(y_2\). On this example, round to odd doesn’t seem too interesting.

However, examining its behavior on a different example reveals a key property of RTO. Let’s zoom out and consider the next representable floating-point number after \(y_2\), which we’ll denote as \(y_3\). In a floating-point number format with one fewer bit of precision \(p - 1\), \(y_1\) and \(y_3\) would be adjacent representable numbers, and \(y_2\) would be the midpoint between them. Rounding with the original precision \(p\), any \(f(x)\) between \(y_1\) and \(y_3\), rounds to \(y_2\) under RTO.

Notice that under precision \(p\), \(y_1\) and \(y_3\) have even significands, while \(y_2\), not representable at precision \(p - 1\), has an odd significand. Thus, the parity of the significand encodes whether the infinitely precise result \(f(x)\) is representable in the lower precision \(p - 1\): the significand of \(f^{*}(x)\) is odd if and only if \(f(x)\) is not representable in precision \(p - 1\). This is a key feature of round-to-odd: parity encodes representability, also called exactness. In floating-point literature, the lowest digit of the significand is often called the sticky bit.

There are a few interpretations of the sticky bit. If we expand the (possibly infinite) significand of \(f(x)\) as \(1X\ldots XYYY\ldots\) where \(1X\ldots X\) are the first \(p - 1\) bits and \(YYY\ldots\) are the trailing digits, then the sticky bit \(S\) summarizes the trailing digits:

and the significand may be written as \(1X\ldots XS\), which is the significand of either \(y_1\) (if \(S = 0\)) or \(y_2\) (if \(S = 1\)). Alternatively, we may view the simplified significand as an interval \(I\): if \(S = 0\), then \(c = 1X\ldots X0\) and \(I = [c, c]\); if \(S = 1\), then \(c = 1X\ldots X1\) and \(I = (c, c + \varepsilon)\), where \(\varepsilon\) is the distance to the next representable floating-point number with precision \(p - 1\). Or, rather than an interval, we can choose an unknown real value \(c \in I\); we cannot capture the exact value of \(f(x)\) since the sticky bit only indicates whether there are trailing digits.

Sticky bits are widely used when implementing floating-point arithmetic in both hardware and software due to their ability to summarize discarded trailing digits. Encoding whether a result has non-zero trailing digits at some precision is essential for correct rounding. In the next section, we’ll see how round to odd preserves enough information through the sticky bit to safely re-round under any standard rounding mode at lower precision, avoiding the double rounding problem.

Properties of Round-to-Odd

Boldo at Melquiond [2] identify several properties of round to odd. It’s worth summarizing them here. The first four are properties shared with other standard rounding modes:

• round to odd is entire : every real number can be rounded to odd;
• round to odd is unique : every real number has a unique rounding to odd;
• round to odd is monotonic : if \(x_1 \leq x_2\), then \(\mathrm{rnd}^{p}_{\text{RTO}}(x_1) \leq \mathrm{rnd}^{p}_{\text{RTO}}(x_2)\);
• round to odd is faithful : if \(y_1 < x < y_2\) are the two representable numbers of precision \(p\) surrounding \(x\), then \(\mathrm{rnd}^{p}_{\text{RTO}}(x)\) is either \(y_1\) or \(y_2\);

The first three properties are fairly straightforward: we want to round any real number; we want the rounding to be deterministic; and we want rounding to preserve order. The fourth property, faithfulness, ensures that rounding does not introduce large errors: if \(\varepsilon\) is the distance between \(y_1\) and \(y_2\), then the absolute error is less than \(\varepsilon\). In addition,

• round to odd is symmetric : for any real number \(x\), \(\mathrm{rnd}^{p}_{\text{RTO}}(-x) = -\mathrm{rnd}^{p}_{\text{RTO}}(x).\)

Boldo and Melquiond [1] prove a key property that distinguishes round to odd from other rounding modes: round to odd permits safe re-rounding. Rounding with round to odd first, then re-rounding under any standard rounding mode at lower precision yields the same result as rounding directly under that standard rounding mode at lower precision, specifically, \(k \geq 2\) digits lower.

Theorem 1. Let \(p, k \geq 2\) be integers; and \(rm\) be a standard rounding mode. Then,

To understand this statement, we return to previous examples covering the different rounding modes. Let \(y_1\) and \(y_3\) be two adjacent floating-point values that are representable with precision \(p\), and let \(y_2\) be the midpoint between them at precision \(p + 1\). For simplicitly, let’s assume that \(y_1\) is positive, so \(y_2\) and \(y_3\) are also positive. Consider rounding an arbitrary real number \(x \in [y_1, y_3)\) under RNE, RTZ, RAZ, and RTO.

For each rounding mode, we color the interval \([y_1, y_3)\) with each segment (or tick) colored according to the rounding result: blue for \(y_1\) and orange for \(y_3\). Dual-coloring indicates a conditional rounding that depends on the value of \(y_1\) (and \(y_3\)). RTO is dual-colored throughout; the midpoint \(y_2\) is also dual-colored for RNE.

Notice that, for these rounding modes, we can distinguish four cases:

• \(x = y_1\): \(x\) is representable, so all rounding modes produce \(y_1\);
• \(y_1 < x < y_2\): \(x\) is closer to \(y_1\), so RNE and RTZ produce \(y_1\), RAZ produces \(y_2\); and RTO chooses based on parity of \(y_1\);
• \(x = y_2\): \(x\) is halfway, so RNE must tie-break based on parity, RTZ produces \(y_1\), RAZ produces \(y_2\); and RTO chooses based on parity of \(y_1\).
• \(y_2 < x < y_3\): \(x\) is closer to \(y_3\), so RNE and RAZ produce \(y_2\), RTZ produces \(y_1\); and RTO chooses based on parity of \(y_1\).

Analyzing these regions at precision \(p + 1\), the significand of \(y_1\) and \(y_3\) are even, since increasing precision adds a trailing zero to their significands. For example, if \(y_1 = 5/32\) with \(p = 3\), then:

By contrast, the significand of \(y_2\) is odd. For the same example, the midpoint \(y_2 = 11/64\) of \(y_1\) and \(y_3\) has the form

The regions \((y_1, y_2)\) and \((y_2, y_3)\) are the intervals between adjacent representable numbers at precision \(p + 1\). Recalling discussion from earlier, these regions are exactly the intervals represented by the sticky bit at precision \(p + 2\). Therefore, rounding to odd at precision \(p + 2\) results in four cases:

• \(x\) is exactly the endpoint \(y_1\), so the last two digits of the rounded significand are \(RS = 01\);

• \(x\) lies in \((y_1, y_2)\), so the last two digits of the rounded significand are \(RS = 01\);

• \(x\) is exactly the midpoint \(y_2\), so the last two digits of the rounded significand are \(RS = 10\);

• \(x\) lies in \((y_2, y_3)\), so the last two digits of the rounded significand and \(RS = 11\).

Notice that these cases correspond exactly to the four cases we identified earlier for standard rounding modes at precision \(p\). After applying round to odd at precision \(p + 2\), representable values (at precision \(p\)) are still representable; midpoints (at precision \(p\)) are still midpoints; and all other values, either on \((y_1, y_2)\) or \((y_2, y_3)\), are rounded to the midpoint of those intervals, preserving their nearness to one endpoint. Therefore, re-rounding under any standard rounding mode at precision \(p\) yields the same result as rounding directly at precision \(p\).

Visually, we can indicate the round to odd step by overlaying gray arrows, representing round to odd at precision \(p + 2\), over the previous figure. Layering the two rounding steps, we see that round to odd at precision \(p + 2\) rounds values in \((y_1, y_2)\) or \((y_2, y_3)\) to the midpoints at precision \(p + 1\); representable values and midpoints remain unchanged. Safe re-rounding corresponds to the coloring of the initial value \(x\) being preserved after rounding to odd.

Therefore, round to odd at precision \(p + 2\) preserves sufficient information so that we can safely re-round under any standard rounding mode at precision \(p\). For precision \(p + k\) where \(k > 2\), the same reasoning applies: values that are neither representable nor midpoints may be rounded differently at precision \(p + k\), but their nearness to one endpoint is preserved.

Applications

Boldo and Melquiond [2, 4] provide potential applications for round-to-odd. They include emulation of FMA, correctly-rounded addition of 3 terms, correctly-rounded sum of \(n\) terms (under certain conditions), compiling the same constant under multiple precisions and rounding modes, and more.

More recent work relies on a corollary of Theorem 1. Combining the definition of a correctly-rounded function with Theorem 1, we get the following result:

Stated otherwise, a correctly-rounded implementation of \(f\) is the composition of (i) a round-to-odd implementation of \(f\); followed by (ii) re-rounding under the desired rounding mode.

Corollary 2. Let \(f\) be a real-valued function, and \(f^{*}\) be a correctly-rounded implementation of \(f\) at precision \(p\) under rounding mode \(rm\). If \(f_{\text{RTO}}^{*}\) is a correctly-rounded implementation of \(f\) at precision \(p + k\) under round to odd (\(k \geq 2\)), then

One successful application of this corollary is found in the RLibm project [5] which automatically generates efficient, correctly-rounded elementary functions by generating a polynomial approximation with additional bits of precision using round-to-odd arithmetic that will be correctly rounded when re-rounded under the desired rounding mode. The general principle of Corollary 2 suggests a modular approach to designing correctly-rounded functions for multiple precisions and rounding modes: first, design a round-to-odd implementation at higher precision; then, apply a re-rounding step to obtain the desired result.

Conclusion

This blog post covered the round to odd including its definitions, properties, and applications. Along the way, we learned about floating-point numbers, correctly-rounded functions, and how rounding works. The key property of round to odd, safe re-rounding, avoid double rounding and suggests a method for designing correctly-rounded functions by separating the concerns of approximating the infinitely precise result and rounding to the target number format. While round to odd is not widely supported in hardware or programming languages today, its unique properties make it a valuable techinique that should be better studied and more widely adopted.

References

1. IEEE. 2019. IEEE Standard for Floating-Point Arithmetic. IEEE Std 754-2019 (Revision of IEEE 754-2008), 1–84. DOI: https://doi.org/10.1109/IEEESTD.2019.8766229.

2. Sylvie Boldo, Guillaume Melquiond. When double rounding is odd. 17th IMACS World Congress, Jul 2005, Paris, France. pp.11. ffinria-00070603v2f.

3. Nicolas Brisebarre, Guillaume Hanrot, Jean-Michel Muller, and Paul Zimmermann. 2025. Correctly Rounded Evaluation of a Function: Why, How, and at What Cost?. ACM Comput. Surv. 58, 1, Article 27 (September 2025), 34 pages. https://doi.org/10.1145/3747840.

4. Sylvie Boldo and Guillaume Melquiond. 2008. Emulation of a FMA and Correctly Rounded Sums: Proved Algorithms Using Rounding to Odd. IEEE Transactions on Computers 57, 4 (2008), 462–471. https://doi.org/10.1109/TC.2007.70819.

5. Jay P. Lim and Santosh Nagarakatte. 2022. One Polynomial Approximation to Produce Correctly Rounded Results of an Elementary Function for Multiple Representations and Rounding Modes. Proc. ACM Program. Lang. 6, POPL, Article 3 (January 2022), 28 pages. https://doi.org/10.1145/3498664.

Number libraries simulate number systems — floating-point, fixed-point, posits, and more — beyond those offered by standard hardware or language runtimes. Well known examples of these libraries include MPFR for arbitrary-precision floating-point numbers, GMP for arbitrary-precision integers and rationals, SoftFloat for emulating IEEE 754 floating-point numbers in software, and Universal for supporting formats across the machine learning landscape. They are essential tools for analyzing numerical error in multi-precision numerical algorithms, exploring different implementation trade-offs, and verifying the correctness of numerical software and hardware.

This flexibility over number systems comes at a cost: number libraries are complex, requiring expert knowledge to build and maintain, and significant effort to provide useful features and ensure correctness. Maintainers of these libraries face significant challenges as their tools serve as trusted reference implementations but must also be efficient for simulation and verification tasks. Due to intense demand from machine learning, there has been a proliferation of new number formats and rounding behaviors. Each new combination of operation, number format, rounding mode, overflow behavior, and special value handling requires careful implementation and testing, multiplying the complexity and maintenance burden. Developers must choose: stick with a smaller set of features or invest significant effort to meet user demand.

I maintain number libraries that fall into the latter category: they aim to support a wide variety of number formats and rounding behaviors. I want to reflect on two design principles that have helped mitigate some of these challenges.

First, the round-to-odd rounding mode [1] allows decoupling of arithmetic operations from rounding. Applying this insight to number libraries, a number library may consist of two independent parts: a core arithmetic engine performing round-to-odd operations, and a core rounding library that safely re-rounds under the desired number format or rounding mode. As a result, each operation provided by the number library is the composition of an operation in the arithmetic engine and the round method of a rounding context instance from the core rounding library. This trick was pioneered by Bill Zorn [2] in his number library found in the Titanic repo [3].

Second, the core rounding library should be organized into rounding contexts which encapsulate number format, rounding mode, and other rounding information; providing a single round method. These rounding contexts are often composable: a rounding context for an IEEE 754 floating-point number can reuse the same logic as a rounding context for a p-bit floating-point number, with added checks for overflow. Thus, the rounding library can be broken down further into composable components that can be assembled to implement rounding for a variety of number formats.

Put together, these two design principles enable significant modularity and code reuse within number libraries, resulting in several benefits. They dramatically improve maintainability: the library is smaller; it consists of smaller, composable components rather than a monolithic implementation where each combination of operation and rounding mode requires separate code. They ensure extensibility: adding features is easier and less error-prone; adding a new operation requires implementing it only once in the arithmetic engine, while adding a new format or rounding mode requires implementing it only once in the rounding library. They improve correctness: testing can be done compositionally; components can be extensively tested in isolation, and their composition inherits the correctness guarantees.

I will explore these two design principles in greater detail and illustrate their benefits through examples. As someone working in this space, I hope this blog post provides some insight into the challenges in this domain and possible solutions.

Separating Rounding from Arithmetic

The first design principle is to separate rounding from arithmetic operations using round-to-odd arithmetic [1]. My blog post covers this topic in detail. I’ll summarize the key ideas here and illustrate how they apply to number libraries.

Theory

For a real number function \(f\), we say that an implementation of \(f\), say \(f^{*}\), is correctly-rounded when it produces the infinitely precise result of \(f\) rounded to the target number format. This rounding operation is usually parameterized by a rounding mode that specifies which representable value to choose when the result is not exactly representable in the target number format; the IEEE 754 standard specifies several such rounding modes. For now, we’ll only consider floating-point numbers with a fixed precision of \(p\) digits, so I’ll denote the rounding operation in the style of Boldo and Melquiond [1] as \(\mathrm{rnd}^{p}_{rm}\). Using this notation, the correctly-rounded implementation of \(f\) in precision \(p\) under rounding mode \(rm\) can be expressed as:

Boldo and Melquiond [1] describe a non-standard rounding mode called round to odd and prove that the rounding mode permits safe re-rounding. Rounding with round to odd at precision \(p + k\) (for \(k \geq 2\)) followed by re-rounding under a standard rounding mode at precision \(p\), yields the same result as rounding directly at precision \(p\) under the desired rounding mode.

Theorem 1. Let \(p, k \geq 2\) be integers; and \(rm\) be a standard rounding mode. Then,

Applying this result to the definition of a correctly-rounded implementation, we can derive that \(f^{*}\) is the composition of (i) a round-to-odd implementation of \(f\) at higher precision, followed by (ii) re-rounding under the desired rounding mode:

The result is not novel: one successful application is found in the RLibm project [4] which automatically generates efficient, correctly-rounded elementary functions by generating a polynomial approximation with additional bits of precision using round-to-odd arithmetic that will be correctly rounded when re-rounded under the desired rounding mode.

For number libraries, the result has significant implications. It suggests that arithmetic and rounding can be decoupled: a number library can consist of two independent components: an arithmetic engine that implements each mathematical operation using round-to-odd arithmetic, and a rounding library that implements rounding operations for various number formats and rounding modes. The only interaction between the two components is that the rounding library must provide the precision \(p + k\) required for safe re-rounding.

Application

To illustrate this approach in practice, consider a number library providing an implementation of multiplication.

module Engine:
    def rto_mul(x, y, p):
        ...

module Round:
    def round(x, p, rm):
        ...

    def rto_prec(p):
        ...

def mul(x, y, p, rm):
    rto_p = Round.rto_prec(p)
    result = Engine.rto_mul(x, y, rto_p)
    return Round.round(result, p, rm)

The Engine module implements arithmetic operations that produce round-to-odd results with at least precision p. The Round module handles rounding operations and precision calculations: round rounds x to precision p using the specified rounding mode, while rto_prec calculates the precision needed for safe re-rounding. The mul function provided by the number library composes these functions by performing round-to-odd multiplication, and re-rounding to the desired precision and rounding mode.

To implement rto_mul, we can use existing libraries like MPFR that have been extensively tested. MPFR provides a narrower interface that performs floating-point arithmetic at a specified precision and rounding mode. Although MPFR does not directly support round-to-odd, we can implement it as described by Boldo and Melquiond [1]: we use MPFR’s implementation at precision p - 1 with round towards zero, followed by an additional step to adapt the result to be round-to-odd at precision p.

def rto_mul(x, y, p):
  r = MPFR.mul(x, y, p - 1, MPFR.RTZ)
  return rto_fixup(r)

Note by Theorem 1, we can request higher precision than p for safe re-rounding at precision p - 2.

Since the arithmetic engine and rounding library are separate, the exact implementation of rto_mul can be changed without affecting the correctness of any mul implementation, as long as it produces correct round-to-odd results. For example, assume that floating-point numbers in our library have the following structure:

class Float: # (-1)^sign * c * 2^exp
  sign: bool # sign
  exp: int # exponent
  c: int # significand (c >= 0)

We can implement rto_mul manually as follows:

def rto_mul(x, y, p):
  s = x.sign != y.sign
  exp = x.exp + y.exp
  c = x.c * y.c
  return Float(s, exp, c)

Notice that this implementation does not perform any rounding, so p is unused. Since the result is infinitely precise, it can be re-rounded under any precision. Extending this implementation to handle special values (\(+\infty\), \(-\infty\), NaN, etc.) just requires careful case analysis.

For the Round module, we need to implement two functions:

• rto_prec(p) computes the precision required for safe re-rounding to precision p; and

• round(x, p, rm) rounds x to precision p using the specified rounding mode rm.

The implementation of rto_prec is straightforward: it returns p + k for a constant k >= 2. Implementing the round function requires care as it’s the core method of the rounding library. The exact implementation is too verbose to include here, but I’ll outline its basic structure. For this example, we’ll ignore special values like NaN and infinity.

def round(x, p, rm):
  n = x.e - p  # compute where to round off digits
  hi, lo = x.split(n) # split into significant and leftover digits
  rbits = lo.round_bits(n) # summarize leftover digits for rounding decision
  increment = decide_increment(hi, rbits, rm) # round away from zero based on rounding mode?
  if increment: # adjust hi if needed
    hi = hi.increment()
  return hi

The round function first determines where to round off digits based on the normalized exponent x.e of the argument and the target precision p. Any digit above this point is significant while digits below must be rounded off. A method split divides the number based on this point, and the lower part is summarized into rounding bits. Since hi represents the round-towards-zero result, we decide whether to round away from zero to the next representable value based on the rounding bits and the specified rounding mode. The correctly-rounded result is hi.

A number library built in this style can be extended to support new operations and rounding modes easily. For a new operation \(g\), we must only implement \(g^{*}_{\text{RTO}}\) in the arithmetic engine, either from existing libraries or manually. The function round in the rounding library implements the function \(\mathrm{rnd}^{p}_{rm}\). Their composition is then the correctly-rounded implementation \(g^{*}\) of \(g\). For a new rounding mode, only the round function needs to be updated; the arithmetic engine remains unchanged. Compare this approach to a monolithic implementation. Each new operation requires implementing operation-specific rounding logic; each new rounding mode requires updating every operation.

To briefly summarize, separating arithmetic from rounding achieves the following benefits:

• Maintainability: The arithmetic engine implements each operation only once, while the rounding library implements rounding logic separately.

• Extensibility: Any new mathematical operation can be composed with the existing rounding library. Similarly, any new rounding logic can be composed with the existing arithmetic engine.

• Correctness: Once a mathematical operation is verified in the arithmetic engine, its correctness applies to any composition with the rounding library. Similarly, once a rounding context is verified, its correctness applies to any mathematical operation. Thus, testing can be done modularly and the results reused.

Rounding Contexts

While separating rounding from arithmetic allows developers to split the number library into two independent components, the rounding library must support a large number of number formats and rounding behaviors. To manage this complexity, we can organize the rounding library into rounding contexts.

A rounding context encapsulates all information required to round a number correctly: the number format (precision, exponent range, etc.), the rounding mode, overflow behavior (saturating, wrapping, exceptions, etc.), and special value handling (NaN, infinity, etc.). Rather than having a single round function that consumes all rounding information as a exhaustive list of parameters, we convert Round into an interface; each rounding context will implement the interface.

interface Round:
  def rto_prec():
    ...

  def round(x):
    ...

  def round_core(x, p, rm): # default method
     ...


def mul(x, y, ctx):
  rto_p = ctx.rto_prec()
  result = Engine.rto_mul(x, y, rto_p)
  return ctx.round(result)

The two interface methods are similar to before:

• rto_prec computes the precision required to safely re-round a number under the rounding context; and
• round rounds x under the rounding context.

The previous round implementation is now round_core. The mul function now accepts a rounding context instance rather than explicit rounding parameters: its implementation must be adapted slightly.

One strategy for implementing rounding contexts is to organize each implementation of Round based on families of number formats. For example, we can implement p-digit floating-point numbers, as a family of rounding contexts.

class MPFloat(Round):
  p: int # p >= 1
  rm: RoundingMode

  def rto_prec(self):
    return self.p + 2

  def round(self, x):
    return self.round_core(x, self.p, self.rm)

Notice that we completely reuse the round_core implementation for the rounding logic.

Critically, we add support for new number formats by simply implementing a new class that implements the Round interface. For example, consider supporting the IEEE 754 floating-point numbers parameterized by es, the size of the exponent field, and nbits, the total number of bits of the representation. While IEEE 754 numbers have restrictions on exponent ranges, its core rounding behavior is similar to p-digit floating-point numbers: we can reuse the MPFloat rounding logic. The exact implementation is too verbose, but an outline of the implementation is as follows:

class IEEEFloat(Round):
  es: int  # exponent size (es >= 2)
  nbits: int  # total number of bits (nbits >= es + 2)
  rm: RoundingMode

  def emin(self): # minimum (normalized) exponent
    return 1 - (1 << (es - 1))

  def rto_prec(self):
    p = self.nbits - self.es
    return MPFloat(p).rto_prec() # need at least this much precision

  def round(self, x):
    max_p = self.nbits - self.es # maximum allowable precision
    e_diff = x.e - self.emin() # e_diff < 0 if subnormal
    p = min(max_p, max_p + e_diff + 1) # adjust precision for subnormals
    r = MPFloat(p, rm).round(x) # re-use rounding logic
    # handle overflow based on rounding mode
    ...
    return r

The implementation of round is more complicated. First, we must deal with subnormal numbers, i.e., numbers with magnitude below \(2^{emin}\) which have reduced precision: the min(max_p, ...) expression computes the effective precision. We adjust the precision accordingly, construct the appropriate MPFloat context, and reuse its round method to round without exponent bounds. Finally, we handle overflow based on the specified rounding mode.

What about fixed-point formats? To support fixed-point numbers, we first alter round_core to accept a parameter n which sets n directly:

def round_core(x, p, n, rm): # added n parameter
  ...

The caller must specify either p or n. The arithmetic engine must also be altered to support a stopping point n rather than precision p when performing fixed-point operations. For example, if we want to round with n=-1, keeping only integer digits, then the arithmetic engine must produce a round-to-odd result with enough arbitrary precision to preserve all integer digits plus extra digits for safe re-rounding. Like before, one of p or n must be specified.

module Engine:
  def rto_mul(x, y, p, n):
    ...

def mul(x, ctx):
  p, n = ctx.rto_params()
  result = Engine.rto_mul(x, y, p, n)
  return ctx.round(result)

Consider implementing a rounding context for an arbitrary-precision fixed-point number that must round any digit less significant than the n+1-th digit:

class MPFixed(Round):
  n: int  # first insignificant digit (drop digits <= n)
  rm: RoundingMode

  def rto_params(self):
    return None, self.n - 2  # 2 additional bits for safe re-rounding

  def round(self, x):
    return self.round_core(x, None, self.n, self.rm)

I could continue to list implementations of rounding contexts for other number formats, but I believe the pattern is clear: to support new number formats, we implement a rounding context class that can often reuse existing rounding logic. Machine integers and fixed-point formats can compose MPFixed and apply the appropriate overflow behavior. Floating-point formats like those described in the OCP MX standard [5] might benefit from a similar approach to IEEE 754 floating-point numbers. Posit numbers [6] are floating-point numbers with tapered precision; one approach might be to use MPFloat with variable precision based on the magnitude of the number.

Organizing the rounding library into rounding contexts achieves the following benefits:

• Maintainability: Rounding contexts encapsulate rounding logic and often compose together existing rounding contexts to implement its rounding behavior.

• Extensibility: Implementing a new number format only requires implementing a new rounding context class, often reusing existing rounding logic; any instance of the new rounding context can be used with all existing mathematical operations.

• Correctness: Rounding can be decomposed into many smaller, often reusable components; verifying each component in isolation increases confidence in the correctness of each rounding context implementation.

Evaluation

To demonstrate the benefits of these design principles, I applied them to the number library underlying the runtime of the FPy language [7]. FPy is an embedded Python DSL for specifying numerical algorithms, with explicit control over rounding via rounding contexts, including first-class rounding context values. The number library supports many families of number formats from IEEE 754 floating-point numbers, OCP MX floating-point numbers, fixed-point numbers, and more. The core arithmetic engine uses MPFR [8] to implement round-to-odd arithmetic operations.

Maintainability

Due to its modular design, the FPy number library remains compact, composing core components to support a wide variety of number formats and operations.

The FPy number library consists of four major components:

• the Number module defines FPy’s number representation;
• the Rounding module implements the round_core;
• the Arithmetic module implements round-to-odd arithmetic using MPFR;
• the Contexts module implements various rounding contexts.

The total code size is approximately 8,000 lines of code (LOC) with 4750 lines dedicated to rounding alone. Since FPy’s arithmetic engine uses MPFR, the arithmetic engine is relatively small at 1500 lines of code. Rounding contexts in FPy implement more than the essential round method with additional methods for encoding and decoding bit patterns, constructing special values, and more. The largest rounding context is almost 850 lines of code, while the smallest is only 60 lines of code.

An exhaustive list of rounding contexts is below:

The MPFloat and MPFixed rounding contexts call the core rounding logic directly, while other rounding contexts compose existing rounding contexts to implement their rounding behavior. Every rounding context may be used with any arithmetic operation provided by the arithmetic engine. This compositional design ensures the codebase remains relatively small, comprising of modular components rather than monolithic implementations where small changes require large modifications or modifications across many parts of the codebase.

Extensibility

To demonstrate extensibility of FPy, I will showcase the implementation of a correctly-rounded implementation of 1/x^2 and the ExpFloat number format.

Implementing 1/x^2

A correctly-rounded implementation 1/x^2 provides additional accuracy compared to composing 1/x with x^2, separately. To implement this operation in FPy, we only need to implement its round-to-odd implementation. To illustrate one such implementation, I implemented a digit recurrence algorithm that iteratively computes the significant digits of 1/x^2.

The implementation adapts the classic reciprocal digit-recurrence algorithm. Assuming that

we know the result is of the form \(q * 2^{-2e}\) where \(q = 1/(1.m)^2\). The algorithm first computes the square of the argument: adding the exponents and squaring the significand. Then, it computes the expected exponent, both \(e\) and \(exp\), and checks for the special case where the (fractional) significand is exactly 1.0, in which case the result is just \(2^{-2e}\). Otherwise, it performs digit-recurrence to compute all but the last significant digit of the result. Finally, the last digit is determined by round-to-odd: if we have a non-zero remainder, we need to round up to the result with odd significand.

def rto_recip_sqr(x, p):
  assert x != 0

  # square the argument
  x = x * x

  e = -x.e # result normalized exponent
  exp = e - p + 1 # result unnormalized exponent

  m = x.c # argument significand (in 1.M)
  one = 1 << (x.p - 1) # representation of 1.0 (fixed-point)

  if m == one:
    # special case: m = 1 => q = 1.0
    q = 1 << (p - 1)
  else:
    # general case: m > 1 => q \in (0.5, 1.0)
    # step 1. digit recurrence algorithm for 1/m
    # trick: skip first iteration since we always extract 0
    q = 0 # quotient
    r = one << 1 # remainder (constant fold first iter)
    for _ in range(1, p): # compute p - 1 bits
      q <<= 1
      if r >= m:
        q |= 1
        r -= m
      r <<= 1

    # step 2. generate last digit by inexactness
    q <<= 1
    if r > 0:
      q |= 1

    # step 3. adjust exponent so that q \in [1.0, 2.0)
    exp -= 1

  # result
  return Float(s, exp, q)

The implementation is clearly naive, but it verifiably satisfies the round-to-odd contract of the arithmetic engine. Composing this implementation with any rounding context in FPy yields a correctly-rounded implementation of 1/x^2 under that number format and rounding mode. For example, computing with single-precision floating-point: the computed result is 0.101321185 compared to 0.101321183 when computed separately as 1/x and x^2.

Implementing ExpFloat

The ExpFloat number format encodes exponential numbers, i.e., 2^{e} where e is an integer exponent, that is, positive, power-of-two numbers. Unlike standard floating-point numbers, ExpFloat numbers cannot be negative, zero, infinity, or NaN. In the OCP MX standard [5], the “E8M0” format is an 8-bit exponential number encoding the possible values of the exponent field of a single-precision IEEE 754 floating-point number. The ExpFloat rounding context is parameterized by nbits, the total number of bits in the representation. To implement ExpFloat in FPy, we compose with the MPFloat rounding context implementation:

class ExpFloat(Round):
  nbits: int  # total number of bits (nbits >= 2)
  rm: RoundingMode # rounding mode
  ... # overflow/underflow behavior

  def mp_ctx(self): # corresponding MPFloat context
    return MPFloat(1, rm)

  def rto_params(self):
    return self.mp_ctx().rto_params()

  def round(self, x):
    # user-defined behavior for negative, zero, Inf, or NaN
    if x.is_nar() or x <= 0:
      ...

    r = self.mp_ctx().round(x) # re-use rounding logic
    # handle overflow/underflow based on rounding mode
    ...
    return r

The implementation of ExpFloat is highly compact: it reuses the MPFloat rounding logic entirely, wrapping it with custom behavior for invalid values, overflow, and underflow behavior. In FPy, the actual implementation of the round method of the ExpFloat context comes out to 50 logical lines of code (110 in total), with most lines dedicated to overflow and underflow handling based on the specified rounding mode; the full context implementation with encoding logic, value constructors, and predicates comes out to 500 lines.

Correctness

Rather than having to test each operator implementation in its entirety, FPy’s design allows testing to be done compositionally. Each arithmetic operation in the arithmetic engine can be tested independently of the rounding library; the rounding library can be tested piecewise by verifying each rounding context or core rounding logic in isolation. In particular, I use property-based testing with the Hypothesis library [10].

Ensuring correctness of the core rounding procedure round_core is especially important, as it is the basis for all rounding contexts. To test round_core, we can verify that it satisfies the expected properties of rounding. For example, we expect round_core to satisfy two properties:

• round_core(x, p, None, rm) has at most p significant digits;
• round_core(x, None, n, rm) has no significant digits below the n+1-th digit.

@given(floats(prec_max=256), st.integers(min_value=0, max_value=1024), rounding_modes())
def test_round_p(x, p, rm):
    y = round_core(x, p, None, rm)
    assert 0 <= y.p <= p

@given(floats(prec_max=256), st.integers(min_value=-1024, max_value=1024), rounding_modes())
def test_round_n(x, n, rm):
    y = round_core(x, None, n, rm)
    _, lo = y.split(n)
    assert lo == 0

The code above checks the two properties. The methods floats and rounding_modes are generators for FPy floating-point number values and rounding modes, respectively. We could test the claim of the round method that for hi, lo = x.split(n), the value of hi is the round to zero result. Assuming that x is finite, we can verify this property as follows:

@given(floats(prec_max=256), st.integers(min_value=-1024, max_value=1024))
def test_round_rtz(x, n):
    hi, _ = x.split(n)
    y = round_core(x, None, n, RoundingMode.RTZ)
    assert y == hi

While these three properties aren’t explicitly checking the numerical correctness of round_core, they provide confidence that the implementation behaves as expected. Similar property tests can be devised to further test the correctness of round_core and combined with concrete test cases or other testing methods to increase confidence in its correctness. Clearly, we would also want to verify the split helper method: that it does in fact split the number into two non-overlapping groups of digits at the specified point.

@given(floats(prec_max=256), st.integers(min_value=-1024, max_value=1024))
def test_split(x, n):
    hi, lo = x.split(n)
    # check we did not lose information
    assert hi + lo == x, "split parts must sum to original"
    # check we did not gain information
    assert hi.p <= x.p, "hi must not have more precision than x"
    assert lo.p <= x.p, "lo must not have more precision than x"
    # check non-overlapping property
    assert hi.exp > n, "LSB of hi must be > n"
    assert lo.e <= n, "MSB of lo must be <= n"

Critically, once round_core is verified, the effort of verifying each rounding context mostly involves verifying the unique behavior of that rounding context; confidence in the core rounding logic carries over to each rounding context implementation.

Testing of the arithmetic engine can be done independently of the rounding library. FPy relies on MPFR for round-to-odd arithmetic, which has been extensively tested; the RLibm system also uses MPFR for verifying its transcendental function implementations [4]. Fully verifying custom transcendental function implementations is difficult work, far beyond the scope of property-based testing.

However, we can still verify that each arithmetic operation round-to-odd implementations meet the round-to-odd contract.

@given(floats(prec_max=256), floats(prec_max=256), st.integers(min_value=2, max_value=1024))
def test_rto_mul(x, y, p):
  r_rtz = MPFR.mul(x, y, p - 1, MPFR.RTZ)
  r_rto = rto_mul(x, y, p)
  assert r_rtz.p == p - 1, "MPFR result must have p - 1 precision"
  assert r_rto.c % 2 == (1 if r_rtz.inexact else 0), "inexact iff LSB is odd"

For arithmetic, our custom rto_mul implementation can be verified against Python’s native Fraction implementation for finite inputs.

@given(floats(prec_max=256), floats(prec_max=256), st.integers(min_value=2, max_value=1024))
def test_rto_mul(x, y, p):
  r_ref = x.as_rational() * y.as_rational()
  r_impl = rto_mul(x, y, p)
  assert r_ref == r_impl

Verifying both the arithmetic engine and rounding library in a compositional manner increases confidence in the correctness of the overall number library.

Conclusion

Number libraries face an explosion of complexity with new formats, rounding modes, and operations. The two design principles presented here — separating arithmetic from rounding via round-to-odd, and organizing rounding logic into composable contexts — offer a practical solution to this challenge. By decoupling these concerns, number library developers can achieve smaller, more maintainable codebases that are more extensible and easier to verify compared to traditional monolithic designs. As the numerical computing landscape continues to evolve, these principles are one approach to providing a foundation for building maintainable, correct, and extensible number libraries that can adapt to future requirements without overwhelming their maintainers.

References

1. Sylvie Boldo, Guillaume Melquiond. When double rounding is odd. 17th IMACS World Congress, Jul 2005, Paris, France. pp.11. ffinria-00070603v2f

2. Bill Zorn. 2021. Rounding. Ph.D. Dissertation. University of Washington, USA. https://hdl.handle.net/1773/48230

3. Bill Zorn. 2017. Titanic [GitHub]. https://github.com/billzorn/titanic. Accessed on November 11, 2025

4. Jay P. Lim and Santosh Nagarakatte. 2022. One Polynomial Approximation to Produce Correctly Rounded Results of an Elementary Function for Multiple Representations and Rounding Modes. Proc. ACM Program. Lang. 6, POPL, Article 3 (January 2022), 28 pages. https://doi.org/10.1145/3498664.

5. Bita Darvish Rouhani, Ritchie Zhao, Ankit More, Mathew Hall, Alireza Khodamoradi, Summer Deng, Dhruv Choudhary, Marius Cornea, Eric Dellinger, Kristof Denolf, Stosic Dusan, Venmugil Elango, Maximilian Golub, Alexander Heinecke, Phil James-Roxby, Dharmesh Jani, Gaurav Kolhe, Martin Langhammer, Ada Li, Levi Melnick, Maral Mesmakhosroshahi, Andres Rodriguez, Michael Schulte, Rasoul Shafipour, Lei Shao, Michael Siu, Pradeep Dubey, Paulius Micikevicius, Maxim Naumov, Colin Verrilli, Ralph Wittig, Doug Burger, and Eric Chung. 2023. Microscaling Data Formats for Deep Learning. arXiv preprint arXiv:2310.10537 (2023). https://arxiv.org/abs/2310.10537

6. John L. Gustafson. 2017. Beating Floating Point at its Own Game: Posit Arithmetic. Supercomputing Frontiers and Innovations 4, 2 (2017), 71–86. https://doi.org/10.14529/jsfi170206

7. Brett Saiki. 2025. FPy [GitHub]. https://github.com/bksaiki/fpy. Accessed on November 12, 2025

8. MPFR Team. 2025. The GNU MPFR Library. https://www.mpfr.org/. Accessed on November 14, 2025.

9. Brett Saiki. 2025. Taxonomy of Small Floating-Point Formats. https://uwplse.org/2025/02/17/Small-Floats.html. Accessed on November 12, 2025

10. Hypothesis Team. 2025. Hypothesis. https://hypothesis.works/. Accessed: 2025-11-12.

1. Expression selection
2. Local error analysis
3. Rewriting
   1. Taylor polynomial approximation ("taylor")
   2. rule-based rewriting ("rr")
4. Simplify
5. Prune

First, Herbie selects a few expressions (1) from its database as a starting point for rewriting. Then, it analyzes (2) the expression to find AST nodes that exhibit high local error. Based on the analysis, the tool utilizes a couple rewriting techniques including polynomial approximation and equality saturation via the egg library. Finally, a second rewriting pass simplifies expressions using egg before merging the new expressions with the current set of alternative implementations, keeping only the best.

This sequential process is repeated for a fixed number of iterations before Herbie extracts and fuses expressions through an algorithm called regime inference. All versions of Herbie utilize this architecture in one way or another. In the last couple versions of the tool, I have noticed a tension building between the rewriting phases within the improve loop.

Herbie employs two primary IRs which we will call SpecIR (programs with real number semantics) and ProgIR (programs with floating-point semantics). We can convert from ProgIR to SpecIR without additional information since we simply replace the rounding operation for every mathematical operator with the identity function. However, translating from SpecIR to ProgIR requires assigning a finite-precision rounding operation to each mathematical function.

Returning to Herbie’s rewriting phases, observer the approximate type signatures of the three rewriting steps:

val taylor : SpecIR -> SpecIR list
val rr : ProgIR -> ProgIR list
val simplify : ProgIR -> ProgIR list

Why the difference? In truth, engineering challenges. The egg-based rewriters rr and simplify are newer and an important part of the Pareto-Herbie (Pherbie) design from 2021. On the other hand, the first version of Herbie included taylor as far back as 2014. Clearly, ProgIR is richer than SpecIR since it contains number format information. For this reason, both rewriting and precision tuning and performed in rr (see Pareto-Herbie) and a shim is required for taylor.

But in light of a recent conversation I had with a fellow PLSE member, I hypothesize that this architecture is likely incorrect, or at the least, problematic. Of course, we could dream of the “sufficiently smart” numerical compiler that resembles the following architecture:

SpecIR, error bound ~~~~~~ magic ~~~~~~> C, machine code, etc.

But just as a compiler achieves its goal via numerous lowering passes, so too should a system like Herbie. In fact, Herbie really is just a messy version of this compiler taking a mathematical specification and number representations producing floating-point code. In light of these observations, I propose rearchitecting Herbie’s improvement loop. Expanding on the numerical compiler diagram, we would implement

SpecIR ~~~~~~ rewrite ~~~~~~> SpecIR ~~~~~~ lowering ~~~~~~> ProgIR

There are two important phases:

• rewrite: either equivalent rewrites or approximations producing a program over real numbers
• lowering: assignment of rounding operations, e.g. precision tuning, and operator selection, and other decisions required for finite-precision computation.

Of course, just as in Herbie’s current architecture, we use this process potentially multiple times with expression selection at the beginning and pruning at the end. Therefore, the whole process looks something like the following:

1. Expression selection
2. Local error analysis
3. Lifting from ProgIR to SpecIR
4. Rewriting
   1. Taylor polynomial approximation ("taylor")
   2. equivalent (real-number semantics) rewrites ("rr")
5. Lowering from SpecIR to ProgIR
   1. operator selection, e.g. reciprocal vs. division
   2. precision tuning
6. Pruning

How (4) and (5) interact is an open question. An interesting proposal would be to seed an egraph with the starting subexpressions from (3) and the approximations from (4a), run rewrites from (4b), and extract from the same egraph in (5a) and possibly (5b). Additionally, it is unclear if separating (4) and (5) will prevent us from finding certain rewrites.

The new design requires that egg be used for both SpecIR and ProgIR which suggests splitting the rules into those over SpecIR and those over ProgIR. The majority of rewriting will be done over the reals, but a small set of rewrites may be used in (5) for operator selection, e.g. posit-quire operations. Constant folding should only be done over SpecIR since these programs are over the reals. If rewriting ProgIR is done in a egraph, it will be minimal (no constant folding) and will just be for extraction via Herbie’s cost model.

Based on this new design, we have a number of insights and engineering improvements:

• if we wish to rewrite expressions over the reals, then the rewrites should be performed over real number programs
• operator (implementation) selection is precision tuning; choosing to approximate 1/x (real number program) with recip_f64(x) is both a choice of syntax and rounding.
• egg becomes a pure syntactic rewriter; it need not know about representations; Herbie’s egg interface can be slimmed down, e.g. no rule expansion and a simpler EggIR.

To summarize, I propose a new design for Herbie’s improvement loop. Problematic expressions should be identified via the normal expression selection and local error analysis phases. These expressions should be lifted to purely real-number programs and passed through two phases. The first phase applies equivalent rewrites (over the real numbers) and polynomial approximations. The second phase lowers to actual floating-point operations via operator selection and precision tuning. Finally, the same pruning operation is performed to shrink the set of useful alternative implementations.

The Early Days

The initial versions of Minim were hacky at best. The goal of development during Minim 0.1.x was expanding the number of available types. By version 0.1.2, the language supported booleans, exact and inexact numbers, symbols, strings, pairs, lists, user-defined functions, hash tables, vectors, and sequences. Nearly all of these types since then have undergone structural changes. The set of procedures to create and modify these procedures was minimal at the time, just enough to be useful.

The worst feature of this era was the owner-reference system that kept track of objects, so it could free resources when objects went out of scope. Initially, new copies were created every time objects were referenced, but with an “improvement”, objects could be set as owners of their data or references of another object’s data. This required unnecessary amounts of copying objects, annoying equality checks, and hours of debugging segmentation faults. What I didn’t know at the time was Scheme implementations usually never copy immutable objects and use garbage collectors to know when to free memory. This issue was not fixed until I finally added my own garbage collector in version 0.3.0.

Expansion

With a plethora of types in the language, the focus for Minim 0.2.x shifted to increasing the number of procedures available. First, file reading was added to support a standard library that was not hard-coded in C. In these versions, file reading occured separately from the existing parser, tracking parentheses and ignoring comments to extract a hopefully parsable string. Initially, reading source code was rough since the reader would spontaneously fail, leading to hours of searching for and fixing bugs, and often to my frustration, the creation of new bugs. In the same release, I added errors with backtracing and syntax with source locations which proved to be helpful when modifying the standard library.

Minim 0.2.1 and 0.2.2 were expansions of the math and list libraries. I put as many of the new procedures to the standard library as I could rather to the already large set of primitive procedures. I also added arity and type errors so that these new procedures could reject arguments upfront and print a more descriptive error. By then, issues with parsing motivated me to remake the parser from scratch which turned out to be a huge improvement for the language. Since then the parser has barely changed and no longer hinders development.

Standardization

Development of Minim 0.3.x felt quite different from the previous versions. Primarily, I began reading the existing standards on Scheme including R4RS, R5RS, and R6RS. For the most part, I ignored these standards before because I wasn’t interested in sticking to an existing blueprint for Minim. However, my design choices began to feel flawed and haphazard and the way forward was becoming unclear. Implementing procedures and features described in the standards became the main goal during this era of development.

The first few changes were performance-related: the addition of a garbage collector and the implementation of tail calling. I detailed the design of the garbage collector in my previous blog post which you can read here. These changes caused a significant slow down in performance, but accelerated the pace of development since I didn’t have to worry about memory management, aside from a few bugs that needed patching.

With garbage collection in place, I turned to important features of any Scheme language: quoting and syntax macros. Syntax macros turned out to be quite the headache; my intial implementation continually caused Minim to crash. I eventually reimplemented macros from scratch, but I found out that even my most recent attempt is still not compliant with the standard. As of today, I have considered that part of the project to be “good enough”, but fixing it is still definitely on my to-do list.

After the syntax macro mess, I moved on and added types like characters, records, and file ports; additional procedures for strings and lists; and more features like multi-valued expressions, continuations, and multi-signature functions with case-lambda.

Today

And with that, we have finally reached the present day. As you have just read, the development of Minim has been a long and winding path, from basic and hacky beginnings to a much more robust implementation. If there’s anything I’ve learned, it’s that a small idea can be fully realized with time and effort.

As for those following my footsteps: I’d highly recommend reading standards for an existing language. You might not be able to implement everything, but I found that following the Scheme standards made Minim a much more robust and sensical language. Standards are an important part of language design no matter how long and dense they might seem.

The Future

What’s next for Minim? As I’ve mentioned before, syntax macros are not Scheme-compliant, but there are many more features of Minim that have deviated from the standard, usually because of my naive choices. A couple examples include the use of def instead of define and the syntax of function definitions. I am still weighing whether or not to resolve these design differences.

More recently, I have implemented caching for Minim source code files: syntax macros are applied and the resulting desguared code is emitted for later use. Testing shows that this decreases the number of expressions executed on boot significantly. On top of caching, I have implemented constant folding since certain expressions can be resolved before runtime. In particular, my implementation of case-lambda is egregious in its use of constant expressions for resolving arity.

My target goal is to implement a native-code compiler for Minim, but having just begun a compilers course this quarter, I have a feeling this may be a long ways away. Until then, I will focus on implementing more of the standard. Please check out the source repository for Minim to see my progress and give the language a try!

The Minim GC is a generational, conservative, mark-and-sweep garbage collector implemented in C based on the Tiny Garbage Collector, a minimal garbage collector. It expands on the TGC by separating allocations into two generations, young and old. The younger generation is marked and swept every cycle, by default every time new allocations total 8 MB in size. Any surviving allocations are moved to the older generation. The older generation is marked and swept every 15 cycles.

The stack is swept from the “bottom” of the stack, provided during initialization, to a local address within the sweeping function. A neat trick to always forcing pointer addresses on the stack no matter the optimization level is to use the setjmp(jmp_buf env) function from setjmp.h which saves the values of registers to set a jump point. It makes sense that this function exists, but I honestly didn’t realize it could be used for this mechanism.

Like the TGC, the Minim GC provides a way to associate a destructor with an allocated block in case memory allocated outside of the garbage collector needs to be freed upon sweeping. However, it also provides a way to associate a “marking” function with each memory block. These functions precisely mark internal pointers so that the garbage collector doesn’t naively mark every possible pointer within the memory block. These functions can cause issues if a developer changes a struct but not its respective marking function; I ran into this issue on a few occasions when switching Minim to use the garbage collector. In addition, atomic allocation macros are provided to allocate data not containing any pointers.

Although naive, the Minim GC has made working with the Minim code base much easier since I don’t have to spend hours dealing with segmentation faults. If I had to make Minim all over again, I definitely would have stuck with using a garbage collector from the beginning. It eliminates my owner-and-reference system that was causing quite a headache when dealing with copies of objects. Now the same object can be in a list, hash table, and vector, all at the same time, since immutability is respected.

As for perfomance issues, I managed to claw back almost all of the slow down by adding tail call optimization which ensures the amount of allocated memory and the size of the stack remains quite small. This garbage collector along with tail call optimization, quasiquoting, and syntax macros will be in the next release, and is currently in the main branch.

From Static to Dynamic Types

My language of choice for developing Minim is C. Not the best choice for a smooth experience, but it’s been quite interesting. The most annoying part is obviousing dealing with pointers; nearly every bug I’ve run into is a segmentation fault. The most interesting concept, however, is the interaction of Minim’s dynamic type system with C’s static type system.

Mainly, how do you define a dynamic type system in a language that is statically typed, especially without classes like in C++ or Java? The best solution that I’ve found is void pointers and enums. Intuitively, store a void pointer and an enumerated value such that the value hints at what is stored at the pointer. With this method, we can store anything from an integer to a string in a unified object, or the MinimObject in the case of the Minim language. Minim can easily verify the type of inputs when invoking a function and can throw an error when necessary.

Of course, all of this is abstracted away when running Minim. The user can still identify the type of an object with procedures such as number? or string?, but they need not worry about its initialization, storage, and deletion. An added benefit is multiple types of objects can be stored in the same list such as (list 1 'a "str"). This concept may be trivial, but it is quite interesting to see that a simple construct can do away with the restrictions of statically-typed code.

Parsing, Parsing, Parsing

The worst experience, by far, has been figuring out how to parse an input string without any fancy libraries. For the most part, if we can munge the string, it’s not too awful. Split the string by looking for spaces and parentheses/brackets. New lines and tabs are really just spaces in Minim, so they’re not too useful like Python. All we have to do is keep track of quoted strings, Lisp-style quotes, and comments.

Unfortunately with version 0.2.0, Minim supports executing expressions from a file, and errors without syntax information are quite unhelpful. Therefore, we need to store the syntax information of where expressions and functions are located. We can’t munge the string before parsing since we lose information about row and column numbers of characters. We must (a) track row and column information, (b) ignore distracting whitespace, and (c) still tokenize words. I managed to pull it with a reader thats around 200 lines long and a parser of similar length, but it’s quite a mess.

Nevertheless, the results have been impressive. Backtraces from errors are quite detailed and they print out “stack frames” with the following format: <file> <row>:<col> <name>. It’s crudeness will definitely be a problem in the future, but for now it works.

Problems to Come

The most broken part of Minim is the blurry distinction between owners and references, and the lack of separation between mutable and immutable objects.

The first problem borrows a concept from C++. In brief, certain objects are the original owners of their information. It’s better to pass a reference of that object to a function rather than the entire copy since it takes less space and is less cumbersome than pointers that are dominant in C. Minim also implements this system, since it’s less resource intensive to not copy lists every time we use them for read-only purposes. However, without a static type system, the use for this is more subtle. Every built-in function in Minim needs to have two separate cases for owners and references, and this parallel strategy causes many issues. It’s not ideal, but it seems to work.

The second problem is a distinction that Racket makes clear: there are immutable objects and there are mutable objects. In Racket, the two different types of objects each have their own set of procedures. Initially, I chose not to care since it seemed cumbersome to have two sets of procedures. Invoking an in-place update of any hash table seemed reasonable, but it’s problematic with function calls and references. There are a number of examples that I can think of that will break Minim.

Conclusion and Future Work

Although, I spent much of this blog talking about what is bad, there has been a lot of good. Most importantly, I’ve learned quite a bit about developing something as complex as a “lightweight” programming language. As of the time I’m writing this, the repository is well over 10,000 lines of code and the language contains over 120 procedures and numerous types like lists, strings, hash tables, vectors, and more.

In the next update, there will be considerably more procedures in the “standard library” I’ve been developing. They will mostly include math functions like gcd and lcm. More list procedures are a must since lists form the backbone of any Lisp/Scheme language. Additionally, I need to resolve the issues mentioned above as well as making procedures proper closures (storing the environment from which they were created).

This blog was long-winded mostly because there was a lot to talk about. I hope to write more about Minim in the near future as it develops from the small fledgling it is today to a language that is full and robust. Stay tuned for more.

While the math/bigfloat interface is sufficient for most high-precision computing, it is lacking in a couple areas. Mainly, it does not properly emulate subnormal arithmetic or allow the exponent range to be changed.

Normally, neither of these problems cause concern. For example, if a user intends to find an approximate value for some computation on the reals, then subnormal arithmetic or a narrower exponent range is not particular useful. However, if a user wants to know the result of a computation specifically in some format, say half-precision, then math/bigfloat is insufficient.

At half-precision, (exp -10) and (exp 20) evaluate to 4.5419e-05 and +inf.0, respectively. On the other hand, evaluating (bfexp (bf -10)) and (bfexp (bf -10)) with (bf-precision 11) returns (bf "4.5389e-5") and (bf "#e4.8523e8"). While the latter results are certainly more accurate, they do not reflect proper behavior in half-precision. The standard bigfloat library does not subnormalize the first result (no subnormal arithmetic), nor does it recognize the overflow in the second result (fixed exponent range).

This library fixes the issues mentioned above by automatically emulating subnormal arithmetic when necessary and providing a way to change the exponent range. In addition, the interface is quite similar to math/bigfloat, so it will feel familiar to anyone who has used the standard bigfloat library before. There are also a few extra operations from the C math library such as gflfma, gflmod, and gflremainder that the bigfloat library does not support.

See math/bigfloat for more information on bigfloats.

To read more of the documentation, please visit here. The source code for the package can be found at this repository. In the future, I plan on integrating this into the FPBench reference interpreter, so we can finally emulate subnormal arithmetic for various floating-point formats correctly. This has been an outstanding issue for a long time.

Life Update: I’m currently in the first quarter of my sophomore year and things are going fairly well considering the state of the world. My work in Herbie has started moving much faster with my exploration of multi-precision expressions and cost/accuracy Pareto curves. Today, I began work on a generic IEEE-754 floating-point plugin for Herbie which will be very useful in the future.

One of the problems I had run into was the lack of subnormals in Racket’s bigflonum library - the language’s MPFR interface. Fortunately, the FFI procedures buried in the math library overall proved to be quite useful, so now I have a parallel set of operations that correctly handles subnormals and checks the range on the result. I emphasize the fact that it is parallel, since I still have access to the original, arguably more practical, set of operations that allow floats to have exponents somewhere on the order of 2^15 and ignore subnormals entirely. On a side note - it would be excellent to see support for limiting the exponent size and doing subnormal arithmetic in Racket’s library; however, I may be the only person in the world currently in need of such features, so… maybe not?

Another problem is the mapping of ordinals since such procedures are done on the Racket side of things, but they have no regard for subnormal numbers and seem to be causing issues on the Herbie side of things. Now that I think about it, it seems that all my problems have to do with subnormal floating point numbers. Go figure.

I see some possibility in separating out this new set of MPFR bindings and creating a generic floating point package that allows a user to specify a float with a given significand and exponent. It would be useful in FPBench since some of the tooling is very broken.

On a separate note: My website is quite new. It’s only been up for one week!! I’ve been working on it recently, and things are looking good. I began using Racket to generate my html pages, so everything is much easier to work with.

Anyways, that’s all my thoughts for today. Maybe next time I’ll write something more technical rather than rambly… Stay tuned for more!!