Tolerance

The Tolerance parameter appears in many transformers that perform spatial relationship tasks such as intersecting, overlaying, and dissolving.

Tolerance is a distance in ground units. Points within that distance of each other may be adjusted to exactly the same location.

This value is often very, very small - but is enough to avoid artifacts and improve results. It is never zero.

In general, using the Automatic setting is recommended.

Why do we need tolerance?

There are multiple reasons for including tolerance in spatial analysis:

• Points representing the same position in the real world may have slightly varying coordinates due to measuring methods, level of accuracy, and error.
• Spatial calculations can produce coordinates with precision well beyond useful.
• Computations are done with 64-bit arithmetic, which can introduce and accumulate error.
• Numbers are often rounded to fit within storage size requirements. This rounding error can propagate through further computation and result in misalignment and gaps between geometries.

The variances are miniscule, but must be accounted for.

Location and Intersections

Coordinates exist on a grid. In most cases, the intersection of two lines will fall somewhere between grid points, with calculated coordinate values that have precision (decimals) well past what is accurate and useful in the coordinate space.

If these results are used for further calculations, tiny differences accumulate and snowball, producing undesirable (and meaningless) fragments of geometry.

Tolerance prevents this, snapping points together where appropriate.

Binary Storage, Math, and Rounding

Differences between coordinates at the same location may be introduced by the nature of computing and number storage.

Storing Decimals in 64-Bit Binary

Consider the result of dividing ten by three (10/3). The result is 3.3333333333… .The number 3 recurs to infinity, and is generally rounded to a useful precision (number of decimal places).

Similarly, some calculations end up recurring in binary.

One divided by five (1/5) in decimal is:

0.2

The result using binary math, however, is:

0.0111 1111 1001 0011 0011 0011 0011 0011…

The 0011 repeats to infinity just as 10/3 does in decimal. The computer rounds it off to the nearest mathematical value which is 0x3FC999999999999A in Hex, or in decimal:

0.200000000000000011102230246252

The difference is minute, but requires that tolerance be applied.

What tolerance value should be used?

If Automatic is an option, it generally should be used. The actual value is written to the log file.

In this log, a Dissolver is processing multiple groups of features in a UTM coordinate system (ground units in meters), and calculating the optimal tolerance value for each group.

Note the values are extremely small, and each one very slightly different.

Explicit values can be entered to override this behavior, but they must be greater than zero, and greater than the automatic tolerance value.

How is tolerance used?

Tolerance is applied using an Anchored Vertex Adjustment.

Features are anchored around existing points. Data is moved as little as possible, and features are more likely to be connected at topologically significant places.