tree 246d7d40cb6b839b9ef560bb7b0ee2de4e0b1fee
parent 3fc05d484e9f77dd51816890e05f2602e4ca4d65
author Richard Musiol <mail@richard-musiol.de> 1554838885 +0200
committer Nigel Tao <nigeltao@golang.org> 1555466981 +0000

vector: fix tests on wasm

On wasm, calculations on float32 values are done with 64 bit precision.
This is allowed by the Go specification, only explicit conversions to
float32 have to round to 32 bit precision. The difference caused by the
additional precision accumulates over several calculations and causes
the test results to not fully match the expectations. Account for this
by giving a 0.1% tolerance.

Fixes golang/go#31281.

Change-Id: I843788f912015600a18ff3d5cf5520c60403b534
Reviewed-on: https://go-review.googlesource.com/c/image/+/171257
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Nigel Tao <nigeltao@golang.org>