modfile: remove trailing newline from comment tokens

In v0.2.0, the go.mod lexer removed trailing LF bytes from comment
tokens. This regressed in v0.3.0. Documentation on Comment.Token says
the trailing newline should not be included.

This CL fixes the lexer to strip trailing newlines again. It will now
strip both LF and CRLF newlines. It also includes a test to ensure
comments are attached at the right place in the syntax tree with the
right content.

Fixes golang/go#39913

Change-Id: I7fba0ed3c85f0a3c23fefc6b7fecfe6df7777aea
Run-TryBot: Jay Conrod <>
TryBot-Result: Gobot Gobot <>
Reviewed-by: Michael Matloob <>
2 files changed