internal/lsp: add gopls semtok command and semantic token tests

The command 'gopls semtok <file>.go' produces a version of <file>.go
decorated with comments describing tokens as semantic tokens. The
format is described towards the top of cmd/semantictokens.go.

There are also tests in testdata/semantic/. a.go is used by lsp_test.go
(with -cover) to show that essentially all of semantic.go is executed.
b.go illustrates the handling of strange, but legal, constructs
like 'const false = nil and non-ascii'. The .golden files show
the expected results.

The implementation will panic, rather than log, on unexpected state.

gopls semtok has been run on all the .go files in x/tools and in
the source distribution. There were no unexpected messages nor panics.

Change-Id: I11715bcc8d6830d1951eb58239978c4a363bbc30
Reviewed-on: https://go-review.googlesource.com/c/tools/+/262198
Run-TryBot: Peter Weinberger <pjw@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Trust: Peter Weinberger <pjw@google.com>
Reviewed-by: Robert Findley <rfindley@google.com>
13 files changed
tree: 9f8a69923d1b7262c65b85c61a633211e9bbee0e
  1. benchmark/
  2. blog/
  3. cmd/
  4. container/
  5. cover/
  6. go/
  7. godoc/
  8. gopls/
  9. imports/
  10. internal/
  11. playground/
  12. present/
  13. refactor/
  14. txtar/
  15. .gitattributes
  16. .gitignore
  17. .prettierrc
  18. AUTHORS
  19. codereview.cfg
  20. CONTRIBUTING.md
  21. CONTRIBUTORS
  22. go.mod
  23. go.sum
  24. LICENSE
  25. PATENTS
  26. README.md
README.md

Go Tools

This subrepository holds the source for various packages and tools that support the Go programming language.

Some of the tools, godoc and vet for example, are included in binary Go distributions.

Others, including the Go guru and the test coverage tool, can be fetched with go get.

Packages include a type-checker for Go and an implementation of the Static Single Assignment form (SSA) representation for Go programs.

Download/Install

The easiest way to install is to run go get -u golang.org/x/tools/.... You can also manually git clone the repository to $GOPATH/src/golang.org/x/tools.

JS/CSS Formatting

This repository uses prettier to format JS and CSS files.

The version of prettier used is 1.18.2.

It is encouraged that all JS and CSS code be run through this before submitting a change. However, it is not a strict requirement enforced by CI.

Report Issues / Send Patches

This repository uses Gerrit for code changes. To learn how to submit changes to this repository, see https://golang.org/doc/contribute.html.

The main issue tracker for the tools repository is located at https://github.com/golang/go/issues. Prefix your issue with “x/tools/(your subdir):” in the subject line, so it is easy to find.