cmd/objdump: fix REP/REPN Plan 9 disassembly

This change contains two fixes:
  1. emit REP/REPNE as separate instructions, not prefixes
  2. don't emit REP/REPNE when the prefix is implied by instruction
     text

Both make the objdump output more consistent with the code in Go tree
and with the output of binutils objdump.

1. According to http://doc.cat-v.org/plan_9/4th_edition/papers/asm
"Only REP and REPN are recognized repeaters. These are not prefixes,
but rather stand-alone opcodes that precede the strings"

For example, before this change "f3a6" was disassembled as:
  REP CMPSB ES:0(DI), DS:0(SI)
After this change it becomes:
  REP; CMPSB ES:0(DI), DS:0(SI)

2. According to Intel Manuals (Vol. 2A)
"Repeat prefixes (F2H, F3H) cause an instruction to be repeated for
each element of a string. Use these prefixes only with string and I/O
instructions (MOVS, CMPS, SCAS, LODS, STOS, INS, and OUTS). Use of
repeat prefixes and/or undefined opcodes with other Intel 64 or IA-32
instructions is reserved; such use may cause unpredictable behavior"

Certain instruction imply repeat prefixes. For example POCNT is
"F30F", which has the "F3" prefix. Before this change we would emit
"REP" prefix for such instructions:
   REP POPCNT 0(CX), DX
After this change
   POPCNT 0(CX), DX

The "REP RET" optimization.

The AMD Software Optimization Guide describes "Two-Byte Near-Return
RET Instruction" (adding REP prefix to RET: "f3c3"; instead of just
RET "c3"). In this case the prefix is not implied by the instruction,
but RET is not one of the string instructions either.  It's unclear
whether it should
(binutils objdump does).

Fixes golang/go#17410

Change-Id: I7ec9de7f99b0e1fd5f7b96d1dc6dcb977f6454a6
Reviewed-on: https://go-review.googlesource.com/36770
Reviewed-by: Keith Randall <khr@golang.org>
2 files changed