medium10 pts · first answer countstransformersquantizationinference

Why 4-bit weights speed up decoding

A server uses W4A16 quantisation (4-bit weights, 16-bit activations) via AWQ. Why can this make single-user decoding faster, not just smaller?

A.Decode is bandwidth-bound, and 4-bit weights are fewer bytesB.4-bit arithmetic is always four times faster than 16-bit arithmeticC.Activations are skipped entirely in 4-bit modeD.The model generates four tokens per forward pass

Discussion 0

Join free to post in the discussion.

Why 4-bit weights speed up decoding

Discussion 0

Related questions