Quantizelinear of float16 -> int8 results unexpected

[SStarver]

Issue with Rounding Behavior in onnxruntime's Quantizelinear Layer

It appears that when processing an independent Quantizelinear layer in onnxruntime, the rounding behavior is consistently rounding to the lower integer instead of the expected 'rounding to nearest even'. Does anyone have an idea about why this might be happening?

Experiment

I have constructed a single-layer Quantizelinear onnx model with nearly random input data. The results are as follows:

• The first row represents the input.
• The second row represents the multiplication result of the input and scale.
• The third row represents the result rounded to the nearest even of the second row.
• The last row represents the related onnxruntime inference result.

Code

import onnx_graphsurgeon as gs
import numpy as np
import onnxruntime
import onnx


def gen_input():
    graph_input = np.array(
        [[[[2.11, 2.5, 2.6], [5.3, 5.5, 5.9], [61.3, 62.5, 80.9]]]], dtype=np.float16
    )
    return graph_input


def run_onnx_runtime(onnx_file, gi):
    onnx_session = onnxruntime.InferenceSession(
        onnx_file, providers=["CPUExecutionProvider"]
    )
    inps = {}
    for inp in onnx_session.get_inputs():
        inps.update({inp.name: gi})
    outname = [onnx_session.get_outputs()[0].name]
    data_output = onnx_session.run(outname, inps)[0]
    return data_output


def get_scale(
    onnx_model,
):
    graph = gs.import_onnx(onnx.load(onnx_model))
    graph.fold_constants().cleanup()
    scale = graph.nodes[0].inputs[1].values
    return scale


def run_hand_infer(gi, scale):
    tmp = gi.astype(np.float32) / np.float32(scale)
    return tmp, np.round(tmp).astype(np.int8)


def print_input(gi):
    print(gi)


def print_outputs(inp, handle_out, rounded_handle_out, onnx_out):
    _, _, rows, cols = inp.shape

    for i in range(rows):
        for j in range(cols):
            print(
                f"{inp[0,0,i, j]:.3f}, {handle_out[0,0,i, j]:.3f},{rounded_handle_out[0,0,i, j]:.3f}, {onnx_out[0,0,i, j]:.3f}"
            )



if __name__ == "__main__":
    onnx_file = "q_1x1x3x3.onnx"
    gi = gen_input()
    onnx_out = run_onnx_runtime(onnx_file, gi)
    scale = get_scale(onnx_file)
    handle_out, rounded_handle_out = run_hand_infer(gi, scale)
    print_outputs(gi, handle_out, rounded_handle_out, onnx_out)

onnx model

Download ONNX Model