Nan logits when performing inference using ModernBERT

[yaswanthg15]

System Info

transformers == 4.48.0.dev0
torch == 2.2.2

Description

I have finetuned ModelBERT model for multi-label task and when performing batched inference I am getting Nan values for logits in a batch except for one sample. So, in each batch except one sample all the remaining logits are Nan

Who can help?

@tomaarsen @ArthurZucker

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Simply passing model(**batch) is resulting in this - while with bs=1 no issue exists.
Code:

model.eval()
labels, predictions, confidences = [], [], []

with torch.no_grad():
    for batch in tqdm(val_loader):
        labels.extend(batch['labels'].cpu().numpy())
        batch.pop('labels')
        batch = {k: v.to(device) for k, v in batch.items()}

        outputs = model(**batch)

        probs = torch.sigmoid(outputs.logits)
        preds = (probs > 0.5).int()
        confs = probs

        predictions.extend(preds.cpu().numpy())
        confidences.extend(confs.cpu().numpy())

Expected behavior

Expecting some values instead of Nan

[tomaarsen]

Could you add a snippet to reproduce? I've done lots of inference and I've never seen nans yet.

• Tom Aarsen

[yaswanthg15]

@tomaarsen While it is not possible for me to make the model public - but coming to the issue I am performing a simple inference using pytorch loop.

[ArthurZucker]

In order to help we would need to know:

• the dtype
• the attention implementation
  This should help us as a lot of nans can come from, sdpa or casting