TEDS score doesnt match the one reported on the paper

[dimitri009]

🔎 Search before asking

• I have searched the PaddleOCR Docs and found no similar bug report.
• I have searched the PaddleOCR Issues and found no similar bug report.
• I have searched the PaddleOCR Discussions and found no similar bug report.

🐛 Bug (问题描述)

I have a problem, when I try to run the TEDS score on PubTabNet with this prompt

python3 ppstructure/table/eval_table.py --det_model_dir=output2/en_ppocr_mobile_v2.0_table_det_infer --rec_model_dir=output2/en_ppocr_mobile_v2.0_table_rec_infer --table_model_dir=output2/en_ppstructure_mobile_v2.0_SLANet_infer --image_dir=/data/Datasets/pubtabnet/pubtabnet/val/ --rec_char_dict_path=ppocr/utils/dict/table_dict.txt --table_char_dict_path=ppocr/utils/dict/table_structure_dict.txt --det_limit_side_len=736 --det_limit_type=min --rec_image_shape=3,32,320 --gt_path=output2/gt.txt

and I get

ppocr INFO: teds: 0.4682228148448854

🏃‍♂️ Environment (运行环境)

paddleocr 2.9.1
paddlepaddle-gpu 3.0.0b1

OS x86_64
CUDA 12.4

🌰 Minimal Reproducible Example (最小可复现问题的Demo)

python3 ppstructure/table/eval_table.py --det_model_dir=output2/en_ppocr_mobile_v2.0_table_det_infer --rec_model_dir=output2/en_ppocr_mobile_v2.0_table_rec_infer --table_model_dir=output2/en_ppstructure_mobile_v2.0_SLANet_infer --image_dir=/data/Datasets/pubtabnet/pubtabnet/val/ --rec_char_dict_path=ppocr/utils/dict/table_dict.txt --table_char_dict_path=ppocr/utils/dict/table_structure_dict.txt --det_limit_side_len=736 --det_limit_type=min --rec_image_shape=3,32,320 --gt_path=output2/gt.txt

[GreatV]

The issue you are encountering—where the TEDS score or the inference results from the exported model do not match those seen during training—is likely due to discrepancies in how the model processes the data during training, evaluation, and inference. Below, we address possible root causes and provide steps to debug and resolve the problem.

---

Potential Causes and Solutions

1. Mismatch Between Training and Inference Configurations

• The exported inference model might use configurations (e.g., preprocessing, postprocessing, dictionary paths) that differ from those used during training or evaluation.
• Specific parameters like rec_char_dict_path, table_char_dict_path, or image resizing settings (det_limit_side_len, max_len, etc.) must be consistent.

Solution:

• Double-check that the parameters used in tools/export_model.py and table/predict_structure.py match the training configuration (SLANet_finetune.yml).
• For example:
  • Ensure --table_char_dict_path in inference uses the same dictionary as character_dict_path in training.
  • Ensure the image processing pipeline (e.g., ResizeTableImage, NormalizeImage) is identical between training and inference.

---

2. Incorrect Handling of Pretrained Weights During Export

• If the pretrained weights are not correctly loaded or saved during the export_model.py step, the exported model may not reflect the state of the trained model.

Solution:

• Verify that the pretrained_model provided during export points to the correct checkpoint:

  python3 tools/export_model.py -c configs/table/SLANet_finetune.yml \
    -o Global.pretrained_model=../translate-pdf-scan/src/weights/table_rec/SLANet_ch/best_accuracy \
    Global.save_inference_dir=../translate-pdf-scan/src/weights/table_rec/ch_ppstructure_mobile_v2.0_SLANet_finetune

• Ensure that best_accuracy.pdparams corresponds to the final checkpoint saved after training.

---

3. Postprocessing Issues in predict_structure.py

• During inference, postprocessing steps like TableLabelDecode may not match the training configuration. This could lead to incorrect predictions (e.g., mismatched HTML structure).

Solution:

• Ensure the PostProcess configuration in SLANet_finetune.yml matches the logic in predict_structure.py.
• Specifically, check the merge_no_span_structure parameter and other settings in TableLabelDecode.

---

4. TEDS Score Calculation Mismatch

• The TEDS score you calculate during evaluation may differ because of inconsistencies in how the ground truth (gt_path) is formatted or how predictions are processed.

Solution:

• Ensure that the ground truth file (output2/gt.txt) used during TEDS evaluation matches the expected format. Refer to the PubTabNet documentation for details on the expected format.
• Verify that the prediction outputs from eval_table.py align with the format of the ground truth.

---

5. Inconsistent Model Behavior in Training and Inference

• If the model behaves differently when running directly on infer_table.py versus predict_structure.py, it may indicate an issue with the export process or runtime.

Solution:

• Directly compare the outputs of the trained model (infer_table.py) and the exported model (predict_structure.py) on the same input image.
• Use the following to debug:

  python3 tools/infer_table.py -c configs/table/SLANet_finetune.yml \
    -o Global.pretrained_model=../translate-pdf-scan/src/weights/table_rec/SLANet_ch/best_accuracy \
    Global.infer_img=ppstructure/docs/table/table.jpg

  python table/predict_structure.py \
    --table_model_dir=../../translate-pdf-scan/src/weights/table_rec/ch_ppstructure_mobile_v2.0_SLANet_finetune \
    --table_char_dict_path=../ppocr/utils/dict/table_structure_dict_ch.txt \
    --image_dir=docs/table/table.jpg \
    --output=../output/table

• Compare the predictions. If discrepancies exist, focus on debugging the exported model.

---

Additional Debugging Steps

1. Visualize Outputs:

   • Save the predicted table structure and bounding boxes from both training and inference for the same input image. This can help identify issues in processing pipelines.

2. Enable Debug Logs:

   • Add debug print statements in predict_structure.py and eval_table.py to log intermediate outputs (e.g., after decoding).

3. Check PaddleOCR Version Compatibility:

   • Ensure that the PaddleOCR version (2.9.1) and PaddlePaddle version (3.0.0b1) are compatible. Incompatibilities can cause subtle bugs.

4. Refer to Similar Issues:

   • The issue Table Recognition: Training model is transferred to the inference model, the prediction effect is inconsistent mentions similar discrepancies. Ensure that the infer_mode parameter is properly set in your exported model.

---

Summary of Key Checks

• Ensure consistent preprocessing and postprocessing pipelines between training and inference.
• Verify that the correct pretrained weights are used during export.
• Debug and compare outputs from infer_table.py and predict_structure.py.
• Validate ground truth and prediction formats for TEDS evaluation.

By systematically addressing these areas, you should be able to resolve the discrepancies in the TEDS score and inference results.

---

Response generated by feifei-bot | chatgpt-4o-latest

[dimitri009]

I have this error even when I try with the model provided here using this prompt:

python3 ppstructure/table/eval_table.py --det_model_dir=output2/en_ppocr_mobile_v2.0_table_det_infer --rec_model_dir=output2/en_ppocr_mobile_v2.0_table_rec_infer --table_model_dir=output2/en_ppstructure_mobile_v2.0_SLANet_infer --image_dir=/data/Datasets/pubtabnet/pubtabnet/val/ --rec_char_dict_path=ppocr/utils/dict/table_dict.txt --table_char_dict_path=ppocr/utils/dict/table_structure_dict.txt --det_limit_side_len=736 --det_limit_type=min --rec_image_shape=3,32,320 --gt_path=output2/gt.txt

the gt.txt file I am using is correct:

PMC5755158_010_01.png <html><body><table><thead><tr><td></td><td><b>Weaning</b></td><td><b>Week 15</b></td><td><b>Off-test</b></td></tr></thead><tbody><tr><td>Weaning</td><td>–</td><td>–</td><td>–</td></tr><tr><td>Week 15</td><td>–</td><td>0.17 ± 0.08</td><td>0.16 ± 0.03</td></tr><tr><td>Off-test</td><td>–</td><td>0.80 ± 0.24</td><td>0.19 ± 0.09</td></tr></tbody></table></body></html> PMC4445578_009_01.png <html><body><table><thead><tr><td rowspan="2"><b>Reactive astrogliois</b></td><td rowspan="2"><b>Changes in astrocytes morphology</b></td><td colspan="2"><b>Changes in molecules expression</b></td></tr><tr><td><b>Upregulated molecules</b></td><td><b>Upregulated or downregulated molecules</b></td></tr></thead><tbody><tr><td rowspan="6">Mild to moderate astrogliosis</td><td>• Hypertrophy of cell body</td><td>• Structural elements: GFAP, nestin, vimentin</td><td>• Inflammatory cell regulators: cytokines, growth factors, glutathione</td></tr><tr><td rowspan="2">• Astrocytes processes are are numerous and thicker</td><td rowspan="2">• Transcriptional regulators: STAT3, NFκB, Rheb-m TOR, cAMP, Olig2, SOX9 [61–65].</td><td>• Transporters and pumps: AQP4 and Na<sup>+</sup>/K<sup>+</sup> transporters [61, 66–69]</td></tr><tr><td>• Glutamate transporter [70–73]</td></tr><tr><td rowspan="3">• The non-overlapping domains of individual astrocytes are preserved</td><td rowspan="3"></td><td>• Vascular regulators: PGE, NO [74, 75]</td></tr><tr><td>• Energy provision: lactate [76]</td></tr><tr><td>• Molecules implicated in synapse formation and</td></tr><tr><td rowspan="5">Severe astrogliosis and glial scar</td><td>• Intense hypertrophy of cell body</td><td></td><td>• Remodeling: thrombospondin and Complement C1q [77, 78]</td></tr><tr><td>• Significant extension of processes</td><td></td><td>• Molecules implicated in oxidative stress and providing protection from oxidative stress: NO, NOS, SOD, Glutathione [67, 68, 79]</td></tr><tr><td>• Proliferation</td><td></td><td></td></tr><tr><td>• Overlapping of individual domains</td><td></td><td></td></tr><tr><td>• Substantial reorganization of tissue architecture [60]</td><td></td><td></td></tr></tbody></table></body></html>

So I don't understand why I get this error, I am using the same models as the models in the documentations table_det and table_rec I still got the TEDS near to Zero.
ppocr INFO: teds: 0.4682228148448854