I Got 100% Accuracy and Spent the Next Hour Trying to Prove It Was Wrong

2026-06-22 · Source: Machine Learning on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Intermediate, short

Summary

An AI exploration to develop a coin-sorting robot initially faced a significant challenge: a ResNet-50 model, trained on mobile-captured images for 20 coin categories, achieved 0% accuracy when deployed on a robot using its own camera for 3 specific categories (AUS_1, AUS_20, AUS_50). This failure was attributed to "domain shift" between the mobile training environment and the robot's operational environment. The solution involved a two-stage fine-tuning process. First, the model learned general coin features from mobile images. Then, it was fine-tuned again using robot-captured images for the target 3 categories. This adaptation successfully bridged the domain gap, resulting in a verified 100% accuracy on a held-out test set for AUS_1, AUS_20, and AUS_50 coins. The high accuracy was confirmed by healthy training curves and the distinct visual characteristics of the target coins.

Key takeaway

For Machine Learning Engineers deploying computer vision models, recognize that domain shift can lead to complete failure, even with well-trained base models. If your model performs poorly in a new environment, consider a two-stage fine-tuning approach: first, train on a broad dataset, then adapt to the specific target environment and distinct classes. Always verify unexpectedly high accuracy by scrutinizing training curves and class separability to ensure legitimate performance.

Key insights

Domain shift can cause complete model failure, but targeted fine-tuning can bridge the gap effectively.

Principles

• Domain shift severely degrades model performance across environments.
• Two-stage fine-tuning can adapt models to new visual contexts.
• Visually distinct classes aid in achieving high classification accuracy.

Method

Fine-tune a base model on a source domain, then fine-tune it again on a target domain's specific visual environment and classes.

In practice

• Adapt pre-trained models to new camera systems via fine-tuning.
• Verify 100% accuracy claims by checking training curves.
• Ensure target classes are visually distinct for robust classification.

Topics

• Domain Shift
• Computer Vision
• ResNet-50
• Fine-tuning
• Model Deployment
• Image Classification

Best for: AI Student, Machine Learning Engineer, Robotics Engineer

Related on AIssential

• Bridging Domain Expertise and Generalization for Performance Estimation — FRAP improves performance estimation under distribution shift by fusing foundation model robustness with base model domain expertise.
• Domain Generalizable Adaptation of 3D Vision-Language Models via Regularized Fine-Tuning — ReFine3D regularizes 3D LMM fine-tuning with multi-view consistency and text diversity to prevent overfitting and enhance domain generalization.
• [Advise] [Help] AI vs Real Image Detection: High Validation Accuracy but Poor Real-World Performance Looking for Insights — High validation accuracy does not guarantee real-world performance, indicating potential overfitting or dataset bias.
• Characterizing and Correcting Effective Target Shift in Online Learning — Online learning's sub-optimality stems from effective target shifts, which can be corrected to match offline performance.
• I Fine-Tuned DistilBERT Twice. It Humbled Me Both Times. — Data characteristics fundamentally determine fine-tuning performance and metric interpretation, not just the model.
• My EfficientNet Scored Worse Than Logistic Regression, Here’s What Changed… — Data pipeline quality and domain matching are more critical than model architecture for robust ML performance.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning on Medium.