Example — an illustrative record showing the WASTE article format. WASTE is pre-launch; this is not a published, peer-reviewed article.
Failure to Replicate Neural Network Performance on ImageNet Subset Using Published Hyperparameters
Supplementary materials are linked from the published record.
Abstract
Despite following the exact methodology described in the original paper, we were unable to replicate the reported 94.2% top-1 accuracy on the stated ImageNet subset, achieving only 87.1% (± 0.4) with identical published hyperparameters across five independent runs.
We document every deviation we could not eliminate — undisclosed data-augmentation ordering, an ambiguous learning-rate warmup, and a likely difference in evaluation-set construction — and quantify the contribution of each to the gap.
The original result is not necessarily wrong, but it is not reproducible from the published description alone. We release our full training and evaluation code to make the replication itself reproducible.
Failure Analysis
Stage: Replication / Reporting completeness
Primary causes
- Under-specified preprocessing and augmentation pipeline
- Ambiguous description of the evaluation subset
Boundary conditions: The original performance may be attainable with the authors' exact (unpublished) pipeline; this report bounds what is achievable from the public description.
Lessons Learned
- Reported accuracy was not reproducible from the paper's description without undisclosed implementation details.
- Data-augmentation ordering and evaluation-set construction accounted for most of the 7-point gap.
- Releasing replication code is as important as releasing the original model code.
WASTE is a fully open-access archive. All articles are free to read, download, and reuse under CC BY 4.0. No account required.
Have a failed experiment or null result to share? Submit your research.