Scrap-SAM-CLIP: Assembling Foundation Models for Typical Shape Recognition in Scrap Classification and Rating
Authors: Guangda Bao, Wenzhi Xia, Haichuan Wang, Zhiyou Liao, Ting Wu, Yun Zhou
Abstract
To address the limitation of 2D methods in inferring absolute scrap dimensions from images, we propose Scrap-SAM-CLIP (SSC), a vision-language model integrating the segment anything model (SAM) and contrastive language-image pre-training in Chinese (CN-CLIP). The model enables identification of canonical scrap shapes, establishing a foundational framework for subsequent 3D reconstruction and dimensional extraction within the 3D recognition pipeline. Individual modules of SSC are fine-tuned on the self-constructed scrap dataset. For segmentation, the combined box-and-point prompt yields optimal performance among various prompting strategies. MobileSAM and SAM-HQ-Tiny serve as effective lightweight alternatives for edge deployment. Fine-tuning the SAM decoder significantly enhances robustness under noisy prompts, improving accuracy by at least 5.55% with a five-positive-points prompt and up to 15.00% with a five-positive-points-and-five-negative-points prompt. In classification, SSC achieves 95.3% accuracy, outperforming Swin Transformer V2_base by 2.9%, with t-SNE visualizations confirming superior feature learning capability. The performance advantages of SSC stem from its modular assembly strategy, enabling component-specific optimization through subtask decoupling and enhancing system interpretability. This work refines the scrap 3D identification pipeline and demonstrates the efficacy of adapted foundation models in industrial vision systems.
Conclusion
The automation and refinement of scrap classification and grading constitute a significant constraint on the green transformation of the steel industry in China. To overcome the limitations in interpretability and robustness inherent in conventional 2D scrap identification, this study proposes a novel 3D analytical framework. To avoid the unreliable fine-grained dimensional inference inherent in direct 2D image analysis, the framework integrates 2D/3D recognition with 3D reconstruction methodologies to recover precise 3D data from multi-view images, which enables accurate classification and grading. This paper investigates instance segmentation for canonical scrap shapes and proposes the SSC model, which leverages the strong generalization capability of foundation models to address the challenges of complex scrap sources and significant shape variations. Through lightweight fine-tuning, SSC achieves higher efficiency and lower computational cost compared to traditional recognition approaches.
The SSC framework utilizes an assembled architecture wherein the SAM and CLIP modules are independently fine-tuned. This design maintains specialized task performance while avoiding end-to-end optimization complexity. The segmentation module employs selective decoder fine-tuning and multi-prompt training, while the classification module uses a Chinese-optimized text encoder. Both components showed stable convergence, with optimal weights selected at epochs 41 and 9, respectively, confirming the efficacy of the decoupled training approach.
In segmentation tasks, EdgeSAM-3X performs best with precise prompts, MobileSAM offers faster inference speeds, and SAM-HQ-Tiny delivers greater robustness. Models at the L and H scales perform similarly. Box prompts achieve higher segmentation accuracy than point prompts, while the addition of an extra point prompt provides the best robustness. Fine-tuning the mask decoder of SAM-B substantially improves accuracy and robustness, especially under noisy prompt conditions, with performance gains of 5% to 15% (min S3: 5.55%↑; max S4: 15.00%↑).
For the classification task, fine-tuning CN-CLIP resulted in substantial performance improvements, with the tuned model significantly outperforming all specialized vision models. It achieved a leading accuracy margin of 2.9% over the second-best model, Swin Transformer V2_base. Visual analysis further confirmed the enhanced feature extraction capability of the SSC model, as evidenced by more distinct and better separated clustering boundaries in the t-SNE projection.
Overall, the proposed SSC model achieves high accuracy and robustness in instance segmentation of typical scrap shapes through assembling and fine-tuning of the base models SAM and CLIP, thereby enhancing the pipeline for 3D scrap recognition and providing effective automatic and semi-automatic annotation tools. Compared to purely 2D inference approaches, the proposed method improves the robustness and interpretability of scrap grading and classification, thus accelerating the automation of the entire scrap steel industry chain and supporting China’s transition toward short-process steelmaking.
Future Work
Within the SSC framework, the segmentation stability of the automated “everything” mode would benefit from enhancement, as its practical application remains heavily reliant on accurate prompt information. Therefore, future work should explore high-quality detection models, potentially by adapting the methodology presented in this study to leverage foundation models for scrap detection tasks, thereby improving overall recognition performance.
Furthermore, the designed Pipeline B for 3D scrap recognition relies on effective cross-view instance tracking to achieve accurate multi-view matching and provide reliable data for 3D reconstruction. While segmentation or classification errors can indeed affect reconstruction accuracy, the redundancy present in multi-view data provides a degree of robustness. Such inaccuracies can be mitigated through subsequent steps such as instance tracking, image retrieval, and image matching. By integrating methods like voting, clustering, or even graph neural networks, it is possible to filter out images or instance data with excessive errors.
The current 3D processing pipeline consists of multiple stages, including at least two key steps: 3D reconstruction and recognition. For the complex, high-dimensional task of extracting 3D scrap dimensions, exploring an end-to-end model may offer greater potential. However, such an end-to-end approach also faces several challenges, including reliance on large-scale, high-quality annotated data; the complexity of model design and training; and the effective integration of multi-view geometric constraints with semantic understanding.