Proposal and study of statistical features for string similarity computation and classification

2026-05-14 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

Researchers propose adaptations of co-occurrence matrix (COM) and run-length matrix (RLM) features, traditionally used in visual computing, for general string similarity computation across words, phrases, codes, and texts. These proposed features are language-agnostic and purely statistical, making them applicable across diverse linguistic and grammatical structures. The study evaluates COM and RLM against established statistical measures like longest common subsequence, maximal consecutive longest common subsequence, mutual information, and various edit distances. In synthetic experiments, COM and RLM features significantly outperformed other state-of-the-art statistical features, demonstrating statistical significance (P-value < 0.001) in three out of four cases. Furthermore, RLM features achieved the best results when applied to a real text plagiarism dataset.

Key takeaway

For research scientists developing string similarity algorithms, you should investigate integrating co-occurrence matrix (COM) and run-length matrix (RLM) features. These methods demonstrated superior performance and language independence in both synthetic and real-world plagiarism detection tasks, potentially offering more robust and versatile solutions than traditional statistical measures for your text analysis and matching applications.

Key insights

Visual computing features, COM and RLM, offer superior, language-agnostic string similarity computation.

Principles

• Statistical features can transcend domain boundaries.
• Language-agnostic methods enhance string analysis.
• COM and RLM improve string similarity metrics.

Method

Adapt co-occurrence matrix (COM) and run-length matrix (RLM) from visual computing to analyze string patterns for similarity, then compare against traditional statistical measures like edit distances.

In practice

• Apply RLM for plagiarism detection.
• Use COM/RLM for cross-language text analysis.
• Integrate these features into string matching algorithms.

Topics

• String Similarity Computation
• Co-occurrence Matrix
• Run-length Matrix
• Statistical Features
• Plagiarism Detection

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• RARE: Redundancy-Aware Retrieval Evaluation Framework for High-Similarity Corpora — RARE framework improves RAG evaluation by accounting for document redundancy in high-similarity real-world corpora.
• Intro to NLP with spaCy (1): Detecting programming languages — spaCy's linguistic features enable robust rule-based NLP systems beyond simple string matching.
• Anatomy of Data Repositories for the Analysis and Detection of Toxicity in Portuguese — Lexical diversity and annotation practices significantly impact cross-domain generalization in hate speech detection datasets.
• Three Buddhist Vocabularies: Computational Stylometry of the English Pali Canon across Sutta, Vinaya, and Abhidhamma — Computational stylometry reveals distinct lexical patterns and significant translation shifts across Buddhist canonical texts.
• Operation-Guided Progressive Human-to-AI Text Transformation Benchmark for Multi-Granularity AI-Text Detection — AI-text detectability is non-monotonic, influenced by edit operations and revision history, not just AI content proportion.
• Discovery of Legal Patterns in Civil Petitions via LLM-Based Fact Extraction and Density Clustering — LLM-based fact extraction significantly improves legal document clustering by reducing noise and enhancing semantic coherence.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.