When I started this assignment I honestly had no idea where to begin.

2026-03-24 · Source: Data Science on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Advanced, quick

Summary

A natural language processing (NLP) system was developed from scratch using pure PyTorch to find similar customer support tickets from a dataset of 8,469 complaints. The system encodes categorical data like ticket priority and channel using Label Encoding and One-Hot Encoding, respectively. It features a custom TF-IDF implementation, including a regex tokenizer, a 5,000-word vocabulary, and bigram/trigram generators, with scores stored as sparse tensors. Semantic understanding is added via 300-dimensional GloVe embeddings, handling out-of-vocabulary words with random normal vectors and weighting embeddings by TF-IDF scores. A hybrid search combines TF-IDF (0.4 weight) and GloVe (0.6 weight) for improved accuracy. Optimized on dual Kaggle T4 GPUs, the system processes 100 queries in 0.141 seconds, achieving an average of 1.41ms per query and a Precision@5 of 21.10%. An interactive Gradio web app allows users to query the system and adjust the hybrid search alpha slider.

Key takeaway

For Machine Learning Engineers building semantic search systems, integrating both TF-IDF and GloVe embeddings offers a robust approach to balance keyword relevance and deep semantic understanding. Your implementation should consider weighting these components, such as 40% TF-IDF and 60% GloVe, to achieve optimal performance. This hybrid strategy can significantly improve query results, as demonstrated by GloVe's 2x higher scores in semantic queries, and can be efficiently deployed using GPU parallelism.

Key insights

Combining TF-IDF and GloVe embeddings improves semantic search accuracy for customer support tickets.

Principles

• Rare words carry higher TF-IDF scores.
• GloVe captures semantic meaning beyond keywords.

Method

Build a hybrid search system by combining TF-IDF and GloVe embeddings, weighting them to balance keyword matching and semantic understanding. Handle OOV words with random vectors.

In practice

• Use `torch.nn.DataParallel` for GPU optimization.
• Implement custom tokenizers for specific data needs.
• Weight word embeddings by TF-IDF for better sentence vectors.

Topics

• Natural Language Processing
• Semantic Search
• TF-IDF
• GloVe Embeddings
• PyTorch

Code references

• hashimrana478-bot/HSRIS-NLP-Assignment

Best for: Machine Learning Engineer, Deep Learning Engineer, AI Student

Related on AIssential

• How I Built a Smart Ticket Search System Using PyTorch and GloVe — Combining TF-IDF with GloVe embeddings creates a robust hybrid search for semantic and keyword matching.
• Building a Hybrid Semantic Search Engine from Scratch: A Deep Dive into TF-IDF, GloVe, and Dual GPU… — Hybrid search combining TF-IDF and GloVe embeddings significantly improves intent-based query relevance over keyword-only methods.
• The Engine Behind Information Retrieval: Exploring TF-IDF — TF-IDF quantifies word importance by balancing local frequency with global rarity, enabling effective information retrieval.
• From TF-IDF to Transformers: Implementing Four Generations of Semantic Search — Semantic search evolved from explicit rules to learned contextual representations, balancing interpretability with semantic depth.
• Building Context-Aware Search in Python with LLM Embeddings + Metadata — Context-aware semantic search combines embedding-based similarity with metadata filtering for relevant results respecting contextual constraints.
• IdioLink: Retrieving Meaning Beyond Words Across Idiomatic and Literal Expressions — Idiom understanding in retrieval requires semantic abstraction beyond lexical overlap.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Data Science on Medium.