InfoQ

Building Embedding Models for Large-Scale Real-World Applications

Sahil Dua discusses the critical role of embedding models in powering search and RAG applications at scale. He explains the ...
GitHub

Cosmos Tokenizer: A suite of image and video neural tokenizers.

Given an image or video, Cosmos Tokenizer outputs either continuous latents or discrete tokens. Cosmos Tokenizer achieves spatial compression rates of 8x or 16x and temporal compression factors of 4x ...
GitHub

HuggingFace WordPiece Tokenizer in C++

This is a C++ implementation of WordPiece (BERT) tokenizer inference. It expects from you a .json file in HuggingFace format that contains all the required ...
IEEE

Spectral–Spatial Feature Tokenization Transformer for Hyperspectral Image Classification

Abstract: In hyperspectral image (HSI) classification, each pixel sample is assigned to a land-cover category. In the recent past, convolutional neural network (CNN)-based HSI classification methods ...