tldt — Too Long, Didn't Tokenize

About

Too Long; Didn't Tokenize (tl;dt) provides a command line interface and library that employs machine learning to condense lengthy texts while preserving context. It addresses API calls, uploaded documents, and crawled websites containing excessive tokens, prompt injection attempts, and extraneous instructions. The tool incorporates LexRank and TextRank methods, supports the OWASP LLM Top 10 guidelines, and protects against Unicode confusables. It also performs HTML to markdown conversion, text sanitization, and cleans personal information and API keys. No external API keys are necessary. It includes a Go library designed for agents that directly call AI APIs, as well as a skill for programming tasks.