Chat Dataset Format Cleaner

Chat Dataset Format Cleaner

Convert local chat JSONL into clean prompt/completion examples with a reproducible split. Supports message arrays, direct pairs, marker cleanup, basic length filters, and JSON summaries. Requires Python 3.10+. Source files are read only.
#الهندسة
التقييم
مطلوب المزيد من التقييمات
المبيعات
0
كيفية الاستخدام
تنزيل

Chat Dataset Format Cleaner

Prepare clean prompt/completion JSONL from local chat exports. The utility supports message arrays and direct pairs, removes common turn wrappers, uses the final assistant response as the target, and creates a reproducible train/validation split.

  • Python standard library only
  • Read-only source handling
  • Configurable split ratio and seed
  • Length filters and record limit
  • Human-readable or JSON summaries

Run python3 scripts/chat_dataset_cleaner.py examples/conversations.jsonl --output-dir /tmp/chat-clean --seed 42.