feat: add 760M inference samples and quality audit

feat: add 760M training logs from MI300X run

docs: update README

Add working directory note to AMD download script docstring

Add FineWeb download script for AMD/US environment (direct HuggingFace, no mirror)

Add deepseek run_lite and fineweb extract/tokenize scripts

The changes update the download plan script to handle .tar.gz files (for Llama 3 models) in addition to the previously supported .tar files. The code checks if the target filename ends with .tar.gz and adjusts the extraction command accordingly by using tar xfzf format gzip files. This ensures compatibility with Llama 3 models which use the .tar.gz format.

feat(download): track shard progress in progress.json for resumability

refactor(download): hardcode hf-mirror endpoint for China access

chore: add ruff pre-commit hooks and apply formatting

feat(download): add hardcoded 100B-token GPT-3 ablation downloader

feat(download): add token-budgeted FineWeb shard planner/downloader

docs: add pending RAG and knowledge base features to README

chore(logs): add gen1.txt log file

build(logs): add FineWeb training logs

chore: add datasets directory with gitignore rules

chore: restore training logs to logs directory

refactor: reorganize project structure

docs: add README.md with project overview and usage

Add FineWeb download and extraction scripts (pyarrow iter_batches)

feat(deps): add requirements.txt for Python dependencies

feat(train): add training duration calculation script