# BitTransformerLM Scripts

This directory contains organized scripts for BitTransformerLM development, training, and evaluation.

## Directory Structure

```
scripts/
├── training/          # Training scripts and experiments
├── examples/          # Example usage and demonstrations
├── testing/           # Test scripts and validation
├── benchmarks/        # Performance benchmarks
└── tools/             # Utility scripts and data processing
```

## Training Scripts (`training/`)

- **basic_training.py** - Simple training setup for small models
- **breakthrough_training.py** - Advanced training with breakthrough techniques
- **cpu_edge_training.py** - CPU-optimized training for edge deployment
- **final_breakthrough_training.py** - Production training pipeline
- **full_attention_training.py** - Full attention mechanism training
- **full_bits_train.py** - Complete bit-level training
- **production_training.py** - Production-ready training script
- **progressive_scaleup.py** - Progressive model scaling
- **quick_training_run.py** - Fast training for development

## Example Scripts (`examples/`)

- **example.py** - Basic usage example
- **better_sampling.py** - Advanced sampling techniques
- **debug_generation.py** - Generation debugging utilities
- **raw_generation.py** - Low-level generation examples
- **simple_test.py** - Simple model testing

## Testing Scripts (`testing/`)

- **code_test.py** - Code functionality testing
- **diffusion_tests.py** - Diffusion mode testing
- **enhanced_generation_test.py** - Advanced generation testing
- **full_attention_inference_test.py** - Attention mechanism tests
- **test_conversation.py** - Conversational AI testing

## Benchmark Scripts (`benchmarks/`)

- **wikitext_benchmark.py** - WikiText dataset benchmarking
- **wikitext_schedule.py** - WikiText training schedule

## Utility Tools (`tools/`)

- **build_full_bits.py** - Bit sequence construction
- **create_dataset.py** - Dataset creation utilities
- **enhanced_checkpoint_system.py** - Advanced checkpointing
- **integration_flow.py** - Integration workflow
- **integration_schedule.py** - Integration scheduling
- **sync_to_hf.py** - HuggingFace synchronization
- **unified_workflow.py** - Unified training workflow
- **watcher.py** - File system monitoring

## Usage

All scripts support the standardized CLI interface provided by `bit_transformer.cli_standards`. Use `--help` with any script to see available options.

### Quick Start

```bash
# Train a small model
python scripts/training/basic_training.py --model-size small --epochs 5

# Run a simple test
python scripts/examples/simple_test.py --d-model 64

# Benchmark on WikiText
python scripts/benchmarks/wikitext_benchmark.py --dataset-name wikitext-2
```

### Environment Variables

Scripts support configuration via environment variables with `BT_` prefix:

```bash
export BT_D_MODEL=128
export BT_NUM_LAYERS=4
export BT_BATCH_SIZE=16
python scripts/training/basic_training.py
```

## Development Guidelines

- All scripts should use `bit_transformer.cli_standards` for argument parsing
- Include proper logging and error handling
- Support both CPU and GPU execution
- Follow the naming conventions established in existing scripts
- Add documentation for any new hyperparameters or features