FlashGPT is a high-performance, modular implementation of the GPT architecture with advanced features like Tree-of-Thought reasoning, algorithmic reasoning modules, and hardware-specific optimizations.
.
├── FlashGPT-Advanced-Modules/ # Advanced modular implementation
├── FlashGPT-Advanced-Thinking-Old/ # Legacy implementation with reasoning modules
├── datasets/ # Training datasets
├── checkpoints/ # Model checkpoints
├── checkpoints_turbo/ # Optimized model checkpoints
├── data/ # Processed data
├── simplebooks_data/ # Book dataset
└── Notebooks/
├── FlashGPT-Base.ipynb # Base implementation
├── FlashGPT-Base-Clean.ipynb # Clean, documented base implementation
├── FlashGPT-Advanced.ipynb # Advanced features implementation
├── FlashGPT-Advanced-Thinking.ipynb # Reasoning-enhanced implementation
├── FlashGPT-Distributed-Training.ipynb # Distributed training setup
└── FlashGPT-distillation-basic.ipynb # Model distillation
- Modern Transformer architecture with enhancements
- RMSNorm for improved stability
- SwiGLU activation functions
- Advanced attention mechanisms (ALiBi, RoPE, GQA)
- Optional Flash Attention support
- Tree of Thought reasoning
- Algorithmic reasoning modules
- Calculator tool integration
- Multi-step reasoning capabilities
- Hardware-specific optimizations (MPS, CUDA)
- Mixed precision training
- Distributed training support
- Model distillation capabilities
- Gradient checkpointing
- Dynamic batch sizing
flashgpt_model.py: Core model implementationprompt_flashgpt.py: Prompt engineering and processingtrain_flashgpt.py: Training utilitiesinfer_flashgpt.py: Inference utilities
Each notebook focuses on specific aspects of the model:
- FlashGPT-Base: Basic implementation and training
- FlashGPT-Base-Clean: Clean, documented implementation
- FlashGPT-Advanced: Advanced features and optimizations
- FlashGPT-Distributed-Training: Distributed training setup
- FlashGPT-distillation-basic: Model distillation techniques
-
Environment Setup
# Clone the repository git clone [repository-url] cd FlashGPT # Install dependencies (requirements.txt to be created) pip install -r requirements.txt
-
Training
- Use notebooks for interactive development
- Use Python scripts for production training
- Checkpoints are saved in
checkpoints/directory
-
Inference
- Load pretrained models from checkpoints
- Use inference scripts or notebooks
- Support for various inference optimizations
- CUDA-enabled GPUs
- Apple Silicon (MPS) optimization
- CPU fallback with optimizations
- Distributed training across multiple devices
-
Base Capabilities
- Language modeling
- Text generation
- Context understanding
-
Advanced Reasoning
- Tree of Thought processing
- Multi-step reasoning
- Calculator integration
- Algorithmic problem solving
-
Optimizations
- Memory efficient attention
- Gradient checkpointing
- Mixed precision training
- Hardware-specific optimizations
Contributions are welcome! Please read the contribution guidelines before submitting pull requests.
[License information to be added]
- Implementation inspired by various transformer architectures
- Incorporates modern ML techniques and optimizations