Massimo Roberto Scamarcia PRO

@maxxafits00 federated learning is definitely the path forward, and it’s something we’ve already begun experimenting with using the flower.ai framework.

Regarding the release, we are currently in mid-training and prioritizing a rigorous "safety-first" pipeline.

We are conducting extensive evaluations on model plasticity, red-teaming for prompt injection, and most importantly, stress-testing for malicious use cases.

We want to ensure the model is robust before it hits the wild.

The current roadmap includes:

• Completing the Knowledge Expansion phase.
• A comprehensive DPO (Direct Preference Optimization) pass to align the "Kurtis" persona and reasoning capabilities.
• Peer review and final validation.

A quick technical spoiler:

The base model pre-training is pure PyTorch and fully multi-GPU compatible. We are utilizing a Curriculum Learning strategy: starting with a small context length and gradually scaling up. This is paired with an enormous batch size and small data chunks.

@maxxafits00 if you are on a budget, I suggest to start small. I unfortunately don't have enough compute to scale right now. To evaluate a pretraining or distillation framework (such as arcee-ai's distillkit), or a new model architecture, you can start from datasets such as TinyStories and move to FineWeb-EDU, cosmopedia, etc later.

Wait for the training and architecture to be stable and validated before moving to a bigger dataset/model. Also, a 7-8B parameters is probably too big for small scale pre-training experiments.

You should try to target 0.5B, max 3B, especially if you use consumer-grade hardware, or a single GPU for rent.

interesting. Yes, as you noticed as well a few billions tokens aren't enough. SmolLM2 360M was trained on 4 trillion tokens.

but I am not sure how to explain those results on piqa and sciq:

uv run lm_eval --model hf   --model_args pretrained=models/Echo-DSRN-Small-Instruct-Kurtis,trust_remote_code=True,device_map="auto"   --tasks hellaswag,winogrande,piqa,sciq --output_path ./results_final

I can share more details in this convo, but this probably uncharted territory for an hybrid RNN with 4 attention heads

Now available at https://huggingface.co/spaces/ethicalabs/Echo-DSRN-Small-Next-Word-Prediction ... on the shared CPU HF resources it runs slow, but on my Macbook M4 and AMD Strix Halo is blazing fast. Memory footprint is low. I am now expanding to 1B using Net2Net and today I tested a SFT run (QLoRA, 4-bit, bf16) on consumer hardware with trl with apparently no catastrophic forgetting.