Trip Venturella built a language model using only out-of-copyright books from the British Library — over 28,000 Victorian texts. It's not going to replace GPT-4, but it's a fascinating experiment in what you can train without modern data. You can run it locally with Simon's
llm-mrchatterbox plugin.