I run quick, cheap experiments on ideas I'm curious about, then write each one up as a question and an answer, with enough detail to reproduce it.
| Question | Answer | Repo |
|---|---|---|
| In a transformer with no MLP, does forcing the residual stream onto a polytope help? | Unclear: the L1 ball may train more reliably but scores no better, and the box fails | polytope |
| Does planning each clause with a discrete "concept" code help a small story model predict arcs? | No: the codes lowered word loss but didn't pay for themselves | concepts |
| Can a polynomial equation solved by Newton replace GELU in a transformer? | No: worse than GELU, barely better than no activation | polydeq |
Each run costs a few dollars on a rented GPU that sets itself up, saves its results to git and deletes itself when done.
