Skip to content
View hextobin's full-sized avatar

Block or report hextobin

Block user

Prevent this user from interacting with your repositories and sending you notifications. Learn more about blocking users.

You must be logged in to block users.

Content in all repositories owned by your account will be closed.
Maximum 250 characters. Please don’t include any personal information such as legal names or email addresses. This note will only be visible to you.
Report abuse

Contact GitHub support about this user’s behavior. Learn more about reporting abuse.

Report abuse
hextobin/README.md

Small ML experiments, answered plainly

I run quick, cheap experiments on ideas I'm curious about, then write each one up as a question and an answer, with enough detail to reproduce it.

Question Answer Repo
In a transformer with no MLP, does forcing the residual stream onto a polytope help? Unclear: the L1 ball may train more reliably but scores no better, and the box fails polytope
Does planning each clause with a discrete "concept" code help a small story model predict arcs? No: the codes lowered word loss but didn't pay for themselves concepts
Can a polynomial equation solved by Newton replace GELU in a transformer? No: worse than GELU, barely better than no activation polydeq

Each run costs a few dollars on a rented GPU that sets itself up, saves its results to git and deletes itself when done.

Pinned Loading

  1. concepts concepts Public

    Q: Do discrete plan codes help a small story model predict arcs? A: Not at this scale. Write-up, code, all 14 checkpoints.

    Python

  2. polydeq polydeq Public

    Q: Can a Newton-solved polynomial layer replace GELU in a transformer? A: No. Write-up, code, 24 runs.

    Python

  3. polytope polytope Public

    Q: In a transformer with no MLP, does forcing the residual stream onto a polytope help? A: Unclear: the L1 ball may train more reliably but scores no better, and the box fails. Write-up, code, test…

    Python