[{"content":"Why is contribution to open source important? Simply, almost all important infra in the world runs on it\u0026hellip; Linux. Without contributors these projects wouldnt be here, or atleast not as good as they are now.\nHowever, this is on the global level. What does it mean to you to be an OSS contributor? Well it may differ between people but it usually revolves around these things:\nIts fulfilling Recognition of skill Fun Like minded community How do you go about your first ever oss contribution? Well I heard people say \u0026ldquo;Start with documentation!!!!\u0026rdquo; and I cannot disagree more :)\nTheir rationale is that as a new contributor no one will trust you to solve issues or look at your pull requests, or maybe its a good first hit of dopamine thats meant to give you momentum. But lets take a step back. Do they really need you for documentation? In this ERA of AI? I dont think so. Moreover, thats not even where my main issue lies. Momentum? The most boring software engineering task I can think of is documentation. Where is the fulfilment part? Fun part?\nWhat I found out to work (For myself atleast) Well first pick a project youre interested in. In my case it was Burn, the leading deep learning framework in Rust.\nThen ask yourself: Do I know the language? If not, learn it. In my case 2 years ago I stumbled upon burn, but I couldnt understand anything because I didnt understand rust. The first year, I stopped midway. This year I made a 2026 resolution to learn rust, and I did.\nAfter you have learnt the language to a level where you can understand the repo, do that! Go read it, until you can use the repo.\nThen use it! You never know, you may stumble upon a bug, an unimplemented functionality you need, or a missing doc\u0026hellip;..\nThen try and solve that pain point, or atleast if you think the repo is useful, give back by checking the issues marked good for beginners.\nThen contribute as much as you can.\npick a project you want to use do you know the language? read it until you can use it now actually use it, for real hit something that annoys you? fix it, thats your first PR grep the good first issues learn it first mine took 6 months yes no yes no My first real contribution I was trying to use argtopk for an int tensor, but I couldnt, so I checked the repo, and voila! My first ever contribution idea came.\nWhat actually happened is that I called argtopk on an Int tensor and got a panic instead of indices:\n1 2 3 let device = Device::ndarray(); let tensor = Tensor::\u0026lt;1, Int\u0026gt;::from_data([1, 2, 3, 4, 5], \u0026amp;device); tensor.argtopk(3, 0); // unimplemented!(\u0026#34;argtopk not implemented for ndarray\u0026#34;) My first assumption was that I was holding it wrong. Thats usually the correct assumption. Not this time. Burn puts every backend behind traits, and a trait method with a default body is free for all backends while a method without one has to be written by each of them separately. int_argtopk had no default, so four backends had \u0026ldquo;implemented\u0026rdquo; it with a panic. Its float twin, float_argtopk, had a default and worked everywhere.\nBefore opening anything I went digging through the history to see if that was a decision or an accident, and it was an accident. Both ops were added together with no defaults, a later PR gave the float one a default and cleaned up its stubs, and the int half was never touched.\nAfter that the code was the easy part. Five lines, written to look exactly like the float version, plus deleting the dead stubs, a native override for tch, and three tests. Opened the issue, opened the PR the same evening, merged a day and a half later.\nThe issue, and the pull request. One detail I like: while I was in there I noticed argtopk was missing a row in the book\u0026rsquo;s tensor op table, so I added it. So I did end up writing documentation. One line of it, at the end, because I was already there fixing the actual bug. That is the right order.\nWhat actually did the work None of the difficulty was in the code:\nUse the thing for real. The bug found me because I wanted argtopk on ints for something, not because I was hunting for a starter task. Read the trait, not the error. The panic tells you where it stopped. The trait tells you why. Check the history before calling it a bug. \u0026ldquo;This is inconsistent with its float twin, here are the two PRs where the int half got dropped\u0026rdquo; is a very different sentence from \u0026ldquo;this panics, please fix\u0026rdquo;. Match what the codebase already does. My impl looks like the float one on purpose. Nobody has to review a new idea. Not one step of that involved starting with documentation.\n","permalink":"https://issabawwab.com/posts/your-first-oss-contribution/","summary":"Why start with documentation is bad advice, the loop I used instead, and the full story of my first merged pull request to Burn, the deep learning framework in Rust.","title":"Your First OSS Contribution"},{"content":"Picking up where I left off At the end of the last post I said I might extend the thing to deal with tensors. This is that. The scalar engine trained a tiny network fine, but it did it one number at a time, and the moment you want to feed it something real like an image that stops being cute. An MNIST digit is 784 pixels. Doing 784 separate scalar Values per image, per sample, is miserable and slow.\nSo the plan was simple to say and annoying to do. Make a Value hold a whole matrix instead of one f32, redo the backward pass so every op pushes matrix gradients, and write my own matmul so the operation actually lands on the graph. Same engine as before, just matrix math instead of scalar math. Most of the real work was in the backward pass and in broadcasting.\nFrom one number to a matrix In the scalar version a node held a single f32. Now it holds an Array2 from ndarray, and the gradient is a matrix of the same shape.\n1 2 3 4 5 6 pub struct ValueData { pub data: Array2\u0026lt;f32\u0026gt;, pub op: Operation, pub gradient: Array2\u0026lt;f32\u0026gt;, pub parents: Vec\u0026lt;Value\u0026gt;, } The nice part is that the graph itself did not change at all. The Rc\u0026lt;RefCell\u0026lt;ValueData\u0026gt;\u0026gt; handle, the parents vector, the topological sort, all of it stayed exactly the same. The only thing that changed is what a node carries and how each op computes its forward value and hands gradients back. That was reassuring, it meant the hard graph stuff I got right the first time was still right.\nWriting my own matmul ndarray gives you two different products. * is elementwise, and .dot() is the actual matrix multiply. The catch is that * already has a trait implementation, and in Rust you can only implement an operator once for a given pair of types, so matrix multiply can not ride on *. That is fine, it just becomes a plain method that builds a node the same way the operators do.\na (1,2) b (2,3) matmul data = a · b (1,3) op = MatMul parents = [a, b] each op builds a node that records how it was made The whole trick lives in that output node. It runs .dot() for the forward value, but it also tags itself with op = MatMul and keeps a and b as its parents, wich is exactly what lets backward work out later how it was built. I also assert that the inner dimensions match, so a bad shape fails with a message I can actually read instead of some panic from deep inside ndarray.\nAddition and the broadcasting headache Addition of two equal sized matrices is the easy case. The forward pass is elementwise, and the local derivative of an add is 1, so the output gradient just flows straight into both parents element by element. The gradient sitting at position ij in the output gets copied into each parents gradient at ij. Nothing clever.\nIt got more involved the moment the shapes did not match. ndarray broadcasts, so a (1, 3) bias added to a (2, 3) matrix stretches its single row across both rows.\n$$ \\begin{bmatrix} b_0 \u0026 b_1 \u0026 b_2 \\end{bmatrix} \\;\\longrightarrow\\; \\begin{bmatrix} b_0 \u0026 b_1 \u0026 b_2 \\\\ b_0 \u0026 b_1 \u0026 b_2 \\end{bmatrix} $$That matters on the way back. That one bias element b_0 now touched two elements of the output, so its gradient is the sum of both of the output gradients it fed into. So the backward pass is just the mirror image of the forward stretch, you take the output gradient and sum the axis that got broadcasted back down to size one. Forward stretches one row into two, backward folds those two rows of gradient back into one.\n$$ \\begin{bmatrix} g_{00} \u0026 g_{01} \u0026 g_{02} \\\\ g_{10} \u0026 g_{11} \u0026 g_{12} \\end{bmatrix} \\;\\longrightarrow\\; \\begin{bmatrix} g_{00}+g_{10} \u0026 g_{01}+g_{11} \u0026 g_{02}+g_{12} \\end{bmatrix} $$I wrapped that up in a little unbroadcast helper, and the pattern I ended up using in litterally every arm is the same. Start with the output gradient in a temp matrix that has the output shape, then unbroadcast it onto the parents shape. If an axis was size one in the parent and bigger in the output, it gets summed back down. If the shapes already lined up, unbroadcast does nothing and you just copy.\nSubtraction and multiply Subtraction is the same shape as addition, the second parent just gets the negative, so a += on the first and a -= on the second, both through unbroadcast.\nMultiply is elementwise, and the derivative of $a \\cdot b$ is $b$ for a and $a$ for b. So each parent recieves the output gradient times the other parents data, still elementwise, then unbroadcast if one side was broadcasted. Its the same coeffecient idea from the scalar post, just done per element across the matrix.\nThe dot product is the one that hurt Matmul backward is not elementwise anymore, its another matrix multiply, and I had to work out the order and wich side gets transposed. Trying to reason about it index by index melted my brain. Thinking in shapes is what made it click.\nSay a is (1, 2) and b is (2, 3), so c = a.b is (1, 3). The output gradient has the same shape as c, so its (1, 3). And the gradient of each input has to match its own data shape, so the gradient of a is (1, 2) and the gradient of b is (2, 3).\nNow you just play shape tetris. To build a (1, 2) out of the (1, 3) output gradient and b wich is (2, 3), the only way the dimensions line up is (1, 3) times (3, 2), wich is the output gradient times b transposed. For the gradient of b, to get a (2, 3) I need (2, 1) times (1, 3), wich is a transposed times the output gradient.\n$$ \\frac{\\partial L}{\\partial a} = \\frac{\\partial L}{\\partial c} \\cdot b^\\top \\qquad\\qquad \\frac{\\partial L}{\\partial b} = a^\\top \\cdot \\frac{\\partial L}{\\partial c} $$ dc (1,3) · bᵀ (3,2) = da (1,2) aᵀ (2,1) · dc (1,3) = db (2,3) inner 3 = 3, ok inner 1 = 1, ok 1 2 3 4 5 6 7 Operation::MatMul =\u0026gt; { let data = self.0.borrow(); let p1 = \u0026amp;data.parents[0]; let p2 = \u0026amp;data.parents[1]; p1.0.borrow_mut().gradient += \u0026amp;(self.gradient().dot(\u0026amp;p2.data().t())); p2.0.borrow_mut().gradient += \u0026amp;(p1.data().t().dot(\u0026amp;*self.gradient())); } The thing I like about this is you dont actually have to rederive it every time. The shapes only fit one way, so they hand you the answer.\nWiring up a layer With the ops in place a layer is boring in a good way. Its a matmul with the weights, then add the bias, then relu.\n1 2 3 4 5 6 7 8 pub fn forward(\u0026amp;self, data: \u0026amp;Value, activation: bool) -\u0026gt; Value { let mut output = data.matmul(\u0026amp;self.weights); output += \u0026amp;self.bias; if activation { output = output.relu(); } output } It worked on the toy, then MNIST happened First I pointed it at the same polynomial from the last post, 2a + 3b - c, three inputs. The loss slid down to basically zero, same as the scalar version. Good, the tensor ops and the backward pass were correct.\nThen MNIST. 784 inputs. The first run the loss started at around three million, and the accuracy came out 10 out of 100. Ten percent is exactly chance for ten classes. And the weird part, the loss kept dropping the whole time while the accuracy never moved off chance.\nThe network just gave up So I printed the actual predictions, and every single image gave nearly the same output, about 0.1 in every slot. The network had collapsed into a constant function that spits out the mean of the targets and completely ignores the input.\nThe 0.1 is not a coincidence. The label is one hot, so across the ten slots the average value in any slot is 0.1. If you take that constant 0.1 and measure mse against a one hot target, nine slots are off by 0.1 and one slot is off by 0.9.\n$$ \\frac{9 \\cdot (0.1)^2 + 1 \\cdot (0.9)^2}{10} = \\frac{0.09 + 0.81}{10} = 0.09 $$And 0.09 is exactly where the loss parked itself. So the number was basically confessing that the net had given up and gone to the mean.\nWhy the weights blew up The weights were drawn uniform from 0 to 1, so all positive, mean 0.5. With 784 positive pixels times positive weights, every term in the sum adds up and nothing ever cancels, so the pre activation grows with the number of inputs.\n$$ \\mathbb{E}[z] \\approx 784 \\cdot 0.13 \\cdot 0.5 \\approx 51 $$One neuron sits around 50 before relu, the next layer pushes that into the thousands, and the target is 0 or 1. So the outputs were wildly out of scale before a single gradient step even happened. With three inputs on the poly this never showed up because the sum was tiny. The bug scaled with the input size, it was there the whole time and 784 inputs is just what made it loud.\nThe gradient side of the same story The forward blowup is only half of it. The part that actually wrecks training is what it does to the gradients. Very roughly, the gradient for a weight is the output error times the input that fed it. When the output is in the thousands, the error is in the thousands, so the gradient is huge too, and then the sgd step weight -= lr * gradient takes a giant leap.\nThe way I think about it is a loss landscape with more than one valley. The good solution sits in a deep valley where the network actually uses the pixels, and right next to it theres a wide shallow one thats just the network outputting the mean and ignoring everything, wich only costs 0.09. The important thing, and the part I got wrong in my head at first, is that the loss never actually blew up. It went down the whole time. The trouble was that the giant early steps skated it straight over the deep valley and dropped it into the shallow one, and once its stuck in that flat basin theres almost no slope left to climb back out, so it just sat there at ten percent. Scaling the init keeps the steps small enough to fall into the deep valley instead of skipping past it.\none loss surface, two minima loss weight shallow: the mean deep: learns uniform init: loss still drops, but into the wrong basin, the mean, 10% scaled init: loss drops into the deep basin that actually learns, 86% Fixing it with better init Two small changes to how the weights start.\n1 2 3 4 5 6 7 8 9 10 11 12 pub fn new(input_size: usize, output_size: usize) -\u0026gt; Layer { let mut rng = rand::rng(); let scale = 1.0 / (input_size as f32).sqrt(); let weights = Array2::from_shape_fn((input_size, output_size), |_| { (rng.random::\u0026lt;f32\u0026gt;() - 0.5) * scale }); let bias = Array2::zeros((1, output_size)); Layer { weights: Value::new(weights), bias: Value::new(bias), } } First, subtract 0.5 so the weights are centered on zero. Now about half of them are negative, so the terms in the sum cancel instead of piling up, and a mean zero sum grows like the square root of the count instead of the count itself. Second, multiply by 1 over the square root of the input size, wich is the factor that keeps the pre activation around one no matter how wide the layer gets. For 784 inputs thats about 0.036. This is basically Xavier and He init, and the whole job of it is keeping the numbers at a sane scale so the gradients arent insane on the first step.\nThe bias I just set to zero. A bias is added once per output, its not summed over the inputs, so it doesnt blow up the same way, but a random positive bias still shoves every output up for no reason, so zero is the clean start.\nAfter that the loss started small and the accuracy jumped to 86 out of 100. Same engine, same data, seperate world just from two lines of init.\nWhat I learned The whole graph layer did not change moving to tensors. Rc\u0026lt;RefCell\u0026lt;..\u0026gt;\u0026gt;, parents, topo sort, all of it survived. Only what a node holds changed, wich felt like a good sign the original design was sound. Shapes are enough to derive the matmul backward. You dont grind indices, you match shapes and there is exactly one arrangement that fits. A dropping loss is not the same as learning. Mine fell seven orders of magnitude while the model was useless. Accuracy was the thing that actually told the truth. Weight init is not something to disregard (your favorite ml framework does it behind the scenes). The same network went from 10 percent to 86 percent from two lines, and the entire reason was just keeping numbers at a sane magnitude. Next steps probably nothing on this project, it was a quick exercise. Based on my microgradrs project. Still just for me, but if it makes the jump from scalars to tensors feel less mysterious then good.\n","permalink":"https://issabawwab.com/posts/tensorizing-micrograd/","summary":"Taking the scalar autograd engine from the last post and making it hold matrices instead of single numbers, then watching it fall apart on MNIST until I fixed the weight init.","title":"Tensorizing Micrograd in Rust"},{"content":"Why I even built this Before anything else I want to be clear about what this project is and what it is not. It is not a library. There is nothing on crates.io, there is no nice API, and honestly nobody should ever use this in real code. It exists for one reason: me.\nWe all reach for the autograd in PyTorch or TensorFlow basically every day and never stop to think about what is actually happening underneath. You call loss.backward() and gradients just appear. I wanted to build that machinery myself, for plain scalar values, from scratch, so that I actually understood how the graph and the backward pass work instead of treating it as magic.\nThe other reason is a bit of a rant. Everyone is letting AI write their code now. I get why, its fast and it mostly works. But I am a big believer in \u0026ldquo;if you dont use it you lose it\u0026rdquo;. Skills are a muscle, and if you never write the hard parts yourself they quietly fade. So I sat down and wrote this by hand, in Rust, specifically because the borrow checker will not let me hand wave my way through shared, mutable graph nodes. That friction is the whole point.\nThe reference for all of this is micrograd, Andrej Karpathy\u0026rsquo;s minimalist autograd engine. It implements backpropagation over a dynamically built computation graph, the same idea the big frameworks use, just shrunk down to scalars.\nThe Value type Everything is built on one type. ValueData is a scalar plus the extra bookkeeping needed to run backprop later:\n1 2 3 4 5 6 7 8 9 pub struct ValueData { pub data: f32, pub op: Operation, pub gradient: f32, pub parents: Vec\u0026lt;Value\u0026gt;, } #[derive(Debug, Clone)] pub struct Value(Rc\u0026lt;RefCell\u0026lt;ValueData\u0026gt;\u0026gt;); The public handle is Value, which wraps Rc\u0026lt;RefCell\u0026lt;ValueData\u0026gt;\u0026gt;. This little combo is the whole reason the thing works in Rust. Rc lets several child nodes point at the same parent (shared ownership), and RefCell lets me mutate a node\u0026rsquo;s gradient later during the backward pass even though its shared. In a language with a garbage collector you get this for free and never notice, in Rust you have to say it out loud.\nThe picture below is the bit that took me a while to feel comfortable with. A Value is not the data, its just a lightweight handle, basically a smart pointer. Cloning a Value does not copy the number, it bumps the Rc count and hands back another pointer to the exact same ValueData. So when an expression like d = (a * b) + (a + b) reuses a, both the multiply node adn the add node store their own Value in parents, but those handles point at one single shared ValueData for a:\nHow Value and ValueData connect the graph Two operation nodes each hold Value handles in their parents list, and those handles are Rc pointers into the same shared ValueData for a and for b. Value = Rc\u0026lt;RefCell\u0026lt;ValueData\u0026gt;\u0026gt;, a cheap handle you can clone and share Rc Rc Rc Rc ValueData data: 2.0 op: Mul parents: [ ] ValueData data: 3.0 op: Add parents: [ ] ValueData (a) data: 1.0 grad: 0.0 op: None two owners, one node ValueData (b) data: 2.0 grad: 0.0 op: None two owners, one node That sharing is the connection. The graph edges are litterally the Value handles sitting inside each node\u0026rsquo;s parents vector, and because they are all Rcs into the same boxes, when backprop writes a gradient into a through one path and then again through the other, both writes land on the same ValueData. The RefCell is what makes those writes legal while everyone is holding a shared reference.\nForward pass A leaf node just stores the number and marks the op as None. Every operation builds a brand new Value and remembers its parents. Addition is the simplest, the local derivative with respect to both inputs is just 1:\n1 2 3 4 5 6 7 8 9 10 11 impl\u0026lt;\u0026#39;a, \u0026#39;b\u0026gt; Add\u0026lt;\u0026amp;\u0026#39;b Value\u0026gt; for \u0026amp;\u0026#39;a Value { type Output = Value; fn add(self, other: \u0026amp;\u0026#39;b Value) -\u0026gt; Value { Value(Rc::new(RefCell::new(ValueData { data: self.data() + other.data(), op: Operation::Add, gradient: 0.0, parents: vec![self.clone(), other.clone()], }))) } } Mul and Sub follow the exact same pattern, only the op tag and the arithmetic change. The one new operation worth showing is relu, since it is the first non linear thing in the crate:\n1 2 3 4 5 6 7 8 pub fn relu(\u0026amp;self) -\u0026gt; Value { Value(Rc::new(RefCell::new(ValueData { data: self.data().max(0.0), op: Operation::Relu, gradient: 0.0, parents: vec![self.clone()], }))) } That is the whole forward story. Build values, combine them, and every node quietly records how it was made.\nBackward pass The backward pass computes gradients with the chain rule:\n$$ \\frac{\\partial L}{\\partial x} = \\frac{\\partial L}{\\partial y} \\cdot \\frac{\\partial y}{\\partial x} $$The idea is simple. Every node knows which operation created it, so it knows how to hand its own gradient down to its parents. Here is that happening one step at a time for the expression\n$$ d = (a \\\\cdot b) + (a + b) $$ Step by step backward pass for d equals a times b plus a plus b A computation graph showing how gradients move backward from d to a and b. d = (a * b) + (a + b) a grad = b + 1 b grad = a + 1 * e = a * b grad = 1 + c = a + b grad = 1 + d grad = 1 * 1 * 1 * b * a * 1 * 1 1. Forward graph\nFirst build the expression graph: e = a * b, c = a + b, and d = e + c.\n2. Seed the output\nThe final value starts with d.grad = 1 because the derivative of d with respect to itself is 1.\n3. Backprop through d = e + c\nAddition passes the gradient unchanged, so e.grad and c.grad both receive 1.\n4. Backprop through e = a * b\nMultiplication sends the other input to each parent: a receives b, and b receives a.\n5. Backprop through c = a + b\nAddition sends 1 to both parents, so a receives another 1 and b receives another 1.\n6. Accumulate gradients\nThe contributions add up: a.grad = b + 1 and b.grad = a + 1.\nPrevious Next d is the output so its gradient starts at 1.0. Then we walk the graph backwards. Since d = e + c, addition passes the gradient through unchanged into both e and c. For $e = a \\cdot b$, multiplication sends the other input to each parent, so a recieves b and b recieves a. And c = a + b sends 1 to both.\nThe detail that trips everyone up is that gradients accumulate. a shows up in both e and c, so its final gradient is the sum of the contribution from the multiply node and the one from the add node. That is why every arm below uses += and never =:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 pub fn backward(\u0026amp;mut self) { match self.op() { Operation::Add =\u0026gt; { for parent in \u0026amp;self.0.borrow().parents { parent.0.borrow_mut().gradient += self.gradient(); } } Operation::Mul =\u0026gt; { let data = self.0.borrow(); let (p1, p2) = (\u0026amp;data.parents[0], \u0026amp;data.parents[1]); let (p1_data, p2_data) = (p1.data(), p2.data()); p1.0.borrow_mut().gradient += self.gradient() * p2_data; p2.0.borrow_mut().gradient += self.gradient() * p1_data; } Operation::Relu =\u0026gt; { let grad = if self.data() \u0026gt; 1e-9 { self.gradient() } else { 0.0 }; self.0.borrow_mut().parents[0].0.borrow_mut().gradient += grad; } // Sub is the same shape as Add, the second parent just gets the negative _ =\u0026gt; {} } } Relu is the interesting one. Since its output is max(0, x), the output is positive only when the input was, so the gradient flows through when the neuron was active and gets killed to zero when it wasnt.\nTopological sort Before running backward on anything I need the nodes in the right order, from the output back through everything it depends on. That is what topo_sort gives me:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 pub fn topo_sort(last: Value) -\u0026gt; Vec\u0026lt;Value\u0026gt; { let mut sorted: Vec\u0026lt;Value\u0026gt; = Vec::new(); let mut visited: Vec\u0026lt;Value\u0026gt; = vec![last.clone()]; let mut stack: Vec\u0026lt;Value\u0026gt; = vec![last.clone()]; while let Some(node) = stack.last().cloned() { let mut pushed_child = false; for parent in \u0026amp;node.0.borrow().parents { if !visited.contains(parent) { visited.push(parent.clone()); stack.push(parent.clone()); pushed_child = true; } } if !pushed_child \u0026amp;\u0026amp; let Some(done) = stack.pop() { sorted.push(done); } } sorted.reverse(); sorted } The sort deliberately does not seed the output gradient itself. Whoever runs the pass sets it, wich keeps the sort doing exactly one job. So back_propagate is just:\n1 2 3 4 5 pub fn back_propagate(last: Value) -\u0026gt; Vec\u0026lt;Value\u0026gt; { let mut list = topo_sort(last); list.iter_mut().for_each(|node| node.backward()); list } A tiny end to end run looks like this. Note the seed gradient is now set by hand:\n1 2 3 4 5 6 7 8 let a = Value::new(1.0); let b = Value::new(2.0); let d = \u0026amp;(\u0026amp;a * \u0026amp;b) + \u0026amp;(\u0026amp;a + \u0026amp;b); // d = (a * b) + (a + b) d.update_gradient(1.0); back_propagate(d); // da/dd = b + 1 = 3, db/dd = a + 1 = 2 There is also a to_dot helper that walks the graph and spits out Graphviz, so I can render a real picture of whatever expression I built with dot -Tsvg graph.dot -o graph.svg. Super handy for sanity checking that the graph looks the way I think it does.\nFrom scalars to a neural net Once the scalar engine works, a neural network is honestly just a lot of Values wired together. A neuron holds a weight per input plus a bias, and its forward pass is the dot product followed by an optional ReLU:\n1 2 3 4 5 6 7 8 pub fn forward(\u0026amp;self, data: \u0026amp;[Value], activation: bool) -\u0026gt; Value { let mut total = self.bias.clone(); for (w, x) in self.weights.iter().zip(data.iter()) { let product = w * x; total += \u0026amp;product; } if activation { total.relu() } else { total } } Every w * x and every += here is building nodes in the same graph from before, so backprop through a neuron comes for free. A Layer is just a Vec\u0026lt;Neuron\u0026gt;, and a Model is a Vec\u0026lt;Layer\u0026gt;. The only real logic in the model is that the hidden layers get an activation and the final output layer does not:\n1 2 3 4 5 6 7 8 pub fn forward(\u0026amp;self, data: \u0026amp;[Value]) -\u0026gt; Vec\u0026lt;Value\u0026gt; { let mut pred = data.to_vec(); for (i, layer) in self.layers.iter().enumerate() { let is_last = i == self.layers.len() - 1; pred = layer.forward(\u0026amp;pred, !is_last); // relu everywhere but the output } pred } Each layer and model also exposes a parameters() method that just collects every weight and bias into one flat Vec\u0026lt;Value\u0026gt;, wich is what the training loop reaches for.\nThe training loop This is the part I was most excited to get working. For each sample we do a forward pass, build the loss as more Value nodes, backprop once through the whole thing, and then nudge every parameter against its gradient. Classic stochastic gradient descent:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 pub fn fit(\u0026amp;mut self, epochs: i32, xs: Vec\u0026lt;Vec\u0026lt;f32\u0026gt;\u0026gt;, ys: Vec\u0026lt;Vec\u0026lt;f32\u0026gt;\u0026gt;, lr: f32) { // xs and ys get wrapped into Values first (left out here) for _ in 0..epochs { for (x, y) in xs.iter().zip(ys.iter()) { let pred = self.forward(x); // mean squared error, built out of Values so it lands in the graph let mut loss = Value::new(0.0); for (p, t) in pred.iter().zip(y) { let diff = p - t; loss += \u0026amp;(\u0026amp;diff * \u0026amp;diff); } loss = \u0026amp;loss * \u0026amp;Value::new(1.0 / pred.len() as f32); // one backward pass over the entire network loss.update_gradient(1.0); back_propagate(loss); // sgd step: push each param downhill, then reset for the next sample let params = self.parameters(); for p in \u0026amp;params { p.subtract_value(lr * p.gradient()); } for p in \u0026amp;params { p.zero_gradient(); } } } } The zero_gradient at the end is easy to forget and definitly matters. Because gradients accumulate with +=, if you dont reset them between samples every step is polluted by the last one and training just falls apart.\nDoes it actually learn? To check that any of this works I made up a boring linear target and asked the network to fit it:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 fn poly(a: f32, b: f32, c: f32) -\u0026gt; f32 { 2.0 * a + 3.0 * b - c } fn main() { let mut rng = rand::rng(); // 64 random samples of the polynomial above let mut data = Vec::new(); let mut truth = Vec::new(); for _ in 0..64 { let a: f32 = rng.random(); let b: f32 = rng.random(); let c: f32 = rng.random(); data.push(vec![a, b, c]); truth.push(vec![poly(a, b, c)]); } // 3 inputs -\u0026gt; 8 hidden (relu) -\u0026gt; 1 output let mut model = Model::new(vec![Layer::new(3, 8), Layer::new(8, 1)]); model.fit(200, data, truth, 0.05); } Running it, the printed loss slides down toward zero over the epochs, wich means the little engine really is computing correct gradients and the weights are moving the right way. Seeing that number drop on a network I wrote every line of, no framework anywhere, was genuinely satisfying.\nWhat I learned Rc\u0026lt;RefCell\u0026lt;T\u0026gt;\u0026gt; is the right tool for a small shared, mutable graph like this. Nodes need shared ownership and delayed mutation and that combo gives you exactly that. Pointer equality matters. PartialEq uses Rc::ptr_eq, so two handles count as equal only when they point at the same node, not when they happen to hold the same number. The topological order is what makes backprop correct. Without it a node can get processed before its gradient has finished accumulating. A neural net really is not a special thing. Its the scalar engine plus a training loop, and once the gradients are right the rest is just plumbing. Next steps Not sure, i may drop it here or extend to deal with tensors and maybe just build on it whenever I read a new paper and want to implement something from scratch. This is based on my microgradrs project. Again, its just for me, but if reading it demystifies backprop a little then even better.\n","permalink":"https://issabawwab.com/posts/micrograd-from-scratch/","summary":"Building a small scalar autograd engine in Rust from scratch, then using it to train a tiny neural network. Based on Karpathy\u0026rsquo;s micrograd.","title":"Building Micrograd from Scratch in Rust"},{"content":"Why a Blog? I\u0026rsquo;ve been meaning to start writing for a while. The idea is simple: if I can explain something clearly, I actually understand it. This blog is my attempt to solidify what I learn by teaching it to my future self (and anyone else who stumbles here).\nWhat to Expect Here\u0026rsquo;s the kind of stuff you\u0026rsquo;ll find here:\nMachine Learning \u0026amp; Deep Learning: from theory to implementation Systems \u0026amp; Infrastructure: performance, deployment, and architecture Dev Tools \u0026amp; Workflows: things that make coding life better Learning Logs: raw notes from my journey through various topics The Philosophy I\u0026rsquo;m a big believer in learning in public. Not everything here will be polished or perfect, and that\u0026rsquo;s the point. Some posts will be deep dives, others will be quick notes. The goal is consistency over perfection.\nCode Will Be Involved Since this is a technical blog, expect code. Lots of it.\n1 2 3 4 5 6 def hello(): \u0026#34;\u0026#34;\u0026#34;The obligatory first function.\u0026#34;\u0026#34;\u0026#34; print(\u0026#34;Hello, World!\u0026#34;) print(\u0026#34;Welcome to the blog.\u0026#34;) hello() And sometimes we\u0026rsquo;ll get into the math too:\n$$ \\mathcal{L}(\\theta) = -\\frac{1}{N} \\sum_{i=1}^{N} \\left[ y_i \\log(\\hat{y}_i) + (1 - y_i) \\log(1 - \\hat{y}_i) \\right] $$ Binary cross-entropy: the loss function that haunts every ML engineer\u0026rsquo;s dreams.\nLet\u0026rsquo;s Go That\u0026rsquo;s it for the intro. The real content starts with the next post. Stay tuned.\nThanks for reading! If you have questions or feedback, feel free to reach out\n","permalink":"https://issabawwab.com/posts/hello-world/","summary":"The first post: why I decided to start writing about what I learn and build.","title":"Hello World: Why I Started This Blog"}]