knowledge distillation is not just compression, ran a few experiments that no one told me to run

30 May, 2026

okay so i have been meaning to write this for a while now. its one of those things where you do some experiments, stare at the outputs for a few days, and then realise the story is actually way more interesting than what your advisor said it would be. so here we go. get a coffee. or chai if you are like me.

i am a phd student working broadly on efficient deep learning. knowledge distillation is one of those topics that everyone sort of knows about but very few people have actually poked at from first principles. if you go and read the original hinton, vinyals, and dean paper ¹ from 2015, it is surprisingly short and handwavy about the mechanism. they show it works, they give the temperature scaling formulation, they coin the term "dark knowledge," and then they mostly move on. the follow-up literature is largely empirical: try this architecture, try this loss weight, try this temperature, report numbers on imagenet.

what is missing, at least as far as i can tell, is a serious mechanistic account of what the student is learning when it learns from soft targets. is it learning a compressed version of the teacher's feature hierarchy? is it learning something about the semantic structure of the label space? is it just doing a fancier version of label smoothing ²? is the internal computation of the distilled student similar to the teacher's or completely different?

the standard story you get in papers is: you have a big teacher network, you have a small student network, you train the student to mimic the teacher's soft probability outputs, and the student learns better than if you just trained it from scratch on hard labels. okay sure. but why does it work? what is actually happening inside the networks when you do this? is the student genuinely learning a compressed version of the teacher's knowledge or is something else going on?

i could not find a satisfying mechanistic answer to this. so i ran some experiments myself. cifar10 as the testbed, resnet18 as the teacher, and a small custom cnn as the student. the experiments are loosely in the spirit of mechanistic interpretability ³: i am trying to understand the internal structure of these networks, not just their accuracy numbers. i want to share what i found and try to build up an actual picture of what distillation is doing, not just that it works.

a quick note on what mechanistic interpretability means in this context. mech interp is a subfield of AI safety and ML research that tries to understand the internal computations of neural networks, not just what they output but what is happening layer by layer, neuron by neuron. the most well known work in this space is probably the circuits work on vision models ⁴ and later the superposition and feature geometry work on language models. the core claim is that neural networks, despite being trained purely by gradient descent with no explicit structure imposed, often develop interpretable internal structure: neurons that respond to specific visual concepts, circuits that implement specific algorithms, features that correspond to human-interpretable concepts ⁵.

applying this lens to knowledge distillation is natural. if distillation changes what the student learns, it should change the network's internal structure. we should be able to see that change in the activations, in the representational geometry, and in what information is encoded at each depth.

the numbers first. teacher resnet18 trained to 94% on cifar10. baseline smallcnn trained with just crossentropy on hard labels got 87.61%. distilled smallcnn trained with knowledge distillation got 88.60%. the student has 242k parameters. the teacher has 11.17 million. you are compressing roughly 46x and losing about 5.4 percentage points of accuracy. that is actually pretty good.

but 1 percent improvement (87.61% to 88.60%) might not sound impressive. so let me put it in context. on cifar10 at this scale, going from 87 to 88.6 with no additional compute at inference and no change in architecture is meaningful. and the principles that produce this 1% gain here produce 2-4% gains on imagenet ⁶ and even larger improvements on harder tasks like machine translation and question answering, where the gap between a small model trained from scratch and a small model distilled from a large one can be substantial ⁷.

but i want to talk about how the improvement happens. not just that it happens. and to understand that, we have to start with the soft targets themselves.

what are the soft targets actually carrying

the fundamental operation in knowledge distillation is temperature scaling ¹. when you compute the softmax of the teacher's logits, you divide by a temperature T before exponentiating. at T=1 you get the standard softmax. at higher T the distribution becomes softer, meaning more evenly spread.

look at what happens to a single ship image as temperature increases. at T=0.5 the distribution is basically a spike. ship gets 100% probability. entropy is 0.01 nats. there is almost no information here about anything other than the correct class. at T=1.0 it loosens up a bit, entropy jumps to 0.15 nats, you can see small bars for automobile and airplane. at T=2.0, entropy is 0.82 nats. at T=4.0 it hits 1.80 nats, and now automobile, bird, airplane, truck all have meaningful probability mass. the model is essentially saying "this is a ship but ships look kind of like trucks and airplanes in some ways." at T=20.0 you are at 2.29 nats, approaching but never quite reaching the uniform maximum of log(10) = 2.30 nats. that last point is important. even at very high temperature the distribution never goes fully flat. the relative ordering of class probabilities is preserved.

the entropy vs temperature curve shows a roughly logarithmic approach toward the uniform upper bound. the blue band showing variance across samples is widest in the T=2 to T=6 range. this heterogeneity matters: the ambiguous images are the ones where the teacher is genuinely uncertain, and those are exactly the images where inter-class similarity information is richest.

this is why the common recommendation in the literature is T=4 to T=6 ¹. too low and you are basically giving hard labels. too high and you are smearing out the structure. the entropy analysis puts that intuition on firmer ground.

the thing hinton called dark knowledge

the term "dark knowledge" was coined in the original 2015 distillation paper ¹. the idea is that the non-maximum probabilities carry a form of knowledge that is invisible when you only look at hard labels. when the teacher says "ship" with 90% confidence but also gives "automobile" 5% and "airplane" 3%, it is communicating something about visual structure that no one-hot label could ever convey.

i ran an analysis specifically on cat images from the cifar10 test set. for each cat image i tracked how the probability assigned to other classes changes as temperature increases.

dog consistently gets the highest probability after cat itself, across all temperatures. at T=1 dog gets about 6.5% average probability on cat images. this rises to about 11.5% by T=5 and saturates there. bird and horse plateau around 9 to 9.5%. airplane plateaus at about 8.5%.

now you might think: airplane is almost as high as dog? that seems wrong semantically. but think about it from a pixel level perspective. cifar10 images are 32x32. a cat and an airplane at 32x32 are both small blurry objects on a background. the low level features are not so different. the teacher has learned something about the distribution of 32x32 images and part of that knowledge is that cats and airplanes share some low level statistical properties ⁸. meanwhile dog gets higher probability because dogs and cats genuinely look similar at a semantic level too.

this brings us to the inter-class similarity matrix, which is the result i find most compelling from experiment 1.

at T=1.0 the matrix is basically an identity matrix. every class has 1.00 with itself and near 0 with everything else. there is a faint cat-dog signal (0.17) and ship-truck (0.03) but nothing meaningful.

at T=4.0 structure starts emerging. cat-dog is now 0.40. automobile-truck is 0.20. bird shows up as similar to cat (0.26) and deer (0.23). the teacher is revealing its knowledge of visual similarity through the soft targets.

at T=10.0 the matrix is much more uniformly orange. most cosine similarities are in the 0.75 to 0.85 range. the relative ordering is still there but everything is compressed into a smaller dynamic range. you lose discriminability.

T=4 is clearly the sweet spot here. the structure is clear without being washed out. this matches the empirical hyperparameter recommendations from the original paper and from subsequent work on temperature calibration ^{9 2}.

the key insight is that the student trained with these soft targets is not just learning "ship = ship." it is learning "ship is most like airplane and truck, somewhat like automobile, not like dog." that is a richer supervisory signal, and it helps the student build better internal representations. this is related to the broader idea in representation learning that inter-sample relationships matter as much as individual labels ^{10 11}.

what happens inside the student after distillation

okay so experiment 1 was about the input to distillation, i.e., what information is in the soft targets. now i want to talk about what happens inside the student network after it is trained with distillation vs hard labels.

i took 32 samples from the cifar10 test set, ran them through teacher, baseline student, and distilled student, extracted the post-relu activations at each layer, and compared the distributions.

the first thing you notice is that the teacher (green, resnet18) has much tighter, lower variance activations than either student. this makes sense because resnet has batch normalization throughout, which constrains activation magnitudes ¹². the small student cnn i built does not have batchnorm.

the second thing you notice is more surprising. the distilled student (red) has notably wider activation distributions than the baseline (blue) in almost every layer. look at layer 1. distilled block1 has activations going up to 6 while teacher and baseline are both under 3.

i expected distillation to make the student more teacher-like. but the activations are actually more spread out, not less.

my hypothesis is that the soft targets provide a richer gradient signal during training. when you are minimizing KL divergence to a soft probability distribution instead of just crossentropy with a one-hot, the gradients carry more information about the relative ranking of classes ¹³. this causes the weights to be pushed harder in more directions, resulting in larger activation magnitudes. alternatively, the distilled student might be trying to compress more information per channel. it has the same number of channels as the baseline but needs to encode more nuanced inter-class structure. one way to do that is to use a wider dynamic range.

there is also a third possibility that i find interesting. the soft targets during training provide a kind of "label with gradient" that is not just about the true class but about the entire probability distribution over classes. this means the network needs to maintain, in its internal state, enough information to reproduce the full soft distribution at the output. that is a richer computational requirement than just "classify correctly." and richer requirements might push the network toward using more of its activation range, similar to how a compression algorithm uses more of its dynamic range when compressing a richer signal.

the activation statistics box in the figure tells the deeper story. at block2, baseline has sparsity 0.482 and kurtosis 9.9. distilled has sparsity 0.484 and kurtosis 7.6. roughly similar. but at block4, baseline sparsity is 0.035 and kurtosis 4.5. distilled sparsity is 0.128 and kurtosis 19.0.

that kurtosis of 19 for the distilled network at block4 is striking. kurtosis measures the peakedness of a distribution. a gaussian has kurtosis 3. kurtosis 19 means very heavy tails: most activations are near zero but a few are extremely large. and the sparsity at block4 is also higher for distilled (0.128) vs baseline (0.035). more zeros, but the nonzero ones are much bigger.

the channel heatmaps make this pattern visual. for the baseline, the heatmap shows fairly uniform moderate activations (mostly teal, around 0.2 to 0.3, scale tops at 0.5). no strong individual channels stand out.

for the distilled model, the scale goes up to ~0.9. there are clearly channels that are almost always bright (channel 63 stands out). but also the floor is darker, more channels are near zero. it is not that all channels are more active. some are much more active, most are less active.

the bar charts make the layer-by-layer comparison explicit. sparsity increases monotonically with depth for both models, but at block4 the distilled student jumps to 0.128 while baseline stays at 0.035. for kurtosis, baseline is higher in early layers (9.0 vs 7.1 at block1) but the pattern reverses sharply at block3 and block4, where distilled hits 19.0 vs baseline's 4.5.

so distillation seems to flatten activation distributions in early layers and then sharply concentrate them in later layers. early layers of both networks are learning similar low level features. the difference shows up later where more abstract and discriminative features are being learned. the distilled network, by block4, has learned a much more selective representation. this looks like what is sometimes called "winner takes all" behavior in neural representations ¹⁴. a sparse code: most channels off, a few channels very strongly on.

this is genuinely interesting from a mechanistic standpoint. distillation is not just helping the student get the right answer more often. it is changing the geometry of how the student organizes information in activation space. this connects to ideas about neural codes being efficient and sparse ^{15 16}, and suggests that the richer training signal from soft targets might be pushing the network toward a more compressed internal code.

the sparse code pattern is also interesting from an efficiency perspective. a network with sparse activations at its later layers is more amenable to hardware acceleration through techniques like pruning and sparse matrix operations ¹⁷. if distillation consistently induces higher sparsity in later layers, there might be a secondary efficiency benefit beyond the model size reduction: the distilled student might be faster to run in practice not just because it has fewer parameters, but because its activations are sparser.

i want to be careful not to overinterpret 32 samples though. the kurtosis and sparsity numbers are striking but they need to be validated on a larger sample. i will do that as a follow-up.

what the layers actually know

experiment 6 is the one i am most excited about and also the one that raised the most questions.

the idea with linear probing is simple. you freeze a trained network, extract activations from some intermediate layer, train a simple logistic regression on those activations, and the accuracy tells you how much linearly accessible class information is encoded there ¹⁸. i ran probes on two tasks: fine-grained (can you classify the 10 cifar10 classes?) and coarse (vehicle vs animal?).

the teacher (resnet18) shows a beautiful monotonically increasing curve. at layer0 probe accuracy is 0.26, at layer2 it is 0.61, at the final pool layer it hits 0.94, essentially matching classification accuracy. this is the expected picture of hierarchical feature learning ^{19 20}. early layers encode low level features, deeper layers progressively build more abstract and discriminative representations.

now look at the baseline and distilled students. both are essentially flat lines from block1 through pool, hovering around 0.09 to 0.12.

that is bizarre. both students achieve 87 to 88% accuracy on cifar10, but linear probes on every single internal layer give only about 10% accuracy. that is barely above chance for a 10-class problem.

how is that possible?

a few things could explain it. the student might encode everything nonlinearly, meaning class information is there but entangled in a way that a linear probe cannot find but the actual trained classifier can because it is trained end-to-end with those specific features. or the student might be doing most of the discriminative work in its final linear head, with the conv layers doing relatively simple preprocessing. this is actually a known phenomenon with small networks, they sometimes find a solution where intermediate representations are not particularly meaningful and the linear head does the heavy lifting ²¹. or global average pooling might be losing spatial information that the network uses but the probe cannot find. or 64 channels might just not be enough capacity to build linearly separable features the way resnet18 can with up to 512 channels.

i think it is probably a combination of the first two. the small network is forced to use a different strategy than the large teacher. it does not have the capacity to build rich hierarchical representations at every layer. instead it learns a relatively simple nonlinear transformation in the conv blocks and does the heavy discriminative work in the linear head ⁶.

the gap is enormous. at the pool layer, teacher probes to 0.937. both students probe to around 0.10. the students finish at 88% classification accuracy but their internal representations barely contain more linearly accessible class information than chance. the student is not a small version of the teacher. it is a fundamentally different computational object that happens to give similar answers. this connects to a broader point in the representation learning literature about how the same output behavior can arise from very different internal mechanisms ²².

now here is where distillation shows its effect at the probing level.

for fine-grained probing (10 classes), distilled is consistently a little above baseline across all blocks: 0.12 vs 0.10, 0.12 vs 0.08, 0.12 vs 0.09, and so on. small but consistent.

for coarse probing (vehicle vs animal), the story is more dramatic. at block2, distilled jumps to 0.62 vs baseline 0.57. at block3, distilled gets 0.61 vs baseline 0.54. the distilled network retains higher coarse semantic accuracy in its intermediate representations.

and the pattern across all blocks is interesting. the distilled student's coarse probe accuracy is not just higher, it also decays less across depth. baseline starts at 0.59 at block1 and drops to 0.53 by the pool layer. distilled starts at 0.60 and drops to 0.57. the distilled network is retaining coarse semantic structure deeper into the network whereas the baseline is losing it as depth increases. this is consistent with the distilled student having internalized a more stable, semantically organized representation of the class space.

this is exactly what you would expect from the dark knowledge story. the teacher's soft targets carry inter-class similarity information that is essentially about coarse structure. cats look like dogs. trucks look like automobiles. ships look like airplanes. this is a vehicle vs animal style organization. the student trained with these soft targets picks up on this coarser level semantic structure and its intermediate representations reflect it ^{23 24}.

the baseline student, trained only with hard labels, gets no signal about which wrong classes are more or less wrong. all wrong answers are equally wrong in a one-hot label. distillation breaks that symmetry. and the asymmetry shows up in the probing experiments as better coarse category representation.

it is also worth noting what the coarse vs fine-grained gap tells us. fine-grained probing (10 classes) gives about the same accuracy for both students everywhere. coarse probing (2 classes: vehicle vs animal) is where they diverge. this means distillation is not broadly improving all aspects of the representation. it is specifically improving the semantic organization of the representation, the part that reflects the relational structure among classes. the part that the soft targets are directly communicating.

putting the story together

experiment 1 established that teacher soft targets at moderate temperature carry genuine inter-class similarity information. the dark knowledge is real. it reflects both semantic similarity (cat-dog) and visual similarity at the pixel level (ship-airplane at 32x32). the inter-class cosine similarity matrix at T=4 is a pretty good proxy for human intuitions about class similarity in cifar10.

experiment 4 showed that the distilled student develops different activation patterns, especially in later layers. the most striking finding is extreme kurtosis (19.0) and higher sparsity at block4. the distilled network is running a sparser code in its deeper layers. a few features activate very strongly, most are zero.

experiment 6 showed that the distilled student's intermediate representations encode better coarse semantic information (vehicle vs animal) than the baseline. fine-grained information is similar in both. but the coarse structure is better preserved throughout the network in the distilled case.

the synthesis is something like this: distillation transfers genuine semantic structure through soft targets. the student picks up on that structure and reflects it in better coarse-grained representations. the student also develops a different computational strategy: sparser, higher-kurtosis activations in later layers, which might be how it encodes a more selective and committed representation of the learned class structure.

and perhaps most importantly, the distilled student is not a small teacher. it is its own thing with its own representational strategy that happens to produce similar outputs. the probing results make this concrete. the teacher builds a clean hierarchy where class information accumulates layer by layer and is linearly accessible at every depth. the student's information is nonlinearly encoded and nearly invisible to linear probes despite giving 88% accuracy. understanding that distinction matters if you want to use distilled models for anything beyond their nominal task.

wait is this dark knowledge or just calibration

i want to engage with a critique that came up when i was thinking through these results. the critic says: you are confusing inter-class similarity with just better calibration. a well-calibrated classifier naturally assigns some probability to confused classes ⁹. any well-trained model would give soft targets with the cat-dog structure you are seeing. there is nothing magical about the teacher, it is just well calibrated.

this is a fair point. some of what i am calling dark knowledge is definitely calibration. a perfectly calibrated model assigns probabilities that match empirical frequencies. if 20% of the time a human says "cat" the image is ambiguous enough that a dog-classifier would also say dog, then a calibrated model would assign ~20% probability to dog for those images.

but i think dark knowledge is more than just calibration for two reasons.

first, the inter-class similarity is systematic in a way that reflects the model's learned feature space, not just calibration. the cat-dog similarity we see is not just "cats and dogs are sometimes confused empirically." it reflects that the model's internal representation of cats and dogs is geometrically close in feature space ²². the cosine similarity between average class representations in the teacher's final layer would give numbers similar to the T=4 inter-class matrix.

second, the calibration story does not fully explain why distillation works better than label smoothing. label smoothing ²⁵ gives you a calibrated soft target too: (1-epsilon) for the correct class and epsilon/(K-1) uniformly for all others. but distillation with class-conditional soft targets works better than label smoothing in most careful comparisons ². the difference is exactly the class-conditional structure, the inter-class similarity, that goes beyond calibration. the fact that the coarse concept probing showed distillation encodes better semantic structure than baseline (which is equivalent to the extreme case of no smoothing) supports the dark knowledge story and is not easily explained by calibration alone.

some things i want to try next

SVD rank analysis. one experiment i planned but hit a bug on was computing the singular value decomposition of weight matrices in both teacher and student to compare effective rank. the hypothesis is that if distillation helps the student use its capacity more efficiently, its weight matrices might have higher effective rank after distillation compared to baseline ²⁶. weight matrix rank is a measure of how many independent directions of information the matrix encodes. i will rerun this properly.

gradient saliency. another experiment i hit a bug on was smoothgrad saliency maps comparing where teacher, baseline, and distilled student look in images. the scientific question is interesting: the teacher, having been trained with the full inter-class similarity structure, should pay attention to different parts of images than the baseline student. the distilled student, having been trained to mimic those soft targets, might develop more teacher-like saliency patterns. this would be direct evidence that dark knowledge shapes where the network looks ²⁷.

nonlinear probing. the flat probing results for the student were surprising. a natural follow-up is to use nonlinear probes (small MLPs) instead of logistic regression. if accuracy jumps with a nonlinear probe, that confirms the information is there but nonlinearly encoded ¹⁸. if it still stays low, something more fundamental is going on.

CKA analysis. centered kernel alignment ²² is a way to measure representational similarity between networks that is invariant to linear transformations. CKA between teacher and student layers would tell us which layers (if any) are encoding similar things even if in different bases. my prediction is that CKA will be higher between the distilled student and the teacher than between the baseline student and the teacher. the distilled student should be more teacher-like in representational geometry even if it is not using the same hierarchical structure.

temperature annealing. what if you started training with high temperature and gradually reduced it? the intuition is that early in training you want the student to learn coarse structure (high T emphasizes inter-class similarity) and later you want to sharpen the targets to push toward correct classifications. there are papers exploring this ¹³ but i have not seen a careful mechanistic study of whether it actually changes internal representations in the expected way.

connections to other work

a few things in the broader literature connect to what i found here and are worth naming explicitly.

knowledge distillation is closely related to metric learning and contrastive learning ^{11 10}. both are about learning to represent data in ways that respect similarity structure. distillation does this implicitly through soft targets. contrastive learning does it explicitly through positive and negative pairs. the coarse concept probing results showing better vehicle vs animal separation in the distilled student suggest that distillation is doing a version of what contrastive methods do: it is organizing the representation space according to semantic similarity.

the capacity gap problem ²⁸ is relevant here. the idea is that if the teacher is much larger than the student, distillation can hurt because the teacher's distributions are too complex for the student to learn. in my case the capacity ratio is about 46x, which is large. the student achieves 88.6% vs teacher's 94%, so there is a meaningful gap. but the activation forensics suggest the student is adapting to this gap by using a different computational strategy rather than trying to replicate the teacher's strategy. it finds its own way to solve the problem, guided but not constrained by the soft targets.

the born-again networks paper ²⁹ is also relevant and philosophically interesting. you train a network, use it as a teacher to train an identical architecture, and repeat. each generation typically improves over the last, even though student and teacher have the same architecture and there is no compression. the final model in a sequence of five generations can match the ensemble of all five. this means distillation is not just useful for compression. even without any size difference, using a pretrained model as a teacher improves performance. the dark knowledge transfers even within the same model family.

and then there is the self-distillation line of work ³⁰ where the network learns from its own earlier training iterations. surprisingly this also works. the model distills knowledge from a version of itself that is in a different part of the loss landscape. this supports the "richer gradient signal" story over the "compressed teacher knowledge" story. you do not even need a separate larger teacher to get the benefit.

a practical note on why these findings matter at scale

i want to close on something that i think does not get discussed enough.

we are in an era where model compression is increasingly important, not just for edge deployment but for cost reasons in cloud inference too. running large language models is expensive. knowledge distillation is one of the main tools for compression. open source models like distilbert ⁷ achieved 97% of bert's performance with 40% fewer parameters and 60% faster inference. the phi models ³¹ were trained with distillation signals from larger models. this is happening at massive scale.

but we barely understand the mechanistic story even for simple cases like what i did here on cifar10. when we compress a 10 billion parameter language model into a 1 billion parameter model through distillation, what is happening inside? which capabilities transfer cleanly? which ones are preserved in output behavior but through a fundamentally different internal mechanism that might fail in unexpected ways?

the fact that in my simple experiment the student achieves 88.6% accuracy through an internal mechanism that is nearly invisible to linear probing (10% probe accuracy vs 94% for the teacher) is a warning sign. the student is doing something we do not understand. it is not just a smaller version of the teacher. at cifar10 scale, if the student fails in some unexpected way, the cost is low. at language model scale, the implications are much larger.

the mechanistic interpretability tools that have been developed for transformers ^{3 5} should be applied to distilled model pairs to understand what changes and what stays the same. that is the research agenda i am building toward. cifar10 is just the starting point.

if you made it to the end, thank you genuinely. these experiments were run on a free colab instance. total training time was about 25 minutes for all three models. i am cleaning up the code and will put it on github soon, will update this post when i do.

and if you have thoughts or have done similar experiments, please reach out. i am always looking for people to argue about this stuff with.

other distillation methods and how they relate

the version of distillation i implemented here (hinton et al. 2015) is sometimes called response-based distillation because it uses only the output layer responses, the logits or probabilities. there are other families of distillation methods that connect to these findings in interesting ways.

feature-based distillation (fitnets ²³) adds regression loss terms that push the student's intermediate activations toward the teacher's. you need either architectural compatibility or extra adaptation layers to do this. the activation forensics here suggest this could be particularly impactful: if you force the student's block4 activations to look like the teacher's layer4 activations, you would be directly constraining the sparse, high-kurtosis pattern we observed. it is an open question whether that constraint helps or hurts. on one hand, it might align the student's computation more closely with the teacher's interpretable hierarchy. on the other hand, it might force the student into a suboptimal representational strategy given its smaller capacity. the capacity gap problem ²⁸ is relevant here: for very large teacher-student size ratios, feature matching can hurt by preventing the student from finding its own optimal solution.

attention transfer ²⁴ is a softer version of feature matching that aligns the spatial attention maps (roughly, which parts of the image each layer is sensitive to) rather than the activation values directly. this preserves the student's freedom to use different magnitudes while still aligning the structural aspects of its computation with the teacher's. given the probing results showing poor linear decodability in the student, attention transfer might be one way to improve the student's interpretability without overly constraining its capacity allocation.

relation-based distillation (RKD ³²) matches the relational structure between samples: if samples A and B are close in the teacher's representation space, they should be close in the student's too. this is more flexible than feature matching because it does not require point-by-point alignment, only structural alignment. it directly connects to the coarse concept probing results: by matching inter-sample distances, you are explicitly encouraging the student to preserve the teacher's semantic organization. the vehicle vs animal clustering that we saw better preserved in the distilled student is exactly the kind of structure relational distillation targets.

born-again networks ²⁹ are philosophically the most interesting to me. you train a network, use it as a teacher to train an identical architecture, and repeat. each generation typically improves over the last, even though student and teacher have the exact same architecture and there is no compression happening. the final model in a sequence of five generations can match the ensemble of all five.

this means distillation is not fundamentally about compression at all. even without any size difference, using a pretrained model as a teacher improves performance. the dark knowledge transfers even within the same model family. why? one interpretation is that the first-generation model, by the time it is used as a teacher, has developed soft targets that are better calibrated than the hard training labels. it has seen every training example many times and has developed a model of which classes are confused with which. training a second network from scratch using these better labels gives it a head start ³³.

another interpretation is that the different training trajectory matters. the first network learned by memorizing hard labels in a particular order. the second network learns by matching the first network's predictions. these are different optimization paths through parameter space, and they can lead to different local minima with different properties ³⁴.

the self-distillation variants ³⁰ push this further: the network learns from a version of itself at a different training stage. the early-epoch model is the teacher for the late-epoch model. this is distillation without any separate teacher at all, and it still works. this suggests the benefit of distillation is at least partly about training dynamics, the richer gradient signal and the smoother loss landscape, rather than specifically about the size gap between teacher and student.

why this matters beyond cifar10 and whether i am overreading it

i want to flag some limitations before closing, partly because it is good practice and partly because i genuinely do not know how much of this generalizes.

first, this is all on cifar10 with a very small student. cifar10 has 10 clean, well-separated classes and a lot of prior work. the teacher-student accuracy gap here (94% vs 88.6%) is modest compared to what you would see with larger models on harder tasks. the qualitative conclusions, that distillation transfers coarse semantic structure, that it induces sparser later-layer activations, that the student's internal representations are hard to linearly decode, might be general. but the specific numbers and the magnitude of effects are probably cifar10-specific.

second, the probing results are the finding i am most uncertain about. that 10% linear probe accuracy for the student while achieving 88.6% classification accuracy is such a striking result that i half-wonder if i made a bug somewhere. i ran it multiple times and got consistent results, but i would feel more confident if someone else replicated it. the most parsimonious explanation is that small networks, even when they work well, build nonlinear representations that are not linearly decodable layer-by-layer. there is some theoretical work on this ²¹ but it is not a settled question.

third, the activation analysis (experiment 4) was on 32 samples. the trends are interesting and the kurtosis=19 result is striking, but 32 samples is too small to draw strong conclusions. i am running this again with 1000 samples and will update this post.

fourth, i only explored one temperature (T=4) for training the distilled student. the entropy analysis suggests T=4 to T=6 is the right range for this teacher, but i did not systematically verify that the mechanistic findings (sparsity, kurtosis, coarse probing) change as you change T. it would be interesting to see if higher T (which gives more uniform soft targets) produces different internal structures in the student.

those are the caveats. now let me say what i think is actually solid here. the inter-class similarity structure in the soft targets (experiment 1) is clean and interpretable and matches semantic priors. that is not an artifact of sample size or the specific architecture. the better coarse concept probing in the distilled student (experiment 6) is a consistent effect across all layers and both probing tasks. and the existence of the training dynamics asymmetry (distilled slower early, better late) is consistent with what other people have reported on different datasets and architectures.

the story that i am confident in: distillation transfers genuine inter-class semantic structure through soft targets. the student absorbs that structure and it shows up as marginally better coarse semantic representations in intermediate layers. the student's internal computational strategy is different from the teacher's, specifically more nonlinear and less hierarchically organized, but the semantic content of its outputs is shaped by the teacher's soft target geometry.

the story i am less confident in: the specific mechanism (sparser activations, winner-takes-all in later layers) by which the student implements a different strategy. that needs more data and more experiments.

looking at the training logs, there are some interesting differences in how the three models trained that are worth talking about because they hint at why distillation works the way it does.

the teacher converges cleanly. by epoch 50 the training loss is 0.0075 and test accuracy is 94%. that very low training loss tells you the teacher is somewhat overconfident, it is fitting almost perfectly on training data. this is actually relevant to the distillation story. if the teacher is overconfident, its soft targets at T=1 would be very sharp, almost as sharp as hard labels. this is part of why you need a higher temperature to extract the inter-class structure. the teacher's "raw" outputs are too confident. temperature scaling is not just a training trick. it is a mechanism for decompressing information that the teacher has learned but is not expressing in its standard output ¹.

the baseline student converges smoothly. loss goes from 1.40 to 0.23 across 50 epochs. accuracy goes from 0.50 to 0.88. nothing surprising.

now look at the distilled student. epoch 1: loss=6.25, accuracy 0.51. epoch 10: loss=2.81, accuracy 0.67. the distilled student is learning slower initially. at epoch 10, the distilled student is at 0.67 vs the baseline at 0.73. the distilled student is behind by 6 percentage points.

by epoch 20 it has largely caught up: 0.76 vs 0.77.

by epoch 30 it has overtaken: 0.82 vs 0.81.

and it finishes ahead: 0.89 vs 0.88.

this delayed but superior convergence is actually theoretically interesting. the distilled student is slower to start but reaches a better final accuracy. one interpretation is that the KL divergence loss to the teacher's soft targets has a more complex landscape than simple cross-entropy. the student is navigating a harder optimization problem early on, but that harder problem leads it toward a better solution. another interpretation is that the soft targets are providing a form of implicit data augmentation: the teacher is essentially generating a richer label for every training example, and learning from that richer label takes longer but generalizes better ³³. a third interpretation connects to ideas about loss landscape geometry: the soft target loss might have a flatter, more benign geometry that makes the optimizer less likely to get stuck in sharp minima ³⁴.

i do not know which of these is right. but the training curve pattern (slower start, better finish) is consistent across multiple runs and seems like a real signal rather than noise.

on the math, briefly

i know some people reading this want to see the equations. here is the minimum viable math.

the standard KD loss is:

L = α · T² · KL( softmax(z_teacher / T) ‖ softmax(z_student / T) ) + (1 − α) · CrossEntropy( softmax(z_student), y )

where z_teacher are teacher logits, z_student are student logits, y is the hard one-hot label, T is temperature, and α is the mixing coefficient. in my experiments i used α = 0.9, meaning 90% of the loss came from distillation and 10% from hard labels.

the T² factor is there because when you compute the KL divergence at temperature T, the gradient magnitude scales as 1/T² relative to the standard cross-entropy ¹. multiplying by T² compensates for this, keeping gradient magnitudes roughly consistent across different temperature settings. without it, high-T experiments would have smaller gradient magnitudes and you would need to retune your learning rate for each T. it is a subtle but important detail that implementations often get wrong.

now here is the mechanistic reason why distillation is different from hard label training. the gradient of the KL term with respect to student logit z_student_i is:

dL_KD / dz_student_i = (1/T) · ( softmax(z_student / T)_i − softmax(z_teacher / T)_i )

this is the difference between the student's soft probability and the teacher's soft probability for class i, divided by T. the student is being pushed to match the teacher probability for every class, not just the most probable one.

for a cat image where dog gets 0.11 probability from the teacher and the student only assigns dog 0.05, there is a gradient pushing the student's dog logit up. the student is being told: you are being too dismissive of dog as a possibility for this cat image. with hard labels, the gradient for the wrong class i is just softmax(z_student)_i − 0 = softmax(z_student)_i. push this logit down. it does not say anything about the relative magnitude of the push across different wrong classes. dog and truck get the same push regardless of which one is visually similar to cat.

this mathematical difference in the gradient is the fundamental reason distillation provides richer supervision than hard labels. the student gets a semantically informed gradient signal at every training step, for every example. over 50 epochs on 50000 training examples, that difference adds up.

is the student actually harder to interpret

i want to come back to the probing results and draw out one implication that i think is important for the mechanistic interpretability community.

the teacher's clean depth-probing curve (0.26, 0.38, 0.61, 0.84, 0.94 as you go deeper) reflects a hierarchical representational structure where class information accumulates layer by layer and is always linearly accessible ¹⁹. this is the structure that makes large networks interpretable. you can probe any layer and understand what it is encoding. you can ablate individual layers and understand what they contribute. circuits work ³ works by exploiting exactly this kind of clean structure.

the students have probe accuracy of 0.09 to 0.12 across all layers. the class information is there (the network achieves 88% accuracy) but it is nonlinearly encoded and nearly invisible to linear analysis. this means the standard mech interp toolkit, probing, linear representation analysis, and conceptually similar tools, does not work well on these small distilled networks.

this is actually a concerning pattern for the broader AI safety adjacent research on model interpretability. if we compress models for deployment, we might be making them harder to interpret. the trade-off between efficiency and interpretability is real. it is not always discussed as such, but the probing results here make it concrete.

i am not saying distillation is bad. the efficiency gains are real and important. i am saying that if you plan to deploy a distilled model in a setting where you need to audit it, check that it is doing the right thing for the right reasons, or detect unexpected failure modes, you should be aware that the internal structure is much harder to read than the teacher's.

one possible response to this is: use feature-based distillation (fitnets ²³) or attention transfer ²⁴ which explicitly constrains the student's intermediate representations to match the teacher's. this would force more teacher-like representational structure into the student and might make it more linearly interpretable. whether this is the right trade-off depends on how much you care about the student's internal representations vs just its output accuracy.

another response is: develop mech interp tools that are explicitly designed for small, nonlinearly-encoded networks. the current tools were largely developed on large networks with clean hierarchical structure. small distilled networks have a different computational character. they need different tools.

one more thing: distillation vs label smoothing

i keep coming back to the label smoothing comparison because it is the cleanest way to isolate what distillation is adding on top of just "softer labels."

label smoothing ²⁵ replaces hard one-hot targets with (1-epsilon) for the correct class and epsilon/(K-1) uniformly for all others. it is a form of regularization that prevents the network from being overconfident. it is simpler than distillation and requires no teacher network.

muller et al. 2019 ² showed that label smoothing actually hurts representation quality in some cases. specifically, they found that label smoothed models have worse representation geometry for knowledge transfer. the representations cluster together in a way that makes the soft targets less informative about inter-class similarity.

this is the opposite of what you might expect. you might think "softer labels = more information about uncertainty = better representations." but label smoothing gives you class-independent softening. every example of every class gets the same epsilon/(K-1) mass distributed to all wrong classes. there is no information about which wrong classes are more plausible.

distillation gives you class-conditional softening. a cat image gets more probability mass on dog than on truck. a ship image gets more probability mass on airplane and truck than on cat. this asymmetry is the inter-class similarity structure. it is the information that label smoothing systematically throws away.

the coarse probing results i found are consistent with this. the distilled student has meaningfully better vehicle vs animal separation in its intermediate representations. that coarse semantic structure comes from the class-conditional soft targets. label smoothing could not produce this because it does not know which wrong classes are similar to the right one.

this also connects to a deeper point about what "information" means in the context of training signals. label smoothing increases entropy of the target distribution, which might seem like more information. but information is context-dependent. a uniform distribution over wrong classes is not actually informative about the task structure. the relevant information is in the asymmetries, in the fact that this input is more like these wrong classes than those wrong classes. distillation preserves that asymmetry. label smoothing destroys it.

what this all means if you just want to use knowledge distillation in practice

okay stepping back from the mechanistic stuff. here is what i would actually tell someone who wants to use knowledge distillation in a real project, based on what i found.

temperature is a real hyperparameter and you should tune it. the entropy analysis gives you a principled way to pick a range. compute the entropy of your teacher's soft outputs at various temperatures. you want to be in the range where mean entropy is roughly 70 to 85% of maximum entropy (log K nats for K classes). too low and you are giving hard labels with extra steps. too high and the near-uniform distribution has lost most of the class-specific structure.

for a well-trained confident teacher, start around T=4. if your teacher is less confident or you are working with more classes (imagenet has 1000), you might need to adjust. the cifar10 numbers here should not be treated as universal.

alpha matters but it is less sensitive than temperature in my experience. the hard label term (1-alpha) is there to keep the student anchored to ground truth when the teacher is wrong. since no teacher is 100% accurate, you always want some hard label component. alpha=0.7 to 0.9 is a reasonable range to search over.

the student will not develop teacher-like internal representations. if you are planning to use the student's intermediate features for anything (visualization, transfer learning, feature extraction), be aware that those features are organized very differently from the teacher's. the student achieves similar output accuracy through a fundamentally different internal mechanism.

if interpretability matters, consider feature-based distillation or relation-based distillation methods ³² instead of just response-based distillation. these constrain the student's internal representations more explicitly and may result in more interpretable intermediate features.

and finally: the distilled student is slower to start but often converges to a better solution than the baseline. if you are comparing distillation against a baseline student, make sure both are trained for enough epochs. cutting training short and seeing that the baseline is ahead early on is not evidence that distillation does not work.

what does this mean for people working on model interpretability at scale

i want to close on this because i think it is the most important implication and also the least discussed.

we are in a moment where model compression is not just a research interest but a commercial necessity. running frontier-scale language models is extremely expensive. there is enormous pressure to find smaller models that preserve the capabilities of larger ones, and distillation is one of the primary tools. models like distilbert ⁷ and the phi family ³¹ represent real, deployed systems where distillation or distillation-adjacent techniques shaped the training process.

the interpretability community has made a lot of progress understanding large models. we have circuit analysis for transformers ³, superposition theory for how features are represented ⁴, and probing techniques for understanding what information is encoded at each layer ¹⁸. all of this work is built on the implicit assumption that the models being analyzed have the capacity to build clean, hierarchical, linearly-decodable representations.

that assumption does not hold for small distilled models. as the probing results here show, a model can achieve 88% accuracy on a 10-class problem while having essentially no linearly-decodable class information in any of its intermediate layers. the information is encoded nonlinearly in a way that the standard probing toolkit cannot find. if someone tried to interpret this student using standard probing methodology, they would conclude the model is not encoding class information and is relying on some mysterious non-semantic features. but that would be wrong. the model works fine. it is just using a different computational style.

this means the interpretability tools need to catch up. we need probing methods that are explicitly designed for nonlinear representations. we need ways to understand what small networks are doing that do not assume large-network representational conventions. and we need to understand, for any given distilled model, how much of the original teacher's interpretable structure has been preserved vs replaced with something opaque.

this is not a hypothetical concern. if a distilled language model is deployed in a high-stakes setting (medical, legal, financial) and something goes wrong, the ability to audit the model matters. if the model's internal representations are opaque to our current interpretability toolkit, we have a problem. the efficiency gain comes at a cost that is currently not well measured.

i do not have a solution to this. but i think naming it clearly is a start. and the kind of mechanistic experiments i have been describing here, comparing probing accuracy, activation geometry, representational structure between teacher and student, are the right kind of analysis to start building that understanding.

so. that is what i have been thinking about. if you are working on something related, i would genuinely love to talk. reach out.