The authors present a "dual representation" of influence functions where computational complexity scales with dataset size rather than model size, making it efficient when models are large relative to datasets. This alternative can estimate how removing a data point changes parameters, outputs, or loss, but is limited to "linearizable models" (models whose behavior can be approximated by their linearization throughout training) and requires materializing a matrix whose size grows with output dimension times dataset size.
Main takeaways:
- Standard influence functions scale with model size; the dual representation scales with dataset size instead.
- Efficient alternative when model size is large relative to dataset size.
- Can estimate changes in parameters, outputs, and loss due to data point removal.
- Only works for linearizable models (models approximable by their linearization during training).
- Requires materializing a matrix of size (output dimension × dataset size), which can be a memory bottleneck.