Overview

Throughout this tutorial, we have seen two very different perspectives on machine unlearning. On one hand, exact and approximate unlearning provide formal guarantees that a model behaves as if particular examples had never been observed during training. On the other hand, modern large-scale systems such as large language models and diffusion models typically rely on heuristic methods that work well empirically but lack comparable guarantees.

This tension between theory and practice motivates many of the central open problems in machine unlearning. In this final section, we discuss some of the current limitations of existing approaches and highlight promising research directions for the future. In particular, we focus on robustness, theoretical gaps, and emerging paradigms such as inference-time and concept-level unlearning.

Current Limitations

Robustness

One of the fundamental challenges in machine unlearning is robustness. Even when an unlearning algorithm appears successful under standard evaluations, forgotten information may still be recoverable through alternative prompts, adversarial attacks, or compositional reasoning.

This issue is particularly pronounced in generative models. Language models may fail direct memorization tests while still reconstructing forgotten information through paraphrasing or multi-step reasoning. Similarly, diffusion models may continue to express supposedly forgotten concepts under modified prompts or stylistic variations.

The problem is compounded by the absence of universally accepted evaluation protocols. Current approaches rely on a combination of querying, membership inference attacks, and hypothesis-testing frameworks, each of which captures only a limited aspect of forgetting. Developing robust evaluation methodologies remains one of the most important challenges for the field.

Two examples of new evaluation paradigms that have emerged as robustness checks are:

  1. Tamper Resistance and Finetuning Attacks: In these evaluations we directly finetune the weights of the model on a subset of the forget set or even unrelated data. We measure how much computation is required to relearn the unforgotten examples.[1] [2] [3]
  2. Data Poisoning: Evaluate whether an unlearned model behaves similarly to one retrained without the poisoned examples, while maintaining performance on clean data. Existing MIAs are actually unable to distinguish between no unlearning and retraining in some cases.[4]

Theoretical Gaps

There is a lot of folklore on some of the limitations and potential impossibilities for unlearning in the non-convex and large-scale settings. For example, it is clear that for generative models our evaluations cannot be centered around ensuring that the probability of a sample being generated is zero. Yet, we are missing rigorous results on whether unlearning is impossible in the non-convex setting and whether other theoretical definitions could be created for moving beyond example-level unlearning.

Future Directions

In essence, our efforts thus far for large-scale models have suppressed information but are not fully eliminating the information from examples. This is exacerbated by our inability to verify whether a completion or output has probability zero of occurring in generative models. This leaves numerous open directions and problems for the field. We list three that are emerging and based on the content presented in this tutorial.

  1. Provable Non-Convex Unlearning: Current theoretical guarantees largely rely on convexity or other simplifying assumptions that do not hold for modern foundation models. Developing meaningful guarantees for non-convex models such as large language models and diffusion models remains one of the central open problems in machine unlearning. Thus far approahces leveraing Langevin dynamics, privacy amplification and gradient clipping have been proposed so far.[5] [6] [7]
  2. Inference-Time Unlearning : Most existing approaches modify model parameters through fine-tuning or retraining. An emerging alternative is inference-time unlearning, where undesirable behaviors are suppressed during generation using decoding strategies or external control mechanisms without modifying the underlying model weights. These are like generalizations of the reinforcement approach from the Harry Potter example. One such extension uses contrastive decoding. [8]
  3. Concept Unlearning: Finally, as we alluded to with applying unlearning to safety, there is a shift towards unlearning of broad concepts rather than individual examples. This is motivated by both safety and copyright. Progress has been pushed forward in both the LLM and diffusion model settings. [9] [10]

References

  1. Hossain, Saad, et al. "TamperBench: Systematically Stress-Testing LLM Safety Under Fine-Tuning and Tampering." arXiv preprint arXiv:2602.06911 (2026).
  2. Siddiqui, Shoaib Ahmed, et al. "From dormant to deleted: Tamper-resistant unlearning through weight-space regularization." Advances in Neural Information Processing Systems 38 (2026): 129326-129357.
  3. Suriyakumar, Vinith Menon, et al. "Unstable unlearning: The hidden risk of concept resurgence in diffusion models." ICLR 2025 Workshop on Navigating and Addressing Data Problems for Foundation Models. 2024.
  4. Pawelczyk, Martin, et al. "Machine unlearning fails to remove data poisoning attacks." International Conference on Learning Representations. Vol. 2025. 2025.
  5. Koloskova, Anastasia, et al. "Certified unlearning for neural networks." arXiv preprint arXiv:2506.06985 (2025).
  6. Mu, Siqiao, and Diego Klabjan. "Rewind-to-delete: Certified machine unlearning for nonconvex functions." Advances in Neural Information Processing Systems 38 (2026): 114911-114947.
  7. Chourasia, Rishav, and Neil Shah. "Forget unlearning: Towards true data-deletion in machine learning." International conference on machine learning. PMLR, 2023.
  8. Suriyakumar, Vinith M., Ayush Sekhari, and Ashia Wilson. "Ucd: Unlearning in llms via contrastive decoding." arXiv preprint arXiv:2506.12097 (2025).
  9. Li, Nathaniel, et al. "The wmdp benchmark: Measuring and reducing malicious use with unlearning." arXiv preprint arXiv:2403.03218 (2024).
  10. Gandikota, Rohit, et al. "Erasing concepts from diffusion models." Proceedings of the IEEE/CVF international conference on computer vision. 2023.