|
Parameter-free
In this note, I plan to explain various parameter-free methods and ideas. Below are some of the topics I intend to cover:
Parameter-free algorithms
[JMLR 2021] Metagrad: Adaptation using multiple learning rates in online learning
Tim van Erven, Wouter M. Koolen, Dirk van der Hoeven
[COLT 2018] Black-Box Reductions for Parameter-free Online Learning in Banach Spaces
A. Cutkosky and F. Orabona
[COLT 2022] Making SGD Parameter-Free
Y Carmon, O Hinder
How to optimize neural networks without tunning
[COLT 2018] Training Deep Networks without Learning Rates Through Coin Betting
F. Orabona and T. Tommasi
[Neurips 2023] Mechanic: A Learning Rate Tuner
Ashok Cutkosky, Aaron Defazio and Harsh Mehta
Theoretical Limits
Push me :)
I must confess—I’m a bit lazy at the moment. If you’re really interested in any of these topics, feel free to give me a nudge (or a push!) via email to expand on them further (even asking for a chinese version).
|