Convex envelopes of complexity controlling penalties: The case against premature envelopment

Vladimir Jojic, Suchi Saria, Daphne Koller

Research output: Contribution to journalConference article

Abstract

Convex envelopes of the cardinality and rank function, l 1 and nuclear norm, have gained immense popularity due to their sparsity inducing properties. This has given rise to a natural approach to building objectives with sparse optima whereby such convex penalties are added to another objective. Such a heuristic approach to objective building does not always work. For example, addition of an L 1 penalty to the KL-divergence fails to induce any sparsity, as the L 1 norm of any vector in a simplex is a constant. However, a convex envelope of KL and a cardinality penalty can be obtained that indeed trades off sparsity and KL-divergence. We consider the cases of two composite penalties, elastic net and fused lasso, which combine multiple desiderata. In both of these cases, we show that a hard objective relaxed to obtain penalties can be more tightly approximated. Further, by construction, it is impossible to get a better convex approximation than the ones we derive. Thus, constructing a joint envelope across different parts of the objective provides a means to trade off tightness and computational cost.

Original languageEnglish (US)
Pages (from-to)399-406
Number of pages8
JournalJournal of Machine Learning Research
Volume15
StatePublished - Dec 1 2011
Externally publishedYes
Event14th International Conference on Artificial Intelligence and Statistics, AISTATS 2011 - Fort Lauderdale, FL, United States
Duration: Apr 11 2011Apr 13 2011

ASJC Scopus subject areas

  • Software
  • Control and Systems Engineering
  • Statistics and Probability
  • Artificial Intelligence

Fingerprint Dive into the research topics of 'Convex envelopes of complexity controlling penalties: The case against premature envelopment'. Together they form a unique fingerprint.

  • Cite this