Rényi Cross-Entropy: Properties and Closed-Form Expressions for Sources with and without Memory

Abstract

Two R ́enyi-type generalizations of the Shannon cross-entropy, the R ́enyi cross-entropy and the Natural R ́enyi cross-entropy, were recently used as loss functions for the improved design of deep learning generative adversarial networks. In this work, we analyse the properties of the R ́enyi and Natural R ́enyi differential cross-entropy measures and derive their expressions in closed form for a wide class of common continuous distributions belonging to the exponential family. We also establish the R ́enyi-type cross-entropy rates between stationary Gaussian processes and between finite-alphabet time-invariant Markov sources.