KL Divergence

Table of Contents

• Introduction
  • Definition and Background
  • Formula and Calculation
  • Nature and characteristics
    • Asymmetry KL
    • Non-negative
• In QPanda3
  • KL divergence for calculating the probability distribution of discrete random variables
  • Calculate the KL divergence of the probability of continuous random variables

Prev Tutorial: Hellinger Fidelity
Next Tutorial: Unitary

---

Introduction

The KL divergence in the quantum domain is an important concept used to measure the difference or information loss between two quantum probability distributions. The following is a detailed introduction to KL divergence in the quantum field:

Definition and Background

KL divergence, also known as information gain or relative entropy, is widely used in information theory and statistics to evaluate the information loss incurred when a theoretical or fitted distribution Q approximates a true distribution P. In the quantum domain, the KL divergence can also be used to measure the difference between two quantum states or quantum probability distributions.

Formula and Calculation

For quantum states ρ and σ, their KL divergence is usually defined as

\[ D_KL(ρ||σ) = tr(ρ log(ρ/σ)) \]

Among them, tr represents the trace of a matrix, which is the sum of the elements on the diagonal of the matrix. This formula is the standard form of the quantum KL divergence, which is used to calculate the difference between two quantum states.

Nature and characteristics

Asymmetry KL

divergence is asymmetric, that is \(D_KL(ρ||σ) ≠ D_KL(σ||ρ)。\) This means that when using KL divergence to measure the difference between two quantum states, one needs to pay attention to the order.

Non-negative

KL divergence is always greater than or equal to 0, that is \(D_KL(ρ||σ) ≥ 0\) .If and only if \(ρ=σ\) , \(D_KL(ρ||σ) = 0\) 。 This shows that when two quantum states are exactly the same, the KL divergence between them is 0. Information loss: KL divergence can be seen as the amount of extra information needed when using distribution σ to encode samples from distribution ρ. It measures the loss of information when using one distribution to approximate another.

In QPanda3

Here is API doc for KL_divergence

KL divergence for calculating the probability distribution of discrete random variables

Python

 
import random
from pyqpanda3.quantum_info import KL_divergence
def test_KL_divergence():
    import numpy as np
    # prepare probability distribution
    
    n = random.randint(1, 50)
    
    alpha = np.ones(n)
 
    # Generate two arrays satisfying probability constraints using Dirichlet distribution, p distribution and q distribution
    p_prob_val = np.random.dirichlet(alpha, size=1)[0]
    q_prob_val = np.random.dirichlet(alpha, size=1)[0]
    print("p_prob:", p_prob_val)
    print("q_prob:", q_prob_val)
    # Calculate KL divergence
    kld = KL_divergence(p_prob_val,q_prob_val)
    print("kld:", kld)
if __name__ == "__main__":
    test_KL_divergence()
 

Output

p_prob: [0.07732759 0.0389159  0.07715404 0.11005092 0.02503134 0.02849995
 0.01620025 0.02423337 0.03629235 0.17035652 0.17097729 0.22496048]
q_prob: [0.19705237 0.04430997 0.28896571 0.09751531 0.03782515 0.00203659
 0.15336831 0.06625373 0.03402398 0.05084955 0.02178681 0.00601251]
kld: 1.2135042033718717

Calculate the KL divergence of the probability of continuous random variables

Python

 
from pyqpanda3.quantum_info import KL_divergence
def test_KL_divergence2():
    import math
    a = -5.0
    b = 6.0
 
    def normal_pdf(x, mu, sigma):
        # probability density function
        # Ensure that the standard deviation is a positive number
        if sigma <= 0:
            raise ValueError("Standard deviation must be a positive number")
 
        # Calculate the probability density of the normal distribution
        coefficient = 1 / (sigma * math.sqrt(2 * math.pi))
        exponent = math.exp(-0.5 * ((x - mu) / sigma) ** 2)
        return coefficient * exponent
    def p_pdf(x):
      # probability density function
        # Normal distribution
        return normal_pdf(x,3,3.14)
 
    def q_pdf(x)->float:
        # The uniform distribution q_pdf(x) approaches 0, which is approximated by the minimum value 0.000001
        return 0.000001
 
    print("p_pdf(3.14)",p_pdf(3.14))
 
    # Calculate KL divergence
    kld = KL_divergence(p_pdf=p_pdf,q_pdf=q_pdf,x_start=a,x_end=b,dx=0.000001)
    print("kld:", kld)
if __name__ == "__main__":
    test_KL_divergence2()
 

Output

p_pdf(3.14) 0.12692546072510916
kld: 9.422541885590975