Deep Learning-based Hybrid Beamforming, Channel Estimation, and Sparse Signal Recovery for Massive MIMO Systems

Abstract

Key technologies that can realize Gbps data rates in sixth-generation (6G) cellular networks include massive multiple-input multiple-output (MIMO), millimeter-wave (mmWave) frequency band, orthogonal frequency division multiplexing (OFDM) and orthogonal time frequency space (OTFS) schemes. In this thesis, deep neural networks (DNNs) are employed to improve the performance of 6G systems. Specifically, three research areas are studied: 1) hybrid analog and digital beamforming based on analytical phase optimization and self-supervised deep learning (DL) for mmWave massive MIMO-OFDM systems, 2) DL-based channel estimation for MIMO-OTFS systems, and 3) DL-based high-dimensional sparse signal recovery with application to high-mobility communications. Hybrid beamforming can address high cost and power consumption of fully digital precoding. A major challenge is solving a high-dimensional and nonconvex optimization problem. To accomplish hybrid precoder design, the optimization problem is transformed into a system of nonlinear equations, and by adopting the Newton method with diagonal Jacobian, an inverse-free phase-optimized analog$/$digital (IFPAD) algorithm is developed that obtains jointly optimal hybrid beamformers. IFPAD features simple implementation and low complexity and avoids iterative matrix inversions. Additionally, a model-driven neural network, AE-HBFnet, is designed based on self-supervised learning to perform the hybrid beamforming. The AE-HBFnet is a lightweight generative model whose architecture is inspired by autoencoders (AEs). The proposed schemes are then extended to mmWave massive MIMO-OFDM systems. The proposed methods achieve improved spectral efficiency (SE) and significantly reduced computational complexity. Accurate channel state information (CSI) of wireless channels is essential for reliable communications. To accurately estimate the CSI of MIMO-OTFS systems, a DL-based framework consisting of three convolutional neural networks (CNNs) is designed: (i) PositionNet obtains the support of the sparse matrix, (ii) PositionNet\textsubscript{$r$} refines the support, and (iii) AmplitudeNet obtains nonzero values. Improvement in bit error rate (BER), normalized mean squared error (NMSE), and a significant reduction in computation are achieved. A robust sparse signal recovery neural network (SSRnet) is designed and applied to CSI acquisition of massive MIMO-OTFS systems. SSRnet can precisely recover the sparse signal, outperforming conventional methods, including least-squares (LS) estimation with perfectly known support, by virtue of its denoising behavior, while offering substantially reduced computational load and pilot overhead.