
Journal of Lie Theory 29 (2019), No. 2, 311341 Copyright Heldermann Verlag 2019 The Polynomial Conjecture for Restrictions of Some Nilpotent Lie Groups Representations Ali Baklouti Faculté des Sciences de Sfax, Dép. de Mathématiques, Sfax 3038, Tunisia ali.baklouti@fss.usf.tn Hidenori Fujiwara Fac. of Science and Technology, University of Kinki, Iizuka 8208555, Japan fujiwara6913@yahoo.co.jp Jean Ludwig Institut Elie Cartan, Université de Lorraine, 57000 Metz, France jean.ludwig@univlorraine.fr [Abstractpdf] Let $G$ be a connected and simply connected nilpotent Lie group, $K$ an analytic subgroup of $G$ and $\pi$ an irreducible unitary representation of $G$ whose coadjoint orbit of $G$ is denoted by $\Omega(\pi)$. Let $\mathcal U(\mathfrak g)$ be the enveloping algebra of ${\mathfrak g}_{\mathbb C}$, $\mathfrak g$ designating the Lie algebra of $G$. We consider the algebra $\left(\mathcal U(\mathfrak g)/\ker \pi\right)^K$ of the $K$invariant elements of $\mathcal U(\mathfrak g)/\ker \pi$. It turns out that this algebra is commutative if and only if the restriction $\pi_K$ of $\pi$ to $K$ has finite multiplicities (cf.\,A.\,Baklouti and H.\,Fujiwara, {\em Commutativit\'{e} des op\'{e}rateurs diff\'{e}rentiels sur l'espace des repr\'{e}sentations restreintes d'un groupe de Lie nilpotent}, J.\,Math.\,Pures\,Appl.\,83 (2004) 137161). In this article we suppose this eventuality and we study the polynomial conjecture asserting that our algebra is isomorphic to the algebra $\mathbb C[\Omega(\pi)]^K$ of the $K$invariant polynomial functions on $\Omega(\pi)$. We give a proof of the conjecture in the case where $\Omega(\pi)$ admits a normal polarization of $G$ and in the case where $K$ is abelian. This problem was partially tackled previously by A.\,Baklouti, H.\,Fujiwara, J.\,Ludwig, {\em Analysis of restrictions of unitary representations of a nilpotent Lie group}, Bull. Sci. Math. 129 (2005) 187209. Keywords: Orbit method, irreducible representations, Penney distribution, Plancherel formula, differential operator. MSC: 22E27 [ Fulltextpdf (251 KB)] for subscribers only. 