Irreducible unitary representation

Let $\pi:G\to U(H)$ be a strongly continuous unitary representation on a nonzero Hilbert space. It is irreducible if its only closed invariant subspaces are $\{0\}$ and $H$ . Closedness is essential in infinite-dimensional Hilbert spaces: the definition is topological, not purely algebraic. Because every $\pi(g)$ is unitary and $\pi(g)^{-1}=\pi(g^{-1})$ , a closed invariant subspace is automatically reducing, so its orthogonal complement is invariant as well.

Commutant characterization

The representation is irreducible if and only if every bounded operator $T$ satisfying $T\pi(g)=\pi(g)T$ for all $g\in G$ is a scalar multiple of the identity. Equivalently, the commutant of $\pi(G)$ is $\mathbb C I$ . This is the unitary form of Schur's lemma. A nontrivial invariant subspace yields a nonscalar commuting orthogonal projection, while the spectral projections of a nonscalar self-adjoint operator in the commutant recover a nontrivial invariant subspace.

Examples and reducible cases

Every one-dimensional unitary representation is irreducible. The trivial representation on $\mathbb C$ is the simplest example. A nontrivial orthogonal direct sum $\pi_1\oplus\pi_2$ is reducible because each summand is a proper closed invariant subspace. An irreducible representation cannot have two orthogonal nonzero invariant summands.

Conventions and scope

Noncompactness does not force irreducible unitary representations to be infinite-dimensional. For instance, $\mathbb R\times\operatorname{SU}(2)$ has the two-dimensional irreducible action $(t,k)\mapsto e^{it}k$ . Conversely, topological irreducibility on a Hilbert space does not imply algebraic irreducibility of the derived action on smooth vectors.

References

Gerald B. Folland, A Course in Abstract Harmonic Analysis, 2nd ed., CRC Press, 2016. DOI record. Relevant: §3.1 on invariant subspaces, irreducibility, and Schur's lemma.