For a π:G→U(H)\pi:G\to\mathcal U(H), an invariant vector is v∈Hv\in H with π(g)v=v\pi(g)v=v for every g∈Gg\in G. The invariant subspace is

HG=⋂g∈Gker⁡(π(g)−I).H^G=\bigcap_{g\in G}\ker(\pi(g)-I).

It is closed, and its orthogonal complement is invariant.

Fixed vector versus invariant subspace

An invariant line may transform by a nontrivial character and contain no nonzero invariant vector. For Koopman representations, constants always form a trivial subrepresentation; ergodicity says the orthogonal complement of constants contains no invariant vector.