One way to proceed is to start with the classical observable and find some suitable Lie algebra containing these as generators. Upon quantization, the observables are promoted to operators, now acting on representations of the Lie algebra. In most cases, one can choose the representations to be unitary and irreducible (or indecomposable, which is the next best thing if you don't need unitary).
Thus, as pointed out by others, if you start with $x$ and $p$ you get the “standard” Hilbert space.
On the other hand, you might be interested in (nuclear) quadrupole deformation, containing the quadrupole moments, which are the traceless symmetric combinations of $x_ix_j$ in 3D, and include angular momenta to describe the associated rotations. You would then naturally “recover” the semi-direct sum $\mathbb{R}^5\rtimes so(3)$ and states in your Hilbert space now span unitary irreps of the associated group.
You might then care to add the kinetic energy as an observable, in which case you must expand $\mathbb{R}^5\rtimes so(3)$ to $\mathfrak{sp}(6,\mathbb{R})$ as the minimal model to include the above. In fact there is a series of such nuclear physics models where the construction process is clear. Some key references are:
- Ui, Haruo. "Quantum Mechanical Rigid Rotator with an Arbitrary Deformation. I: Dynamical Group Approach to Quadratically Deformed Body." Progress of Theoretical Physics 44.1 (1970): 153-171.
- Weaver, L., and L. C. Biedenharn. "Nuclear rotational bands and SL (3, R) symmetry." Nuclear Physics A 185.1 (1972): 1-31,
- and finally Rowe, D. J. "Microscopic theory of the nuclear collective model." Reports on Progress in Physics 48.10 (1985): 1419.
In this way, the Hilbert space emerges quite naturally from the observables in your model as spanned by states in irreducible representations of a Lie algebra.