Spin glasses are canonical examples of complex matter and form a basis for describing artificial neural networks.  Advancing experimental insight into their structure requires repeatable control over microscopic degrees of freedom.  I will present how we achieved this at the atomic level using a quantum-optical system comprised of ultracold gases of atoms coupled via photons resonating within a multimode cavity.  We can realize all-to-all Ising and vector spin glasses whose spin configurations are microscopically detectable and controllable.  This new spin glass system has allowed us to directly observe ultrametricity in a physical system and to realize an associative memory with capacity beyond that of Hopfield’s model.