In-plane anisotropic ground states are ubiquitous in correlated solids such as pnictides, cuprates and manganites. They can arise from doping Mott insulators and compete with phases such as superconductivity; however, their origins are debated. Strong coupling between lattice, charge, orbital and spin degrees of freedom results in simultaneous ordering of multiple parameters, masking the mechanism that drives the transition. Here we demonstrate that the orbital domains in a manganite can be oriented by the polarization of a pulsed THz light field. Through the application of a Hubbard model, we show that domain control can be achieved by enhancing the local Coulomb interactions, which drive domain reorientation. Our results highlight the key role played by the Coulomb interaction in the control and manipulation of orbital order in the manganites and demonstrate a new way to use THz to understand and manipulate anisotropic phases in a potentially broad range of correlated materials.