The HOM problem is decidable

Abstract

We close affirmatively a question which has been open for 35 years: decidability of the HOM problem. The HOM problem consists in deciding, given a tree homomorphism $H$ and a regular tree languagle $L$ represented by a tree automaton, whether $H(L)$ is regular. For deciding the HOM problem, we develop new constructions and techniques which are interesting by themselves, and provide several significant intermediate results. For example, we prove that the universality problem is decidable for languages represented by tree automata with equality constraints, and that the equivalence and inclusion problems are decidable for images of regular languages through tree homomorphisms. Our contributions are based on the following new results. We describe a simple transformation for converting a tree automaton with equality constraints into a tree automaton with disequality constraints recognizing the complementary language. We also define a new class of automaton with arbitrary disequality constraints and a particular kind of equality constraints. This new class essentially recognizes the intersection of a tree automaton with disequality constraints and the image of a regular language through a tree homomorphism. We prove decidability of emptiness and finiteness for this class by a pumping mechanism. The above constructions are combined adequately to provide an algorithm deciding the HOM problem.