In manipulation planning, dynamic interactions between the objects and the robots play a significant role. In this scope, dynamic engines allow to consider them within motion planners, giving rise to physics-based motion planners that consider the purposeful manipulation of objects. In this context, the representation of knowledge regarding how the objects have to be manipulated eases a semantic-based reasoning that reduces the computational cost of physics-based planners. In this work, an ontology framework is proposed to organize the knowledge needed for physics-based manipulation planning, allowing to derive manipulation regions and behaviors. A semantic map is constructed to categorize and assign the manipulation constraints based on the robot, the objects and the type of actions. The ontology framework can be queried using Description Language to obtain the necessary knowledge for the robot to manipulate the objects in its environment.