The accurate timing for collision avoidance actions is crucial for preventing maritime collisions. Traditional methods often rely on collision risk assessments, using quantitative indicators like the Distance to the Closest Point of Approach (DCPA) and the Time to the Closest Point of Approach (TCPA). Ship Officers on Watch (OOWs) are required to execute avoidance maneuvers once these indicators reach or exceed preset safety thresholds. However, the effectiveness of these indicators is limited by uncertainties in the maritime environment and the human behaviors of OOWs. To address these limitations, this study introduces a machine learning method to learn collision avoidance behavior from empirical data of ship collision avoidance, particularly in cross-encounter situations. The research utilizes Automatic Identification System (AIS) data from the open waters around Ningbo Zhoushan Port. After data preprocessing and applying spatio-temporal constraints, this study identifies ship trajectory pairs in crossing scenarios and calculates their relative motion parameters. The Douglas–Peucker algorithm is used to identify the timing of ship collision avoidance actions and a collision avoidance decision dataset is constructed. The Random Forest algorithm was then used to analyze the factors affecting the timing of collision avoidance, and six key factors were identified: the distance, relative speed, relative bearing, DCPA, TCPA, and the ratio of the lengths of the giving-way and stand-on ships. These factors serve as inputs for the XGBoost algorithm model, which is enhanced with Particle Swarm Optimization (PSO), and thus constructing a ship collision avoidance decision model. In addition, considering the inherent errors in any model and the dynamic nature of the ship collision avoidance process, an action time window for collision avoidance is introduced, which provides a more flexible time range for ships to make timely collision avoidance responses based on actual conditions and the specific encounter environment. This model provides OOWs with accurate timing for taking collision avoidance decisions. Case studies have validated the practicality and effectiveness of this model, offering new theoretical foundations and practical guidance for maritime collision avoidance.