The increasing penetration of wind energy conversion systems into modern power grids has introduced significant challenges to transmission line protection schemes. Among these, distance relays equipped with power swing blocking and out-of-step tripping functions are particularly affected due to variations in apparent impedance trajectories during transient disturbances. This study investigates the impact of wind farm penetration on the mal-operation of distance relay power swing blocking and out-of-step functions using a modified IEEE 6-Bus Network integrated with a 10 MW Doubly-Fed Induction Generator (DFIG) wind farm. First, the analysis was performed using the conventional Power Swing Blocking (PSB) and Out-of-Step (OST) protection scheme, which were based on deterministic criteria such as blinders, impedance trajectory analysis, and rate of change of impedance measurements. The analysis was expanded by employing a modified intelligent PSB and OST protection scheme based on a hybrid signal processing and machine learning methodology. The paper focuses on analysing the dynamic behaviour of impedance loci, the resulting relay behaviour and performance under changing degrees of DFIG wind farm penetration. Time-domain simulations were conducted to evaluate relay performance under stable and unstable swing conditions. The obtained results showed that higher levels of wind penetration affected both the magnitude of fault current contribution and the dynamic behaviour of the system, which resulted in changes to the apparent impedance trajectory seen by the distance relay. These changes influenced the operation of the PSB and OST, which led to delayed blocking actions or unwanted relay trips in some scenarios. This study highlighted the importance of protection coordination when integrating large amounts of wind generation into transmission networks.