Kitchen fires pose a significant challenge and threat to people and the environment. Prompt response and accurate classification of fire occurrences are crucial to ensure safety and reduce potential property damage. This study addresses the need for effective fire detection technologies by evaluating the performance of the You Only Look Once version 5 medium (YOLOv5m) model using both hue, saturation, and value (HSV) and visible light (red, green, and blue [RGB]) color spaces. To reduce false positives, the experiment was expanded by including background images in the training dataset. Two kitchen fire datasets, one in RGB and one in HSV, were used to train and evaluate the model. Based on the results, HSV color space offers higher recall and precision for fire-on-pan detection, achieving 0.882 and 0.931, outperforming RGB. The best overall mean performance was in the first experiment, RGB without background images, resulting in the highest mean average precision (mAP)@0.5:0.95 score of 0.651. This performance was limited by lower recall and precision compared to HSV in specific fire scenarios. A key limitation of this study is its focus on kitchen environments, for which the findings may not be directly generalizable to other fire scenarios with different environmental conditions or fire characteristics. Furthermore, the study is limited to the YOLOv5m architecture, where other detection models may yield different results. In terms of kitchen fire detection, this study provides a comprehensive comparison of the RGB and HSV color spaces, offering insights into their benefits and drawbacks. This research shows that the HSV color space is useful in certain fire detection scenarios, and that combining the two color spaces yields an improved detection model for real-time applications.