In order to solve the problems of time‐consuming, low efficiency and difficult identification of microbial colonies, image processing and YOLOv4 algorithm are studied. A microbial colony recognition algorithm based on YOLOv4 is proposed. Firstly, a microbial colony data set should be established, and the images in the original data set should be processed by median filtering and image enhancement. Then labelimg is used to label the targets in the image to obtain the VOC data set. Then, based on the YOLOv4 framework, the CBAM attention mechanism is introduced to reduce the amount of irrelevant information in the calculation, thereby improving the utilization of feature information. The k‐means++ clustering algorithm is used to optimize the target anchor box to improve the accuracy of detection results. Compared with the original YOLOv4 network structure calculation, the improved algorithm improves the detection accuracy from 71 % to 80 % in the detection of microbial colony targets, which has a higher improvement and can more accurately identify microbial colonies.