In order to improve the speed and accuracy of road traffic sign detection and recognition, the classical convolutional neural network architecture Lenet-5 is proposed to be improved and optimized. LeNet-5 network structure was improved, reducing unnecessary link and calculation to improve the real-time performance of algorithm, and combined with the feature of multi-layer fusion method, will be in the two layers of convolution layer extracts the features of the network are sent to all the connection layer, strengthen the characteristic power of expression, on the basis of meet the real time improve the identification accuracy of the algorithm and introduce Dropout mechanism at the same time, to avoid over fitting, improve the network generalization ability of the model. Experimental results show that the improved Lenet-5 convolutional neural network proposed in this paper achieves the desired effect in terms of accuracy and real-time performance.