In recent years, although the topic of unmanned driving continues, it is still a challenge to realize unmanned driving in an unknown environment. In order to realize auto driving in an unknown environment, FCN-BiLSTM framework was proposed and end-to-end training was adopted. Road images collected by the simulator were input into the network, and the data of the steering Angle of the car was output, with the deep learning network in the middle, so as to control the car simulator and realize the smooth driving of the car. BiLSTM network is composed of forward LSTM network and backward LSTM network, which is used to deal with the dependence between before and after images. Experimental results show that in Udacity open source unmanned driving simulator, the combination of convolutional neural network and bidirectional long and short-term memory neural network provides an effective scheme for the control of unmanned driving.