How to accurately forecast stock prices has been a hard nut to crack in the field of quantitative finance. Many scholars have proved that stock prices tend to be nonlinear in previous studies. With the deeper insights in artificial intelligence theory and the unceasing progress of technology, the application of machine learning algorithm with strong nonlinear fitting ability has attracted increasing attention in stock research. This paper primarily introduces the application of machine learning in stock price forecasting and summarizes the relevant research at home and abroad before elaborating the principle of random forest algorithm, gradient enhanced decision tree (GBDT) algorithm and support vector machine algorithm. Then, based on the actual data, this paper compares the outcome of the random forest algorithm, gradient tree (GBDT) algorithm, support vector machine (SVM) algorithm and the traditional time sequence method. The difference among the results shows that machine learning algorithms compared with time series model has better forecasting performance. Additionally random forest algorithm whose predicting results' mean square error is 10.720077538941355 has the best performance, therefore of the most obvious practical value.