用於智慧購物車的動作辨識

Action Recognition for Smart Shopping Carts

紀閎全(Hong-Chuan Chi)

指導教授 ： 易志偉

國立交通大學/資訊學院/資訊科學與工程研究所/碩士(2019年)

摘要

智慧購物車基於深度學習技術分析消費者置入及取出商品的動作，記錄購動車上的商品清單，結合身分識別與電子支付，可實現“拿了就走（Just Walk Out）”的購物體驗。在此研究中，我們設計與實作智慧購物車雛型系統以驗證相關技術，此購物車設有攝影機及通訊介面用以記錄及傳送購物車的影像，使用深度學習網路分析每幀的內容，再進行時間軸上的動作解析，以偵測置物與取物等的購物行為，最終將後採購的商品清單呈現於使用者的虛擬購物車APP上。在效能分析中，基於Faster R-CNN、YOLOv2及YOLOv2-Tiny幀的分類準確度落於93.0%至90.3%間，執行速度分別可達5 fps、39 fps及50 fps；購物動作辨識用於區分無手時段、空手時段及手持物時段，正確率96%、時間誤差0.119秒；購物事件用於偵測消費者對購物車中商品所做的異動，分成：無更動、置物、取物及換物等四種，精確度為97.9%。綜合而言，我們提出一個基於影像動作辨識的智慧購物車方案，使用深度學習影像辨識結合動作解析，以追蹤購物車中商品清單，並實作雛型系統展示研究成果。

關鍵字

智慧購物車 ； 動作辨識 ； Faster R-CNN ； YOLOv2 ； YOLOv2-Tiny ； 智慧商店

並列摘要

In this study, we designed and implemented a smart shopping cart prototype system to verify the related technology. The shopping cart has a camera and communication interface for recording and transmitting images of the shopping cart, and uses a deep learning network to analyze the content of each frame. To detect the shopping behavior of instill and pull it off, perform the mechanism of action analysis on the time axis, and present the list of the purchased products to the user on the virtual shopping Cart-APP. In performance, the classification accuracy based on Faster R-CNN, YOLOv2 and YOLOv2-Tiny frames falls between 93.0% and 90.3%, and the speeds are up to 5 fps, 39 fps and 50 fps respectively. Shopping action recognition used to distinguish among “No hand”, “Empty hand” and “Holding item” time, the correct rate is 96%, and the time error is 0.119s. The shopping event detect the change of items in the shopping cart that made by the consumer. The shopping events are divide into four classes such as: “No Change”, “Insert”, “Take away” and “Change in Items”. The accuracy of shopping events are 97.9%. In summary, Proposed a smart shopping cart solution based on image action recognition, combine the action recognition and deep learning on data to track the list of products in the shopping cart, for the visual aspects implement the prototype system.

並列關鍵字

smart shopping cart ； action recognition ； Faster R-CNN ； YOLOv2 ； YOLOv2-Tiny ； smart store

參考文獻

[1] U.S. Census Bureau, “Quarterly retail e-commerce sales 3rd quarter 2018,” 19 Nov 2018, https://www.census.gov/retail/mrts/www/data/pdf/ec_current.pdf.

[2] 黃有容, “早Amazon一步!阿里巴巴宣布無人零售來了,” 3 July 2017, https://tw.news.yahoo.com/%E6%97%A9amazon-%E6%AD%A5-%E9%98%BF%E9%87%8C%E5%B7%B4%E5%B7%B4%E5%AE%A3%E5%B8%83%E7%84%A1%E4%BA%BA%E9%9B%B6%E5%94%AE%E5%BA%97%E4%BE%86%E4%BA%86-113004086.html.

[3] NEC, “NEC與中國阿里巴巴集團於零售領域戰略合作,” 29 Sep 2017, https://tw.nec.com/zh_TW/press/201709/tw_20170929_01.html.

[4] C. Szegedy, A. Toshev, and D. Erhan, “Deep neural networks for object detection,” in Advances in Neural Information Processing Systems (NIPS 2013) , pp. 2553-2561, 5-10 December 2013, California, USA.

[5] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards real-time object detection with region proposal networks,” in Advances in Neural Information Processing Systems (NIPS 2015), pp. 91-99, 7- 12 December 2015, Montreal, Canada.