深度確定性策略梯度法建構幹道即時號誌控制系統

Constructing Arterial Real-Time Traffic Signal Control System by Deep Deterministic Policy Gradient

黃建皓(Chien-Hao Huang)

指導教授 ： 許添本

國立臺灣大學/工學院/土木工程學研究所/碩士(2022年)

https://doi.org/10.6342/NTU202201730

摘要

交通號誌扮演了都市運輸系統中相當重要的角色，好的號誌時制設計將會有效提升車流紓解效率。然而臺灣多數號誌化交叉口採用定時號誌系統，無法即時應變車流變化為其一大缺點。近年來則因人工智慧蓬勃發展，許多將深度強化學習應用於號誌控制領域之相關研究應運而生，然而過往研究大多僅考慮純汽車流之交通環境，研究結果可能不符臺灣之高混合比之車流環境；即便近年來國內已有相關研究將混合車流納入考量，但這些研究的應用範圍仍僅侷限於獨立交叉口，難以應用於號誌設置密集的都會地區。

基於以上原因，本研究欲將環境拓展至多個交叉口，並考慮混合車流情況下，利用深度強化學習方法建立一即時號誌控制系統。本研究首先於VISSIM® 微觀車流模擬軟體建置一1×3的模擬路網作為強化學習的環境，並於路網中加入機車停等區與機車待轉區等在臺灣常見的交通工程設計；接著利用深度確定性策略梯度法建構號誌代理人，號誌代理人自偵測器取得近端車流密度與速度後，於每一週期動態調整系統中所有交叉口之時比及時差。為使號誌代理人訓練方向更為明確，本研究定義之獎勵函數僅納入路口通過量及等候車隊長度兩項指標；此外考量每回合代理人與環境互動次數不多，因此將獎勵折扣因子調降為0.6，並據此訓練2000回合。最終結果顯示，本研究建構之即時號誌控制系統無論是與同亮式號誌及Synchro® 最佳化時制相比，皆能達到更佳的績效，且支道方向上之車流有更佳的紓解效率；最後再進行流量情境測試及混合比情境測試，驗證了本研究建構之即時號誌控制系統確實具有應變車流變化之能力。

關鍵字

強化學習 ； 深度確定性策略梯度 ； 幹道連鎖 ； 混合車流

並列摘要

Traffic signals play important roles in urban transportation systems. With well-designed signal timing plans, the efficiency of traffic flow dispersing would be increased. However, in Taiwan, most signalized intersections are applied to the pre-timed signal control system, which cannot react to the changes in traffic flow immediately is its shortcomings. With the booming development of artificial intelligence in recent years, lots of studies that applied deep reinforcement learning methods to traffic control areas have come into being. However, in most of those past studies, passenger cars were the only vehicle accounted for in the environment. Although some studies considered mixed traffic flow scenarios in domestic in recent years, they were only able to apply to isolated intersections, which are hard to apply to metropolitan areas with high-density traffic signals.

Based on the reasons mentioned above, we expanded the environment to multi-intersection in this study, and constructed a real-time traffic control system by deep reinforcement learning method under mixed traffic flow. We first constructed a 1×3 virtual network in VISSIM® microscopic traffic simulation software as an environment in reinforcement learning in this study. Also, we added motorbike waiting zones and two-stage left turn waiting areas, which are frequently designed elements for traffic engineering in Taiwan. After that, we built the traffic signal agent by deep deterministic policy gradient. The agent would collect the traffic flow density and the average speed from each approach, and then adjust the splits and offsets dynamically for all intersections in the system after a cycle finishing. To ensure the training process more specific, total throughput and queue length for each intersection are the only two measurements in our reward function. In addition, due to the interaction less frequently between the agent and the environment in this study, we reduced the reward discount factor to 0.6, and then began to train the agent for 2000 episodes with these settings. The final results show that compared to the simultaneous traffic signal system or pre-timed signal timing plans optimized by Synchro®, the real-time traffic signal control system constructed in this study performs better, and reaches better efficiency in dispersing the traffic flow of streets. Finally, we tested the real-time signal control system constructed in this study in different changes in traffic volumes and mixed traffic scenarios. After testing different scenarios, it is proved that our real-time traffic control system does have the ability to react to the changes in traffic flow.

並列關鍵字

Reinforcement Learning ； Deep Deterministic Policy Gradient ； Arterial Signal Coordination ； Mixed Traffic Flow

參考文獻

[1] 「機動車輛登記數」，中華民國交通部。檢索自：https://stat.motc.gov.tw/mocdb/stmain.jsp?sys=100 funid=a3301.

[2] Webster, F. V., "Traffic Signal Settings," Technical Report Road Research Technical Paper No. 39, HerMajerty's Stationery Office, Department of Scientific and Industrial Research Road Research Laboratory, 1958.

[3] Little, J. D. C., "The Synchronization of Traffic Signals by Mixed-Integer Linear Programming.," Operations Research, Jul.-Aug., 1966, Vol.14, No.4, pp. 568-594, 1966.

[4] Little, J. D. C., Kelson, M. D., Gartner, N. H., "MAXBAND: A Program for Setting Signals on Arteries and Triangular Networks.," Transportation Research Record 795, 1981.

[5] Gartner, N. H., Assmann, S. F., Lasaga, F., Hou, D. L., "Multi-Band Approach to Arterial Traffic Signal Optimization.," Transportation Res-B, Vol 25B, No. 1, pp. 55-74, 1991.