[1]刘少博,黄颖杰,何伟铭.面向能源自洽的港口AGV充电策略强化学习建模与优化[J].长安大学学报(自然科学版),2024,44(5):164-176.[doi:10.19721/j.cnki.1671-8879.2024.05.014]
　LIU Shao-bo,HUANG Ying-jie,HE Wei-ming.Reinforcement learning modeling and optimization for AGV charging strategy in energy self-consistent port[J].Journal of Chang’an University (Natural Science Edition),2024,44(5):164-176.[doi:10.19721/j.cnki.1671-8879.2024.05.014]
点击复制

面向能源自洽的港口AGV充电策略强化学习建模与优化()
分享到：

长安大学学报（自然科学版）[ISSN:1006-6977/CN:61-1281/TN]

卷:
第44卷
期数:
2024年5期
页码:
164-176
栏目:
交通能源融合技术专题
出版日期:
2024-10-10

文章信息/Info

Title:
Reinforcement learning modeling and optimization for AGV charging strategy in energy self-consistent port
文章编号:
1671-8879(2024)05-0164-13
作者:
刘少博1; 2; 3; 黄颖杰1; 2; 3; 何伟铭1; 2; 3
(1. 武汉理工大学 智能交通系统研究中心,湖北 武汉 430063; 2. 武汉理工大学水路交通控制全国重点实验室,湖北 武汉 430063; 3. 武汉理工大学 三亚科教创新园,海南 三亚 572000)
Author(s):
LIU Shao-bo1; 2; 3;  HUANG Ying-jie1; 2; 3;  HE Wei-ming1; 2; 3
(1. Intelligent Transportation Systems Research Center, Wuhan University of Technology, Wuhan 430063,Hubei, China; 2. State Key Laboratory of Maritime Technology and Safety, Wuhan 430063, Hubei, China; 3. Sanya Science and Education Innovation Park, Wuhan University of Technology, Sanya 572000, Hainan, China)
关键词:
交通工程;  能源自洽港口;  清洁能源消纳;  AGV充电策略;  强化学习;  仿真场景
Keywords:
traffic engineering;  energy self-consistent port;  clean energy consumption;  AGV charging strategy;  reinforcement learning;  simulation scenarios
分类号:
U691.3
DOI:
10.19721/j.cnki.1671-8879.2024.05.014
文献标志码:
A
摘要:
为优化港口自动导引运输车(AGV)充电策略以提升港口清洁能源消纳的效果,提出一种仿真场景下的能源自洽港口AGV充电策略强化学习建模与优化方法。通过分析AGV运输与充电的调度流程,构建基于仿真的强化学习策略优化框架,将AGV的充电决策建模为马尔科夫决策过程(MDP),通过对环境空间、动作空间、奖励函数的设置来描述AGV充电决策对于整体运输作业的影响; 根据中国某港口布局,构建包含集装箱进出口运输、AGV充电等流程的仿真模型,并对水平运输环节中,涉及的AGV选取、调度流程及充电规则进行定义,表达出实际运输场景中涉及的能源约束、设备数量约束、物流调度规则约束; 分别将2种不同发力特征的新能源曲线作为充能奖励系数,利用深度Q值网络(DQN)算法与仿真模型的交互,实现充电策略的优化,并将优化充电策略与2种不同区间的规则充电策略在不同初始电量场景中进行比较分析。研究结果表明:在4种不同的初始电量情况下,2种优化充电策略的清洁能源消纳率分别保持在74%～94%和69%～72%,而规则充电策略则是在38%～62%和33%～55%; 在总能耗上,优化充电策略的清洁能源占比为35%～47%,而规则充电策略则为17%～37%; 在能耗成本上,优化充电策略普遍优于规则充电策略; 4种策略在4种不同初始电量场景下的仿真时间均为60 000～62 000 s,没有明显差距,说明该方法在提高消纳能力的同时并没有增加整体作业时间,验证了该方法的有效性和经济性。
Abstract:
To optimize the charging strategy of automated guided vehicles(AGVs)in energy self-consistency ports and enhance the consumption of clean energy, a reinforcement learning modeling and optimization method for AGV charging strategy in simulation scenarios was proposed. By analyzing the scheduling process of AGV transportation and charging, a simulation-based framework for reinforcement learning strategy optimization was established. The charging decision-making process of AGVs was modeled as the Markov decision process(MDP), with the influence of AGV charging decisions on overall transportation operations articulated through the configuration of environmental space, action space, and reward function. A simulation model, reflective of a Chinese port's layout, was constructed to simulate the processes of container import and export transportation and AGV charging. The horizontal transportation link, including AGV selection, scheduling procedures, and charging protocols was defined, while capturing the constraints of energy, equipment quantity, and logistics scheduling rules present in actual transportation scenarios. Two clean energy generation curves with different feature were utilized as reward coefficients for charging, facilitating the optimization of charging strategies through the interaction of the DQN algorithm with the simulation model. The optimized charging strategies were compared and analyzed against two different rule-based charging strategies in different initial battery power scenarios. The results show that the clean energy consumption rates of the two optimized charging strategies remain 74% to 94% and 69% to 72% while the rule-based charging strategies are 38% to 62% and 33% to 55%. In terms of total energy consumption, the clean energy share of the optimized charging strategy is 35% to 47%, compared to 17% to 37% for the rule-based charging strategy. The optimized charging strategy is generally more favorable in terms of energy cost. The simulation durations for all four strategies across four different initial battery power scenarios are consistent, ranging from approximately 60,000 to 62,000 s, indicating that the proposed method does not prolong overall operational time while enhancing clean energy consumption capacity, thus confirming its effectiveness and economic efficiency.6 tabs, 11 figs, 25 refs.

参考文献/References:

[1] 邰能灵,王萧博,黄文焘,等.港口综合能源系统低碳化技术综述[J].电网技术,2022,46(10):3749-3764.
TAI Neng-ling,WANG Xiao-bo,HUANG Wen-tao,et al.Review of lowcarbon technology for integrated port energy systems[J].Power System Technology,2022,46(10):3749-3764.
[2]方斯顿,赵常宏,丁肇豪,等.面向碳中和的港口综合能源系统(二):能源-交通融合中的柔性资源与关键技术[J].中国电机工程学报,2023,43(3):950-969.
FANG Si-dun,ZHAO Chang-hong,DING Zhao-hao,et al.Portintegrated energy systems toward carbon neutrality(Ⅱ):Flexible resources and key technologies in energy-transportation integration[J].Proceedings of the CSEE,2023,43(3):950-969.
[3]ZHAO Q R,JI S W,GUO D,et al.Research on cooperative scheduling of automated quayside cranes and automatic guided vehicles in automated container terminal[J].Mathematical Problems in Engineering,2019,2019:6574582.
[4]张亚琦,杨 斌,胡志华,等.自动化码头AGV充电与作业的集成调度研究[J].计算机工程与应用,2017,53(18):257-262,270.
ZHANG Ya-qi,YANG Bin,HU Zhi-hua,et al.Research of AGV charging and job integrated scheduling at automated container terminal[J].Computer Engineering and Applications,2017,53(18):257-262,270.
[5]范厚明,郭振峰,杨 宇.预约机制下送箱集卡多集装箱码头调度问题[J].系统仿真学报,2017,29(12):3051-3060.
FAN Hou-ming,GUO Zhen-feng,YANG Yu.Truck scheduling for delivering containers among multiple container terminals based on truck appointment system[J].Journal of System Simulation,2017,29(12):3051-3060.
[6]范厚明,郭振峰,李 阳.考虑碳排放和预约机制的送箱集卡多码头调度问题[J].同济大学学报(自然科学版),2018,46(9):1241-1252,1280.
FAN Hou-ming,GUO Zhen-feng,LI Yang.Truck scheduling in delivering containers from an outside yard to multiple container terminals considering carbon emission and the truck appointment system[J].Journal of Tongji University(Natural Science),2018,46(9):1241-1252,1280.
[7]李林蔓,李雨青,王孟雅,等.基于启发式规则的自动化码头换电式AGV调度优化方法[J].运筹与管理,2023,32(10):9-15.
LI Lin-man,LI Yu-qing,WANG Meng-ya,et al.Optimization method for powerchanging AGV scheduling of automatic terminal based on heuristic rules[J].Operations Research and Management Science,2023,32(10):9-15.
[8]HU H T,CHEN X Z,WANG T S,et al.A threestage decomposition method for the joint vehicle dispatching and storage allocation problem in automated container terminals[J].Computers & Industrial Engineering,2019,129:90-101.
[9]秦 琴,梁承姬.自动化码头考虑缓冲区的设备协调调度优化[J].计算机工程与应用,2020,56(6):262-270.
QIN Qin,LIANG Cheng-ji.Optimization of equipment coordination scheduling with considering buffer space in automated container terminal[J].Computer Engineering and Applications,2020,56(6):262-270.
[10]滕 浩,庄子龙,黄子钊,等.自动化码头水平运输调度的改进人工蜂群算法[J].计算机集成制造系统,2023,29(7):2449-2460.
TENG Hao,ZHUANG Zi-long,HUANG Zi-zhao,et al.Improved artificial bee colony algorithm for automatic terminal horizontal transportation scheduling[J].Computer Integrated Manufacturing Systems,2023,29(7):2449-2460.
[11]XU Y X,QI L,LUAN W J,et al.Load-in-load-out AGV route planning in automatic container terminal[J].IEEE Access,2020,8:157081-157088.
[12]初良勇,周于佩,姚艺飞,等.自动化码头多资源协同调度优化模型与算法[J].哈尔滨工程大学学报,2022,43(10):1539-1546.
CHU Liang-yong,ZHOU Yu-pei,YAO Yi-fei,et al.Co-scheduling optimization of multiple resources in the automated terminal considering AGV path conflict[J].Journal of Harbin Engineering University,2022,43(10):1539-1546.
[13]王 轩,汪 宝,陈艳萍,等.低碳自动化码头AGV物流调度和有序充电的联合经济优化[J/OL].上海交通大学学报,2023:1-24[2023-06-01].https://doi.org/10.16183/j.cnki.jsjtu.2023.027.
WANG Xuan,WANG Bao,CHEN Yan-ping.Joint economic optimization of AGV logistics scheduling and orderly charging in low-carbon automated terminal[J/OL].Journal of Shanghai Jiaotong University,2023:1-24[2023-06-01].https://doi.org/10.16183/j.cnki.jsjtu.2023.027.
[14]SINGH N,DANG Q V,AKCAY A,et al.A matheuristic for AGV scheduling with battery constraints[J].European Journal of Operational Research,2022,298(3):855-873.
[15]梁 晶,王超恒.基于改进NSGA-Ⅱ算法的自动化集装箱码头AGV调度优化[J].中国水运(下半月),2023,23(5):28-30.
LIANG Jing,WANG Chao-heng.AGV scheduling optimization of automated container terminal based on improved NSGA-Ⅱ algorithm[J].China Water Transport,2023,23(5):28-30.
[16]LI L M,LI Y Q,LIU R,et al.A two-stage stochastic programming for AGV scheduling with random tasks and battery swapping in automated container terminals[J].Transportation Research Part E,2023,174:103110.
[17]MA N,ZHOU C H,STEPHEN A.Simulation model and performance evaluation of battery-powered AGV systems in automated container terminals[J].Simulation Modelling Practice and Theory,2021,106:102146.
[18]彭云辉,张 朋,张 煜,等.考虑自动化集装箱码头充电桩分配策略的动态调度仿真分析[J].武汉理工大学学报(交通科学与工程版),2023,47(1):180-184.
PENG Yun-hui,ZHANG Peng,ZHANG Yu,et al.Dynamic scheduling simulation analysis considering charging pile allocation strategy of automated container terminal[J].Journal of Wuhan University of Technology(Transportation Science & Engineering),2023,47(1):180-184.
[19]马宁丽,胡志华.基于拥堵感知的自动化集装箱码头AGV充电策略[J/OL].计算机集成制造系统,2022:1-22[2022-03-08].https://kns.cnki.net/kcms/detail/11.5946.TP.20220305.1541.010.html.
MA Ning-li,HU Zhi-hua.Congestion-aware AGV charging strategy in automated container terminal[J/OL].Computer Integrated Manufacturing Systems,2022:1-22[2022-03-08].https://kns.cnki.net/kcms/detail/11.5946.TP.20220305.1541.010.html.
[20]傅正堂,胡志华,宗 康.集装箱码头AGV电量非饱和状态下的调度优化[J].大连海事大学学报,2017,43(3):58-62.
FU Zheng-tang,HU Zhi-hua,ZONG Kang.Scheduling optimization of container terminal AGV under electricity unsaturation condition[J].Journal of Dalian Maritime University,2017,43(3):58-62.
[21]ZHENG X Y,LIANG C J,WANG Y,et al.Multi-AGV dynamic scheduling in an automated container terminal:A deep reinforcement learning approach[J].Mathematics,2022,10(23):4575.
[22]CHOE R,KIM J,RYU K R.Online preference learning for adaptive dispatching of AGVs in an automated container terminal[J].Applied Soft Computing,2016,38:647-660.
[23]ANGELOUDIS P,BELL M G H.An uncertainty-aware AGV assignment algorithm for automated container terminals[J].Transportation Research Part E,2010,46(3):354-366.
[24]李 玉,苌道方,高银萍,等.基于数字孪生的自动化集装箱码头多AGV动态调度[J].计算机集成制造系统,2023,29(12):4175-4190.
LI Yu,CHANG Dao-fang,GAO Yin-ping,et al.Multi-AGV dynamic scheduling of automated container terminal based on digital twin[J].Computer Integrated Manufacturing Systems,2023,29(12):4175-4190.
[25]王盼龙,梁承姬,王 钰.自动化集装箱码头多层设备调度及仿真分析[J].计算机工程与应用,2023,59(24):345-359.
WANG Pan-long,LIANG Cheng-ji,WANG Yu.Multilayer equipment scheduling and simulation analysis of automated container terminal[J].Computer Engineering and Applications,2023,59(24):345-359.

相似文献/References:

[1]王建伟,李娉,高洁,等.中国交通运输碳减排区域划分[J].长安大学学报(自然科学版),2012,32(01):0.
[2]李曙光,周庆华.具有破坏排队的离散时间动态网络装载算法[J].长安大学学报(自然科学版),2012,32(01):0.
[3]凌海兰,郗恩崇.基于随机波动条件的公交客运量预测模型[J].长安大学学报(自然科学版),2012,32(01):0.
[4]田娥,肖庆,陆小佳,等.安全驾驶的横向安全预警报警阈值的确定[J].长安大学学报(自然科学版),2012,32(01):0.
[5]侯贻栋,赵炜华,魏 朗,等.驾驶人空间距离判识规律心理学分析[J].长安大学学报(自然科学版),2012,32(03):86.
　HOU Yi-dong,ZHAO Wei-hua,WEI Lang,et al.Analysis on psychology in cognitive distance about drivers[J].Journal of Chang’an University (Natural Science Edition),2012,32(5):86.
[6]赵跃峰,张生瑞,魏 华.隧道群路段运行速度特性分析[J].长安大学学报(自然科学版),2012,32(06):67.
　ZHAO Yue-feng,ZHANG Sheng-rui,WEI hua.Operating speed characteristics of tunnel group section[J].Journal of Chang’an University (Natural Science Edition),2012,32(5):67.
[7]林 杉,许宏科,刘占文.一种高速公路隧道交通流元胞自动机模型[J].长安大学学报(自然科学版),2012,32(06):73.
　LIN Shan,XU Hong-ke,LIU Zhan-wen.One cellular automaton traffic flow model for expressway tunnel[J].Journal of Chang’an University (Natural Science Edition),2012,32(5):73.
[8]刘俊德,徐 兵,梁永东,等.交通事故下高速公路行车安全评估[J].长安大学学报(自然科学版),2012,32(06):78.
　LIU Jun-de,XU bing,LIANG Yong-dong,et al.Traffic safety assessment of expressway in the accident[J].Journal of Chang’an University (Natural Science Edition),2012,32(5):78.
[9]芮海田,吴群琪,赵跃峰,等.公路建设对区域经济发展的影响分析——以陕西省为例[J].长安大学学报(自然科学版),2012,32(06):83.
　RUI Hai-tian,WU Qun-qi,ZHAO Yue-feng,et al.Influence of highway construction on regional economy development——taking Shaanxi as an example[J].Journal of Chang’an University (Natural Science Edition),2012,32(5):83.
[10]彭 辉,续宗芳,韩永启,等.城市群城际运输结构配置客流分担率模型[J].长安大学学报(自然科学版),2012,32(02):91.
　PENG Hui,XU Zong-fang,HAN Yong-qi,et al.Sharing ratios model of passenger flows in intercity transportation structure configuration among urban agglomeration[J].Journal of Chang’an University (Natural Science Edition),2012,32(5):91.

备注/Memo

备注/Memo:
收稿日期:2024-04-27
基金项目:国家重点研发计划项目(2021YFB2601300)
作者简介:刘少博(1985-),男,河南洛阳人,副教授,工学博士,E-mail:shaobo@whut.edu.cn。
更新日期/Last Update: 2024-10-20