Development of Sea Route Display System by Using Augmented Reality

2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC) anff Center, anff, Canada, October 5-8, 2017 Development of Sea oute Display System by Using ugmented eality Tadatsugi Okazaki, ei Takaseki, uri Shoji, and Kazushi Matsubara, Department of Maritime System Engineering Tokyo University of Marine Science and Technology Tokyo, Japan okazaki@kaiyoai.ac.jp bstract navigator draws a route on a chart to navigate a ship from an origin port to a destination port. Then, the navigator sails the ship based on the route. In other word, the navigator misidentifies the route there is a possibility of running aground. In order to prevent navigator s misidentifies, this study proposed a sea route display system which displays the sea route on the surface of the sea by using augmented reality so that navigator may grasp the ship s position from the sea route easily. Effectiveness of the developed system was carried out with actual ship experiment. Keywords navigation support system; augmented reality; sea route I. INTODUCTION In the sea, there is no route that can be saw like the road of land transportation. Navigator makes the route from the departure port to the arriving port on the chart. The navigator measures the position of own ship and navigates the ship on the route described in the chart. There is Electric Chart Display System (ECDIS)[1] as a nautical instrument for the navigator to grasp the ship s position from the route. ECDIS displays the route planed on the electric chart just like the car navigation system and displays the position of the own ship measured with GS[2]. Therefore, the navigator can easily understand own ship s position from the route by using ECDIS. However, the installing obligation of ECDIS is tanker, freighter of 3000 tons or more, and international passenger boat of 500 tons or more[3]. Many of freighters of 3000 tons or less do not equip with ECDIS because they do not have obligation for installing ECDIS and it costs to high. In order to navigate the ship on the route without using ECDIS, navigators required navigation skill. For instance, the ship under the strong wind moves in a direction different from the ship s heading angle because the ship is pressed by the wind force. Therefore, the navigator navigates the ship in consideration of the difference between the moving direction and the ship s heading angle. These navigation skill are obtained by the experience. However, the number of well experienced navigators is expected to be going to decrease in the future, because seafarers aging problem progresses[4]. Then, young navigators should navigate the ship without the support of the experienced navigators. On the other This work was financially supported by JSS KKENHI Grant Number 17H01356. hand, the sea route in bay is indicated by buoys or beacons. The sea route shows a safe route for which enough depth is secured. In other words, the ship runs aground when the ship comes off from the sea route. There is a possibility of misidentifying the buoy even by an experienced navigator when he navigates in the sea route which passes for the first time at nighttime. In such a situation, ECDIS is effective to make the ship navigated safely. However, a lot of ships did not equip with ECDIS as we mentioned above. Therefore, there is needs of navigation support system for recognizing the ship s position from the route with low cost. This study is aiming to develop a sea route display system which displays the sea route on the surface of the sea so that navigator may grasp the ship s position from the sea route easily. In order to show the sea route like as road, augmented reality () technique is used in this study[5]. The image could be made from landscape image from the bridge taken by web camera and 3D object of sea route by using toolkit [6]. In order to indicate the sea route at the described position in the chart, information of latitude and longitude of waypoints and ship s position obtained from GS are needed by this system. Then, the sea route display system consists of a mobile C, web camera, and GS. There are two method for making image such as location base and vision base. In the location base technique, a lot of measuring instruments are needed to display the 3D object at the right position. On the other hand, the vision base technique required only web camera and GS for the purpose. In the vision base technique, there are two way, one is marker base and another is marker-less [7]. If marker-less method was used in this system, the horizon and hull in the image should be recognized but it couldn t be recognize at nighttime. Therefore, marker base technique is used in this system. However, if the marker cannot keep the horizontal by the ship s rolling, the 3D object in the image is not displayed at right position. Therefore, a horizontal stabilizer for the marker is also developed in this study. Effectiveness of developed system was carried out with actual ship experiment. In this paper, outline of sea route was introduced at section II. Then, the sea route display system which developed in this study was described in section III and experimental result of developed system was shown in section IV. Then, 978-1-5386-1644-4/17/$31.00 2017 IEEE 3403

consideration and conclusion were indicated in section V and VI respectively. II. SE OUTE Driving a ship on the sea has many difficulties more than driving a car on the land, e.g. grasping location, sea traffic laws, pass only permitted area, considering high fuel costs, wind and tide, and risk of enormous accident cost. Here to navigate the ship to destination port we present the necessary course line and the sea route with the factor which makes sea route the sea route.. Course line Locating the position of the ship is significant or else the ship wonders off the itinerary may lead the vessel to enter dangerous area like the shallows with the result of running aground. n important element to sail through visual information less vast water field efficiently as possible are azimuth direction, which works as the substitute of the road to be the only indicator showing course. The navigation is taken on following process. 1. Setting course line connecting port to destination port. 2. Drawing lines and degrees based on bearing method on every veering point. 3. Correctly locating the position and maneuver obediently to the course line. Course line are as shown on Fig.1 the straight line planned before the voyage, drawn on the chart which shows the route ship takes, and must be followed unless unexpected occasions. Navigator judges whether the ships is on course line from the scenery around the ship when the ship does not equip ECDIS. Then, the navigator uses two major methods such as bearing method, and transit method. The process of bearing method is explained in Fig.1 in three steps. First the navigator targets three close object visible from the ship written in the chart as well, e.g. lighthouse, mountain peak, building with height, etc. Then using the compass set on the ship, navigator measures the direction of the each object. Then on the chart navigator draws the lines from each of the objects according to the degrees of the object he/she has just measured. Finally the position where the three lines crossed on the chart is the location of the ship. Transit is the second important navigation method. This navigation method is used when two objects are seen on the line stretched straight forward extended from the bow. Navigator prefers to use this method because of usability of transit line. The method of transit are shown in Fig.2. If the position of the ship is right of the course line, the navigator is seen the crest at the right of the lighthouse. If the position of the ship is left of the course line, the navigator is seen the crest at the left of the lighthouse. Finally if the position of the ship is on the course line, the navigator is seen on the crest at the back of the lighthouse.. Sea route Sea route exists in sea areas near the bay along the coast on purpose of guaranteeing ships the safety of the area. s shown Crest Ship Ship C Shallows Ship Course line View from Ship Fig1. Example of bearing method Course line Shallows! View from Ship C Crest Lighthouse Fig.2 Example of transit method on chart View from Ship in Fig.1 surrounded by thick pale colored lines. The ship entering the bay must pass this sea route obeying major sea rules and local sea rules. uoys and beacons are set by the local to indicate the sea route and the obstacles such as shallow area to avoid the ship to go aground. uoy and beacons are used by navigators to know whether the ship being correctly located inside the sea route. The navigator operates navigation by seeing those buoys during the daytime. The navigation after sunset are operated using beacons light. The visible objects 3404

above the sea surface in the night time are mostly only ship`s light and the beacon`s light. The navigator has to recognize the beacon`s light from the jungle of lights and do so by checking the light`s blinking frequency. Each beacon has own official blinking cycle and are written formally on the chart. From these information navigator distinguish the right beacon indicating the right sea route. In many cases the beacons expressing the same sea route has same blink cycle, but on some cases the blink cycles of beacons on the same sea route are not synchronized. The beacons of the other sea route may appear to be synchronized with one of the beacons of the original sea route, frequently leads the first time arrival navigators to misidentifying beacons. C. equired sea traffic sign More explicit traffic signs are desired in these regions to prevent an accident. Sea route display system has drawn distinct line between the sea route indicating buoys, displaying the transit line until the veering point, and at the veering point set an beacon like monument on the sea to make simple the veering at the right location. Sea route display system developed in this study to creates three kinds of object. The first is to draw sea route boundary line between the buoys,. Second is drawing clear course line on the sea. The last aim is to set up an monument at the aimed veering point, and a object enabling easy to use transit method to be implemented without having two real world visible object lined up on the course line. III. SE OUTE DISLY SYSTEM. Outline of the system The outline of the sea route display system using is described in Fig.3. System are consisted from position calculation system, visual system to display navigation assisting image, and the stabilizer to deter the image from being deviated.. osition calculation system The purpose of the calculation system is to provide the position data of 3D model object for the visual system. Marker base recognizes the world axis by having the view from web-camera instead of using longitude and latitude. So the latitude, longitude data must be turned into axis data in order to have the navigation assisting objects to be displayed in effective place. Fig.4 indicates translation to axis. In the figure, oint and shows waypoint of sea rout and red line indicates sea route. lso, point indicates marker s position and it is origin of axis. In this system, data of latitude and longitude of waypoints is translating to axis based data as follows, d la t ( la t la t ) 6 0 1 8 5 2.2 d lo n ( lo n lo n ) c o s ( la t ) 6 0 1 8 5 2.2 x c o s s in d la t y s in c o s d lo n (1) (2) Lat dlat Fig.3 Outline of the sea route display system where x and y represents lateral and longitudinal distance of object drawn by axis, la t and lo n represents the latitude and longitude of the place objects being drawn, la t and lo n represents the marker`s latitude and longitude, denotes ship s heading angle from the lateral line. In order to inform the ship s position to the navigator, deviation from the sea route and distance to next waypoint are calculated in this system. Fig.5 shows coordinate system for calculating the deviation and distance to veering point. Definition in the figure is same as Fig.4. These distance were obtained by solving following equations, w x ( la t la t ) 6 0 1 8 5 2.2 w y ( lo n lo n ) c o s ( la t ) 6 0 1 8 5 2.2 d x ( la t la t ) 6 0 1 8 5 2.2 x X dlon Fig.4 Translation to axis d y ( lo n lo n ) c o s ( la t ) 6 0 1 8 5 2.2 (3) (4) 1 ta n ( / ) (5) w y w x y Lon Y 3405

Lat D Web-camera Marker dx wx l d dy wy Lon Fig.5 Coordinate system for position calculating system On-board C Gyro sensor Motor d c o s s in d la t l s in c o s d lo n (6) 2 2 D w x w y l (7) where d is deviation from the sea route, l is advance distance from previous waypoint, D is distance to veering point, is course of sea route, lat and lo n represents the latitude and longitude of waypoint. C. visual system The purpose of the visual system is to create an objects that could help the navigator navigate in more visual comfort and safety. The visual system are consisted of web-camera, marker, mobile C having program installed, and a monitor which provides image to the mariner. The program are created using the Toolkit application. On the visual system we have created three groups of objects to assist the navigator. The first object is the sea route indicating line. Drawing lines between the buoys to equip the navigator with right information of sea route and prevent getting out of the course. The second momentous object implemented in the system is the course line drawn on the sea. y this the navigator could visually see where the ship must head. Finally the third object is the pole like object which stands on the veering point and the object enabling transit method. y using these objects the navigator knows exactly where to veer without working with the chart drawing bearing lines and be equipped with transit method. lso the object changes color according to the distance from the ship to give navigator more sure sense of distance by clarifying the deviation from position calculation system. The color of the objects depending on the distance are shown in TLE Ⅰ and Ⅱ. D. Stabilizer The origin of the image created by the system relies on the position of the camera. So if the camera moves the objects moves unexpectedly. To operate marker-based Fig6. Stabilizer set on the ship technology on the continuously pitching and rolling ship, stabilizer must be implemented to hold the camera and the marker horizontal. Fig.6 shows snapshot of the stabilizer. The stabilizer is consisted of an on-board C functioning as control system, a gyro-sensor and an accelerometer sending the TLE Ⅰ. COLO CHT OF COUSE LINE Deviation 0~50m 50~100m 100~150m 150~200m 200m~ TLE Ⅱ COLO CHT OF WEEING OINT OJECT Distance position information to the on-board C, two motors being controlled by the on-board C, and the plate to uphold the whole system. On obtaining the values from the gyro-sensor and the accelerometer the system activates the motors to negate the movement. lacing the web-camera and maker on top of the system, stabilizer works to deter the pitching and rolling movement on the camera and keeps the object displayed in the appropriate place. IV. EXEIMENT Veering point object ~2000m light blue 2000~1600m 1600~1200m 1200~800m 800~400m 400m~0m Course line color pink yellow white black blue blue orange purple yellow black In order to examine the effectiveness of the sea route display system which we developed we had actual ship experiment. 3406

. Setting of experiment The sea route experiment took place was Uraga Channel and Nakanose Channel located in between oso peninsula and Miura peninsula. It is known as the one of the world`s most overcrowded sea area for being the only connecting seaway to Tokyo bay, Yokohama bay, and Chiba bay from the pacific ocean[20]. Maneuvering in these channel are said to be difficult due to the traffic, the shape of the area being twisted route and the red line drawn between the buoys shows the right TLE Ⅲ. INCIL DEMENSION OF SHIOJI-MU arameter Value Length 49.93[m] readth 10.00[m] Draft 2.80[m] Displaced weight 783.1[tons] as the shape of an S, and keeping track of the navigation traffic sign. We have operated two actual ship experiments the one is to project a course line, veering point object, and the transit object. The other is projecting a sea route line on the sea. Target ship was training ship Shioji-Maru that belonging to faculty of marine technology in Tokyo University of Marine Science and Technology. rincipal dimension of the training ship was described in Table III. This training ship was used for training exercise for students and researching maritime field for faculties. Therefore, the ship was installed with local area network (LN) system which provide various ship s information such as ship s position, ship s motion, and ship s engine motion at real-time. Thereby, we could obtain the ship s position and ship s heading when the mobile C was connected to the LN system.. esults The lines and the objects were drawn by the system in Fig.3. Fig.8 shows image during the experiment. On the upper picture yellow course line successfully, indicating the distance from the course line was 50~100m, tall blue colored lighthouse like veering point object indicating the remainder distance was 2000~1600m, and the transit object. On the second picture color of course line was white, it means the distance from the course line was 100~150m, and the veering point object distance to the veering point was 1600~1200m as indicated. Finally as seen in bottom picture the distance to the course line was 100~150m, distance to the veering point was 1200~800m as indicated by the. Short white object adjacent to the tall colored light-house objects represents the ship`s positional affiliating side`s by acting as a catalyst to enable transit method displayed 500m front reinforcing the sense of distance to an significant point of navigation. Visually recognizing the transit object on the right means the ship is on the left of the course line, the object recognized on the left indicates the ship is on the right. Fig.9 shows the result of second experiment. The system properly drew sea route line between the buoys. Green line drawn between the buoys shows the left boundary of the sea Fig8. images during the course line experiment Fig9. image during the sea route experiment boundary of the sea route. V. CONSIDETION The experiment result ascertained the system could effectively assist navigators on grasping the operating ship position. Unexpectedly the sea area was clouded by mist on the day we have processed the experiment. This fortunately showed that the developed system works properly in vision limited cases as well. Studies of virtual reality is expanding rapidly like seen in many fields. Last year the maritime safety agency started to run virtual IS traffic signs which displays traffic sign that does not exist in the actual sea area on the instruments map. This technology are called virtual reality and 3407

this similar to the technology may solved the problem in place having too much depth that the buoys cannot be placed. Sea route display system developed in this study could aid the navigators further grasp of the situation displaying the course line and sea route boundary on the sea using technology. Navigator`s may gain more assist by further study on technology. VI. CONCLUSION This paper presents a result of technology being used to assist navigators grasping the situation on the sea without using expensive instruments like ECDIS. The system shown in this paper are consisted from calculation system replacing the position information to based position information, visual system to create objects to display, and the stabilizer to hold the whole system in the right place. For the future work, function to help maneuvering the ship could be added, displaying shallow area and displaying predicted ship s position under the weather condition such as wind and current. EFEENCES [1] International Electro technical Commission, Maritime navigation and radiocommunication equipment and systems - Electronic chart display and information system (ECDIS) - Operational and performance requirements, methods of testing and required test results, IEC61174, 2015. [2] International Electro technical Commission, Maritime navigation and radiocommunication equipment and systems Global navigation satellite system- art1 Global positioning system(gs) -, IEC61108-1, 2003. [3] International Maritime Organization, egulation 19 Carriage requirements for shipborne navigational systems ans equipment, IMO esolution MSC.282(86), 2009. [4] N.Nishii, N.Yamatani, D. Hara, H. Mikami, Factors that lead to early turnover of young Seafarers, The Journal of Japan Institute of Navigation, Vol.130, pp.114-121, 2014. [5] Kato, H., illinghurst, M. "Marker tracking and hmd calibration for a video-based augmented reality conferencing system.", In roceedings of the 2nd IEEE and CM International Workshop on ugmented eality (IW 99), October 1999. [6] ToolKit, http://artoolkit.sourceforge.net/, (2017/4/7). [7] Georg Klein, and David Murray, arallel Tracking and Mapping for smal Worlspaces, roceedings ISM 2007, pp.225-235, 2007. 3408