THe rip currents are very fast moving narrow channels,

1 Rip Current Detection using Optical Flow Shweta Philip sphilip@ucsc.edu Abstract Rip currents are narrow currents of fast moving water that are strongest near the beach. These type of currents are dangerous for swimmers. Many people get stuck in rip currents and are responsible for majority of the lifeguard rescues near beaches [1] Not only novice but also experienced swimmers get stuck in these currents. These also cause deaths, since people try to swim towards the beach for safety but the velocity of these currents are so strong, that they die of exhaustion. Lifeguards and experienced people can spot the rip currents and hence steer clear of them[4]. They are a threat to people who cannot spot them, which is not an easy task. This project talks about using optical flow to detect rip currents. We use the Lucas-Kanade optical flow algorithm to get the velocity of the motion of the water. We alter the color mapping algorithm for optical flow to identify the rip currents. Index Terms Computer vision, optical flow, Lucas-Kanade, color-mapping and rip-currents. 1 INTRODUCTION THe rip currents are very fast moving narrow channels, which if caught in can cause a person to drown. The currents pull a person towards the water body. Instinctively, people try to swim against the current, but it results in exhaustion and eventually drowning. According to University of Delaware Sea Grant College, there are four ways to identify rip currents. They are as follows: 1) A channel of churning, choppy water; 2) A line of sea foam, seaweed, or debris moving steadily seaward; 3) Different colored water beyond the surf zone; and 4) A break in the incoming wave pattern as waves roll into shore.[2] The rip currents are caused along the coastline where break waters occur. Certain circulation cells are formed when the waves break strong at some regions and weakly in others. These generally occur at beaches with sand bars and channel system nearby. This water flows back towards off-shore following a narrow path, forming a rip current.[12] Fig 1. Formation of Rip Currents [3] This project uses Lucas-Kanade algorithm for optical flow to find the motion of the water near the shore leading to the detection of rip currents. 2 MOTIVATION According to a study by United States LifeSaving Association, the annual number of deaths caused by rip current exceeds 100. There are about 80% of the rescues due to people almost drowning in rip currents.[12][4] These deaths usually not highlighted in the newspapers. Rip currents usually flow at the speed of 0.5 m/s. Some of them can also speed up to 2.5 m/s.[1] There are around 100 people that die due to a combination of fear, panic and exhaustion. This is faster than most people can swim[4]. From expert swimmers to first-timers, rip currents have killed more Americans than hurricanes, in an average year. Most lifeguards, experienced people are able to detect the position of the rip currents, they look for various signs as mentioned in the Introduction.[4] An mobile application that can detect rip currents in oceans, can help inexperienced people identify these currents. This will help in avoiding deaths of swimmers as well reduce the number of rescues performed by the lifeguards. 3 RELATED WORK A patented device has been designed to detect the existence of rip tide in public swimming beaches. The device captures images and applies pre-filtering to highlight the characteristics of rip tides. The data thus obtained is used to classify as Rip-Tide or Normal waves. The classification is done by simulating how a human classifies them. Other methods such as building a neural network can also be used along with the device.[5] This device is not as readily available as a mobile phone application even though this as well uses image processing to detect rip currents. Another method developed by Tuba Ozkan-Haller, an associate professor at Oregon State University, where she analyzes the topography of the ocean floor, to gather information about the water flow above it. This gives information on how the water will escape in the form of rip currents.[4] All the above mentioned methods use different approaches, which require special devices to identify. We use a very popular algorithm in Computer Vision known as Optical Flow algorithm. Optical Flow algorithm is widely used with motion estimation and video compression. It helps to determine the apparent motion of an object in a visual scene caused by the relative motion between the

2 camera and the scene.[13] This algorithm uses a sequence of ordered images to determine the instantaneous velocity of the images. One of the most popular optical flow algorithms was suggested by Bruce D. Lucas and Takeo Kanade, known as the Lucas-Kanade Algorithm. They presented an image registration algorithm that uses spatial intensity gradient information to direct the search for position that yields the best match.[6] Applying Lucas-Kanade on objects that are rigid and have constant texture is fairly easy and gives great results. There have been research done to apply Optical flow algorithm, to estimate the motion of moving water, clouds and so on. Particularly, of objects that have non-rigid surfaces and dynamic texture. Vidal et al suggested that the dynamic texture can be modeled as a time varying, Linear Dynamical System plus a 2-D translational motion model. This model is used along with the optical flow algorithm to capture the motion of the non rigid, dynamic texture surfaces.[7] Another method was suggested by Sakaino, where the image intensity was represented in the form of a wave model. It is assumed, that image intensities will change according to the function specified by the wave model. This wave model is considered as a function of different frequencies. Each frequency combined gives a model that is assumed to act similar to waves in the ocean.[8] These algorithms suggested are too complex to be operated on a mobile device. Here, we implement original Lucas-Kanade Optical flow algorithm and use a colormap to represent the instantaneous velocities. The proposed methodology is explained in the following section. 4 PROPOSED METHODOLOGY The Optical flow as mentioned is used to capture the apparent motion of the object. There are many different algorithms to perform Optical flow. They all basically consider the same two assumptions; they are as follows[9]: 1) Pixel intensity of an object between consecutive frames remain the same 2) Neighboring pixels follow the same motion. One of such Optical flow algorithms, is Lucas-Kanade. The algorithm is explained below: Consider a pixel I(x, y, t) in the first frame, which moves a distance of dx, dy in time dt in the second frame. A time dimension is added to represent how much the pixel moves in a given time. Taking the first assumption into account we can write the following equation[9]: I(x, y, t) = I(x + dx, y + dy, t + dt) (1) Assuming that dx, dy is small, the image constraint at I(x, y, t) with Taylor series leads to following equation: δi δx δx δt + δi δy δy δt + δi δt = 0 (2) δi δx u + δi δy v + δi δt = 0 (3) I x u + I y v + I t = 0 (4) Based on the second assumption, that the neighboring pixels have a similar motion, the Lucas-Kanade algorithm uses a 3x3 matrix and obtains the values of (I x, I y, I t ). There are 9 points in consideration, hence we get 9 equations and we have our two unknowns (u, v) which is our velocity vector. This results in over-determining of the unknown variables. A least square fit method is used to find the velocity vector. We use the following equation to determine the velocity vector[9]: [ [ u 2 Σi I = xi v] Σ i I xi I yi Σ i I xi I yi Σ i I yi 2 ] 1 [ ] Σi I xi I ti Σ i I yi I ti This algorithm is really good at finding small motions, which is great for our problem statement. The water surface changes texture hence pixel intensity instantly, hence the pixel intensity will vanish or change just as fast. This algorithm helps us to obtain the details of the small motions of the pixels intensity before it vanishes. We use an implementation provided by University of Central Florida, Center for Research in Computer Vision for Lucas-Kanade Algorithm[10]. The platform used is MATLAB R2014b T M. There are again many ways to represent the motion of the objects. After experimenting with two types of representation, we found that the best way to represent these small motions is through color mapping the velocity vector. We use a colorwheel shown in Figure 2. Here, we use an HSV model for mapping. Every direction is mapped to a different Hue as shown in the Figure2. The magnitude of the velocity is mapped with the saturation. The more the magnitude of the velocity the more the saturation. We use the code provided by Deqing Sun, Department of Computer Science, Brown University. Fig 2. The color wheel [11] We first test this algorithm on objects that are rigid and have a constant shape and size. This algorithm is good enough in finding the motion of the object and gives a different color to each direction. In Figure 3, we see an object moving to its left and in Figure 4 we see the same object moving to its right. Without the color, both results will look the same. The color map helps us to distinguish between the direction in which the object is moving. If we go back to the color wheel, we see that blue and red are on the opposite sides of the color wheel, hence we can say about the below two figures, is that both of them are moving in opposite directions from each other. (5)

3 Fig 3. Color map of object moving left Fig 4. Color map of object moving right Now, we apply the algorithm, to the video taken at a beach in California. The two input frames is given in Figure 5 and the output flow map is given in Figure 6. This mapping gives movement in all the directions. Fig 5. Input consecutive frames Fig 6. Color map of movement of waves We now need to find a way to segment out motion of water moving backwards. We used thresholding on the direction of the motion. We got the following results: [Note the following result in Figure7 is not the same result as the color map shown in Figure 6. This color map is obtained between a different set of consecutive frames.] Fig 7. Color map after thresholding The algorithm does a certain amount of computation, to create the output shown in Figure 5, but after thresholding, the data motion that is not used is still computed. We can reduce this computation. It is explained in the following paragraph. The algorithm for computing the color map stores the color values of the pixels in three channels which are the red, green and the blue channel. We compared the different channels of the flow map, and also with the complete flow map. The comparison is shown in Figure 8 [Last Page]. The image in the first row, first column represents the blue channel, the image in the second column represents the green channel and lastly we have the red channel. If we compare all of these channels, we see that the blue channel covers all of the backward motion along with some noise. Also comparing it to Figure 7, we know the we only need to compute the values for the blue channel. Hence, reducing the computation. Another question that arises is, if this video is taken from another direction, the motion that is moving backwards into the ocean will be represented by a different color channel. Hence, for simplification, we restrict the direction from which the video of the waves can be taken. We need the video to be taken in which such that the observer is on the left side of the water body and the water body is at the center of the camera. Hence, we computed optical flow and only blue channel of the flow map for every consecutive frame pair in the input video. Now we have flow maps that has segmented backward motion. Now, we need to overlay this new flow map on the original frames. This will give us areas in the original video, that may be our potential rip currents. We take the flow map images and create a logical mask that has 1 value for which the pixel value is less than 255 and greater than 0 and the mask value will be 0 otherwise. We also apply a noise mask which ignores the first 85 rows of pixels and the last 90 rows of pixels. This is done only for this particular video. We will replace this by segmentation, which will segment the frames such that the optical flow is performed only on the water body. We use this noise mask because, even though there is no movement in the sky and in the sand in the video, the algorithm still captures some motion. This is because, we do not have a steady capture of the video. The output obtained my the mask is overlaid as red (255, 0, 0) on the original frame. Now, the output represents a single direction. The output does not yet represent the magnitude of the motion. Figure 9 shows the output of the

4 algorithm. 5 CONCLUSION Fig 9. Color map after thresholding The algorithm though robust enough to find the small motions in the ocean, it also is highly sensitive to noise. The output of this algorithm, gives a video which marks all the water that is moving backwards towards the ocean as red area. We were successful in segmenting out the motion. The original code obtained from University of Central Florida and Brown University were slightly modified to improve the time taken to calculate the optical flow. We separated the construction of the color wheel from the flow to color mapping function, so that the color wheel does not get recreated each time we perform optical flow on two consecutive images. REFERENCES [1] https : //en.wikipedia.org/wiki/rip current [2] http : //www.ceoe.udel.edu/ripcurrents/identify/index.html [3] http : //www.ripcurrents.noaa.gov/science.shtml [4] http : //www.psmag.com/nature and technology/how to detect dangerous rip currents riptide 52211 [5] http://www.google.com/patents/us20050271266 [6] Bruce D. Lucas, Takeo Kanade. In Proceedings of Image Understanding Workshop, pp.121-130, 1981 [7] Rene Vidal and Avinash Ravichandran, Optical Flow Estimation & Segmentation of Multiple Moving Dynamic Textures, Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference. [8] H. Sakaino, Fluid Motion Estimation Method based on Physical Properties of Waves, Computer Vision and Pattern Recognition, 2008. CVPR 2008 [9] http : //docs.opencv.org/master/d7/d8b/tutorial py l ucas k anade.html [10] http : //crcv.ucf.edu/reu/2014/p4 opticalf low.pdf [11] http : //members.shaw.ca/quadibloc/other/colint.htm [12] http : //www.usla.org/?page = RIP CURRENT S [13] https : //en.wikipedia.org/wiki/optical f low 6 FUTURE WORK As mentioned in the conclusion, the algorithm is very sensitive to the noise. The solution for this is to stabilize the video before performing the segmentation. Another application of Optical Flow algorithm is, Video Stabilization. We can further use the Lucas Kanade algorithm to stabilize the individual frames of the video before generating the color map and thus the final output. Another possible improvement to the algorithm, will be to remove the color surface area that represents the backward motion and replace it with arrows. Arrows are easier to understand than the surface area marked by the algorithm. This algorithm still works on a Windows operated Personal Computer, we need to optimize the code such that we get the results in an Android OS equipped smart phone in real time. The algorithm is still not efficient enough to identify a rip current. It marks every backward motion as a potential rip current. We can probably narrow down the rip current by thresholding as per the magnitude of the velocity. The algorithm also needs to be independent of the direction the video is taken, as it is a restriction on the application. We need to generalize the algorithm to be able to work from any direction and still not increase the computation.

Fig 8. Different channels of the color map 5