PEDEST: Parallel Environment-Driven Evolution of Soft Robots.

MSc Artificial Intelligence Track: Machine Learning Master Thesis PEDEST: Parallel Environment-Driven Evolution of Soft Robots. by Luka Stout 10616713 August 7, 2016 42 ECTS September 2015 August 2016 Supervisor: Dr E W Haasdijk Assessor: Prof Dr AE Eiben

ii

Abstract This thesis is exploratory research of an environment-driven evolutionary system. This evolutionary system is a system where the fitness of an individual is not calculated by a fitness function, but evolution is driven by environmental properties. Not much is known about environmentdriven evolutionary algorithms and their effect on the morphological complexity of individuals wandering around in them. The performance and complexity of soft-bodied robots in large populations in an environment-driven evolutionary system is investigated in different environments. Multiple measures of morphological complexity are proposed to study it over the course of the experiments. These large populations are simulated in parallel using the VoxCAD simulator. To create additional environments and possibly multiple equilibria, two mechanisms have been introduced to change the evolutionary landscape of the pedestrians. We have found that under all conditions the distance that the robots move increases over time. We also have found that, for the small timeframe that the robots have been simulated, the morphological complexity does not necessarily grow over time. This contradicts previous work. iii

iv

Acknowledgements First I want to thank Evert Haasdijk for his discussions, patience, and getting me up to speed on statistics. Florian Golemo and Marjeta Markovic for their help. As this thesis is a continuation of their work, I thank them for letting me use their work and helping me in using it to get to this result. The Skype-discussions were immensely helpful. I want to thank Nick Cheney for providing us with the Voxelyze simulator. It is a part of the VoxCAD GUI for this simulator made by Jon Hiller who I also want to thank. I also want to thank Onno Huygen for helping me get the complexity measures right. Eric de Kruijf for reading through all of this and correcting numerous mistakes. I thank SURFsara (www.surfsara.nl) for the support in using the Lisa Compute Cluster. Lastly, I want to thank my family and girlfriend for putting up with me these last few months. Their support has helped me get through it all.

Contents 1 Introduction 1 2 Background 5 2.1 Environment-Driven Evolution............................ 6 2.2 NEAT.......................................... 8 2.3 CPPN.......................................... 9 2.4 Soft-bodied Robots................................... 10 3 Environmental Setup 13 3.1 Pedestrians....................................... 13 3.2 Meeting & Mating................................... 14 3.3 Cost-mechanism.................................... 15 3.4 Disease......................................... 16 4 Complexity Measures 17 4.1 Material-complexity.................................. 17 4.2 Mean-material-complexity............................... 18 4.3 Shape-complexity.................................... 18 5 Implementation details 23 5.1 Simulator........................................ 24 5.2 Parallelization...................................... 24 5.3 Workers......................................... 25 6 Experimental Setup 27 6.1 Environment...................................... 27 6.2 Features......................................... 28 7 Analysis 31 7.1 Distance......................................... 32 7.2 Number of offspring.................................. 32 7.3 Shape.......................................... 35 7.4 Materials........................................ 36 7.5 Best performing pedestrians.............................. 46 8 Discussion and Future Work 49 8.1 Discussion and Future work.............................. 50

2 CONTENTS

Chapter 1 Introduction Since the beginning of artificial intelligence, the end goal has been to create human intelligence in computers such as deductive reasoning, learning and communicating in natural language. Most current roboticists, however, want to create much simpler behavior in robots such as walking or object manipulation. The algorithms creating this simple behavior are hoped to scale to create a bigger and more complex behavior until the behavior shown by a given robot is seen by an observer as intelligent behavior. This is a variant of the Turing Test: if a robot looks as if it is acting intelligently; then it is intelligent [9]. In search of such intelligent behavior roboticist have used the principles of biological evolution to create an artificial analog called Evolutionary Algorithms [25, 32, 36, 37, 53]. In natural evolution evaluation of the organisms is implicit. This is in contrast to artificial evolution where a pre-defined fitness function is used to evaluate organisms. According to Darwin s theory on the Survival of the Fittest fitter organisms produce more offspring. Over multiple generations this will, hopefully, drive the population to be better performing. The artificial life literature provides several examples where no such fitness function is used [26, 52, 64, 66]. Here, the reproductive success of a robot only depends on the survival of the robot and the spreading of their genes instead of depending on a performance measure. The properties of the environment now implicitly determine the rate at which genes spread. More recently there has been the field of environment-driven evolution [11, 65]. In this field robots with evolvable bodies and minds wander around in an arena without the presence of a central system controlling evolution or a user-defined fitness function. In 1994 Sims showed the potential of using evolutionary algorithms when evolving morphologies. They are capable of producing natural, complex morphologies and behaviors [57]. Despite the time that has passed since then, with all its improvements in computational speed and evolutionary algorithms, the morphologies produced by evolutionary algorithms are not obviously more natural, complex or interesting. Figure 1.1 shows a comparison of the complexity of organisms over the last two decades. 1

2 CHAPTER 1. INTRODUCTION (a) The organisms used by Sims in 1994 [57] (b) The organisms used by Lehman and Stanley in 2011 [39] Figure 1.1 Visual representations of the individuals used in two different experiments, 17 years apart. Since the experiments of Sims in 1994 no obvious increase has been made in the complexity of solutions. Images taken from the respective papers. One hypothesis as to why there is no clear increase in complexity is the widespread use of a limited set of rigid components as the building blocks for organisms, for example [5, 33, 38]. In contrast, nature utilizes a vast array of materials, ranging from hard bones to soft muscle cells, to create the building blocks for organisms. The use of heterogeneous materials enables nature to produce more complex and high-performing bodies [63]. An open question is whether access to more material types that are in principle less complex than the rigid components, enables artificial evolution to create more complex forms. This hypothesis is tested by using soft-bodied robots as the organisms in our evolutionary algorithm. Using the VoxCAD simulator [30], we use the multi-material soft-bodied robots that also have been used in [13, 14, 15]. Quantifying complexity is notoriously difficult [15] and how it changes over time is an open question. The general belief in this area is that in more difficult or more complex environments individuals that are more complex perform better [5]. In this thesis, we have created three measures the complexity of these soft-bodied robots. To understand the question about whether and why there is an increase complexity better we introduce two types of complexity to the environment. The first is adding a negative effect to the presence of particular materials. The second is a disease. This disease affects the active voxels of the robots by turning them into passive voxels. The disease only affects the phenotype of the robots while leaving the genotype alone. A difference between existing work on environment-driven evolution [11] is that we do not evolve the control program of individuals but their morphology. The evolution of morphologies is an interesting addition to environment-driven evolution as it more closely resembles how nature evolves. Due to evolution of morphologies, we can investigate the evolution of the morphologies and its complexity. To examine the morphological complexity of the pedestrian novel methods of calculating them are proposed in this thesis.

3 Figure 1.2 A pedestrian (the soft-robot used in this work) in motion. The figure is taken from [15]. The goal of this thesis is to introduce PEDEST a Parallel Environment-Driven Evolution of Soft Robots and show initial experiments. This system combines environment-driven evolution of morphologies with soft-bodied robots to get a close representation of natural evolution. It is an exploration of the dynamics of an environment-driven evolutionary system and its impact on the morphological complexity of its robots called pedestrians using initial experiments. We make several contributions. The first contribution is our proposal of multiple different ways to compute the morphological complexity of the pedestrians. The second contribution is that we show that the simulation can be done in a parallel fashion. The speed up that is gained from parallelizing the simulations is necessary to accommodate for the large populations that help the performance of an environment-driven evolutionary system [11]. Background literature is reviewed in Chapter 2 providing an understanding of the current developments in Environment-Driven Evolutionary Algorithms. In Chapter 3 the environment is defined and in Chapter 4 the complexity measures are shown. Chapter 5 discusses implementation details and the use of the cluster. Chapter 6 outlines the experiments and their parameters. An analysis of the evolutionary system is done in Chapter 7. A final discussion and future work in Chapter 8 concludes this research.

4 CHAPTER 1. INTRODUCTION

Chapter 2 Background The field of evolutionary algorithms started in the 1960s. Independent from each other multiple variations of evolutionary algorithms were developed. Genetic Algorithms by Holland [32], Genetic Programming by Koza [36, 37], Evolutionary Programming by Fogel [25] and Evolutionary Strategies by Rechenberg [53]. In the 1990s they merged into what is now known as evolutionary algorithms. They are all part of the class of search techniques inspired by natural evolution, and they have been highly successful in this search across a wide range of tasks [4, 18, 21]. In natural selection fit individuals that are more likely to survive and find a suitable mate. Because they are more likely to produce offspring, their genes are more likely to spread through the population. Over multiple generations, this will adapt the population to their living conditions and their needs. Genetic algorithms apply this principle to machine learning problems with the individuals being possible solutions. The better solutions to a problem of a population will create more offspring than the worse-performing ones. This property of evolution drives the population towards more desirable solutions. The algorithms start with the first generation of candidate solutions. These first solutions are usually randomly generated. A fitness function then evaluates each candidate and selects the parents for the next generation. This fitness function is used as a representation of the tasks that the candidate solutions have to solve. A solution with a higher fitness (or lower if the task is to minimize a fitness function) has a higher probability to be selected as a parent. These parents recombine their genotypes (their DNA) using recombination and variation to produce a new set of candidate solutions. This process continues until some stopping condition is reached. These stopping conditions might be a certain number of generations that has been reached or a pre-defined fitness level that has been found. The first two columns of Table 2.1 show the differences between natural evolution and conventional artificial evolution. The first difference has already been shown. In natural evolution, the fitness of an individual is implicitly defined by their reproductive success, whereas in artifi- 5

6 CHAPTER 2. BACKGROUND cial evolution an individual s fitness defines their reproductive success. Another difference is the selection mechanisms. In nature, this process is very complex while in artificial evolution it is a simple randomized selection from the population with every individual having a probability based on its fitness. The mapping from genotype to phenotype is another difference. This mapping in nature, of course, is based on DNA and environmental influences. In artificial evolution, this is a simple mathematical transformation or parameterized procedure. Another difference shows itself in the execution of the evolutionary strategies. In nature, there are desynchronized birth and death events, whereas in artificial evolution these events are synchronized for every generation. No mating is possible between individuals of different generations. The last difference shown in Table 2.1 is the influence of the population-structure on the evolutionary process. In nature, these populations live together and the individuals only mate within these populations. These populations go extinct when the fitness of the individuals are too low. In traditional evolutionary algorithms, this is impossible because for every generation a new population of the same size is generated. In the early 1990s the first generation of experiments of what is now known as evolutionary robotics were carried out in computer simulations [7, 23, 34, 48] and soon after in real robots [24, 29, 40]. The term evolutionary robotics was coined shortly after these experiments [16]. It is the application of artificial evolution on a robot physical or simulated where the fitness is given by the performance of the robot. Initially, research was primarily interested in individual robots because the algorithms from other evolutionary algorithms transferred over nicely, but since then an effort has been made into working out the specific problems and advantages that evolutionary robotics bring. One of the natural extensions to evolutionary robotics is that a population can exist in a world where they can move around and interact with each other. These approaches depend on the fitness of individuals. Determining this fitness is usually done with a fitness function (e.g. the distance a robot pushes the block or the distance it moved from its starting location). Assumptions that are made while constructing a fitness function The way this fitness can place restrictions on the final form of the robot or the kind of behavior that is generated. Designing a fitness function in a way that it is unbiased towards a particular solution is a difficult problem. 2.1 Environment-Driven Evolution Paraphrasing [65], a system that implements natural selection without the use of a fitness function enables research into the evolution of mind and body in interesting ways [3, 31, 49]. This offers opportunities for novel research in artificial life and embodied artificial intelligence with implications for evolutionary robotics and evolutionary biology [41]. These open-ended approaches to evolutionary algorithms have been considered in multiple settings. Mouret and Doncieux [45] have used it to promote behavioral diversity in multi-objective

2.1. ENVIRONMENT-DRIVEN EVOLUTION 7 Natural Evolution Artificial Evolution PEDEST Fitness Observed quantity: a posteriori effect of selection and reproduction ( in the eye of the observer ). Selection Complex multifactor force based on environmental conditions, other individuals of the same species and those of other species (predators). Genotype-phenotype mapping Highly complex biochemical and developmental process influenced by the environment. Execution Parallel, decentralized execution; birth and death events are desynchronized. Population Spatial embedding implies structured populations. Population size varies according to the relative number of birth and death events. Populations can and do go extinct. Predefined a priori quantity that drives selection and reproduction. Randomized operator with selection probabilities based on given fitness. Typically a simple mathematical transformation or parameterized procedure. A few systems use generative and developmental genotype-phenotype maps. Typically centralized with synchronized birth and death. Typically unstructured and panmictic (all individuals are potential partners). Population size is usually kept constant by synchronizing time and number of birth and death events. Observed quantity: a posteriori effect of selection and reproduction ( in the eye of the observer ). Complex force based on environmental conditions. A mathematical transformation using the generative Compositional Pattern- Producing Networks. Parallel, decentralized execution; birth and death events are desynchronized. Population size varies according to the relative number of birth and death events. Individuals can only mate with others that are alive at the same time. Table 2.1 Differences between Natural Evolution, general Artificial Evolution and PEDEST. The first two columns are an adaptation of the same table found in [22]

8 CHAPTER 2. BACKGROUND settings. Novelty search by Lehman and Stanley [38] use the open-endedness to tackle problems where a normal objective function leads to sub-optimal behavior. However, these approaches still define an objective function. These objective functions are the notion of novelty for Lehman and Stanley and a secondary objective for Mouret and Doncieux. Objective-free evolution has been studied since the start of the 1990s [51, 52]. Its primary focus is to look at the evolutionary system in the absence of a task. The primary evolutionary drive in such a system is an implicit fitness function or some environmental criterium that boosts or dampens the chance of an individual to spread its genome. These approaches have been steadily gaining interest from the community [8, 11, 55, 65]. It is thought that implicit fitness is a strong addition or alternative to classical approaches with an explicit fitness function [55]. In 2011 Bredeche et al. [11] proposed medea (minimal Environment Driven Evolutionary Algorithm) as an embodied evolutionary algorithm that uses an implicit fitness function. In the experiment, the agents start with a random genome. In its lifetime the agents move in the arena while broadcasting their genome and collecting the genomes of other agents. At the end of its lifetime, the robot will randomly select a collected genome as its mate. By mutating this genome, it introduces variation to the system. It will use this mutated genome to start a new lifecycle. There is no explicit fitness function, but the agents are still stimulated to move because agents that move further meet more agents. A pedestrian that meets more more agents than others increases its chances to spread its genome. The spreading of genomes of well-performing organisms will increase the average performance of the population. 2.2 NEAT Neuroevolution, the artificial evolution of neural networks using evolutionary algorithms, had shown great promise in complex reinforcement learning tasks [27, 28, 44, 50, 67]. Because Neuroevolution searches for a behavior instead of a value function it is effective in problems with continuous and high-dimensional state spaces. Traditionally neuroevolution used a fixed neural network topology. Most commonly this is a fully connected neural network with a single hidden layer. Evolution was used to search for the best connection weights for this network. It did this by letting the better-performing networks breed to create new high-performing networks. The solution space is explored by using crossover of the weight vectors and mutation of connection weights. A problem with using traditional neuroevolution is that it uses a fixed topology. The topology of a network has a significant influence on the performance and behavior of a network. Modifying the structure of the networks in combination with the connection weights has proven to be an effective strategy [12]. Nuroevolution of Augmenting Topologies (NEAT) [60] is a way that combines these two ideas. It encodes and evolves the structure and weights of neural networks. The approach uses genetic operators that introduce new genes and removes old ones in addition to crossover and mutation. The big problem that NEAT solves is the problem of competing conventions. This issue occurs

2.3. CPPN 9 when the same genes in two different networks with the same behavior have a different impact on that behavior. Using crossover on the two genes with two different functions can have a big impact on the performance child as a result. The insight that genes with the same origin tend to have a similar function is NEAT s big contribution. Whenever a new gene is created by using one of the genetic operators it is assigned a number that marks the order in which the gene has appeared in the population. These genetic markers are retained when the genes are used in the reproduction process. They are used to align genotypes of varying lengths. This is done to prevent crossover on competing conventions and to detect genetic similarity to create subpopulations. Reproduction only operates on individuals that are in the same subpopulation. This ensures less competition for reproduction between old individuals with a high fitness and newer individuals that show innovative structures with probably a lower fitness. The fitness of innovative individuals is generally lower as these innovative parts are not yet optimized. Because these individuals are genetically different, they are reproduced separately. NEAT starts with an initial population of genotypes that represent minimal neural networks. The genetic operators mutate the genotypes by inserting or removing genes and adjusting the weights of the networks. If these larger networks provide an advantage, they are kept and will compete with other networks that are genetically similar. 2.3 CPPN The human genome consists of about 30 000 genes. Something they encode is the human brain with 100 trillion connections [19, 20, 35]. The only way that this is possible is if there is a mapping that translates several dimensions into many. This is done through an indirect encoding. Because the structures in the human body often occur in patterns that repeat themselves the same gene group can provide the blueprints for multiple parts. One way to create this indirect encoding for neural networks is a compositional pattern producing network (CPPN) [58] This CPPN encodes the solution as a network of simple activation functions. In a traditional neural network, the activation functions that are commonly used are sigmoid, tanh, or the soft-max functions. Common activation functions in a CPPN include sigmoidal, Gaussian, sine, absolute value, linear, step functions, and with other possibilities as well. These functions loosely correspond to a geometric regularity. By using a Gaussian function, the output of the CPPN can become symmetric. A periodic function such as the sine function creates repetition. Combining these regular functions with irregular ones creates imperfect symmetry. These imperfect symmetric patterns are regular patterns with subtle variations. This information reuse allows the evolutionary algorithm to search a smaller search space and speeds up the search process where such regularities are common. CPPNs are functions of geometry (i.e. locations in space). Consider a CPPN that takes three inputs (x, y, and z). These labels represent points in a 3D-space. In our experiments, the output of that input represents the type of material used at that point. Using that information

10 CHAPTER 2. BACKGROUND entire structures can be made, see for example the EndlessForms website 1 [17]. CPPN will be used as the representation of the soft-bodied robots that are used in this work. As these CPPNs are essentially just extended artificial neural networks, NEAT can be used to create a new CPPN from multiple parent CPPNs. This strategy is called CPPN-NEAT [58] and it is a trivial modification of NEAT. CPPN-NEAT will be used to evolve the CPPNs in the experiments in this work. 2.4 Soft-bodied Robots Traditionally robots are made up of rigid and hard materials. Most robots used in Evolutionary Robotics are these type of robots. They have been studied at length [46]. The robots ability to interact with their environment is limited by these rigid components, especially when compared to their natural analogs in muscles and connective tissues. While soft robots that have an excellent performance can be designed, the design process is complicated by competing and hard-to-define objectives [63]. These problems are areas where evolutionary algorithms excel, but historically they have not been able to scale to larger robotic designs. Only a few attempts have been made largely to evolve these soft-bodied robots [13, 14, 15, 54] due to the computational costs involved in the simulation of the soft-voxels. In 2011 Hiller and Lipson developed a simulator for soft-voxel robots [30] called VoxCAD. It has been used in multiple studies involving evolution before [13, 14, 15, 30]. However, this work is the first time that they are examined in the context of environment-driven evolution. An open question is whether the evolutionary process will produce more complex forms if it is able to create organisms out of multiple materials. These materials can be hard or soft, analogous to bone or soft tissue, and inert or expandable, analogous to supportive tissue or muscle. These materials can be represented by the VoxCAD simulator. Figure 2.1 shows a sample of the shapes that these robots can have. 1 http://www.endlessforms.com

2.4. SOFT-BODIED ROBOTS 11 Figure 2.1 A pedestrian. The green and red voxels are the positive and negative oscillating materials (muscles) respectively. The light blue voxels are the soft passive material (fat) and the blue voxels are the hard passive voxels (bone).

12 CHAPTER 2. BACKGROUND

Chapter 3 Environmental Setup As seen in the Table 2.1 PEDEST is different from both natural and artificial evolution. PEDEST is a parallel environment-driven evolutionary system with soft-bodied robots. The individuals in PEDEST are called pedestrians. The difference between generic artificial evolution and PEDEST is that no fitness function is used to measure the performance of the individuals. Without this measure, the traditional evolutionary algorithms can no longer be used. Another difference is the parallel nature of the execution. A desynchronized selection and reproduction step is present at the end of a pedestrian s lifetime instead of a synchronized step where the new agents are being generated. The difference with natural selection is the organisms that are used. The pedestrians are less complex than natural organisms. Due to this, there are also differences between the genotype-phenotype mappings. 3.1 Pedestrians Most robots in Evolutionary Robotics are made out of rigid materials [46]. The robots ability to interact with their environment is limited by these rigid components, especially when compared to their natural analogs in muscles and connective tissues. While soft robots that have an excellent performance can be designed, the design process is complicated by competing and hard-to-define objectives [63]. These problems are areas where evolutionary algorithms excel, but historically they have not been able to scale to larger robotic designs. Only a few attempts have been made largely to evolve these soft-bodied robots [13, 14, 15, 54] due to the computational costs involved in the simulation of the soft-voxels. The pedestrians are based on the work by Cheney, Hiller, and Lipson [15, 30]. They are softbodied robots made of four possible tissue types; hard passive tissue, soft passive tissue, and two kinds of active tissues (positive and negative oscillating). Hard passive tissue and soft passive tissue are described as bone or fat respectively. The two kinds of active tissues are analogous to muscles. The difference between them is that they are in opposite periods of movement of each 13

14 CHAPTER 3. ENVIRONMENTAL SETUP other. Using these cells a robot is formed from their genome with a maximum size of 10 by 10 by 10 cells. Only the most central body is used when producing the robot morphology if there are multiple disconnected parts. The transition from a CPPN to a pedestrian is done by iteratively querying the CPPN for every voxel with its x-coordinate, y-coordinate, z-coordinate and the voxel s distance from the center. The first output of the network is used to determine whether any material is present in that voxel, while the other 4 output nodes represent the materials. The maximum value is used to specify the material present at that location. If there are multiple disconnected bodies only the most central patch is used as the pedestrian. One of the pedestrians in motion is shown in Figure 3.1. The evolution of morphologies is an interesting addition to environment-driven evolution as it more closely resembles how nature evolves. In nature, the brain and the body together define the behavior of the individuals. In most traditional evolutionary algorithms either the control program or the morphology of the robots is evolved. In previous environment-driven evolutionary algorithms, the evolution of morphologies has not yet been examined. Due to evolution of morphologies, we can investigate the evolution of the morphologies and its complexity. To examine the morphological complexity of the pedestrian novel methods of calculating them are proposed in Chapter 4. It is important to note that while most traditional evolutionary algorithms start with a population of random individuals NEAT and by extension CPPN-NEAT start with a population of minimal networks with random weights. More complex networks gradually appear in the population allowing diverse and complex patterns in the phenotypes to be represented. Because the size of the networks is unlimited, NEAT can start minimally and grow the necessary structure over generations. The new structures only stay if they are found to be useful through fitness evaluations. These fitness evaluations are implicit in an environment-driven evolutionary algorithm, like PEDEST and medea, but they are present nonetheless through the need to meet to be able to mate. Another advantage of this increasing complexity of the networks is that NEAT searches through minimal networks. So it does not need to evaluate needlessly complex networks. One distinction that needs to be made is the difference between the phenotype and the genotype of the pedestrians. The genotype is the information that is used in the reproductive system, their DNA. The genotype of the pedestrians is their CPPN. The phenotype of an individual are the observed properties of that individual. In PEDEST the phenotype of a pedestrian is the shape of the pedestrian and the composition of its materials. 3.2 Meeting & Mating The mating process is done in the same way as medea [11]. To produce offspring pedestrians have to meet other pedestrians in the arena. Because the pedestrians have no sensors, the only

3.3. COST-MECHANISM 15 Figure 3.1 A pedestrian in motion. way to meet more pedestrians is to walk further. The pedestrians meet each other by getting close to each other. This is checked by comparing the paths of a pedestrian with all other pedestrians that have already been simulated and were in the same area around the same time. All these meeting events are then recorded. A single random event is chosen from these meeting events as the basis of a new child. If a pedestrian never meets another pedestrian during its lifetime, a random pair of parents that have met each other in the experiment so far is chosen. This random pair is selected from among the last 500 meeting events to ensure that individuals born in the first few generations do not have a much higher chance to become a parent. The networks of the two parents of a new pedestrian are combined using the CPPN-NEAT strategy of reproduction [59, 60]. The CPPNs are combined using recombination of the genes of the parents with random genes from either parent. Mutation of the new pedestrian is done by adjusting the weights of the network and introducing or removing new nodes to it. The new nodes are assigned a random activation function from the available functions, e.g. sigmoidal, Gaussian, sine, absolute value, linear, and step functions. Newborn pedestrians are placed at random locations in the arena instead of at the meeting point of their parents. This was done to prevent clusters of stationary or near-stationary robots with no incentive to move because partners are close-by. Because PEDEST uses the meeting of two pedestrians as the basis for reproduction, there is an incentive to move for the pedestrians. By moving further, they will have a higher chance to reproduce and spread their genes through the population. Early experiments showed that without the presence of a mechanism to stabilize the population size, it would often increase or decrease very fast. To combat this not only the choice was made not to create offspring for every other pedestrian that was met, but also to still create offspring when the pedestrian did not meet another in its lifetime. 3.3 Cost-mechanism To create interesting dynamics in the environment and add the possibility that the evolutionary system will find different equilibria we have added an energy system to the environment. Using these energy units a varying lifetime is assigned to the pedestrians. This is done to introduce different equilibria to the environment. There is a tradeoff between being slow and steady or fast and furious. Every muscle cell a pedestrian has decreases the number of energy units a pedestrian has. The number of energy units of a pedestrian is calculated using

16 CHAPTER 3. ENVIRONMENTAL SETUP Lifespan(p) = 0.0025 (1000 ( cost muscle Count muscle (p) )) where Count muscle (p) is the number of muscles pedestrian p has and cost muscle the number of energy units lost for every muscle cell. Every energy unit adds 0.002 5 seconds to the lifetime of a pedestrian. Without this mechanism, this translates to a lifetime of 2.5 seconds. 3.4 Disease Different environments have various kinds of difficulty. Evolutionary algorithms are such good search techniques because they can handle these different environments. To test the resiliency to these environments of an environment-driven system such as PEDEST we have introduced disease to the pedestrians. The disease will atrophy muscles, i.e. the soft active voxels, in the pedestrian by turning them into bone, i.e. the hard passive voxel-type. The probability of atrophying a specific muscle is relative to the number of fat cells a pedestrian has. probability is calculated as follows: Disease Probability(p) = Count fat(p) 1000 diseasestrength where Count fat (p) is the number of fat cells pedestrian p has and the diseasestrength can be set to influence the difficulty of the disease. Every fat cell adds a small chance of applying the disease. We chose to give small pedestrians an advantage by dividing by 1000 instead of the size of the pedestrians because it was shown in earlier trials of PEDEST that the evolutionary system tends to generate large pedestrians. The idea for this disease comes from nature where the muscles atrophy if they are not being used enough. Something to note about the disease is that it is applied at the birth of a pedestrian and that the transformation is performed on the phenotype of the pedestrians, not on their genotype. It does not have an effect on the CPPNs of the pedestrian but does have an effect on the pedestrian s ability to move. Two different strategies have been identified to make the impact of the disease as small as possible for the pedestrians. The first is having few fat cells. This ensures that the probability is tiny and as such only a few muscle cells will turn into bone. The second way is to have clusters of muscles. A cluster of muscles will be able to continue the movement even if part of the cluster turns into bone. If the system is resilient to the disease, there will be a small difference between pedestrians from experiments where the disease was present and ones where it was absent. An additional benefit of the disease is that it adds an interesting dynamic to the environment. It also adds the possibility to find types of pedestrians that perform well under the different conditions. This

Chapter 4 Complexity Measures The way complexity evolves is a longstanding open question. In general, it is believed, and it has been shown in some environments that more difficult environments are better handled by individuals that are more complex [5]. In [5] a traditional evolutionary algorithm was used to evolve a population of robots for locomotion in different environments. These environments differed in the amount of difficulty they presented to the robots. These populations were then investigated to answer this open question. It was demonstrated that the complexity of the individuals grew over evolutionary time. It is also shown that individuals in more complex, more difficult, environments evolved to have a greater complexity. Quantifying this morphological complexity is a notoriously hard problem [15]. The most commonly used measure has been the variability of components [10]. These components can range from organelles [43] to limbs [1]. This has inspired the way the morphological complexity will be quantified in this work. In [5] a geometric complexity measure is proposed. It uses the curvature of the body and its entropy to define a measure for morphological complexity. However, this measure cannot be used to measure the complexity of the individuals in PEDEST because every curve of the soft-bodied robot is a 90 degrees angle. Other measures that are dismissed by Auerbach and Bongard [5] prove to be useful in our setting because of the different bodies and shapes of our agents. 4.1 Material-complexity The first measure of morphological complexity that will be employed is material-complexity. It gives a quantifiable measure to the complexity of the cell types in the softbots. A pedestrian made out of only one material is less complex than a pedestrian made out of all three. The distribution of the amounts of materials also plays a role in the complexity, e.g. a pedestrian with all but one cells of material A and one cell of material B is less complex than a pedestrian where the number of both material A and material B is equal. Shannon s Entropy [56] is used to 17

18 CHAPTER 4. COMPLEXITY MEASURES calculate it. The equation used to calculate the probability of having a certain number of cells of a specific type is p i (x) = C i(x) size(x) where C i (x) is an indicator function how many cells in pedestrian x are of type i and size(x) a function that returns the number of non-empty voxels in pedestrian x. The possible values for i are fat, bone, muscle + and muscle. Empty voxels are not used in this measure of complexity as they are taken into account by using the shape-complexity (Shown in Section 4.3). To calculate the material-complexity from these probabilities is material complexity(x) = i log 2 p i (x) if p i (x) > 0 p i (x) 0 if p i (x) = 0 One thing to note about the material-complexity is that it does not take the location of any of the voxels into account. 4.2 Mean-material-complexity Not only the amounts of materials is part of the complexity of the materials. Another part is the locations of the materials and how they are clustered together. One strength of CPPNs is their ability to capture regularity because of this we expect there to be large clusters of the same materials. The material-complexity has been extended to calculate what we call the mean-material-complexity. The material-complexity is not calculated on the entire body but on smaller parts of it. These parts are 2 by 2 by 2. The mean of all these complexities is taken as the mean-material-complexity. Parts, where only a single material is present, give a very low complexity. The complexity will become higher when the different cell types are scattered around the body. The combination of this will show us whether the regularity of the CPPNs influences the evolutionary process. 4.3 Shape-complexity A trivial way of measuring complexity would be to count the number of cells a pedestrian has. One of the problems with this is that a full 10 by 10 by 10 cube is less complex than an intricate shape of fewer than 1000 voxels. It can even be argued that a simple 2 by 2 by 2 cube has the same complexity as the full 10 by 10 by 10 cube. Auerbach and Bongard describe multiple geometric measures to calculate complexity [5]. One of them, a measure to calculate concavity, is useful in this research. However, it is discarded as it can be deceived by relatively simple body shapes, (e.g. a C shape would have a high concavity but does not show a high complexity). However, in PEDEST these simple body shapes are

4.3. SHAPE-COMPLEXITY 19 among the more complex body shapes that can be found. Some work [47, 61] has shown how Shannon Entropy can be used to measure complexity in a 3D object by using the curvature of the object as the random variable. Due to differences in the body types, these measures cannot be used. The following three ways of quantifying the complexity of the shape of the pedestrians have been identified and combined into a single measure. Complexity of the convex hull The first measure of shape-complexity that has been identified is the complexity of the convex hull of the pedestrian. The convex hull is the smallest convex set that contains the pedestrian. A convex set is a set of points where all lines between two points are inside that set. This can be imagined as taking a deflated balloon and putting the pedestrian inside of it. A simple cuboid pedestrian inside a balloon creates a balloon where only the eight corners would be visible. However, a more complex pedestrian, with lots of protrusion, would create a lot more points on the balloon that are visible from the outside. The triangles that can be made from the lines from point to point on the outside of the balloon are called the simplices of the convex hull. The amount of these simplices is what constitutes the complexity of a convex hull to us. The convex hulls of the pedestrian are calculated using the QuickHull algorithm [6]. Figure 4.1a shows a convex hull that is more complex than the convex hulls of Figure 4.1b or Figure 4.1c. Concavity The second way to quantify the shape-complexity of a pedestrian is its concavity. It is a way to calculate how hollow a shape is. A simple cuboid pedestrian would not be hollow at all, while a shape like the pedestrian in Figure 4.1c would be very concave. The concavity is calculated as the ratio of the volume of the pedestrian versus the volume of its convex hull or Concavity = 1 V p V h where V p is the volume of the pedestrian and V h the volume of the convex hull. The volume of the convex hull is computed as one would compute the volume of any irregular 3D-shape. The volume of the pedestrian is the number of voxels as the volume of a single voxel is 1. Figure 4.1 shows the concavity of three different 2D pedestrians. Limbs The third and last way of quantifying shape-complexity is by using the number of limbs a pedestrian has. As no existing way of calculating this was found a novel method to detect the number of limbs is proposed, which is shown in Algorithm 1. By hollowing out the pedestrian from its center of mass, the body will fall apart into disconnected parts. These disconnected parts are then counted to find the number of limbs a pedestrian has. The maximum number of

20 CHAPTER 4. COMPLEXITY MEASURES (a) A 2d pedestrian (b) A 2d pedestrian with a hull-complexitywith a hull-complexity of 7 and a concavity of 5 and a concavity of 1 15 /18 = 0.21 of 1 21 /23 = 0.09. (c) A 2d pedestrian with a hull-complexity of 5 and a concavity of 1 13 /23 = 0.43 Figure 4.1 A 2d representation of the complexity of the convex hulls and the concavity of the pedestrians. The colored regions represent cells of different materials. The empty regions represent empty voxels. The black lines shown are the convex hull. In 2D the simplices are lines instead of triangles, as such the complexity of convex hull in 2D is the number of line-segments on it. parts that have been counted while hollowing out the body is then used as the number of limbs. The hollowing out procedure is shown in Figure 4.2. Merging of sub-components shape-complexity The three subcomponents; the complexity of the convex hull, the concavity and the number of limbs are combined to create a single measure for shape-complexity. The combination is done using PCA [62]. This makes it easier to reason about the shape-complexity as a measure than three separate measures. Due to the high correlation between the three measures, the amount of information that is lost in this process is low. The three components are standardized to have a mean of 0 and a standard deviation of 1. The first component of the PCA explains 81.4% of the variance found in the data of our experiments. We can still reason about the shape-complexity using this measure without a large amount of information because the variance explained by the PCA is so high. The shape-complexity is normalized to be between 0 and 1 so it is easier to reason about it, and it is closer in numbers to the (mean-)material-complexity.

4.3. SHAPE-COMPLEXITY 21 C C C C. (a) Before hollowing out (b) First step; hollowing out with a radius of 0 (c) Second step; hollowing out with a radius of 1 (d) Third step; hollowing out with a radius of 2 Figure 4.2 A schematic representation showing the hollowing out process of a 2d pedestrian. The limb-detection algorithm would find two limbs here because of the two disconnected parts in the second step. The colored regions represent cells of different materials. The empty regions represent empty or hollowed out voxels. C represent the center of mass. Data: A body Result: Number of limbs in that body count = 0 i=1 while body not empty do Hollow out body with range i c = number of disconnected parts if c > count then count = c end i = i + 1 end return count Algorithm 1: Limb detection algorithm

22 CHAPTER 4. COMPLEXITY MEASURES (a) Shape-complexity 0.540 Number of limbs 3 Hull-complexity 52 Concavity 0.445 Material-complexity 0.265 Mean-material-complexity 0.057 (b) Shape-complexity 0.209 Number of limbs 2 Hull-complexity 22 Concavity 0.202 Material-complexity 1.469 Mean-material-complexity 0.310 Figure 4.3 The complexity-values of two pedestrians. The left figure shows a pedestrian that has a more complex shape but a less complex composition. The right figure shows a pedestrian with a less complex shape but a more complex composition.

Chapter 5 Implementation details In Chapter 3 we explained how the environment-driven evolutionary system works. In this chapter we will explain how we implemented PEDEST. The general outline of the experiments is as follows: the initial population is created at the start of the experiments. The pedestrians of the populations are then simulated in parallel and independently of each other. Then a process places the simulated pedestrians into the environment and finds the meeting points from which new pedestrians are being generated. These pedestrians are then also simulated and placed into the world, and the process repeats itself until the experiment is terminated. The entire project is open-source and located at the authors GitHub page 1. Due to the size of the experiments it is unfeasible to run the experiments on a simple desktop. Even with the parallelizations implemented we needed the help of a cluster. The Lisa Compute Cluster 2 was used to execute the experiments. Lisa is part of the computing clusters at SurfSARA. The use of a cluster is necessary because the simulation of the pedestrians is computationally intensive. It also allows us to parallelize these simulations more easily. To not overload the computing cluster the queue system of Lisa is used. Tasks can be submitted to this queue so that can be computed. This queue is used extensively to simulate the pedestrians. The Lisa Cluster is meant for people who are in need of large computing capacities, but do not need the facilities of a real supercomputer. The system offers a large number of multi-core nodes, many connected with a fast InfiniBand Network enabling efficient parallel programs. Taken from the SURFsara website 2 1 https://github.com/lstout/large-softbot-simulation 2 https://userinfo.surfsara.nl/systems/lisa 23

24 CHAPTER 5. IMPLEMENTATION DETAILS 5.1 Simulator The simulator that is used is the soft-voxel simulator created by Hiller and Lipson called Vox- CAD [30]. VoxCAD is a general purpose multi-material voxel simulation library. Using a command-line version of VoxCAD called Voxelyze the pedestrians are simulated. Voxelyze has provided by Cheney at the Creative Machines Lab s GitHub page 3. The simulator takes a VXA-file as input. This file contains the materials of which the pedestrians are made, their properties and the composition they are in. It also contains parameters for the physical nature of the environment such as temperature and gravity. The simulator produces as output a trace-file. This trace-file contains the position of the gravitational center of the pedestrians recorded at every hundredth time-step of the simulation. This trace-file will be used to put the pedestrian into the environment and find other pedestrians it met along the way. 5.2 Parallelization Physically realistic simulations of robot simulations have to live with the limitations of simulators. These simulators have problems with simulating more than a couple of dozen individuals [65]. Parallelizing an environment with large populations is a challenging task, especially given that PEDEST has populations of soft-bodied robots. Even without a multi-robot system, soft-bodied robots are expensive to simulate. It is expected that the development of interesting morphologies in the experiments of this thesis greatly benefits from having larger populations, which can be hundreds or thousands of individuals in size. In traditional evolutionary systems, the control loop can be trivially parallelized [2]. This is done by the parallelization of the evaluation and mating steps of the control loop. The environment-driven system that is explored here does not implement such a classic control loop. Instead, the evolution is implemented through local interactions between individuals. The evolution is not centrally led by an ominous system, and the individuals are born asynchronously. This makes a parallelization of the system much less straightforward. Instead of simulation all the pedestrians one step at a time, every pedestrian is simulated in parallel and without any knowledge of the other pedestrians in the arena. Afterward, there is a step that places the pedestrian into the arena and finds the pedestrians that it has met. A problem with this approach is how to deal with collisions between the pedestrians. If two pedestrians would collide with each other in the arena, both pedestrians would need to be simulated, including the collision and the effect it has on both pedestrians. The choice has been made to remove collision between pedestrians to alleviate this issue. We think that this simplification can be made without reducing the validity of the results presented in this work because these collisions would only reduce the effectiveness of the system. They reduce the effectiveness as the pedestrians have no way of inspecting their surroundings and thus could not 3 https://github.com/creativemachineslab/softbotevolution/

5.3. WORKERS 25 Governs and starts Leader. Starts experiments Creates database Termination Governs and starts Governs and starts NEATWorker Create initial pedestrians Mate pedestrians Apply disease Apply cost-mechanism VoxelyzeWorker Simulating pedestrians Simulated pedestrians New pedestrians PostProcessingWorker Place the pedestrians Find meeting-events Select mates Parents Figure 5.1 The processes of the experiments and their responsibilities. The dashed lines show control flow and the solid lines show the flow of pedestrians and their state. The workers do not communicate with each other but communicate by finding the pedestrians that they need to process in the database or in files in the directory that they watch. avoid collisions. This simplification also makes the parallelization almost trivial. We not only implemented the parallelization of the simulations but also the processing of the pedestrians. New pedestrians are made during the simulations and the post-processing work that places them into the environment and finds mates. 5.3 Workers The experiments are performed using a leader/follower protocol. One main leader sets up the experiment and starts the three workers: the NEATWorker, the VoxelyzeWorker, and the PostProcessingWorker. Their tasks are explained here but are also shown in Figure 5.1. The NEATWorker is responsible for the mating process and the application of the disease and costmechanism, the VoxelyzeWorker is in charge of the simulation of the pedestrians, and the Post- ProcessingWorker is responsible for the meeting of the pedestrians. The leader is the main process that governs the other workers and determines the termination. It also sets up a database that is used by the workers to keep track of the experiment. If the cluster is overcrowded there might not be any simulations of pedestrians for quite some time. If this happens, the leader is programmed to shut down one hour after the last simulation freeing a node on the cluster to increase the throughput of other jobs. The first worker is called the NEATWorker and is responsible for the creation of new pedestrians. The worker queries the database every few seconds to collect pedestrians that are to be created. They already have two parents who are used by NEAT to create the genome of the new

26 CHAPTER 5. IMPLEMENTATION DETAILS pedestrian. It then marks them as created in the database. Another task of the NEATWorker is to apply the disease to individuals. It does this by altering the VXA-files of the pedestrians while saving the originals to a backup. The job of the VoxelyzeWorker is to collect the genomes of the newly created pedestrians and submit them to the job system to be simulated. It polls the database to find pedestrians that have a genome and that are not simulated yet. The soft-bots are submitted to Lisa s queue system to be simulated in groups of 16 because the Lisa processors have 16 cores each. After they have been simulated they are marked as such in the database and their trace-files are moved to the appropriate directory for further processing. The last worker is called the PostProcessingWorker. It watches the directory where the tracefiles of simulated pedestrians are placed. These trace-files get adjusted to start at a random point in the arena instead of at coordinate (0, 0). These traces are placed into the database. The traces of other pedestrians that were in the same area at around the same time are retrieved from the database. The worker selects the ones that are actually meeting events. One of these events is randomly chosen as the basis for a new child. A random meeting event from the last 500 meeting events is selected if a pedestrian was unable to meet another pedestrian in the arena. This is done to give pedestrians that meet a higher chance to produce offspring. If no meeting event exists yet, two random pedestrians are chosen as parents for the new pedestrian. Another task of the PostProcessingWorker is to clean the traces database. This database fills up with all the points pedestrians have walked as the experiment runs. The SQL-queries to collect all traces around a pedestrian becomes slower as a result. To combat this, all traces that cannot be encountered anymore are removed from the database. Traces are not used anymore after all individuals that are born before it are simulated.

Chapter 6 Experimental Setup Twenty experiments have been done to explore PEDEST. The experiments look into the effect of the introduction of two different mechanisms and the change in morphological complexity over time. Five experiments without the cost-mechanism and disease, five experiments with the costmechanism turned on but without the disease present, five experiments with the cost-mechanism turned off but with the disease present and five with both the disease and the cost-mechanism have been done. The experiments terminate when all pedestrians that are born before the 120 th second have been simulated. Due to the parallel nature of the execution of the simulations, it is advised not to simulate only a fixed number of individuals because pedestrians that are simulated later are not necessarily in the world at a later time. In total 111624 pedestrians have been simulated. This is an average of 5581 pedestrians per experiment. The smallest experiment had 5084 pedestrians, while the biggest had 6462. 6.1 Environment The arena is a toroid of 0.5 meter by 0.5 meter. Having an unbounded world is a necessity as the simulator does not have the option to add bounds. Adding these bounds would also increase the complexity of the simulations, checking for collisions at every time step. Every cell is 1mm 3 and as such every pedestrian has a maximum size of 1cm 3. The arena has a width of 50 body No. of experiments Disease cost-mechanism 5 Off Off 5 Off On 5 On Off 5 On On Table 6.1 The number of experiments with the parameters used. 27

28 CHAPTER 6. EXPERIMENTAL SETUP lengths. The pedestrians have a meeting range of 2.5 centimeter, which is 2.5 body-lengths and 5% of the width of the arena. medea [11] uses a meeting range of 20 /7 body-lengths which is comparable. The area within the meeting range is 39 cm 2. This is only 1.6% of the 2500 cm 2 arena. However, in medea, the arena is smaller than the one in PEDEST, w.r.t the size of individuals. This might mean that it is harder for pedestrians to find each other than it is for the e-pucks in medea. The first population of 100 pedestrians is placed in the arena at a random place and a random time-step between the 0 th and 1 st second. Because every pedestrian has a maximum lifetime of 2.5 seconds and the experiments end at the 120 th second, there is a minimum of 48 generations and a minimum of 4800 pedestrians simulated. The cost-mechanism has a negative effect on the lifetime of a pedestrian. Every muscle incurs a small penalty on the lifetime of a pedestrian. This minor penalty has been set to a penalty of 0.5 energy unit. Every pedestrian starts with 1000 energy units corresponding to a lifetime of 2.5 seconds. A pedestrian made of 1000 muscles will have a 10% penalty to his lifetime if the costmechanism is turned on. To give the pedestrians enough time to find mates this base-lifetime is 10 times longer than the evaluation time used in [15]. The disease strength has been set to 0.5. This means that the every fat cell adds 0.5% to the chance of every muscle cell to become bone. The muscle cells of a pedestrian made out of 500 fat cells have a 25% chance to become bone. 6.2 Features The distance is calculated from the trace-files produced by the VoxCAD-simulator. The distance between two points is less trivial in a wraparound arena because the shortest distance might be across one of the borders. As such the distance between to points of the trace-file in a certain direction is calculated as d(p i, p i+1, arena p ) = min ( p i+1 p i, arena p p i+1 p i ) where p i is the coordinate of point i in the direction p and arena p is the length of the arena in that direction. The distance between the start and end point of the pedestrian is then calculated as the sum of the distance between the consecutive points on the path of the pedestrian or d(x, y) = d(x 0, x N 1, arena x ) 2 + d(y 0, y N 1, arena y ) 2 where x a vector of all the x-coordinates of the trace-file is, y a vector of the y-coordinates and N the number of points on the path. Future research might use a measure that uses the exact area covered as the basis for analysis. Another measure that is used to measure the performance of the evolutionary algorithm is the number of children every pedestrian has. It collected after the experiments finish from the

6.2. FEATURES 29 database as the output files do not contain this information. In PEDEST the order of creation of pedestrians does not mean necessarily mean that the order is the same with respect to the time of birth. Because of this, the birthtime of every pedestrian is also collected to look at the evolutionary timeline. The three different subcomponents of the shape-complexity, the complexity of the convexhull, the concavity, and the number of limbs, are collected by the feature collector. The shapecomplexity is calculated during analysis as the full data is necessary to use PCA to combine the subcomponents. The size of the pedestrians is also collected. The size is counted as the number of non-empty voxels. The material-complexity and the mean-material-complexity are collected by the featurecollector by using the VXA-files that are used to simulate the pedestrians. For every pedestrian a number of features are collected regarding the materials it is made out of. For the three different types of materials, bone, fat, and muscle, the amount of voxels of that type is collected and this amount relative to the size of the pedestrian. The amounts are counted before the disease affects the pedestrians.

30 CHAPTER 6. EXPERIMENTAL SETUP

Chapter 7 Analysis In this chapter, the experiments of the previous chapter will be analyzed. This will be done by looking at the values of the different features and their change over time under different conditions. To accurately compare their values over time the pedestrians are grouped based on their birthtime. Every group spans 5 seconds. This is 24 groups per experiment as we have 120 seconds of simulation time. They might be seen as generations, although in PEDEST birthtime is not always a direct indicator of the generation a pedestrian is born in, because of varying birthtimes and lifespans. The Wilcoxon signed-rank test [68] is used to determine whether significant differences exist between the different conditions. A p-value of 0.05 is used as the cutoff to confirm statistical significance. The median of a feature for the last generation of an experiment is calculated and used to compare the different conditions with this test. To test for correlation between two different features Kendall s tau-b is used. Again, a p-value of 0.05 is used as the cutoff to determine correlation. Kendall s tau-b is used because it is a non-parametric statistical test from which we can test statistical dependance. Something to note about the results and analysis shown here is that the evolutionary system in PEDEST still is in its infancy at the end of an experiment because only about 50 generations are simulated. This is far less than the usual number of generations in traditional evolutionary algorithms. By definition, environment-driven approaches are slower in terms of evolutionary progress as there is less guarantee that successful individuals procreate. This means that, for the entire population to become as good as a traditional genetic algorithm s population, more generations have to be simulated. This heavily influences the analysis done here as not all effects might have appeared yet or they might change over time. 31

32 CHAPTER 7. ANALYSIS 7.1 Distance As seen in Figure 7.1 there is a very slight linear trend where the distance that the pedestrians have moved goes up over time. This trend is easy to miss in Figure 7.2 where the median over time is plotted. The increase over time indicates that the population as a whole improves itself which shows that the evolutionary system works. Table 7.1 shows some statistics about the experiments under different conditions. In all circumstances, the distance being moved by the pedestrians is increasing over time. Significant differences were found between the experiments without any additional mechanisms turned on and the ones where any mechanisms are present. The cost-mechanism and disease-mechanism do not have a significant difference. The absence of significant difference is interesting because it indicates that the mechanisms have the same kind of impact on the distance moved. The Kendall s tau-b coefficients show almost no correlation (between 0.015 and 0.025) between the birthtime of a pedestrian and the distance it has moved. The Wilcoxon signed-ranked test shows no significant difference between the experiments where the disease was present versus those where it was not present. The pedestrians of the experiments without any mechanisms have a median distance moved of 1.640 10 2 while those of the experiments with the disease turned on have a median distance moved of 1.599 10 2. The fact that no significant difference is found is surprising as it was expected that the disease would have a negative effect on the performance of the pedestrians. However, it does reinforce the idea that evolutionary systems are resilient to small, disadvantageous changes in the environment. Turning on the cost-mechanism does have a significant impact on the distance traveled (p < 0.05). The pedestrians of the experiments where the cost-mechanism is turned on have a median distance moved of 1.518 10 2 m. This is less than the median distance of the pedestrians of the experiments without it. This difference is significant. This impact is expected because the same pedestrian has less time to walk around in the experiments where the cost-mechanism is turned on than in experiments where it was turned off. Interestingly the experiments where both the disease is present and the cost-mechanism is turned on are closest to the base experiments in terms of distance moved. The pedestrians of those experiments have a median distance of 1.605 10 2 m moved. 7.2 Number of offspring Something that inhibits the evolution of the system is the low number of meeting events. 67% of the pedestrians never meet another pedestrian and thus have no offspring at all. Assuming that this percentage would be the same in a traditional evolutionary algorithm, the remaining 33% would be heavily skewed towards the best-performing individuals. As seen in the green distribution of Figure 7.3, this distribution is not skewed towards the best-performing pedestrians in PEDEST. This might be because it is too hard to meet others even when walking far or because the pedestrian did not walk very far. This fact has a negative impact on the performance of the system because well-performing pedestrians are also in this 67%. Due to this some well-

7.2. NUMBER OF OFFSPRING 33 Condition d ˆd Slope ˆd first gen. ˆd last gen. τ-b None 2.353 10 2 m 1.640 10 2 m 1.417 10 3 1.459 10 2 m 1.693 10 2 m 0.025 Disease only 2.308 10 2 m 1.599 10 2 m 9.290 10 4 1.587 10 2 m 1.593 10 2 m 0.015 Cost only 2.221 10 2 m 1.518 10 2 m 1.269 10 3 1.252 10 2 m 1.702 10 2 m 0.024 Both 2.241 10 2 m 1.605 10 2 m 1.168 10 3 1.308 10 2 m 1.721 10 2 m 0.021 Table 7.1 This table shows certain statistics of the different types of experiments in terms of distance that the pedestrians that have moved. d is the mean distanc, ˆd is the median distance, ˆd first gen. is the median distance of the first generation and ˆd last gen. is the median distance of the last generation. The slope is the slope of the linear regression of the distance versus the birthtime. τ-b is Kendall s tau-b correlation between a pedestrian s birthtime and the distance it has walked. All conditions show an increase in median distance from the first generation to the last. All tau-b correlations are statistically significant (p < 0.05). This indicates that for all experiments the distance increases over time. Figure 7.1 These plots show how the distance that has been walked by the pedestrians evolves over time. A linear regression line has been plotted through the data. All linear regression lines show a tiny increase over time. Both the cost-mechanism and the disease have a negative effect on the performance of the pedestrians in terms of the distance traveled, but the effect is subtle.

34 CHAPTER 7. ANALYSIS Figure 7.2 The figures show how the distance that has been walked by the pedestrians evolves over time. The line plotted is a cubic spline through the median over the generations. The filled region shows the interquartile range. Because the medians go up only very slightly, we can see that a big part of the population does not improve over the course of an experiment. This is due to the speed of the evolutionary process. The evolutionary system is only in its infancy and world should be simulated for longer to get better results.

7.3. SHAPE 35 Figure 7.3 The fitness (measured in distance moved) of a pedestrian against the number of offspring. There is a small positive correlation (τ-b = 0.2), as expected. However, it is hard for well performing pedestrians to convert their performance into more children. Figure 7.4 The distribution of distances divided by the number of offspring a pedestrian has. The pedestrians without any children (in blue) have significantly (p < 0.001) less distance traveled than the pedestrians that do have children (in green). The vertical lines show the means of the two groups. The mean of the group with children is higher than the mean of the group without any. performing genomes are not used to evolve a new pedestrian. Possible solutions to this problem are proposed in Section 8.1. Figure 7.3 shows the distance that a pedestrian has walked against the number of offspring that they have. There is a small positive correlation (0.2) between the distance and the number of offspring. It is hard for well-performing individuals to convert this performance into offspring. Figure 7.4 shows the distribution of the distance traveled with the pedestrians grouped on whether the pedestrian has any offspring. It can be seen that the group with at least one child has a distribution that is more skewed towards a higher distance moved. A pressure exists from within the environment to have the pedestrians walk further to spread their genes. 7.3 Shape As seen in Figure 7.5 the shape-complexity goes down over time under all conditions. It goes down mostly due to the increase in the size of the pedestrians as seen in Figure 7.7. This increase in size reduces all three components of the shape-complexity: the complexity of the convex hull, the concavity and the number of limbs of the pedestrian. No significant differences were found between the different conditions in term of size, shape-complexity, concavity, the number of triangles or the number of limbs. It can be concluded that the shape of the pedestrians is not affected by either the cost mechanism or the disease.

36 CHAPTER 7. ANALYSIS There is a negative correlation between the shape-complexity and the distance moved by the pedestrians (Figure 7.6). This negative correlation is interesting because it is believed that a higher complexity leads to a higher average performance [5]. The pedestrians that have almost no shape-complexity are pedestrians that are almost entirely square. This shape has a wide variety of distance walked because the materials play a prominent role there. The best performing pedestrians have a slightly bigger shape-complexity (between 0.2 and 0.4.) There is a steep decline in performance with a shape-complexity of more than 0.4. The reason for this might be that pedestrians at those shape-complexities are under-explored by the evolutionary system, and the right shapes have not been found yet. Figure 7.9 shows the individual with the highest shape-complexity found in our experiments. It did not perform well (a distance of 0.008). The complexity of the convex hull is calculated as the number of triangles of the hull. The shown pedestrian has 60 triangles while a cuboid pedestrian has 12 triangles. The concavity is calculated as a ratio of volumes. This pedestrian has a concavity of 0.619. 8 limbs were detected by our proposed limb-detection algorithm. Figure 7.8 shows the size of the pedestrians versus the distance that they have walked. There is a small positive correlation. Tiny pedestrians (size < 50) do not walk very far. The best individuals are between 400 and 700 voxels. This size corresponds to the individuals with some shape-complexity and high performance. They can be seen in the sparse cloud above the dense areas in Figure 7.6. 7.4 Materials The complexity of the composition of its body is measured by the material-complexity, as explained in Section 4.1. The material-complexity goes up slightly over the course of the experiments, as shown in Figure 7.10. Figure 7.11 shows that a large number of pedestrians have a material-complexity of 1.0 or 1.5. These values correspond to pedestrians made of 2 or 3 materials in equal proportions. This range of material-complexity also corresponds to the best performing individuals. No significant differences were found between the different conditions w.r.t material-complexity. Because there are significant differences found in the number of specific materials (Section 7.4.1), it is unexpected that none are found here. Figure 7.12 shows the pedestrian with the highest material-complexity. It can be seen that the pedestrian has all materials and about the same amount of each of them. The pedestrian evolved with the cost-mechanism turned on and the disease present in the environment. The material-complexity of this pedestrian is 1.999. The pedestrian has a distance moved of 0.035. That is around twice the median performance of other pedestrians under these conditions. The mean-material-complexity becomes smaller over time, as seen in Figure 7.13. This lower mean-material-complexity means that the smaller subcomponents of the pedestrians get more uniform, i.e., the materials are clustered together instead of being scattered throughout the body.

7.4. MATERIALS 37 Figure 7.5 The shape-complexity plotted over time under the various conditions. It was thought that more interesting shapes would appear over time. However the opposite is the case, the median shapecomplexity goes down over time under all conditions.

38 CHAPTER 7. ANALYSIS Figure 7.6 The shape-complexity of the pedestrians plotted against the distance that they have walked. A lower shape-complexity has a higher chance to perform well and walk far. The individuals with a high shape-complexity do not perform well. It was thought that more interesting shapes correspond to a better ability to walk. Figure 7.7 The size of the pedestrians, measured in the number of voxels, and how the median changes over time. Over time the median size goes up. This increase is interesting as there is a negative correlation between the size and the performance of pedestrians.

7.4. MATERIALS 39 Figure 7.8 The size of the pedestrians plotted against the distance they have traveled. There is a small positive (0.14) correlation. A slight increase in performance is seen around a size of around 500 voxels. However, the same performance is gotten by pedestrians that are entirely filled (a size of 1000 voxels). The performance of the big pedestrians is unexpected as the best soft-bodied robots shown in [15] are not this big. Figure 7.9 The pedestrian with the highest shape-complexity. It did not perform well (a distance of 0.008). It cannot be seen clearly from this picture, but there are 4 upside-down pyramids. These pyramids point to the regularity with variation, for which CPPNs are famous. The complexity of the convex hull is calculated as the number of triangles of the hull. This pedestrian has 60 while a cuboid pedestrian has 12. The concavity is calculated as a ratio of volumes. This pedestrian has a concavity of 0.619. 8 limbs were detected by our proposed limb-detection algorithm.

40 CHAPTER 7. ANALYSIS Figure 7.10 Under all conditions the material-complexity goes up over time very slightly, indicating that fewer pedestrians are made out of only, or mostly, a single material. No significant differences were found between the different conditions. The larger part of the populations of pedestrians have a mean-material-complexity between 0.2 and 0.3, which also gives the best performance (Seen in Figure 7.14). The same kind of distribution can also be seen in Figure 7.11, where the material-complexity is shown. This shows that the evolutionary process focuses on some of the behaviors that produce better pedestrians. There are no significant differences found between environments with the disease present or the cost-mechanism turned on and without them. Figure 7.15 shows the pedestrian with the highest mean-material-complexity found. The pedestrian has a mean-material-complexity of 0.641. It moved 0.022 m or 1.5 times the median distance of pedestrians under the same conditions. 7.4.1 Muscle Figure 7.16 shows that over time the number of muscles increases under all conditions. Interestingly, there is no significant difference between experiments with or without disease present. Because the disease alters the muscles and turns them into bones, it was thought that the evolu-

7.4. MATERIALS 41 Figure 7.11 The relation between distance walked and the material-complexity of individuals. There are two interesting spikes in the distribution of the material-complexity at 1.0 and 1.5 (seen in the top histogram). These values correspond to pedestrians made out of 2 and 3 materials equally. As explained in 2.3, CPPNs promote regularity. This regularity is shown in these spikes. The small positive correlation (0.091) between the material-complexity and the distance moved, in combination with the small increase in material-complexity over time, indicates that the evolutionary system learns what make effective pedestrians. tionary system would react to this by creating pedestrians with more muscle cells or by evolving away the fat cells because it will reduce the impact of the disease in both cases. The medians differ between the different conditions but not enough to be significantly different. The cost mechanism does have an impact on the number of muscle cells (Figure 7.16 top right). The number of muscle cells is significantly less for experiments where the mechanism was turned on. As expected the number of muscle cells is less when there is a disadvantage to having them. The disadvantage of having muscle cells is larger when the cost mechanism is on than it is necessary to have more of them with the disease present. This can be deduced from the fact that the costmechanism does have a significant effect while the presence of the disease does not. The increase in muscle cells does not mean that the pedestrians also perform better, shown in Figure 7.17. No clear relation between the number of muscle cells and the distance moved by the pedestrians is found. 7.4.2 Fat The number of fat cells in the pedestrians stays roughly equal over time under all conditions. The addition of the cost mechanism to the environment increases the number of fat cells in the pedestrians, as seen in Figure 7.18. The decrease in muscle cells in combination with the growth of the pedestrians leaves the system with some holes to fill. These holes are filled with fat. No significant differences between the experiments where the disease is present or not were found, most likely because there is not enough incentive to have a large number of fat cells without the disease present. In Figure 7.18 in can be seen that in the experiments without disease present

42 CHAPTER 7. ANALYSIS Figure 7.12 The pedestrian with the highest material-complexity. It can be seen that the pedestrian has all materials and about the same amount of each of them. The material-complexity of this pedestrian is 1.999. The pedestrian has a distance moved of 0.035. This is around twice the median performance of the pedestrians. Figure 7.13 The medians of the mean-material-complexities go up slightly over time (A linear regression slope of 5.5 10 3 ± 5 10 4 ). This means that the pedestrians get less uniform over time, there will be multiple different materials in the smaller parts of the pedestrians.

7.4. MATERIALS 43 Figure 7.14 The mean-material-complexity of pedestrians plotted against the distance that they have moved. Having a very uniform body, indicated by a low mean-material-complexity, leads to a worse performing pedestrian, but so does having a high mean-material-complexity. A clear tradeoff can be seen here. Note that moderate mean-material-complexity does not necessarily mean that the pedestrian has different cells scattered throughout their body. It could also have a highly complex area and a uniform area because the mean-material-complexity is still a mean value. Figure 7.15 The pedestrian with the highest mean-material-complexity. The pedestrian has a meanmaterial-complexity of 1.860. It moved 0.022 m. This is above the median distance.

44 CHAPTER 7. ANALYSIS Figure 7.16 Under all conditions, the number of muscle cells goes up over time. The cost-mechanism has a significant effect on the number of muscle cells, dampening their growth over time. The experiments with the disadvantageous effects have lower medians than the base experiments. Figure 7.17 A very small positive correlation (0.08) is present between the distance traveled by the pedestrians and their number of muscles. No clear pattern can be seen in the figure. Pedestrians of all different amounts of muscle cells perform well.

7.4. MATERIALS 45 Figure 7.18 The number of fat cells and the median plotted over time under the different conditions. The medians stay roughly equal over time. A significant difference is found between the experiments where the cost-mechanism is on versus when it is off. there are almost no fat cells. Due to this, there is almost no room for a change in the number of fat cells. A clear pattern can be seen in Figure 7.19. The number of fat cells has a clear negative effect on the distance traveled by the pedestrians. It was thought that having fat would be an advantage to the performance of the pedestrians, but the opposite proved to be the case. 7.4.3 Bone The median number of bone cells does not change much over time in the experiments (seen in Figure 7.20). The upper quantile goes up over time, showing that part of the population gets more bone cells. This trend is what is in line with the expectations as the number of bone cells shows a negative relation with the distance moved seen in Figure 7.21. The number of bone cells stays the same as it is already at the optimum level for the best performance. It can now be seen that even though muscle cells do not have a high positive relation with the distance moved,

46 CHAPTER 7. ANALYSIS Figure 7.19 The relation between the number of fat cells and the distance traveled by pedestrians. Even though almost no correlation (0.039) is present, a clear negative relation can be seen. The very small correlation is attributed to the large variance in terms of distance moved at 0 fat cells. the increase of muscle cells is because that the number of bone and fat cells are already close to the optimum value and increasing them would give a decrease in performance. Both the fat cells and the bone cells have a very negative relation with the distance moved. 7.5 Best performing pedestrians The best-performing individual is shown in Figure 7.22. It moved 0.189m (almost 19 body lengths). It has a caterpillar-like movement where it is using its backside to push itself forward. It was observed in an experiment where the disease was present, but the cost-mechanism was turned off. The best distance traveled in [15] is 2.5 body lengths. However, because the lifespan of the pedestrians is about 10 times longer than the individuals used in [15] the best individuals found by generic evolutionary algorithms are one order of magnitude better. The big difference is mostly due to the number of generations that have been simulated. Cheney simulated 1000 generations to get to his results where we have only simulated about 50 generations have been simulated. However, due to the nature of PEDEST high-performing pedestrians can be discarded from evolution if they do not meet others. In the current experiments, this happens more often than in a traditional evolutionary algorithm. Because of this, the evolutionary process is slower in an environment-driven system than in a traditional one. Therefore it is hypothesized that even when PEDEST would simulate 1000 generations, the pedestrians would still perform a little bit worse than the individuals found in by Cheney.

7.5. BEST PERFORMING PEDESTRIANS 47 Figure 7.20 The number of bone cells in the pedestrians and the medians of the generations plotted over time. The medians stay roughly the same. The upper quantile increases its number of bone cells, showing that the evolutionary process is slowly increasing the number of bone cells in the population. Figure 7.21 The number of bone cells of the pedestrians plotted against the distance walked by those pedestrians. A clear negative relation can be seen. The biggest part of the pedestrians has no bone cells; these pedestrians have a very mixed performance.

48 CHAPTER 7. ANALYSIS Figure 7.22 The best pedestrian that was found in the experiments. It moved 0.189m (almost 19 body lengths). The behavior shown is the behavior shown by all high-performing pedestrians, it is a caterpillar-like movement where the back of the pedestrian is used to push itself forward.