A society of hill-climbers

The paper is concerned with function optimisation in binary search spaces. It focuses on how hill climbers can work together and/or use their past trials in order to speed up the search. A hill climber is viewed as a set of mutations. The challenge is twofold: one must determine how many bits should be mutated, and which bits should preferably be mutated, or in other words, which climbing directions are to be preferred. The latter question is addressed by recording the last worst trials of the hill climbers within an individual, called repoussoir. The hill climbers further explore the neighborhood of their current point so as to get away from the repoussoir. As to the former question, no definite answer is proposed. Nevertheless, we experimentally show that hill climbers behave quite differently depending on whether one sets a mutation rate p/sub m/ per bit, or sets the exact number M of bits to mutate per individual. Two algorithms describing societies of hill climbers, with or without memory of the past trials, are described. These are experimented on several 900-bit problems, and significantly outperform standard genetic algorithms and evolution strategies.


I. I NTRODUCTION
Most works in Evolutionary Computation (EC) explic itly refer to Darwinian evolution and biologic metaphors.The transmission of biologic information follows complex schemes, most of which seem to be designed for the species to survive changing environments [32].However, as most applications of EC consider fixed fitness landscapes, bio logic metaphors may be unnecessarily complex.
This leads to investigate other modes of transmitting information from one population to the next one.Let us distinguish three levels of information transmitted along generations: . .• The basic level is that of genotyp1c material.The transmission of genotypic material is ensured by stan dard mutation and crossover operators, and genotypic material relevant to the fitness of individuals is on av erage transmitted proportionally to its relevance [14], [21].However, this transmission happens to fail due to the disruptiveness of evolution operators, which leads to the "fleeting discovery" phenomenon [9]: relevant building blocks appear and disappear several times be fore being effectively exploited.• The second level is that of individuals themselves: when elitist selection is used (the fittest individuals always survive, the phenomenon of fleeting discover ies is avoided.Though theoretical results in the GA framework endorse elitism [25], elitist strategy is not theoretically optimal in term of progress rate in the ES context [5].Nevertheless, a wide range of EC al gorithms use elitist selection schemes.• The third level is that of control of evolution, intended as all means (bias) for improving the course of evolu tion and the transmission of relevant genotypic mate rial.The information relative to control can be car ried by individuals themselves, through genotypic yet non phenotypic-ally relevant material (e.g.type of crossover [30], [31], mask of crossover [26], mutation step size [27], [2], encoded within each individual).
In the former case, referred to as adaptive biological evolution, the control information is part of genotypic material; it is evolved and transmitted by the same mechanism.
In the latter case, the control information is determin istically updated from the current population using exogenous heuristics: it represents an abstraction of the current (or the few past) population(s).As the evolution of an individual is here biased according to an abstraction of the other ones, this case is referred to as sociological evolution.This paper focuses on sociological evolution and more precisely studies how a society of hill-climbers could evolve.Only binary search spaces are considered throughout the paper.In terms of genetic algorithms, a hill-climber per forms a sequence of mutations.Most works in GAs have considered small mutation rates (e.g.10-3); and the control of binary mutation is mostly concerned with determining the average number of bits to mutate [7].Still, mutation with strong rate can be much desirable [16].But strong mutation asks the further ques tion of which bits should be mutated: if it indifferently affects all bits, mutation gets intolerably disruptive as evo lution goes on; this contrast with the disruptiveness of crossover decreasing as evolution goes on and population gets homogeneous.This paper investigates how hill-climbers can exchange information in order to determine which bits should be preferably mutated, that is, which climbing directions are to be preferred in the current state of the population.This paper investigates two kinds of hill-climber soci eties, respectively termed natural societies (NS) and his torical societies (HS).These are presented in section 2, and briefly compared to some related works [20], [10], [11], [29], [4].Section 3 presents and discuss an experimental valida tion of these schemes on six large-sized problems, and some perspectives for future research build up the final section IV.

II. N ATURAL AND H ISTORICAL S OCIETIES
This section describes two organizations for a hill-climber society.In both cases, a generation consists of several hill-climbers exploring the neighborhood of their respec tive current starting point, or current seeds.In natural societies, a hill-climber only knows its current seed.In his torical societies, a hill-climber is further provided with a summary of the individuals explored in the previous gener ations.These schemes are compared to Evolution Strategy [27], Tabu Search [10] and Population-Based Incremental Learning [4].

A. Natural Society
In natural societies (NS), each hill-climber explores the neighborhood of its current seed with uniform distribution.
With the addition of selection at the population level, this scheme can be viewed as a variant of (µ + >.) Evolution Strategy (ES) [27], [2], where µ stands for the number of hill-climbers and >. for the total number of trials allotted to the hill-climbers.However, ESs mostly consider continuous search spaces, and use a mutation rate per coordinate: This rate depends on the coordinate and the individual in Adaptive ES [2] or is the same for all coordinates of all individuals, and is ad justed according to the 1/5th rule in Traditional ES [23]).These powerful adaptive mechanisms are, as such, ill-suited to binary spaces, which violate the strong causality prin-ciple [23]: if the mutation rate is allowed to become small enough (less than 1/ N * >., if N denotes the size of the problem), mutation has no effect.On the other hand, if a lower bound on the mutation rate is enforced, there is no such thing as a "very small mutation".In opposition, Gaussian mutation in continuous search spaces always has some effect as long as the mutation rate is not exactly 0.
In NS, the user sets the number M of bits to mutate per individual; a hill-climber randomly explores the indi viduals differing from its seed by exactly M bits.Rather surprisingly, and for a rather wide range of values for M, NS outperform Traditional ES [23] as well as Adaptive ES [20], [1] (see section III).Note however that such decision can definitely prevent the algorithm to reach the actual optimum ... NS involves three parameters: the number µ of hill climbers, the number >. of trials allowed to all hill-climbers and the number M of bits to mutate.

B. Historical Society
In historical societies (HS), hill-climbers are also pro vided with some memory of the past generations.This memory is first designed to avoid repeating past unfruitful trials, i.e. individuals which don't pass the selection step: Indeed, if the fitness landscape is fixed, and provided the selection scheme is that of (µ + >.) ES, an individual that was not selected at a given generation will never be selected in the future.Further, this memory is used to determine preferred directions for hill-climbing.
How could low-fitness individuals contribute to a better exploitation of highly fit individuals ?The answer origi nates from a Machine Learning perspective [19], [18].Con sider individuals x, y, z and t, where x has a (compara tively) high fitness, and y, z and t all have a low fitness (Table 1).
Bit 1 takes different values for the highly fit x and the low fit y, z and t.By induction, this difference in their genotypes may be considered a "cause" for the difference in their fitness; that is, from these examples, one could consider the rule If bit 1 is 0, then the fitness is high.In consequence, when exploring the neighborhood of x, one should preferably mutate bit 5 (which has no visible effects whatsoever) than bit 1 (a possible cause of high fitness).More generally, the more difference a bit makes between the seed and the unfruitful trials, the less this bit should be mutated.Practically, the past unfruitful trials are stored as an av erage individual termed repoussoir (French for foil, or re peller), which belongs to [O, l]N if N denotes the dimension of the problem.This repoussoir is updated every genera tion by relaxation (linear combination of average of current worse individuals and previous repoussoir, see Figure 1).Each hill-climber selects the bits to mutate by means of repoussoir-based tournament: the winner is the bit which takes the most similar value for the seed and the repoussoir.Thereby, the offspring will be still farther from the repous soir than the seed was: the climbing direction consists to get away from the repoussoir.
HS involves six parameters (Figure 1): besides the three parameters µ, ,\ and M of NS, it also depends on the frac tion R of individuals used to construct the repoussoir, the relaxation factor a, and the tournament size T used to se lect the bits to mutate.Only M was found critical (see section III).

C. Related works
Historical hill-climber societies continue a previous work of the authors devoted to "mutation by imitation" [29]; two modifications were needed to address large sized problems.First, the repoussoir now reflects the worst individuals in all previous generations, according to the relaxation factor a, whereas it previously depended on the last population only (e.g.a= 1).Second, the bits to mutate were determined on the basis of a roulette wheel selection, the probability of mutating the k-th bit being a decreasing function of Ix( k )-Repoussoir( k) I• It turned out that this function was quite difficult to adjust.The roulette wheel mechanism was therefore replaced by a tournament-based selection, more robust and less computationally expensive.
The idea of storing the past trials in order not to repeat them is borrowed to Tahu Search [10] and has been already explored in induction-guided evolution [22].The first dif ference lies in the fact that the memory of the past search is stored as a distribution rather than as a list of previous tri als [10] or schemas of individuals [22]; further, this memory is used stochastically rather than deterministically.
In the framework of continuous parameter optimization, Hansen & al. [12] use a memory made of a basis of nr, in which the adaptive mutation is computed.The adaptive mechanism then becomes independent of the underlying system of coordinates.This strategy, however, does not transpose immediately to binary spaces.
In the context of binary spaces, historical societies can also be compared to the Population-Based Incremental Learning (PBIL) scheme [4].PBIL constructs an "ideal individual", summarizing the best individuals in the pre vious populations (exactly as the repoussoir summarizes the worst individuals in HS).This ideal individual can be viewed as a distribution; every generation, the population is constructed from scratch according to this distribution.In particular, there exists no transmission of genetic ma terial between one population and the next one (the only transmitted information is that of the ideal individual).Therefore, there is little chance indeed to rediscover the previous best individual.This obviously is an advantage when the fitness landscape presents many local optima.
But PBIL should be adversely affected, when the region of high fitness is "narrow" and more resembles a path than a hill: if you ever leave the path you are not certain to find it again.Evidence for this is given by experimentations on the 91-bit Long Path problem [15]:  This problem admits a unique global optimum, no local optima, and was purposely designed to discourage stan dard hill-climbers.The fitness landscape is composed of two different regions; the first one resembles the OneMax landscape, and culminates in x = (0, .. 0); the second re gion is composed of a path of exponential length, starting at x.The fitness of an individual either is its rank on the path, if it belongs to the path; or its number of zeros oth-erwise.To solve this problem, an evolutionary algorithm must have different skills, e.g.find the top of a hill (in the OneMax part of the landscape) and find shortcuts on an otherwise exponentially long path [13].
The results obtained by NS, HS and PBIL are averaged on 30 runs. Figure 2 and 3 plots the best fitness reached for a given number of generations.The four parameters of PBIL are set as in [3]; in particular the number of trials per generation is 100.To permit a fair comparison, >. is set to 100 for NS and HS too.NS parameters finally are >.= 100, µ = J.O and M = 2. HS parameters are >.= 100, µ = 1, M = 2, T = 20 (very similar results are obtained for T = 10 and T = 30) and R = 50%.The relaxation factor a (for HS and PBIL) varies from 1 to 0.01; it is shown to have much influence on PBIL results for the LongPath problem (Figure 2), in contrast with HS (Figure 3).

III. E XPERIMENTAL RESULTS
For the sake of convenience, experimentations have con sidered numerical problems which have already been used to evaluate PBIL (3].The maximum of both F 1 and F2 is reached at point (0, ... ,0) and has value 107 while the maximum of function F3 is reached for Xi = 0.024 * (i + 1).However, due to the coarse discretization (9 bits), the maximum value here for F3 is 416.64.

B. Reference algorithms
Multiple restart Hill-Climbers denoted HCl and HC2, as well as two genetic algorithms denoted SGA and GA-scale, have been used in [3] as reference algorithms: HCl randomly considers the neighbors of the seed (differing from the seed by one bit); the seed is replaced by the first neighbor which strictly improves on the seed.If all neigh bors of the seed have been considered, HCl is restarted with another seed.HC2 differs from HCl in that it allows to replace the seed by a neighbor having similar fitness.SGA is a standard GA which uses two point crossover, with crossover rate 100%, mutation rate 10-3, population size 100 and elitist selection.GA-scale differs from SGA in two respects: it uses uniform crossover with rate 80%, and the fitness of the worst individual is subtracted from the fit ness of all individuals in the population before selection.A more detailed description of these algorithms, as well as their results on the considered problems, can be found in (3]. Two additional reference algorithms have also been con sidered: TES (for Traditional evolution strategy) is a binary (µ+.X) ES involving a single mutation rate per bit O" ; O" is modified according to the Rechenberg ' s 1/5 rule [23].The geomet rical factor used to increase O" ranges from 1.1 to 2. AES (for Adaptive ES) is a boolean (µ+>.)-ES that uses the adaptive mutation of (20] as described in (1] with minor differences: Each individual is attached one mutation rate p (the probability of mutation of any single bit) which it self undergoes mutation according to the following Obalek's rule: with a lower bound of 1 / N on p to guarantee effective mutation.In both cases, µ is 10 and >. is 100.

C. Results
The parameters for NS (and HS) areµ= 10 and>.=50 (R = 50%, T = 30 and a= .01),which were found rather robust for all problems through systematical trials.On the opposite, the value of M seems critical depending on the fitness landscape.
All algorithms are allowed 200,000 fitness calculations per run.Results averaged over 20 independant runs, are shown on Table 2: The results of HCi, GAs and PBIL are taken directly from [4].
NS and HS significantly outperform standard GAs and ESs, and obtain similar or better results than PBIL on all problems but F3 Binary.However, the case of F3 is to be handled separately: the easy F3 Gray is solved exactly by the Hill-Climbers HCl and HC2, as well as by NS and HS with M = 1.As a consequence, F3 Binary requires jumps of variable length (e.g. in order to pass from 10 .... 0 to 011...1), which is harder to achieve within a single fixed M-bit mutation than in the PBIL scheme.
Letting aside both F3 functions, these results asks for three comments.First of all, the efficiency of the sim ple mechanism of NS, which obtains better results than all other methods but PBIL, is surprising, but requires that M is well adjusted.Moreover, NS obtains really bad results for M = 1 comapred to all values of M > 1 (results not shown on Table 2).Indeed, for M = 1, NS gets trapped in local minima as any standard hill-climber.But this distinc tion between exploiting the seed and exploring new regions of the search space vanishes as M increases.
Second, HS significantly improves on NS, even with the same value of M (results not shown on Ta ble 2).But fur ther, the repoussoir-based choice of the bits to mutate al lows to modify more bits simultaneously in the same mu tation: the best value of M for HS is in most cases larger than for NS.This might be one reason for the greater over all efficiency observed in the results of Last and overall, the main weakness of both NS and HS is that parameter M must be supplied by the user.We therefore made some attempts for adjusting M online, either at the level of individuals as in ES [27], or at the level of the population, on the basis of rewards a la Davis [7].
But both heuristics seemingly suffer from myopia.An option which often gets small rewards (as happens for M = 1) will be favored over an option which seldom gets big rewards (as happens for higher values of M), no mat ter how the cumulated rewards of both options compare.Besides, if an option is rarely selected, it likely gets no re wards, and it is still more rarely selected ... Further, complementary experiments (not reported here) show that it is preferable to adjust the mutation rate per individual, than per bit: both NS and HS obtain bet ter results when the mutation rate is per individual (i.e.mutate exactly M bits) than when the mutation rate is M / (total nb of bits) per bit (i.e.mutate M bits on aver age), for wide range of values of M.

IV. C ONCLUSION AND P ERSPECTIVES
This work focuses on the adjustment of the mutation rate and the mutation distribution, in the framework of binary "Hill-Climber Societies".
In terms of hill-climbing, the mutation rate governs the length and the variance of the jumps allowed.Experimen tal validation on several large-sized problems strongly sug gests that one should rather fix the length of the jumps, and use a fixed number M of bits to mutate per individ ual, than adaptively adjust a mutation rate per bit: be the adjustment based on the 1/5th or the Obalek rules (section III-B), it tends to favor small frequent steps over big rare steps 1 .Further, for optimal and near-optimal values of M (the length of the jumps), a low variance is preferable: bet ter using a number M of bits to mutate per individual than a mutation rate per bit of M / (total nb of bits).It remains to determine the desirable value of M. The adjustment of M based on rewards a la Davis [7] suffers from the same limitations as the adaptive adjustment of a mutation rate per bit.Further work is concerned with de termining M off line, for instance by measuring the fitness distance correlation (1 7).
In what regards the mutation distribution, the goal con sists in determining judicious climbing directions.The orig-1 This is to be related to the trend of GAs, toward optimizing the average rather than the maximal fitness in the population [8].
inality of our approach is to reverse the question and ask where not to go, rather than where to go.Finding where to go is amenable to find the "gradient" of the fitness function -which is difficult in binary spaces.But finding where to go can be partially answered on the basis of the past history of the search: one should not go in the regions where un fruitful explorations have been done.This remark leads to construct a repoussoir summarizing the worst individuals explored in the previous generations.The preferred climb ing direction thereafter consists in getting away from the repoussoir.This HS approach resembles Tahu search [10] in that it memorizes the past trials in order not to repeat them.The difference lies in the coding and use of the mem ory, which is deterministic in Tahu search and stochastic in HS.Many other uses of the past history of search remain to be invented; still, HS outperforms standard GAs and the non-recombinant ( µ, >.)-ESs on all 6 problems experi mented so far, and outperforms the PBIL scheme on 5 out of these 6 problems.Further work is concerned with extending HS to continuous search spaces.

A CKNOWLEDGEMENTS
The authors want to thank the anonymous reviewers whose comments help to improve the readibility of this pa per.

-Figure 2 .
Figure 2. PEIL on the 91-bit Long Path problem.Hill-Climber Societies on the Lon Path problem

Figure 3 .
Figure 3. HS and NS on the 91-bit Long Path problem.
A. ProblemsAll three functions involve 100 numerical variables X i coded on 9 bits each and varying in [-2.56,2.56(.Both stan dard binary and Gray encoding have been considered,

Table 1 :
What can be gamed by comparing low and high fit individuals ?