Random Search and Grid Search for Function Optimization

Function optimization requires the selection of an algorithm to efficiently sample the search space and locate a good or best solution.
There are many algorithms to choose from, although it is important to establish a baseline for what types of solutions are feasible or possible for a problem. This can be achieved using a naive optimization algorithm, such as a random search or a grid search .
The results achieved by a naive optimization algorithm are computationally efficient to generate and provide a point of comparison for more sophisticated optimization algorithms. Sometimes, naive algorithms are found to achieve the best performance, particularly on those problems that are noisy or non-smooth and those problems where domain expertise typically biases the choice of optimization algorithm.
In this tutorial, you will discover naive algorithms for function optimization.
After completing this tutorial, you will know:

The role of naive algorithms in function optimization projects.
How to generate and evaluate a random search for function optimization.
How to generate and evaluate a grid search for function optimization.

Let’s get started.

Random Search and Grid Search for Function Optimization Photo by Kosala Bandara , some rights reserved.

Tutorial Overview
This tutorial is divided into three parts; they are:

Naive Function Optimization Algorithms
Random Search for Function Optimization
Grid Search for Function Optimization

Naive Function Optimization Algorithms
There are many different algorithms you can use for optimization, but how do you know whether the results you get are any good?
One approach to solving this problem is to establish a baseline in performance using a naive optimization algorithm.
A naive optimization algorithm is an algorithm that assumes nothing about the objective function that is being optimized.
It can be applied with very little effort and the best result achieved by the algorithm can be used as a point of reference to compare more sophisticated algorithms. If a more sophisticated algorithm cannot achieve a better result than a naive algorithm on average, then it does not have skill on your problem and should be abandoned.
There are two naive algorithms that can be used for function optimization; they are:

Random Search
Grid Search

These algorithms are referred to as “ search ” algorithms because, at base, optimization can be framed as a search problem. E.g. find the inputs that minimize or maximize the output of the objective function.
There is another algorithm that can be used called “ exhaustive search ” that enumerates all possible inputs. This is rarely used in practice as enumerating all possible inputs is not feasible, e.g. would require too much time to run.
Nevertheless, if you...