Hyperparameter Tuning
Hyperparameters against learned parameters; grid, random and Bayesian search; the budget; and the rule that nothing is ever tuned on the test set.
Parameters are learned from the data. Hyperparameters are chosen before training, judged on validation, and belong in the experiment record — because they decide what the learning is allowed to do.
Grid search is exhaustive and exponential. Random search covers each important setting better per trial, because most settings turn out not to matter. Neither is allowed anywhere near the test set.
When a trial costs hours, spend the trials sequentially: model the objective from the trials so far, choose the next one to balance exploration and exploitation, and kill trials that are clearly losing before they finish.
A search costs compute, time and validation-set credibility, and returns less with every trial. Tune the learning rate first, stop when the curve flattens, and remember that a fixed leak or a better feature usually beats any amount of tuning.
AutoML runs a search over pipelines and hyperparameters against a validation metric and hands you the winner. What it hides is the search space, the preprocessing leakage it may have committed inside the loop, the validation set it has now overfitted, and the serving cost of the pipeline it chose.