Remarks on Locally Weighted Regression in Software

Generating Denso QR Bar Code in Software Remarks on Locally Weighted Regression

Above we considered using a linear function to approximate f in the neighborhood of the query instance x, The literature on locally weighted regression contains a broad range of alternative methods for distance weighting the training examples, and a range of methods for locally approximating the target function In most cases, the target function is approximated by a constant, linear, or quadratic function More complex functional forms are not often found because (1) the cost of fitting more complex functions for each query instance is prohibitively high, and (2) these simple approximations model the target function quite well over a sufficiently small subregion of the instance space

One approach to function approximation that is closely related to distance-weighted regression and also to artificial neural networks is learning with radial basis functions (Powell 1987; Broomhead and Lowe 1988; Moody and Darken 1989) In this approach, the learned hypothesis is a function of the form

where each xu is an instance from X and where the kernel function K,(d(x,, x ) ) is defined so that it decreases as the distance d ( x , , x ) increases Here k is a userprovided constant that specifies the number of kernel functions to be included Even though f ( x ) is a global approximation to f ( x ) , the contribution from each of the Ku(d (xu,x ) ) terms is localized to a region nearby the point xu It is common

to choose each function K, (d (xu,x ) ) to be a Gaussian function (see Table 54) centered at the point xu with some variance a ;

K,(d(x,, x ) ) = e2"

+d2(xu,x)

We will restrict our discussion here to this common Gaussian kernel function As shown by Hartman et al (1990), the functional form of Equation (88) can approximate any function with arbitrarily small error, provided a sufficiently large number k of such Gaussian kernels and provided the width a2 of each kernel can be separately specified The function given by Equation (88) can be viewed as describing a twolayer network where the first layer of units computes the values of the various K,(d(x,, x ) ) and where the second layer computes a linear combination of these first-layer unit values An example radial basis function (RBF) network is illustrated in Figure 82 Given a set of training examples of the target function, RBF networks are typically trained in a two-stage process First, the number k of hidden units is a determined and each hidden unit u is defined by choosing the values of xu and : that define its kernel function K,(d(x,, x ) ) Second, the weights w , are trained to maximize the fit of the network to the training data, using the global error criterion given by Equation (85) Because the kernel functions are held fixed during this second stage, the linear weight values w , can be trained very efficiently Several alternative methods have been proposed for choosing an appropriate number of hidden units or, equivalently, kernel functions One approach is to allocate a Gaussian kernel function for each training example (xi, f (xi)),centering this Gaussian at the point x i Each of these kernels may be assigned the same width a2Given this approach, the RBF network learns a global approximation to the target function in which each training example ( x i ,f ( x i ) ) can influence the value of f only in the neighborhood of xi One advantage of this choice of kernel functions is that it allows the RBF network to fit the training data exactly That is, for any set of m training examples the weights wo w, for combining the m Gaussian kernel functions can be set so that f ( x i ) = f (xi) for each training

FIGURE 82 A radial basis function network Each hidden unit produces an activation determined by a Gaussian function centered at some instance xu Therefore, its activation will be close to zero unless the input x is near xu The output unit produces a linear combination of the hidden unit activations Although the network shown here has just one output, multiple output units can also be included

A second approach is to choose a set of kernel functions that is smaller than the number of training examples This approach can be much more efficient than the first approach, especially when the number of training examples is large The set of kernel functions may be distributed with centers spaced uniformly throughout the instance space X Alternatively, we may wish to distribute the centers nonuniformly, especially if the instances themselves are found to be distributed nonuniformly over X In this later case, we can pick kernel function centers by randomly selecting a subset of the training instances, thereby sampling the underlying distribution of instances Alternatively, we may identify prototypical clusters of instances, then add a kernel function centered at each cluster The placement of the kernel functions in this fashion can be accomplished using unsupervised clustering algorithms that fit the training instances (but not their target values) to a mixture of Gaussians The EM algorithm discussed in Section 6121 provides one algorithm for choosing the means of a mixture of k Gaussians to best fit the observed instances In the case of the EM algorithm, the means are chosen to maximize the probability of observing the instances xi, given the k estimated means Note the target function value f (xi) of the instance does not enter into the calculation of kernel centers by unsupervised clustering methods The only role of the target values f (xi) in this case is to determine the output layer weights w, To summarize, radial basis function networks provide a global approximation to the target function, represented by a linear combination of many local kernel functions The value for any given kernel function is non-negligible only when the input x falls into the region defined by its particular center and width Thus, the network can be viewed as a smooth linear combination of many local approximations to the target function One key advantage to RBF networks is that they can be trained much more efficiently than feedforward networks trained with BACKPROPAGATIONfollows from the fact that the input layer and the output This layer of an RBF are trained separately