Design Choices in Software

Print QR Code JIS X 0510 in Software Design Choices

In applying BACKPROPAGATIONany given task, a number of design choices to must be made We summarize these choices below for our task of learning the direction in which a person is facing Although no attempt was made to determine the precise optimal design choices for this task, the design described here learns

the target function quite well After training on a set of 260 images, classification accuracy over a separate test set is 90% In contrast, the default accuracy achieved by randomly guessing one of the four possible face directions is 25%

Input encoding Given that the ANN input is to be some representation of the image, one key design choice is how to encode this image For example, we could preprocess the image to extract edges, regions of uniform intensity, or other local image features, then input these features to the network One difficulty with this design option is that it would lead to a variable number of features (eg, edges) per image, whereas the ANN has a fixed number of input units The design option chosen in this case was instead to encode the image as a fixed set of 30 x 32 pixel intensity values, with one network input per pixel The pixel intensity values ranging from 0 to 255 were linearly scaled to range from 0 to 1 so that network inputs would have values in the same interval as the hidden unit and output unit activations The 30 x 32 pixel image is, in fact, a coarse resolution summary of the original 120 x 128 captured image, with each coarse pixel intensity calculated as the mean of the corresponding high-resolution pixel intensities Using this coarse-resolution image reduces the number of inputs and network weights to a much more manageable size, thereby reducing computational demands, while maintaining sufficient resolution to correctly classify the images Recall from Figure 41 that the ALVINN system uses a similar coarse-resolution image as input to the network One interesting difference is that in ALVINN, each coarse resolution pixel intensity is obtained by selecting the intensity of a single pixel at random from the appropriate region within the high-resolution image, rather than taking the mean of all pixel intensities within this region The motivation for this ic ALVINN is that it significantly reduces the computation required to produce the coarse-resolution image from the available high-resolution image This efficiency is especially important when the network must be used to process many images per second while autonomously driving the vehicle Output encoding The ANN must output one of four values indicating the direction in which the person is looking (left, right, up, or straight) Note we could encode this four-way classification using a single output unit, assigning outputs of, say, 02,04,06, and 08 to encode these four possible values Instead, we use four distinct output units, each representing one of the four possible face directions, with the highest-valued output taken as the network prediction This is often called a 1-0f-n output encoding There are two motivations for choosing the 1-of-n output encoding over the single unit option First, it provides more degrees of freedom to the network for representing the target function (ie, there are n times as many weights available in the output layer of units) Second, in the 1-of-n encoding the difference between the highest-valued output and the second-highest can be used as a measure of the confidence in the network prediction (ambiguous classifications may result in near or exact ties) A further design choice here is "what should be the target values for these four output units ' One obvious choice would be to use the four target values (1,0,0,O) to encode a face looking to the

left, (0,1,0,O) to encode a face looking straight, etc Instead of 0 and 1 values, we use values of 01 and 09, so that (09,O1,01,01) is the target output vector for a face looking to the left The reason for avoiding target values of 0 and 1 is that sigmoid units cannot produce these output values given finite weights If we attempt to train the network to fit target values of exactly 0 and 1, gradient descent will force the weights to grow without bound On the other hand, values of 01 and 09 are achievable using a sigmoid unit with finite weights

Network graph structure As described earlier, BACKPROPAGATION be apcan plied to any acyclic directed graph of sigmoid units Therefore, another design choice we face is how many units to include in the network and how to interconnect them The most common network structure is a layered network with feedforward connections from every unit in one layer to every unit in the next In the current design we chose this standard structure, using two layers of sigmoid units (one hidden layer and one output layer) It is common to use one or two layers of sigmoid units and, occasionally, three layers It is not common to use more layers than this because training times become very long and because networks with three layers of sigmoid units can already express a rich variety of target functions (see Section 462) Given our choice of a layered feedforward network with one hidden layer, how many hidden units should we include In the results reported in Figure 410, only three hidden units were used, yielding a test set accuracy of 90% In other experiments 30 hidden units were used, yielding a test set accuracy one to two percent higher Although the generalization accuracy varied only a small amount between these two experiments, the second experiment required significantly more training time Using 260 training images, the training time was approximately 1 hour on a Sun Sparc5 workstation for the 30 hidden unit network, compared to approximately 5 minutes for the 3 hidden unit network In many applications it has been found that some minimum number of hidden units is required in order to learn the target function accurately and that extra hidden units above this number do not dramatically affect generalization accuracy, provided cross-validation methods are used to determine how many gradient descent iterations should be performed If such methods are not used, then increasing the number of hidden units often increases the tendency to overfit the training data, thereby reducing generalization accuracy Other learning algorithm parameters In these learning experiments the learning rate r] was set to 03, and the momentum a! was set to 03 Lower values for both parameters produced roughly equivalent generalization accuracy, but longer training times If these values are set too high, training fails to converge to a network with acceptable error over the training set Full gradient descent was used in all these experiments (in contrast to the stochastic approximation to gradient descent in the algorithm of Table 42) Network weights in the output units were initialized to small random values However, input unit weights were initialized to zero, because this yields much more intelligible visualizations of the learned weights (see Figure 410), without any noticeable impact on generalization accuracy The

number of training iterations was selected by partitioning the available data into a training set and a separate validation set Gradient descent was used to minimize the error over the training set, and after every 50 gradient descent steps the performance of the network was evaluated over the validation set The final selected network was the one with the highest accuracy over the validation set See Section 465 for an explanation and justification of this procedure The final reported accuracy (e-g, 90% for the network in Figure 410) was measured over yet a third set of test examples that were not used in any way to influence training