CS231n 2017 Assignment2 layers.py以及FullyConnectedNet部分

Se_ren_di_pity

969人浏览 · 2025-08-03 15:39:48

Se_ren_di_pity · 2025-08-03 15:39:48 发布

layers.py

affine_forward:

仿射前向传播，接收x with shape(N,d_1,d_2,...,d_k)，将其reshape成(N, D)，并与W(D, M)作矩阵乘法，得到输出out (N, M)

$out = xW+b$

def affine_forward(x, w, b):
    """
    Computes the forward pass for an affine (fully-connected) layer.

    The input x has shape (N, d_1, ..., d_k) and contains a minibatch of N
    examples, where each example x[i] has shape (d_1, ..., d_k). We will
    reshape each input into a vector of dimension D = d_1 * ... * d_k, and
    then transform it to an output vector of dimension M.

    Inputs:
    - x: A numpy array containing input data, of shape (N, d_1, ..., d_k)
    - w: A numpy array of weights, of shape (D, M)
    - b: A numpy array of biases, of shape (M,)

    Returns a tuple of:
    - out: output, of shape (N, M)
    - cache: (x, w, b)
    """
    out = None
    ###########################################################################
    # TODO: Implement the affine forward pass. Store the result in out. You   #
    # will need to reshape the input into rows.                               #
    ###########################################################################
    # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

    reshaped_x = x.reshape((len(x), -1)) # 将X重塑为(N, D)，使用-1让他自动计算D
    out = reshaped_x.dot(w) + b.reshape(1,len(b)) #重塑b使其自动broadcast

    # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
    ###########################################################################
    #                             END OF YOUR CODE                            #
    ###########################################################################
    cache = (x, w, b)
    return out, cache

affine_backward:

对上述的前向传播进行反向传播，计算dx,dW,db，已给出dout

$\frac{\partial L}{\partial b} = \frac{\partial L}{\partial out}\frac{\partial out}{\partial b}$ ， $\frac{\partial out}{\partial b} = 1$

$\frac{\partial L}{\partial x} = \frac{\partial L}{\partial out}\frac{\partial out}{\partial x}$ ， $\frac{\partial out}{\partial x} = W^T$

$\frac{\partial L}{\partial W} = \frac{\partial L}{\partial out}\frac{\partial out}{\partial W}$ ， $\frac{\partial out}{\partial W} = x^T$

def affine_backward(dout, cache):
    """
    Computes the backward pass for an affine layer.

    Inputs:
    - dout: Upstream derivative, of shape (N, M)
    - cache: Tuple of:
      - x: Input data, of shape (N, d_1, ... d_k)
      - w: Weights, of shape (D, M)
      - b: Biases, of shape (M,)

    Returns a tuple of:
    - dx: Gradient with respect to x, of shape (N, d1, ..., d_k)
    - dw: Gradient with respect to w, of shape (D, M)
    - db: Gradient with respect to b, of shape (M,)
    """
    x, w, b = cache
    dx, dw, db = None, None, None
    ###########################################################################
    # TODO: Implement the affine backward pass.                               #
    ###########################################################################
    # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
    reshaped_x = x.reshape(len(x), -1) # 将X重塑为(N, D)，使用-1让他自动计算D
    db = dout.sum(axis = 0) # b与out的每一行相加，因此梯度是dout的每一列之和(因为dout/db = 1)
    dx = dout.dot(w.T).reshape(x.shape) # dout与w的转置相乘，得到梯度dx，再重塑为原来的形状
    dw = reshaped_x.T.dot(dout) # 重塑后的x与dout相乘，得到梯度dw

    # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
    ###########################################################################
    #                             END OF YOUR CODE                            #
    ###########################################################################
    return dx, dw, db

relu_forward:

使用ReLU函数进行前向传播

$out = ReLU(x)$

def relu_forward(x):
    """
    Computes the forward pass for a layer of rectified linear units (ReLUs).

    Input:
    - x: Inputs, of any shape

    Returns a tuple of:
    - out: Output, of the same shape as x
    - cache: x
    """
    out = None
    ###########################################################################
    # TODO: Implement the ReLU forward pass.                                  #
    ###########################################################################
    # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

    out = np.clip(x, 0, None)

    # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
    ###########################################################################
    #                             END OF YOUR CODE                            #
    ###########################################################################
    cache = x
    return out, cache

relu_backward:

反向传播，只保留大于0的输入所对应的权重的梯度

$\frac{\partial L}{\partial x} = \frac{\partial L}{\partial out} \frac{\partial out}{\partial x}$ ， $\frac{\partial out}{\partial x} = 1[x>0]$

def relu_backward(dout, cache):
    """
    Computes the backward pass for a layer of rectified linear units (ReLUs).

    Input:
    - dout: Upstream derivatives, of any shape
    - cache: Input x, of same shape as dout

    Returns:
    - dx: Gradient with respect to x
    """
    dx, x = None, cache
    ###########################################################################
    # TODO: Implement the ReLU backward pass.                                 #
    ###########################################################################
    # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

    dx = dout*(x>0) #x>0的部分才保留梯度
   
    # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
    ###########################################################################
    #                             END OF YOUR CODE                            #
    ###########################################################################
    return dx

batchnorm_forward:

batch normalize的前向传播，分为train和test阶段两种模式，train的时候mean和var就采用batch里的，然后加权累积平均running_mean和running_var，在test的时候直接使用train阶段累积的running_mean和running_var

$out = \gamma\frac{x-sample\_mean}{sample\_std} + \beta$

def batchnorm_forward(x, gamma, beta, bn_param):
    """
    Forward pass for batch normalization.

    During training the sample mean and (uncorrected) sample variance are
    computed from minibatch statistics and used to normalize the incoming data.
    During training we also keep an exponentially decaying running mean of the
    mean and variance of each feature, and these averages are used to normalize
    data at test-time.

    At each timestep we update the running averages for mean and variance using
    an exponential decay based on the momentum parameter:

    running_mean = momentum * running_mean + (1 - momentum) * sample_mean
    running_var = momentum * running_var + (1 - momentum) * sample_var

    Note that the batch normalization paper suggests a different test-time
    behavior: they compute sample mean and variance for each feature using a
    large number of training images rather than using a running average. For
    this implementation we have chosen to use running averages instead since
    they do not require an additional estimation step; the torch7
    implementation of batch normalization also uses running averages.

    Input:
    - x: Data of shape (N, D)
    - gamma: Scale parameter of shape (D,)
    - beta: Shift paremeter of shape (D,)
    - bn_param: Dictionary with the following keys:
      - mode: 'train' or 'test'; required
      - eps: Constant for numeric stability
      - momentum: Constant for running mean / variance.
      - running_mean: Array of shape (D,) giving running mean of features
      - running_var Array of shape (D,) giving running variance of features

    Returns a tuple of:
    - out: of shape (N, D)
    - cache: A tuple of values needed in the backward pass
    """
    mode = bn_param['mode']
    eps = bn_param.get('eps', 1e-5)
    momentum = bn_param.get('momentum', 0.9)

    N, D = x.shape
    running_mean = bn_param.get('running_mean', np.zeros(D, dtype=x.dtype))
    running_var = bn_param.get('running_var', np.zeros(D, dtype=x.dtype))

    out, cache = None, None
    if mode == 'train':
        #######################################################################
        # TODO: Implement the training-time forward pass for batch norm.      #
        # Use minibatch statistics to compute the mean and variance, use      #
        # these statistics to normalize the incoming data, and scale and      #
        # shift the normalized data using gamma and beta.                     #
        #                                                                     #
        # You should store the output in the variable out. Any intermediates  #
        # that you need for the backward pass should be stored in the cache   #
        # variable.                                                           #
        #                                                                     #
        # You should also use your computed sample mean and variance together #
        # with the momentum variable to update the running mean and running   #
        # variance, storing your result in the running_mean and running_var   #
        # variables.                                                          #
        #                                                                     #
        # Note that though you should be keeping track of the running         #
        # variance, you should normalize the data based on the standard       #
        # deviation (square root of variance) instead!                        # 
        # Referencing the original paper (https://arxiv.org/abs/1502.03167)   #
        # might prove to be helpful.                                          #
        #######################################################################
        # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

        sample_mean = np.mean(x, axis = 0)
        sample_var = np.var(x, axis = 0)
        std = np.sqrt(sample_var+eps) #防止除0
        u_out = (x - sample_mean)/std
        out = gamma*u_out + beta
        
        running_mean = momentum*running_mean + (1 - momentum)*sample_mean
        running_var = momentum*running_var + (1 - momentum)*sample_var
        
        cache = {'x':x, 'gamma':gamma, 'mean':sample_mean,
                 'std':std, 'u_out':u_out, 'out':out}
        

        # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
        #######################################################################
        #                           END OF YOUR CODE                          #
        #######################################################################
    elif mode == 'test':
        #######################################################################
        # TODO: Implement the test-time forward pass for batch normalization. #
        # Use the running mean and variance to normalize the incoming data,   #
        # then scale and shift the normalized data using gamma and beta.      #
        # Store the result in the out variable.                               #
        #######################################################################
        # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

        out = gamma*(x - running_mean)/np.sqrt(running_var + eps) + beta

        # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
        #######################################################################
        #                          END OF YOUR CODE                           #
        #######################################################################
    else:
        raise ValueError('Invalid forward batchnorm mode "%s"' % mode)

    # Store the updated running means back into bn_param
    bn_param['running_mean'] = running_mean
    bn_param['running_var'] = running_var

    return out, cache

batchnorm_backward:

反向传播

$\frac{\partial L}{\partial \beta} = \frac{\partial L}{\partial out} \frac{\partial out}{\partial \beta}$ ， $\frac{\partial out}{\partial \beta} = 1$

$\frac{\partial L}{\partial \gamma} = \frac{\partial L}{\partial out}\frac{\partial out}{\partial \gamma}, \frac{\partial out}{\partial \gamma} = u\_out, u\_out = \frac{x-mean}{std}$

$\frac{\partial L}{\partial x} = \frac{\partial L}{\partial out}\frac{\partial out}{\partial u\_out} \frac{\partial u\_out}{\partial (x-mean)} \frac{\partial (x-mean)}{\partial x}$

$\frac{\partial u\_out}{\partial (x-mean)} = \frac{\partial u\_out}{\partial (x-mean)}(direct\,gradient)+\frac{\partial u\_out}{\partial std}\frac{\partial std}{\partial (x-mean)}$

而 $\frac{\partial std}{\partial (x-mean)} = \frac{x-mean}{std}$ ， $\frac{\partial u\_out}{\partial std} = -\sum_N\frac{x-mean}{std^2}$ ， $\frac{\partial u\_out}{\partial (x-mean)}(direct\, gradient) = \frac{1}{std}$

$\frac{\partial (x-mean)}{\partial x} = 1 - \frac{\partial (x-mean)}{\partial mean} \frac{\partial mean}{\partial x}, \frac{\partial mean}{\partial x} = \frac{1}{N}$

这里要讲一下 $\frac{\partial (x-mean)}{\partial mean}$ ，因为 $x-mean$ 是每个样本的x对应的维度都减去了其对应维度的mean，所以其对应维度的梯度应是所有样本的梯度的sum

def batchnorm_backward(dout, cache):
    """
    Backward pass for batch normalization.

    For this implementation, you should write out a computation graph for
    batch normalization on paper and propagate gradients backward through
    intermediate nodes.

    Inputs:
    - dout: Upstream derivatives, of shape (N, D)
    - cache: Variable of intermediates from batchnorm_forward.

    Returns a tuple of:
    - dx: Gradient with respect to inputs x, of shape (N, D)
    - dgamma: Gradient with respect to scale parameter gamma, of shape (D,)
    - dbeta: Gradient with respect to shift parameter beta, of shape (D,)
    """
    dx, dgamma, dbeta = None, None, None
    ###########################################################################
    # TODO: Implement the backward pass for batch normalization. Store the    #
    # results in the dx, dgamma, and dbeta variables.                         #
    # Referencing the original paper (https://arxiv.org/abs/1502.03167)       #
    # might prove to be helpful.                                              #
    ###########################################################################
    # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

    x = cache["x"]
    mean = cache["mean"]
    std = cache["std"] 
    u_out = cache["u_out"]
    gamma = cache["gamma"]
    N = len(x)
    
    dbeta  = dout.sum(axis = 0) # dbeta和之前的db同理
    dgamma = (u_out*dout).sum(axis = 0) #dgamma是u_out和dout的逐元素乘积的和                                                                     
    
    du_out = gamma*dout 
    dminus_2 = du_out/std # dminus_2 是x-mean的直接梯度
    dstd = -((x - mean)*du_out/np.square(std)).sum(axis = 0) #对std的直接梯度
    dminus_1 = (x-mean)*dstd/std # 对x-mean通过std的间接梯度
    dminus = dminus_1 + dminus_2 # 这是对x-mean的总梯度
    dx = dminus - dminus.sum(axis=0)/N # dx = d(x-mean) - (dL/dmean)*(dmean/dx)
    

    # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
    ###########################################################################
    #                             END OF YOUR CODE                            #
    ###########################################################################

    return dx, dgamma, dbeta

batchnorm_backward_alt:

上面的反向传播是按照链式法则一步一步计算的，我们可以先整理上述的公式，直接联立并得出dx的公式，由于相乘之间会有一定的式子约分，可以减少计算量

$\frac{\partial L}{\partial x} = \frac{\partial L}{\partial out}\frac{\partial out}{\partial u\_out} \frac{\partial u\_out}{\partial (x-mean)}\frac{\partial (x-mean)}{\partial x} = \frac{\partial L}{\partial u\_out} (\frac{1}{std}-\frac{x-mean}{std}\sum_N\frac{x-mean}{std^2})\frac{\partial (x-mean)}{x}$

我们可以将这个公式拆成四个部分：

$\frac{\partial L}{\partial u\_out}\frac{1}{std}$ ，这一项不作变形

$-\frac{1}{N}\sum_N\frac{\partial L}{\partial u\_out}\frac{1}{std}$ ，这一项不作变形

$-\frac{x-mean}{std}\sum_N\frac{\partial L}{\partial u\_out}\frac{x-mean}{std^2}$ ，由 $u\_out = \frac{x-mean}{std}$ ，将该式子化简为 $-u\_out\sum_N\frac{\partial L}{\partial u\_out}\frac{u\_out}{std}$

$-\frac{1}{N}\sum_N\frac{x-mean}{std}\sum_N\frac{\partial L}{\partial u\_out}\frac{x-mean}{std^2}$ ，这里由于broadcast按每个维度相乘的因子均相同，则遍历每个样本的 $\sum_N(x-mean) = 0$ ，因此这一整项为0

因此 $\frac{\partial L}{\partial x} = \frac{\partial L}{\partial u\_out} - \frac{1}{N}\sum_N\frac{\partial L}{\partial u\_out}\frac{1}{std} - u\_out\sum_N\frac{\partial L}{\partial u\_out}\frac{u\_out}{std}$

def batchnorm_backward_alt(dout, cache):
    """
    Alternative backward pass for batch normalization.

    For this implementation you should work out the derivatives for the batch
    normalizaton backward pass on paper and simplify as much as possible. You
    should be able to derive a simple expression for the backward pass. 
    See the jupyter notebook for more hints.
     
    Note: This implementation should expect to receive the same cache variable
    as batchnorm_backward, but might not use all of the values in the cache.

    Inputs / outputs: Same as batchnorm_backward
    """
    dx, dgamma, dbeta = None, None, None
    ###########################################################################
    # TODO: Implement the backward pass for batch normalization. Store the    #
    # results in the dx, dgamma, and dbeta variables.                         #
    #                                                                         #
    # After computing the gradient with respect to the centered inputs, you   #
    # should be able to compute gradients with respect to the inputs in a     #
    # single statement; our implementation fits on a single 80-character line.#
    ###########################################################################
    # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

    x = cache["x"]
    mean = cache["mean"]
    std = cache["std"] 
    u_out = cache["u_out"]
    gamma = cache["gamma"]
    N = len(x)
    
    #此处同上
    dbeta  = dout.sum(axis = 0)
    dgamma = (u_out*dout).sum(axis = 0)
    du_out = gamma*dout
    
    dx = du_out/std - du_out.sum(0)/(N*std) - np.sum(du_out*u_out/std, 0)*u_out/N 

    # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
    ###########################################################################
    #                             END OF YOUR CODE                            #
    ###########################################################################

    return dx, dgamma, dbeta

layernorm_forward

之前batchnorm使用的mean和var是cross样本的，layernorm使用的mean和var是cross特征的

def layernorm_forward(x, gamma, beta, ln_param):
    """
    Forward pass for layer normalization.

    During both training and test-time, the incoming data is normalized per data-point,
    before being scaled by gamma and beta parameters identical to that of batch normalization.
    
    Note that in contrast to batch normalization, the behavior during train and test-time for
    layer normalization are identical, and we do not need to keep track of running averages
    of any sort.

    Input:
    - x: Data of shape (N, D)
    - gamma: Scale parameter of shape (D,)
    - beta: Shift paremeter of shape (D,)
    - ln_param: Dictionary with the following keys:
        - eps: Constant for numeric stability

    Returns a tuple of:
    - out: of shape (N, D)
    - cache: A tuple of values needed in the backward pass
    """
    out, cache = None, None
    eps = ln_param.get('eps', 1e-5)
    ###########################################################################
    # TODO: Implement the training-time forward pass for layer norm.          #
    # Normalize the incoming data, and scale and  shift the normalized data   #
    #  using gamma and beta.                                                  #
    # HINT: this can be done by slightly modifying your training-time         #
    # implementation of  batch normalization, and inserting a line or two of  #
    # well-placed code. In particular, can you think of any matrix            #
    # transformations you could perform, that would enable you to copy over   #
    # the batch norm code and leave it almost unchanged?                      #
    ###########################################################################
    # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

    mean = x.mean(axis = 1, keepdims=True) #保持二维
    var = x.var(axis = 1, keepdims=True)
    std = np.sqrt(var + eps)
    u_out = (x-mean)/std 
    out = gamma*u_out + beta
    
    cache = {'x':x, 'gamma':gamma, 'mean':mean,
                 'std':std, 'u_out':u_out, 'out':out}

    # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
    ###########################################################################
    #                             END OF YOUR CODE                            #
    ###########################################################################
    return out, cache

layernorm_backward

反向传播，公式化推导同batchnorm的反向传播

def layernorm_backward(dout, cache):
    """
    Backward pass for layer normalization.

    For this implementation, you can heavily rely on the work you've done already
    for batch normalization.

    Inputs:
    - dout: Upstream derivatives, of shape (N, D)
    - cache: Variable of intermediates from layernorm_forward.

    Returns a tuple of:
    - dx: Gradient with respect to inputs x, of shape (N, D)
    - dgamma: Gradient with respect to scale parameter gamma, of shape (D,)
    - dbeta: Gradient with respect to shift parameter beta, of shape (D,)
    """
    dx, dgamma, dbeta = None, None, None
    ###########################################################################
    # TODO: Implement the backward pass for layer norm.                       #
    #                                                                         #
    # HINT: this can be done by slightly modifying your training-time         #
    # implementation of batch normalization. The hints to the forward pass    #
    # still apply!                                                            #
    ###########################################################################
    # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

    x = cache["x"]
    mean = cache["mean"]
    std = cache["std"] 
    u_out = cache["u_out"]
    gamma = cache["gamma"]
    _, D = x.shape
    
    dbeta  = dout.sum(axis = 0)
    dgamma = (u_out*dout).sum(axis = 0)
    du_out = gamma*dout
    
    dx = du_out/std - du_out.sum(axis=1, keepdims=True)/(D*std) - np.sum(du_out*u_out/std, axis=1, keepdims=True)*u_out/D

    # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
    ###########################################################################
    #                             END OF YOUR CODE                            #
    ###########################################################################
    return dx, dgamma, dbeta

dropout_forward

两种前向传播模式，train和test，train时，神经元有概率p不失活，test时，神经元不失活

def dropout_forward(x, dropout_param):
    """
    Performs the forward pass for (inverted) dropout.

    Inputs:
    - x: Input data, of any shape
    - dropout_param: A dictionary with the following keys:
      - p: Dropout parameter. We keep each neuron output with probability p.
      - mode: 'test' or 'train'. If the mode is train, then perform dropout;
        if the mode is test, then just return the input.
      - seed: Seed for the random number generator. Passing seed makes this
        function deterministic, which is needed for gradient checking but not
        in real networks.

    Outputs:
    - out: Array of the same shape as x.
    - cache: tuple (dropout_param, mask). In training mode, mask is the dropout
      mask that was used to multiply the input; in test mode, mask is None.

    NOTE: Please implement **inverted** dropout, not the vanilla version of dropout.
    See http://cs231n.github.io/neural-networks-2/#reg for more details.

    NOTE 2: Keep in mind that p is the probability of **keep** a neuron
    output; this might be contrary to some sources, where it is referred to
    as the probability of dropping a neuron output.
    """
    p, mode = dropout_param['p'], dropout_param['mode']
    if 'seed' in dropout_param:
        np.random.seed(dropout_param['seed'])

    mask = None
    out = None

    if mode == 'train':
        #######################################################################
        # TODO: Implement training phase forward pass for inverted dropout.   #
        # Store the dropout mask in the mask variable.                        #
        #######################################################################
        # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

        mask = (np.random.rand(*x.shape) < p)/p #这里除以p是为了在test的时候不用乘p作期望
        out = x * mask

        # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
        #######################################################################
        #                           END OF YOUR CODE                          #
        #######################################################################
    elif mode == 'test':
        #######################################################################
        # TODO: Implement the test phase forward pass for inverted dropout.   #uuu
        #######################################################################
        # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

        out = x

        # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
        #######################################################################
        #                            END OF YOUR CODE                         #
        #######################################################################

    cache = (dropout_param, mask)
    out = out.astype(x.dtype, copy=False)

    return out, cache

dropout_backward:

train模式时，反向传播时保留未失活的梯度

test模式时，由于不存在失活，直接返回梯度

def dropout_backward(dout, cache):
    """
    Perform the backward pass for (inverted) dropout.

    Inputs:
    - dout: Upstream derivatives, of any shape
    - cache: (dropout_param, mask) from dropout_forward.
    """
    dropout_param, mask = cache
    mode = dropout_param['mode']

    dx = None
    if mode == 'train':
        #######################################################################
        # TODO: Implement training phase backward pass for inverted dropout   #
        #######################################################################
        # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

        dx = dout*mask

        # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
        #######################################################################
        #                          END OF YOUR CODE                           #
        #######################################################################
    elif mode == 'test':
        dx = dout
    return dx

conv_forward_naive:

naive方式实现conv层的前向传播，即给输入x打上padding后，使用卷积核按照stride进行卷积

def conv_forward_naive(x, w, b, conv_param):
    """
    A naive implementation of the forward pass for a convolutional layer.

    The input consists of N data points, each with C channels, height H and
    width W. We convolve each input with F different filters, where each filter
    spans all C channels and has height HH and width WW.

    Input:
    - x: Input data of shape (N, C, H, W)
    - w: Filter weights of shape (F, C, HH, WW)
    - b: Biases, of shape (F,)
    - conv_param: A dictionary with the following keys:
      - 'stride': The number of pixels between adjacent receptive fields in the
        horizontal and vertical directions.
      - 'pad': The number of pixels that will be used to zero-pad the input. 
        

    During padding, 'pad' zeros should be placed symmetrically (i.e equally on both sides)
    along the height and width axes of the input. Be careful not to modfiy the original
    input x directly.

    Returns a tuple of:
    - out: Output data, of shape (N, F, H', W') where H' and W' are given by
      H' = 1 + (H + 2 * pad - HH) / stride
      W' = 1 + (W + 2 * pad - WW) / stride
    - cache: (x, w, b, conv_param)
    """
    out = None
    ###########################################################################
    # TODO: Implement the convolutional forward pass.                         #
    # Hint: you can use the function np.pad for padding.                      #
    ###########################################################################
    # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

    N, C, H, W = x.shape
    F, _, HH, WW = w.shape
    stride = conv_param['stride']
    pad = conv_param['pad']
    H_ = 1 + (H + 2 * pad - HH) // stride
    W_ = 1 + (W + 2 * pad - WW) // stride
    out = np.zeros((N, F, H_, W_))
    #第一维、第二维保持不变，第三维前面填充pad个，后面填充pad个，第四维同理，填充 方式为'constant'，即填充0 
    x_padded = np.pad(x, ((0,0),(0,0),(pad,pad),(pad,pad)), 'constant')
    
    for n in range(N):
      for f in range(F): #F个filter
        for h in range(H_):
          for i in range(W_):
            # 进行内积：n个样本，:全部通道
            out[n, f, h, i] = np.sum(x_padded[n, :, h*stride:h*stride+HH, i*stride:i*stride+WW] * w[f, ...]) + b[f]

    # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
    ###########################################################################
    #                             END OF YOUR CODE                            #
    ###########################################################################
    cache = (x, w, b, conv_param)
    return out, cache

conv_backward_naive:

上述前向传播相应的反向传播

第f个filter的偏置b需要累加n个样本的梯度

$\frac{\partial L}{\partial b} = \frac{\partial L}{\partial out}\frac{\partial out}{\partial b}$

第f个filter的权重W需要累加其扫过n个样本的所有卷积过的像素的梯度

$\frac{\partial L}{\partial W} = \frac{\partial L}{\partial out}\frac{\partial out}{\partial W}$

对x_padded的梯度同理，每被filter扫过一次，所经过的像素对应的梯度就要累加W

$\frac{\partial L}{\partial x} = \frac{\partial L}{\partial out}\frac{\partial out}{\partial x}$

最后对于x的梯度，我们截取x_padded来自于x的部分的对应梯度即可

def conv_backward_naive(dout, cache):
    """
    A naive implementation of the backward pass for a convolutional layer.

    Inputs:
    - dout: Upstream derivatives.
    - cache: A tuple of (x, w, b, conv_param) as in conv_forward_naive

    Returns a tuple of:
    - dx: Gradient with respect to x
    - dw: Gradient with respect to w
    - db: Gradient with respect to b
    """
    dx, dw, db = None, None, None
    ###########################################################################
    # TODO: Implement the convolutional backward pass.                        #
    ###########################################################################
    # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

    (x, w, b, conv_param) = cache
    N, C, H, W = x.shape
    F, _, HH, WW = w.shape
    stride = conv_param['stride']
    pad = conv_param['pad']
    H_ = 1 + (H + 2 * pad - HH) // stride
    W_ = 1 + (W + 2 * pad - WW) // stride
    x_padded = np.pad(x, ((0,0),(0,0),(pad,pad),(pad,pad)), 'constant')
    
    dx_padded = np.zeros_like(x_padded)
    dw = np.zeros_like(w)
    db = np.zeros_like(b)
    
    for n in range(N):
      for f in range(F):
        db[f] += np.sum(dout[n,f])
        for h in range(H_):
          for i in range(W_):
            dw[f] += dout[n, f, h, i]* x_padded[n, :, h*stride:h*stride+HH, i*stride:i*stride+WW]
            dx_padded[n, :, h*stride:h*stride+HH, i*stride:i*stride+WW] += w[f] * dout[n, f, h, i]
            
    dx = dx_padded[..., pad:pad+H, pad:pad+W]

max_pool_forward_naive:

max pool的前向传播的naive实现，即取max操作

def max_pool_forward_naive(x, pool_param):
    """
    A naive implementation of the forward pass for a max-pooling layer.

    Inputs:
    - x: Input data, of shape (N, C, H, W)
    - pool_param: dictionary with the following keys:
      - 'pool_height': The height of each pooling region
      - 'pool_width': The width of each pooling region
      - 'stride': The distance between adjacent pooling regions

    No padding is necessary here. Output size is given by 

    Returns a tuple of:
    - out: Output data, of shape (N, C, H', W') where H' and W' are given by
      H' = 1 + (H - pool_height) / stride
      W' = 1 + (W - pool_width) / stride
    - cache: (x, pool_param)
    """
    out = None
    ###########################################################################
    # TODO: Implement the max-pooling forward pass                            #
    ###########################################################################
    # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

    N, C, H, W = x.shape
    HH, WW, stride = pool_param['pool_height'], pool_param['pool_width'], pool_param['stride']
    H_ = 1 + (H - HH) // stride
    W_ = 1 + (W - WW) // stride
    
    out =np.zeros((N, C, H_, W_))
    for h in range(H_):
      for w in range(W_):
        out[..., h, w] = np.max(x[..., h*stride:h*stride+HH, w*stride: w*stride+WW], axis=(2,3))
    
    # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
    ###########################################################################
    #                             END OF YOUR CODE                            #
    ###########################################################################
    cache = (x, pool_param)
    return out, cache

max_pool_backward_naive:

max pool的反向传播，这里cache有原本的输入x，那么我们就不在前向传播计算掩码矩阵了，直接在反向传播这里用x计算保留梯度的index，out中的元素只会来源于唯一的x，所以这里不用累加

def max_pool_backward_naive(dout, cache):
    """
    A naive implementation of the backward pass for a max-pooling layer.

    Inputs:
    - dout: Upstream derivatives
    - cache: A tuple of (x, pool_param) as in the forward pass.

    Returns:
    - dx: Gradient with respect to x
    """
    dx = None
    ###########################################################################
    # TODO: Implement the max-pooling backward pass                           #
    ###########################################################################
    # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

    x, pool_param = cache
    N, C, H_, W_ = dout.shape
    HH, WW, stride = pool_param['pool_height'], pool_param['pool_width'], pool_param['stride']
    
    dx = np.zeros_like(x)
    for n in range(N):
        for c in range(C):
            for h in range(H_):
                for w in range(W_):
                    ind = np.argmax(x[n, c, h*stride:h*stride+HH, w*stride:w*stride+WW])
                    ind_ = np.unravel_index(ind, (HH, WW))
                    dx[n, c, h*stride+ind_[0], w*stride+ind_[1]] = dout[n, c, h, w]

    
    # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
    ###########################################################################
    #                             END OF YOUR CODE                            #
    ###########################################################################
    return dx

spatial_batchnorm_forward:

对输入的batch对每个通道C进行batch norm，因此要现将输入x重塑为(N*H*W, C)，然后直接调用之前写过的batchnorm_forward即可，调用完后，再重塑回4维尺寸

def spatial_batchnorm_forward(x, gamma, beta, bn_param):
    """
    Computes the forward pass for spatial batch normalization.

    Inputs:
    - x: Input data of shape (N, C, H, W)
    - gamma: Scale parameter, of shape (C,)
    - beta: Shift parameter, of shape (C,)
    - bn_param: Dictionary with the following keys:
      - mode: 'train' or 'test'; required
      - eps: Constant for numeric stability
      - momentum: Constant for running mean / variance. momentum=0 means that
        old information is discarded completely at every time step, while
        momentum=1 means that new information is never incorporated. The
        default of momentum=0.9 should work well in most situations.
      - running_mean: Array of shape (D,) giving running mean of features
      - running_var Array of shape (D,) giving running variance of features

    Returns a tuple of:
    - out: Output data, of shape (N, C, H, W)
    - cache: Values needed for the backward pass
    """
    out, cache = None, None

    ###########################################################################
    # TODO: Implement the forward pass for spatial batch normalization.       #
    #                                                                         #
    # HINT: You can implement spatial batch normalization by calling the      #
    # vanilla version of batch normalization you implemented above.           #
    # Your implementation should be very short; ours is less than five lines. #
    ###########################################################################
    # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

    N, C, H, W = x.shape
    x = x.transpose(0,2,3,1).reshape(N*H*W, C)
    out, cache = batchnorm_forward(x, gamma, beta, bn_param)
    out = out.reshape(N, H, W, C).transpose(0,3,1,2)

    # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
    ###########################################################################
    #                             END OF YOUR CODE                            #
    ###########################################################################

    return out, cache

spatial_batchnorm_backward

上述前向传播相应的反向传播过程

同理，对dout重塑后，调用batchnorm_backward，再将dx重塑即可

def spatial_batchnorm_backward(dout, cache):
    """
    Computes the backward pass for spatial batch normalization.

    Inputs:
    - dout: Upstream derivatives, of shape (N, C, H, W)
    - cache: Values from the forward pass

    Returns a tuple of:
    - dx: Gradient with respect to inputs, of shape (N, C, H, W)
    - dgamma: Gradient with respect to scale parameter, of shape (C,)
    - dbeta: Gradient with respect to shift parameter, of shape (C,)
    """
    dx, dgamma, dbeta = None, None, None

    ###########################################################################
    # TODO: Implement the backward pass for spatial batch normalization.      #
    #                                                                         #
    # HINT: You can implement spatial batch normalization by calling the      #
    # vanilla version of batch normalization you implemented above.           #
    # Your implementation should be very short; ours is less than five lines. #
    ###########################################################################
    # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

    N, C, H, W = dout.shape
    dout = dout.transpose(0,2,3,1).reshape(N*H*W, C)
    dx, dgamma, dbeta = batchnorm_backward_alt(dout, cache)
    dx = dx.reshape(N, H, W, C).transpose(0,3,1,2)

    # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
    ###########################################################################
    #                             END OF YOUR CODE                            #
    ###########################################################################

    return dx, dgamma, dbeta

spatial_groupnorm_forward

将通道分为G个组，每个组内独立进行归一化

def spatial_groupnorm_forward(x, gamma, beta, G, gn_param):
    """
    Computes the forward pass for spatial group normalization.
    In contrast to layer normalization, group normalization splits each entry 
    in the data into G contiguous pieces, which it then normalizes independently.
    Per feature shifting and scaling are then applied to the data, in a manner identical to that of batch normalization and layer normalization.

    Inputs:
    - x: Input data of shape (N, C, H, W)
    - gamma: Scale parameter, of shape (C,)
    - beta: Shift parameter, of shape (C,)
    - G: Integer mumber of groups to split into, should be a divisor of C
    - gn_param: Dictionary with the following keys:
      - eps: Constant for numeric stability

    Returns a tuple of:
    - out: Output data, of shape (N, C, H, W)
    - cache: Values needed for the backward pass
    """
    out, cache = None, None
    eps = gn_param.get('eps',1e-5)
    ###########################################################################
    # TODO: Implement the forward pass for spatial group normalization.       #
    # This will be extremely similar to the layer norm implementation.        #
    # In particular, think about how you could transform the matrix so that   #
    # the bulk of the code is similar to both train-time batch normalization  #
    # and layer normalization!                                                # 
    ###########################################################################
    # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

    N, C, H, W = x.shape
    size = (N*G, C//G *H*W)
    temp = x.reshape(size).T
    mean = np.mean(temp, 0)
    var = np.var(temp, 0) + eps
    std = np.sqrt(var)
    temp = (temp - mean)/std
    temp = temp.T.reshape(N, C, H, W)
    out = gamma * temp + beta
    cache = {'x':x, 'gamma':gamma, 'mean':mean,
                 'std':std, 'u_out':temp, 'size':size}

    # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
    ###########################################################################
    #                             END OF YOUR CODE                            #
    ###########################################################################
    return out, cache

spatial_groupnorm_backward:

上述前向传播对应的反向传播

def spatial_groupnorm_backward(dout, cache):
    """
    Computes the backward pass for spatial group normalization.

    Inputs:
    - dout: Upstream derivatives, of shape (N, C, H, W)
    - cache: Values from the forward pass

    Returns a tuple of:
    - dx: Gradient with respect to inputs, of shape (N, C, H, W)
    - dgamma: Gradient with respect to scale parameter, of shape (C,)
    - dbeta: Gradient with respect to shift parameter, of shape (C,)
    """
    dx, dgamma, dbeta = None, None, None

    ###########################################################################
    # TODO: Implement the backward pass for spatial group normalization.      #
    # This will be extremely similar to the layer norm implementation.        #
    ###########################################################################
    # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

    x = cache['x']
    mean = cache['mean']
    std = cache['std'] 
    u_out = cache['u_out']
    gamma = cache['gamma']
    size = cache['size']

    dbeta  = dout.sum(axis = (0,2,3), keepdims = True)
    dgamma = (u_out*dout).sum(axis = (0,2,3), keepdims = True)
    du_out = (gamma*dout).reshape(size).T
    u_out = u_out.reshape(size).T
    M = size[1]
    
    dx = du_out/std - du_out.sum(0)/(M*std) - np.sum(du_out*u_out/std, 0)*u_out/M 
    dx = dx.T.reshape(x.shape)

    # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
    ###########################################################################
    #                             END OF YOUR CODE                            #
    ###########################################################################
    return dx, dgamma, dbeta

fc_net.py

TwoLayerNet:

__init__

初始化权重以及偏置

    def __init__(self, input_dim=3*32*32, hidden_dim=100, num_classes=10,
                 weight_scale=1e-3, reg=0.0):
        """
        Initialize a new network.

        Inputs:
        - input_dim: An integer giving the size of the input
        - hidden_dim: An integer giving the size of the hidden layer
        - num_classes: An integer giving the number of classes to classify
        - weight_scale: Scalar giving the standard deviation for random
          initialization of the weights.
        - reg: Scalar giving L2 regularization strength.
        """
        self.params = {}
        self.reg = reg

        ############################################################################
        # TODO: Initialize the weights and biases of the two-layer net. Weights    #
        # should be initialized from a Gaussian centered at 0.0 with               #
        # standard deviation equal to weight_scale, and biases should be           #
        # initialized to zero. All weights and biases should be stored in the      #
        # dictionary self.params, with first layer weights                         #
        # and biases using the keys 'W1' and 'b1' and second layer                 #
        # weights and biases using the keys 'W2' and 'b2'.                         #
        ############################################################################
        # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

        self.params['W1'] = np.random.randn(input_dim, hidden_dim)*weight_scale
        self.params['W2'] = np.random.randn(hidden_dim, num_classes)*weight_scale
        self.params['b1'] = np.zeros(hidden_dim)
        self.params['b2'] = np.zeros(num_classes)

        # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
        ############################################################################
        #                             END OF YOUR CODE                             #
        ############################################################################

loss:

这里直接使用我们在上面封装好的api即可

    def loss(self, X, y=None):
        """
        Compute loss and gradient for a minibatch of data.

        Inputs:
        - X: Array of input data of shape (N, d_1, ..., d_k)
        - y: Array of labels, of shape (N,). y[i] gives the label for X[i].

        Returns:
        If y is None, then run a test-time forward pass of the model and return:
        - scores: Array of shape (N, C) giving classification scores, where
          scores[i, c] is the classification score for X[i] and class c.

        If y is not None, then run a training-time forward and backward pass and
        return a tuple of:
        - loss: Scalar value giving the loss
        - grads: Dictionary with the same keys as self.params, mapping parameter
          names to gradients of the loss with respect to those parameters.
        """
        scores = None
        ############################################################################
        # TODO: Implement the forward pass for the two-layer net, computing the    #
        # class scores for X and storing them in the scores variable.              #
        ############################################################################
        # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

        h, cache_h = affine_forward(X, self.params['W1'], self.params['b1'])
        h_act, cache_act = relu_forward(h)
        scores, cache_out = affine_forward(h_act, self.params['W2'], self.params['b2'])

        # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
        ############################################################################
        #                             END OF YOUR CODE                             #
        ############################################################################

        # If y is None then we are in test mode so just return scores
        if y is None:
            return scores

        loss, grads = 0, {}
        ############################################################################
        # TODO: Implement the backward pass for the two-layer net. Store the loss  #
        # in the loss variable and gradients in the grads dictionary. Compute data #
        # loss using softmax, and make sure that grads[k] holds the gradients for  #
        # self.params[k]. Don't forget to add L2 regularization!                   #
        #                                                                          #
        # NOTE: To ensure that your implementation matches ours and you pass the   #
        # automated tests, make sure that your L2 regularization includes a factor #
        # of 0.5 to simplify the expression for the gradient.                      #
        ############################################################################
        # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

        """stable_scores = np.exp(scores-np.max(scores,axis=1).reshape(-1,1))
        total = np.sum(stable_scores,1).reshape(-1,1)
        loss = np.sum(-np.log(stable_scores[range(n),y]/total))/n
        loss+= 0.5*self.reg*(np.sum(np.square(self.params['W1'])) + np.sum(np.square(self.params['W2'])))"""
        
        loss, grad_loss = softmax_loss(scores, y)
        loss+= 0.5*self.reg*(np.sum(np.square(self.params['W1'])) + np.sum(np.square(self.params['W2'])))
        grad_act, grads['W2'], grads['b2'] = affine_backward(grad_loss, cache_out)
        grads['W2'] += self.reg*self.params['W2'] # 正则化梯度
        grad_h = relu_backward(grad_act, cache_act)
        grad_x, grads['W1'], grads['b1'] = affine_backward(grad_h, cache_h)
        grads['W1'] += self.reg*self.params['W1']

        # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
        ############################################################################
        #                             END OF YOUR CODE                             #
        ############################################################################

        return loss, grads

使用Sovler API来训练我们的TwoLayerNet

model = TwoLayerNet()
solver = None

##############################################################################
# TODO: Use a Solver instance to train a TwoLayerNet that achieves at least  #
# 50% accuracy on the validation set.                                        #
##############################################################################
# *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

model = TwoLayerNet(hidden_dim=100)
solver = Solver(model, data, optim_config={'learning_rate': 1e-3},
                batch_size=200,lr_decay=0.95, num_epochs=10, 
                print_every=200)
solver.train()

# *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
##############################################################################
#                             END OF YOUR CODE                               #
##############################################################################

训练的loss以及acc可视化：

FullyConnectedNet:

class FullyConnectedNet(object):
    """
    A fully-connected neural network with an arbitrary number of hidden layers,
    ReLU nonlinearities, and a softmax loss function. This will also implement
    dropout and batch/layer normalization as options. For a network with L layers,
    the architecture will be

    {affine - [batch/layer norm] - relu - [dropout]} x (L - 1) - affine - softmax

    where batch/layer normalization and dropout are optional, and the {...} block is
    repeated L - 1 times.

    Similar to the TwoLayerNet above, learnable parameters are stored in the
    self.params dictionary and will be learned using the Solver class.
    """

    def __init__(self, hidden_dims, input_dim=3*32*32, num_classes=10,
                 dropout=1, normalization=None, reg=0.0,
                 weight_scale=1e-2, dtype=np.float32, seed=None):
        """
        Initialize a new FullyConnectedNet.

        Inputs:
        - hidden_dims: A list of integers giving the size of each hidden layer.
        - input_dim: An integer giving the size of the input.
        - num_classes: An integer giving the number of classes to classify.
        - dropout: Scalar between 0 and 1 giving dropout strength. If dropout=1 then
          the network should not use dropout at all.
        - normalization: What type of normalization the network should use. Valid values
          are "batchnorm", "layernorm", or None for no normalization (the default).
        - reg: Scalar giving L2 regularization strength.
        - weight_scale: Scalar giving the standard deviation for random
          initialization of the weights.
        - dtype: A numpy datatype object; all computations will be performed using
          this datatype. float32 is faster but less accurate, so you should use
          float64 for numeric gradient checking.
        - seed: If not None, then pass this random seed to the dropout layers. This
          will make the dropout layers deteriminstic so we can gradient check the
          model.
        """
        self.normalization = normalization
        self.use_dropout = dropout != 1
        self.reg = reg
        self.num_layers = 1 + len(hidden_dims)
        self.dtype = dtype
        self.params = {}

        ############################################################################
        # TODO: Initialize the parameters of the network, storing all values in    #
        # the self.params dictionary. Store weights and biases for the first layer #
        # in W1 and b1; for the second layer use W2 and b2, etc. Weights should be #
        # initialized from a normal distribution centered at 0 with standard       #
        # deviation equal to weight_scale. Biases should be initialized to zero.   #
        #                                                                          #
        # When using batch normalization, store scale and shift parameters for the #
        # first layer in gamma1 and beta1; for the second layer use gamma2 and     #
        # beta2, etc. Scale parameters should be initialized to ones and shift     #
        # parameters should be initialized to zeros.                               #
        ############################################################################
        # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

        h = [input_dim] + hidden_dims + [num_classes]
        for i in range(self.num_layers):
          self.params['W'+str(i+1)] = np.random.randn(h[i], h[i+1])*weight_scale
          self.params['b'+str(i+1)] = np.zeros(h[i+1])
        
        if self.normalization != None:
          for i in range(self.num_layers-1):
            self.params['gamma'+str(i+1)] = np.ones(h[i+1])
            self.params['beta' +str(i+1)] = np.zeros(h[i+1])

        # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
        ############################################################################
        #                             END OF YOUR CODE                             #
        ############################################################################

        # When using dropout we need to pass a dropout_param dictionary to each
        # dropout layer so that the layer knows the dropout probability and the mode
        # (train / test). You can pass the same dropout_param to each dropout layer.
        self.dropout_param = {}
        if self.use_dropout:
            self.dropout_param = {'mode': 'train', 'p': dropout}
            if seed is not None:
                self.dropout_param['seed'] = seed

        # With batch normalization we need to keep track of running means and
        # variances, so we need to pass a special bn_param object to each batch
        # normalization layer. You should pass self.bn_params[0] to the forward pass
        # of the first batch normalization layer, self.bn_params[1] to the forward
        # pass of the second batch normalization layer, etc.
        self.bn_params = []
        if self.normalization=='batchnorm':
            self.bn_params = [{'mode': 'train'} for i in range(self.num_layers - 1)]
        if self.normalization=='layernorm':
            self.bn_params = [{} for i in range(self.num_layers - 1)]

        # Cast all parameters to the correct datatype
        for k, v in self.params.items():
            self.params[k] = v.astype(dtype)


    def loss(self, X, y=None):
        """
        Compute loss and gradient for the fully-connected net.

        Input / output: Same as TwoLayerNet above.
        """
        X = X.astype(self.dtype)
        mode = 'test' if y is None else 'train'

        # Set train/test mode for batchnorm params and dropout param since they
        # behave differently during training and testing.
        if self.use_dropout:
            self.dropout_param['mode'] = mode
        if self.normalization=='batchnorm':
            for bn_param in self.bn_params:
                bn_param['mode'] = mode
        scores = None
        ############################################################################
        # TODO: Implement the forward pass for the fully-connected net, computing  #
        # the class scores for X and storing them in the scores variable.          #
        #                                                                          #
        # When using dropout, you'll need to pass self.dropout_param to each       #
        # dropout forward pass.                                                    #
        #                                                                          #
        # When using batch normalization, you'll need to pass self.bn_params[0] to #
        # the forward pass for the first batch normalization layer, pass           #
        # self.bn_params[1] to the forward pass for the second batch normalization #
        # layer, etc.                                                              #
        ############################################################################
        # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

        cache = {}
        out = X
        for i in range(self.num_layers-1):
          out, cache[str(i+1)] = affine_forward(out, self.params['W'+str(i+1)], 
                                                self.params['b'+str(i+1)])
          if(self.normalization == 'batchnorm'):
            out, cache[str(i+1)+'_bn' ] = batchnorm_forward(out, self.params['gamma'+str(i+1)], self.params['beta'+str(i+1)], self.bn_params[i])
          if self.normalization=='layernorm':
            in_, cache[str(i+1)+'_ln'] = layernorm_forward(out, self.params['gamma'+str(i+1)], self.params['beta'+str(i+1)], self.bn_params[i])
          out, cache[str(i+1)+'_act'] = relu_forward(out)
          if self.use_dropout:
            out, cache[str(i+1)+'_dropout'] = dropout_forward(out,  self.dropout_param)
        out, cache[str(self.num_layers)] = affine_forward(out, self.params['W'+str(self.num_layers)], self.params['b'+str(self.num_layers)])
        
        scores = out    

        # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
        ############################################################################
        #                             END OF YOUR CODE                             #
        ############################################################################

        # If test mode return early
        if mode == 'test':
            return scores

        loss, grads = 0.0, {}
        ############################################################################
        # TODO: Implement the backward pass for the fully-connected net. Store the #
        # loss in the loss variable and gradients in the grads dictionary. Compute #
        # data loss using softmax, and make sure that grads[k] holds the gradients #
        # for self.params[k]. Don't forget to add L2 regularization!               #
        #                                                                          #
        # When using batch/layer normalization, you don't need to regularize the scale   #
        # and shift parameters.                                                    #
        #                                                                          #
        # NOTE: To ensure that your implementation matches ours and you pass the   #
        # automated tests, make sure that your L2 regularization includes a factor #
        # of 0.5 to simplify the expression for the gradient.                      #
        ############################################################################
        # *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****

        loss, dout = softmax_loss(scores, y)
        for i in range(self.num_layers):
          loss+= 0.5*self.reg*np.sum(np.square(self.params['W'+str(i+1)]))
          
        dout, dw, db = affine_backward(dout, cache[str(self.num_layers)])
        grads['W' + str(self.num_layers)] = dw + self.reg * self.params['W' + str(self.num_layers)]
        grads['b' + str(self.num_layers)] = db
        for i in range(self.num_layers-1, 0, -1):
          if(self.use_dropout):
            dout = dropout_backward(dout, cache[str(i)+'_dropout'])
          dout = relu_backward(dout, cache[str(i)+'_act'])
          if(self.normalization=='batchnorm'):
            dout, grads['gamma'+str(i)], grads['beta'+str(i)] = batchnorm_backward_alt(dout, cache[str(i)+'_bn'])
          if(self.normalization=='layernorm'):
            dout, grads['gamma'+str(i)], grads['beta'+str(i)] = layernorm_backward(dout, cache[str(i)+'_ln'])
          dout, dw, grads['b'+str(i)] = affine_backward(dout, cache[str(i)])
          grads['W'+str(i)] = dw + self.reg*self.params['W'+str(i)]
          
          

        # *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
        ############################################################################
        #                             END OF YOUR CODE                             #
        ############################################################################

        return loss, grads

2048 AI社区

有“AI”的1024 = 2048，欢迎大家加入2048 AI社区

更多推荐

视觉 AI界面控件自动化测试：TestComplete 能测什么，不能测什么

2048 AI社区

在 bisheng 平台搭建「主-子智能体」协同：弱模型适配、执行效率与问题记录

在Bisheng平台构建“主-子智能体”协同系统，通过将智能决策与确定性执行分离，实现弱模型下的高效稳定运行。核心策略包括：模型仅负责判断，复杂逻辑由Java引擎统一处理；长文本通过落盘文件传递，聊天链路仅保留路径引用；严格适配ReAct协议约束，避免解析失败与死循环。最终达成总智能体规划驱动、子智能体调用工具执行的闭环流程，部署方案定稿为deploy.json，全程可追溯、可复现，崩溃面被设计性