neograd.autograd.ops package

Submodules

neograd.autograd.ops.basics module

class neograd.autograd.ops.basics.Add[source]

Bases: Operation

Element wise addition between two Tensors or Tensor-like

backward(tens1, tens2)[source]

Sets grad_fn of operands

Local gradient is an identity matrix, that should be dotted with the upper gradient which results in upper gradient

Parameters

  • tens1 (Tensor) – First operand

  • tens2 (Tensor) – Second operand

forward(tens1, tens2)[source]

Calculates element wise addition

Parameters

  • tens1 (Tensor or int or float or list or np.ndarray) – First operand

  • tens2 (Tensor or int or float or list or np.ndarray) – Second operand

Returns

Tensor of the result

class neograd.autograd.ops.basics.Div[source]

Bases: Operation

Element wise division between two Tensors or Tensor-like

backward(tens1, tens2)[source]

Sets grad_fn of operands

Local gradient of tens1 is 1/tens2.data, local gradient of tens2 is -1*tens1.data/tens2.data^2, which is element wise multiplied with upper gradient

Parameters

  • tens1 (Tensor) – First operand

  • tens2 (Tensor) – Second operand

forward(tens1, tens2)[source]

Calculates element wise division

Parameters

  • tens1 (Tensor or int or float or list or np.ndarray) – First operand

  • tens2 (Tensor or int or float or list or np.ndarray) – Second operand

Returns

Tensor of the result

class neograd.autograd.ops.basics.Dot[source]

Bases: Operation

Dot product between two Tensors or Tensor-like

backward(tens1, tens2)[source]

Sets grad_fn of operands

Local gradient of tens1 is transpose of tens2.data, local gradient of tens2 is transpose of tens1.data, which is dotted with upper gradient

Parameters

  • tens1 (Tensor) – First operand

  • tens2 (Tensor) – Second operand

forward(tens1, tens2)[source]

Calculates dot product

Parameters

  • tens1 (Tensor or int or float or list or np.ndarray) – First operand

  • tens2 (Tensor or int or float or list or np.ndarray) – Second operand

Returns

Tensor of the result

class neograd.autograd.ops.basics.Exp[source]

Bases: Operation

Exponentiates the Tensor or Tensor-like

backward(tens)[source]

Sets grad_fn of operand

Local gradient is exponentiation of tens.data itself

Parameters

tens (Tensor or int or float or list or np.ndarray) – Operand

forward(tens)[source]

Calculates exponentiation

Parameters

tens (Tensor or int or float or list or np.ndarray) – Operand

Returns

Tensor of the result

class neograd.autograd.ops.basics.Flatten[source]

Bases: Operation

Performs flattening of Tensor or Tensor-like

backward(tens)[source]

Sets grad_fn of operand

No local gradient, upper gradient is reshaped to original shape

Parameters

tens (Tensor or int or float or list or np.ndarray) – Operand

forward(tens)[source]

Performs flattening from any dimension to 1D

Parameters

tens (Tensor or int or float or list or np.ndarray) – Operand

Returns

Tensor of the result

class neograd.autograd.ops.basics.Log[source]

Bases: Operation

Natural Logarithm of the Tensor or Tensor-like

backward(tens)[source]

Sets grad_fn of operand

Local gradient is exponentiation of 1/tens.data itself

Parameters

tens (Tensor or int or float or list or np.ndarray) – Operand

forward(tens)[source]

Calculates natural logarithm

Parameters

tens (Tensor or int or float or list or np.ndarray) – Operand

Returns

Tensor of the result

class neograd.autograd.ops.basics.Mul[source]

Bases: Operation

Element wise multiplication between two Tensors or Tensor-like

backward(tens1, tens2)[source]

Sets grad_fn of operands

Local gradient for each Tensor is the other Tensor’s data, which is element-wise multiplied with upper gradient

Parameters

  • tens1 (Tensor) – First operand

  • tens2 (Tensor) – Second operand

forward(tens1, tens2)[source]

Calculates element wise multiplication

Parameters

  • tens1 (Tensor or int or float or list or np.ndarray) – First operand

  • tens2 (Tensor or int or float or list or np.ndarray) – Second operand

Returns

Tensor of the result

class neograd.autograd.ops.basics.Pow[source]

Bases: Operation

Raises one Tensor or Tensor-like to the power of another Tensor or Tensor-like

backward(tens1, tens2)[source]

Sets grad_fn of operands

Local gradient of tens1 is tens1.data^(tens2.data-1), local gradient of tens2 is log(tens1.data), which is element wise multiplied with upper gradient

Parameters

  • tens1 (Tensor) – First operand

  • tens2 (Tensor) – Second operand

forward(tens1, tens2)[source]

Calculates raising to a power

Parameters

  • tens1 (Tensor or int or float or list or np.ndarray) – First operand

  • tens2 (Tensor or int or float or list or np.ndarray) – Second operand

Returns

Tensor of the result

class neograd.autograd.ops.basics.Reshape[source]

Bases: Operation

Performs reshaping of Tensor or Tensor-like

backward(tens)[source]

Sets grad_fn of operand

No local gradient, upper gradient is reshaped to original shape

Parameters

tens (Tensor or int or float or list or np.ndarray) – Operand

forward(tens, new_shape)[source]

Performs reshaping of Tensor to a new shape

Parameters

  • tens (Tensor or int or float or list or np.ndarray) – Operand

  • new_shape (tuple) – New shape to be reshaped into

Returns

Tensor of the result

class neograd.autograd.ops.basics.Sub[source]

Bases: Operation

Element wise subtraction between two Tensors or Tensor-like

backward(tens1, tens2)[source]

Sets grad_fn of operands

Local gradient is an identity matrix, that should be dotted with the upper gradient which results in upper gradient, for the other one local gradient is a negative identity matrix which results in negative upper gradient

Parameters

  • tens1 (Tensor) – First operand

  • tens2 (Tensor) – Second operand

forward(tens1, tens2)[source]

Calculates element wise subtraction

Parameters

  • tens1 (Tensor or int or float or list or np.ndarray) – First operand

  • tens2 (Tensor or int or float or list or np.ndarray) – Second operand

Returns

Tensor of the result

class neograd.autograd.ops.basics.Sum(axis=None)[source]

Bases: Operation

Performs sum along a specified axis

If axis is None, then the sum of the entire Tensor is calculated

Parameters

axis (None or int or tuple of int) – Axis along which it should be summed

backward(tens)[source]

Sets grad_fn of operand

Local gradient is all ones and the upper gradient must be added a new axis along the axis attribute if axis is not None, for broadcasting of upper_gradient as during forward pass the dimension will be reduced along the axis it is summed

Parameters

tens (Tensor or int or float or list or np.ndarray) – Operand

forward(tens)[source]

Calculates sum along an axis

Parameters

tens (Tensor or int or float or list or np.ndarray) – Operand

Returns

Tensor of the result

class neograd.autograd.ops.basics.Transpose[source]

Bases: Operation

Performs transpose of Tensor or Tensor-like

backward(tens)[source]

Sets grad_fn of operand

No local gradient, upper gradient is just transposed

Parameters

tens (Tensor or int or float or list or np.ndarray) – Operand

forward(tens)[source]

Performs transpose

Parameters

tens (Tensor or int or float or list or np.ndarray) – Operand

Returns

Tensor of the result

neograd.autograd.ops.basics.add(tens1, tens2)[source]

Abstraction for Add.forward

Parameters

  • tens1 (Tensor or int or float or list or np.ndarray) – First operand

  • tens2 (Tensor or int or float or list or np.ndarray) – Second operand

Returns

Tensor of the result

neograd.autograd.ops.basics.div(tens1, tens2)[source]

Abstraction for Div.forward

Parameters

  • tens1 (Tensor or int or float or list or np.ndarray) – First operand

  • tens2 (Tensor or int or float or list or np.ndarray) – Second operand

Returns

Tensor of the result

neograd.autograd.ops.basics.dot(tens1, tens2)[source]

Abstraction for Dot.forward

Parameters

  • tens1 (Tensor or int or float or list or np.ndarray) – First operand

  • tens2 (Tensor or int or float or list or np.ndarray) – Second operand

Returns

Tensor of the result

neograd.autograd.ops.basics.exp(tens)[source]

Abstraction for Exp.forward

Parameters

tens (Tensor) – Operand

Returns

Tensor of the result

neograd.autograd.ops.basics.flatten(tens)[source]

Abstraction for Flatten.forward

Parameters

tens (Tensor) – Operand

Returns

Tensor of the result

neograd.autograd.ops.basics.log(tens)[source]

Abstraction for Log.forward

Parameters

tens (Tensor) – Operand

Returns

Tensor of the result

neograd.autograd.ops.basics.mul(tens1, tens2)[source]

Abstraction for Mul.forward

Parameters

  • tens1 (Tensor or int or float or list or np.ndarray) – First operand

  • tens2 (Tensor or int or float or list or np.ndarray) – Second operand

Returns

Tensor of the result

neograd.autograd.ops.basics.pow(tens1, tens2)[source]

Abstraction for Pow.forward

Parameters

  • tens1 (Tensor or int or float or list or np.ndarray) – First operand

  • tens2 (Tensor or int or float or list or np.ndarray) – Second operand

Returns

Tensor of the result

neograd.autograd.ops.basics.reshape(tens, new_shape)[source]

Abstraction for Reshape.forward

Parameters

tens (Tensor) – Operand

Returns

Tensor of the result

neograd.autograd.ops.basics.sub(tens1, tens2)[source]

Abstraction for Sub.forward

Parameters

  • tens1 (Tensor or int or float or list or np.ndarray) – First operand

  • tens2 (Tensor or int or float or list or np.ndarray) – Second operand

Returns

Tensor of the result

neograd.autograd.ops.basics.sum(tens, axis=None)[source]

Abstraction for Sum.forward

Parameters

  • tens (Tensor) – Operand

  • axis (None or int or tuple of int) – Axis along which it should be summed Defaults to None

Returns

Tensor of the result

neograd.autograd.ops.basics.transpose(tens)[source]

Abstraction for Transpose.forward

Parameters

tens (Tensor) – Operand

Returns

Tensor of the result

neograd.autograd.ops.conv module

class neograd.autograd.ops.conv.Conv[source]

Bases: object

Base class for Convolution and Pooling operations

Parameters

  • padding (int) – Padding value to be applied

  • stride (int) – Stride to be taken

fragment_iterator(padded_inputs, kernel_shape, *args)[source]

Zips the fragments with any other args

Parameters

  • padded_inputs (np.ndarray) – Inputs that are padded

  • kernel_shape (tuple) – Shape of the kernel

  • *args – Any other objects that should be zipped together, this is usually the upper gradient chunks that are the result of the convolution

Returns

zip of fragments and ony other objects in args

generate_fragments(padded_data, kernel_shape)[source]

Generates fragments of data

Takes the data and slices it to the shape of kernel_shape in x and y axis (the last two dims), including all the other dimensions, effectively taking a chunk/fragment of the data.

Each time a chunk is taken, the x index and y index are incremented by the stride

Parameters

  • padded_data (np.ndarray) – Data that is already padded, to be convolved on

  • kernel_shape (tuple) – Shape of kernel to be convolved with

Yields

sliced inputs_data(fragment), row slice(start and end indices of fragment among rows) and column slice(start and end indices of fragment among columns)

Raises

  • AssertionError – if stride isn’t greater than or equal to 1

  • AssertionError – if padding isn’t greater than or equal to 0

get_result_shape(inputs_shape, kernel_shape)[source]

Calculates the x and y dimensions of result of convolution

Takes the inputs_shape and kernel_shape and returns the x and y dims of the result of convolving inputs with the kernel as a 2D convolution

Parameters

  • inputs_shape (tuple) – Shape of inputs

  • kernel_shape (tuple) – Shape of kernel

Returns

tuple of x and y dims of result

pad(data)[source]

Pads the data in x and y dimensions (last two dims)

Only the last two dimensions are padded rest aren’t padded(padded with 0, has no effect)

Parameters

data (np.ndarray) – Data to be padded

Returns

data that is padded

unpad(padded_data)[source]

Unpads the padded data

Slices the padded_data in last two dimensions where it is padded, by rejecting the padding and only extracting the original data

Parameters

padded_data (np.ndarray) – Data that needs to be unpadded

Returns

Unpadded data

class neograd.autograd.ops.conv.Conv2D(padding, stride)[source]

Bases: Operation, Conv

Implements 2D convolution

2D convolution where the inputs has only 1 channel and its shape is of the form (num_examples, x_dim, y_dim) is convolved with a 2D kernel

backward(inputs, kernel, bias)[source]

Sets the grad_fn of inputs, kernel and bias

Since each convolution of a fragment with kernel results in a Tensor, the corresponding upper gradient values of all the examples are taken.

inputs_grads are initialized to zero of shape of padded_inputs, then the corresponding gradient that is calculated is tucked into the inputs_grads based on the row slice and the column slice. The inputs_grads are then unpadded to reject the gradients of pads and only keeps the gradients of the original inputs

kernel_grads are intitialized to zero and since they are used multiple times, its gradient is added each time

Since bias is added to all outputs, its gradient is just the sum of the upper gradient

Parameters

  • inputs (Tensor) – Tensor that is convolved on

  • kernel (Tensor) – Tensor that is convolved with(weights)

  • bias (Tensor) – bias value

forward(inputs, kernel, bias)[source]

Implements the forward pass

Each fragment of shape (num_examples, kernel_shape[0], kernel_shape[1]) is element wise multipled with the kernel and then summed along its x and y axis and then adds it with the bias

Parameters

  • inputs (Tensor or int or float or list or np.ndarray) – Tensor to be convolved on

  • kernel (Tensor or int or float or list or np.ndarray) – Tensor to be convolved with(weights)

  • bias (Tensor or int or float or list or np.ndarray) – bias value

Returns

Tensor of the result that is convolved

validate_inputs(inputs)[source]

Validates the inputs

Parameters

inputs (Tensor or np.ndarray) – Data to be validated

Raises

ValueError – If inputs aren’t of format (num_examples, x_dim, y_dim)

class neograd.autograd.ops.conv.Conv3D(padding, stride)[source]

Bases: Operation, Conv

Implements 3D convolution

3D convolution over a colume where the inputs has multiple channels and its shape is of the form (num_examples, num_channels, x_dim, y_dim) is convolved with a 3D kernel of shape (num_channels, kernel_shape[0], kernel_shape[1])

backward(inputs, kernel, bias)[source]

Sets the grad_fn of inputs, kernel and bias

Since each convolution of a fragment with kernel results in a Tensor, the corresponding upper gradient values of all the examples, across all channels are taken.

To calculate sum_grad which is the gradient of the sum operation during forward pass, the fragment needs to be expanded along first axis to allow for broadcasting of upper gradient slice.

inputs_grads are initialized to zero of shape of padded_inputs, then the corresponding gradient that is calculated is tucked into the inputs_grads based on the row slice and the column slice. The inputs_grads are then unpadded to reject the gradients of pads and only keeps the gradients of the original inputs

kernel_grads are intitialized to zero and since they are used multiple times, its gradient is added each time

Since bias is a vector here and is not added to all the outputs, it is only summed across all the examples and the first and second axis

Parameters

  • inputs (Tensor) – Tensor that is convolved on

  • kernel (Tensor) – Tensor that is convolved with(weights)

  • bias (Tensor) – bias value

forward(inputs, kernel, bias)[source]

Implements the forward pass

Each fragment of shape (num_examples, num_channels, kernel_shape[0], kernel_shape[1]) is element wise multipled with the kernel and then summed along its x, y and z axis and then adds it with the bias

Parameters

  • inputs (Tensor or int or float or list or np.ndarray) – Tensor to be convolved on

  • kernel (Tensor or int or float or list or np.ndarray) – Tensor to be convolved with(weights)

  • bias (Tensor or int or float or list or np.ndarray) – bias value

Returns

Tensor of the result that is convolved

validate_inputs(inputs)[source]

Validates the inputs

Parameters

inputs (Tensor or np.ndarray) – Data to be validated

Raises

ValueError – If inputs aren’t of format (num_examples, num_channels, x_dim, y_dim)

class neograd.autograd.ops.conv.MaxPool2D(kernel_shape, padding, stride)[source]

Bases: Operation, Conv

Implements 2D MaxPooling

In MaxPooling, it is differentiable only if kernel_shape along with the stride covers the entire input if not it is not differentiable and fails gradient checking

Parameters

kernel_shape (tuple) – Shape of the kernel

backward(inputs)[source]

Sets the grad_fn of inputs

Since argmax operates only on one axis, the fragment is first flattened across x and y dims (last two dims) Then the one hot encoding of the max indices are taken that are the multiplied with the corresponding upper grad slice fragment_grad is reshaped to original fragment shape

Parameters

inputs (Tensor) – Tensor that is maxpooled

forward(inputs)[source]

Forward pass of max pooling 2d

The fragments are generated, for each of it, the maximum value in the x and y dims is returned for all examples

Parameters

inputs (Tensor or int or float or list or np.ndarray) – Data to be maxpooled

Returns

Tensor of the result

validate_inputs(inputs)[source]

Validates the inputs

Parameters

inputs (Tensor or np.ndarray) – Data to be validated

Raises

ValueError – If inputs aren’t of format (num_examples, x_dim, y_dim)

class neograd.autograd.ops.conv.MaxPool3D(kernel_shape, padding, stride)[source]

Bases: Operation, Conv

Implements 3D MaxPooling

In MaxPooling, it is differentiable only if kernel_shape along with the stride covers the entire input if not it is not differentiable and fails gradient checking

Parameters

kernel_shape (tuple) – Shape of the kernel

backward(inputs)[source]

Sets the grad_fn of inputs

Since argmax operates only on one axis, the fragment is first flattened across x and y dims (last two dims) Then the one hot encoding of the max indices are taken that are the multiplied with the corresponding upper grad slice fragment_grad is reshaped to original fragment shape

Parameters

inputs (Tensor) – Tensor that is maxpooled

forward(inputs)[source]

Forward pass of max pooling 3d

The fragments are generated, for each of it, the maximum value in the x and y dims is returned for all examples across all channels

Parameters

inputs (Tensor or int or float or list or np.ndarray) – Data to be maxpooled

Returns

Tensor of the result

validate_inputs(inputs)[source]

Validates the inputs

Parameters

inputs (Tensor or np.ndarray) – Data to be validated

Raises

ValueError – If inputs aren’t of format (num_examples, num_channels, x_dim, y_dim)

neograd.autograd.ops.conv.conv2d(inputs, kernel, bias, padding, stride)[source]

Abstraction of Conv2D.forward

Parameters

  • inputs (Tensor or int or float or list or np.ndarray) – Tensor to be convolved on

  • kernel (Tensor or int or float or list or np.ndarray) – Tensor to be convolved with(weights)

  • bias (Tensor or int or float or list or np.ndarray) – bias value

  • padding (int) – Padding value to be applied

  • stride (int) – Stride to be taken

Returns

Tensor of the result

neograd.autograd.ops.conv.conv3d(inputs, kernel, bias, padding, stride)[source]

Abstraction of Conv3D.forward

Parameters

  • inputs (Tensor or int or float or list or np.ndarray) – Tensor to be convolved on

  • kernel (Tensor or int or float or list or np.ndarray) – Tensor to be convolved with(weights)

  • bias (Tensor or int or float or list or np.ndarray) – bias value

  • padding (int) – Padding value to be applied

  • stride (int) – Stride to be taken

Returns

Tensor of the result

neograd.autograd.ops.conv.maxpool2d(inputs, kernel_shape, padding, stride)[source]

Abstraction of MaxPool2D.forward

Parameters

  • inputs (Tensor or int or float or list or np.ndarray) – Tensor to be convolved on

  • kernel_shape (tuple) – Shape of the kernel

  • padding (int) – Padding value to be applied

  • stride (int) – Stride to be taken

Returns

Tensor of the result

neograd.autograd.ops.conv.maxpool3d(inputs, kernel_shape, padding, stride)[source]

Abstraction of MaxPool3D.forward

Parameters

  • inputs (Tensor or int or float or list or np.ndarray) – Tensor to be convolved on

  • kernel_shape (tuple) – Shape of the kernel

  • padding (int) – Padding value to be applied

  • stride (int) – Stride to be taken

Returns

Tensor of the result

neograd.autograd.ops.operation module

class neograd.autograd.ops.operation.Operation[source]

Bases: object

Transforms Tensors by applying some function

Used when some input is getting transformed into an output, for functions where gradient calculation is required with the forward pass and the backward pass defined

backward(*args)[source]

Abstract backward method

Raises

NotImplementedError – If backward method isn’t overridden

get_broadcast_shape(*tensors)[source]

Return broadcasted shape of Tensors

If the tensors can be broadcasted, then the broadcasted shape is returned , else None.

Parameters

*tensors (Tensor) – Tensors that should be broadcasted

Returns

Broadcasted shape if it can be broadcasted, if not None Also even if atleast one of the Tensors has requires_broadcasting set to False, it returns None

get_result_tensor(result, *tensors)[source]

Returns the result tensor of the Operation

If tracking is enabled, then, it creates a Node for the result_tensor with parent_broadcast_shape and adds edges to the graph

If tracking is disabled, then no Node creation and edge addition occurs

Parameters

  • result (np object) – Result after performing a raw numpy operation

  • *tensors (Tensor) – Operands of the operation

Returns

Tensor of the result

get_tensors(*operands)[source]

Returns the processed operands as tuple of Tensors

Parameters

*operands (Tensor or int or float or list or np.ndarray) – Operands of the Operation

Returns

tuple of Tensors if len(tuple)>1 else returns the first Tensor

process_operands(operands)[source]

All operands are converted to Tensors

Parameters

operands (Tensor or int or float or list or np.ndarray) – Operands of the Operation

Returns

tuple of Tensors

result_requires_grad(tensors)[source]

Checks if the result requires grad

Checks if the result requires gradient to be calculated given the operands of the Operation, if atleast one operand requires_grad to True, then result will also have requires_grad to True

Parameters

tensors (Tensor) – Tensors that are operated on

Module contents