neograd.nn package

Submodules

neograd.nn.activations module

class neograd.nn.activations.LeakyReLU(leak=0.01)

Bases: Layer, Operation

LeakyReLU Layer

backward(inputs)

Sets the grad_fn of the Tensor

If element in data is greater than zero, its local gradient will be 1 else will be leak value

Parameters

inputs (Tensor) – Operand

forward(inputs)

Calculates LeakyReLU of inputs

Parameters

inputs (Tensor) – Inputs to the Layer

Returns

Tensor of result

class neograd.nn.activations.ReLU

Bases: Layer, Operation

ReLU Layer

backward(inputs)

Sets the grad_fn of the Tensor

If element in data is greater than zero, its local gradient will be 1 else 0

Parameters

inputs (Tensor) – Operand

forward(inputs)

Calculates ReLU of inputs

Parameters

inputs (Tensor) – Inputs to the Layer

Returns

Tensor of result

class neograd.nn.activations.Sigmoid

Bases: Layer, Operation

Sigmoid Layer

backward(inputs)

Sets the grad_fn of the Tensor

Parameters

inputs (Tensor) – Operand

forward(inputs)

Calculates Sigmoid of inputs

Parameters

inputs (Tensor) – Inputs to the Layer

Returns

Tensor of result

class neograd.nn.activations.Softmax(axis)

Bases: Layer, Operation

Softmax Layer

Parameters

axis (None or int or tuple of int) – Axis along which it should be calculated

backward(inputs)

Sets the grad_fn of the Tensor

Quite a tricky one, first the Jacobian of each of the slices along the specified axis of the result is taken, which is then dotted with the corresponding slice of the upper gradient

Parameters

inputs (Tensor) – Operand

static calc_softmax(arr, axis=None)

Calculates stable Softmax

Parameters

• arr (np.ndarray) – Array whose Softmax is to be calculated

• axis (int or tuple of int) – Axis along which to calculate the Softmax Defaults to None

Returns

Softmax of the array

forward(inputs)

Calculates Softmax of inputs

Parameters

inputs (Tensor) – Inputs to the Layer

Returns

Tensor of result

class neograd.nn.activations.Tanh

Bases: Layer, Operation

Tanh Layer

backward(inputs)

Sets the grad_fn of the Tensor

Parameters

inputs (Tensor) – Operand

forward(inputs)

Calculates Tanh of inputs

Parameters

inputs (Tensor) – Inputs to the Layer

Returns

Tensor of result

neograd.nn.checkpoint module

class neograd.nn.checkpoint.Checkpoint(model, dirpath, hash_length=16)

Bases: object

Creates and initializes files for checkpoints

A JSON file checkpoints.json is created which contains the dict which has the tracked values and also the params file name at the time of adding a checkpoint, operates in append mode A new params file is created at each checkpoint and the JSON file is updated

Parameters

• session (str) – Current session that is in use

• dirpath (str) – Directory in which checkpoints must be saved

• model (Model) – Model to be checkpointed

• hash_length (int) – Character length of session identifiers. Defaults to 16

_generate_hash()

Generates 64 hex digit sha256 hash of a random number

Returns

sha256 hash

_init_files(dirpath)

Initializes files required for Checkpoint

Creates a new folder at dirpath, if it doesn’t exist A checkpoint.json file is created, if it is empty, then a new session is created if self.session is None, then automatically the last session is initialized as self.session

Parameters

dirpath (str) – Directory in which checkpoints must be saved

_save(updated_checkpoint, params_fname_hash)

Saves the checkpoint

Saves the checkpoint details onto checkpoints.json and creates a new file with the params of the model

if self.session isn’t already in existing checkpoints, then it creates a new dict and adds the checkpoint there.

Parameters

• updated_checkpoint (Checkpoint) – Checkpoint that is updated

• params_fname_hash (str) – Hash that is generated to be the name of filename

_update(new_checkpoint)

Updates the new_checkpoint

Updates the checkpoint by including the datetime of adding new checkpoint Generates the hash that’ll be used as the filename for the params file that’ll be saved

Parameters

new_checkpoint (Checkpoint) – New Checkpoint to be updated

Returns

New Checkpoint, hash that is used as fname

add(**tracked)

Adds a new checkpoint

Parameters

**tracked – All the data that needs to be tracked in checkpoints.json

Raises

• ValueError – If forbidden_attrs (‘datetime’) are used as keys in tracked, because the same key is used by neograd to add key of the same value, which might get overwritten

• ValueError – If values in tracked aren’t serializable and don’t belong to builtin classes

load(params_fname, load_params=True)

Retrieves the Checkpoint

Returns the checkpoint based on the params_fname and loads the params onto the model if load_params is True

Parameters

• params_fname (str) – Filename to load params from

• load_params (bool) – Whether params should be loaded from the file onto the model

Returns

Checkpoint desired

Raises

ValueError – If the current session is not present in checkpoints.py

new_session()

Creates a new session in checkpoints.json

Also creates a new directory for the session

Returns

self

specify_session(session)

Used to specify a particular session to use for checkpoints.json

Parameters

session (str) – The session to be used

Raises

ValueError – If session is not already in checkpoints.json

neograd.nn.layers module

class neograd.nn.layers.Container

Bases: object

Contains many Layers

Parameters

• eval (bool) – Whether the Container is in eval mode

• layers (list of Layer/Container) – Layer to be included in the container

freeze()

Freezes all the layers present in the Container

parameters(as_dict=False)

Recursively goes through all the layers in the Container and gets the params of each Layer

Parameters

as_dict (bool) – Whether params need to be returned as a dict, Defaults to False

Returns

list of Params

set_eval(eval)

Sets eval

Sets its eval to the eval argument also recursively sets the eval of its layers

Parameters

eval (bool) – Whether Container is in eval mode or not

set_params(container_params)

Sets the params for all the layers

unfreeze()

Unfreezes all the layers present in the Container

class neograd.nn.layers.Layer

Bases: object

Fundamental building block of the model

Parameters

eval (bool) – Whether the Container is in eval mode

freeze()

Freezes all the Params in the Layer

parameters(as_dict=False)

Returns the parameters in the Layer

If any of the attributes in a Layer is instance of Param, then it is automatically considered as a param for the model

Parameters

as_dict (bool) – Whether params need to be returned as a dict, Defaults to False

Returns

list of Params or dict

set_eval(eval)

Sets eval

Sets its eval to the eval argument

Parameters

eval (bool) – Whether Container is in eval mode or not

set_params(layer_params)

Sets the params for the current layer

unfreeze()

Unfreezes all the Params in the Layer

class neograd.nn.layers.Param(data, requires_grad=False, requires_broadcasting=True)

Bases: Tensor

Alias for Tensor

Just an alias for Tensor, so that when params are gathered for a Layer, only these are automatically considered for param, while ignoring some helper Tensors which aren’t necessarily param

Parameters

__frozen (bool) – Whether current Param is frozen or not. This is required because we need to know if it has been frozen before unfreeze is called

freeze()

Sets requires_grad=False

unfreeze()

Sets requires_grad=True only if its frozen

frozen condition is checked because only if it was previously frozen, we can set requires_grad = True, if we don’t check for it and requires_grad is False originally, then we might set it to True, which would be incorrect

neograd.nn.loss module

class neograd.nn.loss.BCE

Bases: Loss

Binary Cross Entropy

forward(outputs, targets, epsilon=1e-09)

Forward pass of BCE

epsilon used to prevent log0

Parameters

• outputs (Tensor) – Outputs of Layer/Container/Model/Operation

• targets (Tensor) – Targets to be evaluated against

• epsilon (float) – For numerical stability of log Defaults to 1e-9

Returns

Tensor of the result

class neograd.nn.loss.CE

Bases: Loss

Cross Entropy

forward(outputs, targets, epsilon=1e-09)

Forward pass of CE

epsilon used to prevent log0

Parameters

• outputs (Tensor) – Outputs of Layer/Container/Model/Operation

• targets (Tensor) – Targets to be evaluated against

• epsilon (float) – For numerical stability of log Defaults to 1e-9

Returns

Tensor of the result

class neograd.nn.loss.Loss

Bases: object

Base class of all loss functions

get_num_examples(outputs_shape)

Gathers the number of examples

If dimensions of outputs_shape is 0, ie it is a scalar, then num_examples is 1 Else the first dimension value of outputs_shape is taken as num_examples

Parameters

outputs_shape (tuple of int) – Shape of the outputs

Returns

Number of examples

class neograd.nn.loss.MSE

Bases: Loss

Mean Squared Error

forward(outputs, targets)

Forward pass of MSE

Parameters

• outputs (Tensor) – Outputs of Layer/Container/Model/Operation

• targets (Tensor) – Targets to be evaluated against

Returns

Tensor of the result

class neograd.nn.loss.SoftmaxCE(axis)

Bases: Operation, Loss

Implements Softmax activation with CrossEntropyLoss

Purpose of this is to eliminate costly Jacobian calculation involved with vanilla softmax activation. Since Softmax is most commonly used with Cross Entropy loss, if both are combined in one single Operation, then the derivative is a very minimal subtraction between the softmax output and the targets. So many intermediate backward calculations can be prevented with this.

Parameters

axis (int or tuple of int) – Axis along which to calculate the Softmax Defaults to None

epsilon to prevent log0

backward(outputs, targets)

Sets the grad_fn of outputs

Parameters

• outputs (Tensor) – Tensor which is usually the outputs of the last layer of the network

• targets (Tensor) – Targets to be evaluated against

forward(outputs, targets, epsilon=1e-09)

Calculates Softmax of inputs and the Cross Entropy loss

Parameters

• outputs (Tensor) – Outputs of Layer/Container/Model/Operation

• targets (Tensor) – Targets to be evaluated against

• epsilon (float) – For numerical stability of log Defaults to 1e-9

Returns

Tensor of the result

neograd.nn.model module

class neograd.nn.model.EvalMode(model, no_track)

Bases: object

ContextManager for handling eval

A ContextManager to run the model in eval mode, ie while testing the model. Use of this is that some layers like Dropout need to be turned off while testing

Parameters

• model (Model) – Model to be put into eval

• no_track (bool) – If True, then the backward graph is not created and the tensors aren’t tracked

• graph (Graph) – Graph that’s currently in use

class neograd.nn.model.Model

Bases: object

eval(no_track=True)

Invokes EvalMode ContextManager

Parameters

no_track (bool) – If Tensors shouldn’t be tracked, Defaults to False

Returns

EvalMode ContextManager

get_layers()

Gathers all the layers in the Model

Accomplishes by going through all its attributes and if their values are instances of Container/Layer it is taken as a layer

Returns

Dict with attributes as key and their objects as value

load(fpath)

Loads the params from the filepath onto the model

Parameters

fpath (str) – File path

parameters(as_dict=False)

Gathers the params of the whole Model

Accomplishes this by iterating through all layers and getting their params

Parameters

as_dict (bool) – Whether to return the params as a dict. Defaults to False

save(fpath)

Saves the params of the model in the specified file path

Parameters

fpath (str) – File path

set_eval(eval)

Sets eval

Sets the eval of its layers to eval argument

Parameters

eval (bool) – Whether in eval mode or not

neograd.nn.optim module

class neograd.nn.optim.Adam(params, lr, beta1=0.9, beta2=0.999, epsilon=1e-08)

Bases: Optimizer

https://youtu.be/JXQT_vxqwIs

Parameters

• iter (int) – The number of iterations that has occurred, used for bias correction

• beta1 (float) – Value of beta1

• beta2 (float) – Value of beta2

• epsilon (float) – Value of epsilon

init_adam_grads()

Initializes rms grads and momentum grads

For each param, it sets its rms_grad to 0, momentum_grad to 0

reset_iter()

Resets iter to 0

step()

Updates the params

Uses the rms_grads and momentum_grads to update the params

update_adam_grads()

Updates rms grads and momentum_grads

class neograd.nn.optim.GD(params, lr)

Bases: Optimizer

Vanilla Gradient Descent

step()

Updates the params

class neograd.nn.optim.Momentum(params, lr, beta=0.9)

Bases: Optimizer

Gradient Descent with Momentum

https://youtu.be/k8fTYJPd3_I

Parameters

beta (float) – Value of Beta

init_momentum_grads()

Initializes momentum grads

For each param, it sets its momentum_grad to 0

step()

Updates the params

Uses the momentum_grads to update the params

update_momentum_grads()

Updates momentum grads

class neograd.nn.optim.Optimizer(params, lr)

Bases: object

Base class for all optimizers

Parameters

• params (list of Param) – Params that need to be updated

• lr (float) – Learning rate

zero_grad(all_members=False)

Resets the grads of tensors

By default, since after loss.backward, only Tensors in memory are the params, only their gradients are reset since everytime a new graph is dynamically created

However if retain_graph=True in backward, then all the members in the graph, need to be zero_grad-ed to get the correct gradients, to prevent this all_members can be set to True

Parameters

all_members (bool) – If all the members in the graph should be zero_grad-ed. Defaults to False

class neograd.nn.optim.RMSProp(params, lr, beta=0.9, epsilon=1e-08)

Bases: Optimizer

https://youtu.be/_e-LFe_igno

Parameters

• beta (float) – Value of Beta

• epsilon (float) – Value of epsilon

init_rms_grads()

Initializes rms grads

For each param, it sets its rms_grad to 0

step()

Updates the params

Uses the rms_grads to update the params

update_rms_grads()

Updates rms grads

neograd.nn.utils module

neograd.nn.utils.get_batches(inputs, targets=None, batch_size=None)

Returns batches of inputs and targets

Split the inputs and their corresponding targets into batches for efficient training

Parameters

• inputs (Tensor) – Inputs to be batched

• targets (Tensor) – Targets to be batched. Defaults to None

• batch_size (int) – Size of the batches. Defaults to None meaning batch_size will be same as number of examples

Yields

Batches of inputs and their corresponding targets

Raises

• AssertionError – If first dimensions of inputs and targets don’t match

• ValueError – If batch_size is greater than number of examples

• ValueError – If batch_size is negative

• ValueError – If batch_size is 0

neograd.nn.utils.load_model(fpath)

Loads the model

Parameters

fpath (str) – Path from which to load the model

Returns

Model object that is loaded

neograd.nn.utils.save_model(fpath, model)

Saves the model

Saves the model by pickling it onto a file

Parameters

• fpath (str) – Path in which to save the model

• model (Model) – Model to be saved

Raises

TypeError – if model isn’t an instance of Model

Module contents