mmrazor.engine¶

hooks¶

optimizers¶

runner¶

mmrazor.models¶

algorithms¶

architectures¶

distillers¶

losses¶

mutables¶

class mmrazor.models.mutables.BaseMutable(alias: Optional[str] = None, init_cfg: Optional[Dict] = None)[source]¶

Base Class for mutables. Mutable means a searchable module widely used in Neural Architecture Search(NAS).

It mainly consists of some optional operations, and achieving searchable function by handling choice with MUTATOR.

All subclass should implement the following APIs:

fix_chosen()
dump_chosen()
current_choice.setter()
current_choice.getter()

Parameters

alias (str, optional) – alias of the MUTABLE.
init_cfg (dict, optional) – initialization configuration dict for BaseModule. OpenMMLab has implement 5 initializer including Constant, Xavier, Normal, Uniform, Kaiming, and Pretrained.

abstract property current_choice¶: Current choice will affect forward() and will be used in mmrazor.core.subnet.utils.export_fix_subnet() or mutator.

abstract dump_chosen() → mmrazor.utils.typing.DumpChosen[source]¶

Save the current state of the mutable as a dictionary.

DumpChosen has chosen and meta fields. chosen is necessary, fix_chosen will use the chosen . meta is used to store some non-essential information.

abstract fix_chosen(chosen) → None[source]¶: Fix mutable with chosen. This function would fix the chosen of mutable. The is_fixed will be set to True and only the selected operations can be retained. All subclasses must implement this method.

Note

This operation is irreversible.

property is_fixed: bool¶

whether the mutable is fixed.

Note

If a mutable is fixed, it is no longer a searchable module, just: a normal fixed module.

If a mutable is not fixed, it still is a searchable module.

Type: bool

class mmrazor.models.mutables.BaseMutableChannel(num_channels: int, **kwargs)[source]¶

BaseMutableChannel works as a channel mask for DynamicOps to select channels.

|---------------------------------------| |mutable_in_channel(BaseMutableChannel) | |---------------------------------------| | DynamicOp | |---------------------------------------| |mutable_out_channel(BaseMutableChannel)| |---------------------------------------|

All subclasses should implement the following APIs and the other abstract method in BaseMutable

current_mask

Parameters: num_channels (int) – number(dimension) of channels(mask).

property activated_channels: int¶: Number of activated channels.

abstract property current_mask: torch.Tensor¶: Return a mask indicating the channel selection.

dump_chosen() → mmrazor.utils.typing.DumpChosen[source]¶: Dump chosen.

fix_chosen(chosen=None)[source]¶: Fix the mutable with chosen.

num_choices() → int[source]¶: Number of available choices.

class mmrazor.models.mutables.DCFFChannelUnit(num_channels: int, candidate_choices: List[Union[int, float]] = [1.0], choice_mode: str = 'ratio', divisor: int = 1, min_value: int = 1, min_ratio: float = 0.9)[source]¶

DCFFChannelUnit is for supernet DCFF and based on OneShotMutableChannelUnit. In DCFF supernet, each module only has one choice. The channel choice is fixed before training.

Parameters

num_channels (int) – The raw number of channels.
candidate_choices (List[Union[int, float]], optional) – A list of candidate width numbers or ratios. Each candidate indicates how many channels to be reserved. Defaults to [1.0](choice_mode=’number’).
choice_mode (str, optional) – Mode of candidates. One of “ratio” or “number”. Defaults to ‘ratio’.
divisor (int) – Used to make choice divisible.
min_value (int) – the minimal value used when make divisible.
min_ratio (float) – the minimal ratio used when make divisible.

prepare_for_pruning(model: torch.nn.modules.module.Module)[source]¶: In DCFFChannelGroup nn.Conv2d is replaced with FuseConv2d.

class mmrazor.models.mutables.DMCPChannelUnit(num_channels: int, choice_mode: str = 'number', divisor: int = 1, min_value: int = 1, min_ratio: float = 0.5)[source]¶

DMCPChannelUnit is for supernet DMCP and based on OneShotMutableChannelUnit. In DMCP supernet, each module only has one choice. The channel choice is fixed before training.

Note

In dmcpunit, a new attribute activated_tensor_channels is defined

in self.mutable_channel, which is specifically used to store the number of channels in the form of tensor. Defaults to None.

Parameters

num_channels (int) – The raw number of channels.
choice_mode (str, optional) – Mode of candidates. One of “ratio” or “number”. Defaults to ‘ratio’.
divisor (int) – Used to make choice divisible.
min_value (int) – the minimal value used when make divisible.
min_ratio (float) – the minimal ratio used when make divisible.

prepare_for_pruning(model: torch.nn.modules.module.Module)[source]¶: In DMCPChannelGroup nn.BatchNorm2d is replaced with DMCPBatchNorm2d.

class mmrazor.models.mutables.DerivedMutable(choice_fn: Callable, mask_fn: Optional[Callable] = None, source_mutables: Optional[Iterable[mmrazor.models.mutables.base_mutable.BaseMutable]] = None, alias: Optional[str] = None, init_cfg: Optional[Dict] = None)[source]¶

Class for derived mutable.

A derived mutable is a mutable derived from other mutables that has current_choice and current_mask attributes (if any).

Note

A derived mutable does not have its own search space, so it is not legal to modify its current_choice or current_mask directly. And the only way to modify them is by modifying current_choice or current_mask in corresponding source mutables.

Parameters

choice_fn (callable) – A closure that controls how to generate current_choice.
mask_fn (callable, optional) – A closure that controls how to generate current_mask. Defaults to None.
source_mutables (iterable, optional) – Specify source mutables for this derived mutable. If the argument is None, source mutables will be traced automatically by parsing mutables in closure variables. Defaults to None.
alias (str, optional) – alias of the MUTABLE. Defaults to None.
init_cfg (dict, optional) – initialization configuration dict for BaseModule. OpenMMLab has implement 5 initializer including Constant, Xavier, Normal, Uniform, Kaiming, and Pretrained. Defaults to None.

Examples

>>> from mmrazor.models.mutables import SquentialMutableChannel
>>> mutable_channel = SquentialMutableChannel(num_channels=3)
>>> # derive expand mutable
>>> derived_mutable_channel = mutable_channel * 2
>>> # source mutables will be traced automatically
>>> derived_mutable_channel.source_mutables
{SquentialMutableChannel(name=unbind, num_channels=3, current_choice=3)}  # noqa: E501
>>> # modify `current_choice` of `mutable_channel`
>>> mutable_channel.current_choice = 2
>>> # `current_choice` and `current_mask` of derived mutable will be modified automatically  # noqa: E501
>>> derived_mutable_channel
DerivedMutable(current_choice=4, activated_channels=4, source_mutables={SquentialMutableChannel(name=unbind, num_channels=3, current_choice=2)}, is_fixed=False)  # noqa: E501

property current_choice¶: Current choice of derived mutable.

property current_mask: torch.Tensor¶: Current mask of derived mutable.

dump_chosen() → mmrazor.utils.typing.DumpChosen[source]¶

Dump information of chosen.

Returns: Dumped information.
Return type: Dict

fix_chosen(chosen) → None[source]¶: Fix mutable with subnet config.

Warning

Fix derived mutable will have no actually effect.

property is_fixed: bool¶: Whether the derived mutable is fixed.

Note

Depends on whether all source mutables are already fixed.

static is_source_mutable(mutable: object) → bool[source]¶

Judge whether an object is source mutable(not derived mutable).

Parameters: mutable (object) – An object.
Returns: Indicate whether the object is source mutable or not.
Return type: bool

property num_choices: int¶

Number of all choices.

Note

Since derive mutable does not have its own search space, the number of choices will always be 1.

Returns: Number of choices.
Return type: int

class mmrazor.models.mutables.DiffChoiceRoute(edges: torch.nn.modules.container.ModuleDict, num_chosen: int = 2, with_arch_param: bool = False, alias: Optional[str] = None, init_cfg: Optional[Dict] = None)[source]¶

A type of MUTABLES for Neural Architecture Search, which can select inputs from different edges in a differentiable or non-differentiable way. It is commonly used in DARTS.

Parameters

edges (nn.ModuleDict) – the key of edges is the name of different edges. The value of edges can be nn.Module or DiffMutableModule.
with_arch_param (bool) – whether forward with arch_param. When set to True, a differentiable way is adopted. When set to False, a non-differentiable way is adopted.
alias (str, optional) – alias of the DiffChoiceRoute.
init_cfg (dict, optional) – initialization configuration dict for BaseModule. OpenMMLab has implement 6 initializers including Constant, Xavier, Normal, Uniform, Kaiming, and Pretrained.

Examples

>>> import torch
>>> import torch.nn as nn
>>> edges_dict=nn.ModuleDict()
>>> edges_dict.add_module('first_edge', nn.Conv2d(32, 32, 3, 1, 1))
>>> edges_dict.add_module('second_edge', nn.Conv2d(32, 32, 5, 1, 2))
>>> edges_dict.add_module('third_edge', nn.MaxPool2d(3, 1, 1))
>>> edges_dict.add_module('fourth_edge', nn.MaxPool2d(5, 1, 2))
>>> edges_dict.add_module('fifth_edge', nn.MaxPool2d(7, 1, 3))
>>> diff_choice_route_cfg = dict(
...     type="DiffChoiceRoute",
...     edges=edges_dict,
...     with_arch_param=True,
... )
>>> arch_param
Parameter containing:
tensor([-6.1426e-04,  2.3596e-04,  1.4427e-03,  7.1668e-05,
    -8.9739e-04], requires_grad=True)
>>> x = [torch.randn(4, 32, 64, 64) for _ in range(5)]
>>> output=diffchoiceroute.forward_arch_param(x, arch_param)
>>> output.shape
torch.Size([4, 32, 64, 64])

property choices: List[str]¶

all choices.

Type: list

dump_chosen() → mmrazor.utils.typing.DumpChosen[source]¶

Save the current state of the mutable as a dictionary.

DumpChosen has chosen and meta fields. chosen is necessary, fix_chosen will use the chosen . meta is used to store some non-essential information.

fix_chosen(chosen: List[str]) → None[source]¶

Fix mutable with choice. This operation would convert to fixed mode. The is_fixed will be set to True and only the selected operations can be retained.

Parameters: chosen (list(str)) – the chosen key in MUTABLE.

forward(x: Any, arch_param: Optional[torch.nn.parameter.Parameter] = None)[source]¶

Calls either forward_fixed() or forward_arch_param() depending on whether is_fixed() is True and whether arch_param() is None.

To reduce the coupling between Mutable and Mutator, the arch_param is generated by the Mutator and is passed to the forward function as an argument.

Note

forward_fixed() is called when in fixed mode. forward_arch_param() is called when in unfixed mode.

Parameters

x (Any) – input data for forward computation.
arch_param (nn.Parameter, optional) – the architecture parameters for DiffMutableModule.

Returns

the result of forward

Return type

Any

forward_all(x)[source]¶

Forward all choices.

Parameters: x (Any) – x could be a Torch.tensor or a tuple of Torch.tensor, containing input data for forward computation.
Returns: the result of forward all of the choice operation.
Return type: Tensor

forward_arch_param(x, arch_param: torch.nn.parameter.Parameter) → torch.Tensor[source]¶

Forward with architecture parameters.

Parameters

x (list[Any] | tuple[Any]]) – x could be a list or a tuple of Torch.tensor, containing input data for forward selection.
arch_param (nn.Parameter) – architecture parameters for for DiffMutableModule.

Returns

the result of forward with arch_param.

Return type

Tensor

forward_fixed(inputs: Union[List, Tuple]) → torch.Tensor[source]¶

Forward when the mutable is in fixed mode.

Parameters: inputs (Union[List[Any], Tuple[Any]]) – inputs could be a list or a tuple of Torch.tensor, containing input data for forward computation.
Returns: the result of forward the fixed operation.
Return type: Tensor

sample_choice(arch_param: torch.Tensor) → List[str][source]¶: sample choice based on arch_param.

class mmrazor.models.mutables.DiffMutableModule(**kwargs)[source]¶

Base class for differentiable mutables.

Parameters

module_kwargs (dict[str, dict], optional) – Module initialization named arguments. Defaults to None.
alias (str, optional) – alias of the MUTABLE.
init_cfg (dict, optional) – initialization configuration dict for BaseModule. OpenMMLab has implement 5 initializer including Constant, Xavier, Normal, Uniform, Kaiming, and Pretrained.

Note

forward_all() is called when calculating FLOPs.

compute_arch_probs(arch_param: torch.nn.parameter.Parameter) → torch.Tensor[source]¶: compute chosen probs according to architecture params.

forward(x: Any, arch_param: Optional[torch.nn.parameter.Parameter] = None)[source]¶

Calls either forward_fixed() or forward_arch_param() depending on whether is_fixed() is True and whether arch_param() is None.

To reduce the coupling between Mutable and Mutator, the arch_param is generated by the Mutator and is passed to the forward function as an argument.

Note

forward_fixed() is called when in fixed mode. forward_arch_param() is called when in unfixed mode.

Parameters

x (Any) – input data for forward computation.
arch_param (nn.Parameter, optional) – the architecture parameters for DiffMutableModule.

Returns

the result of forward

Return type

Any

abstract forward_arch_param(x, arch_param: torch.nn.parameter.Parameter)[source]¶

Forward when the mutable is not fixed.

All subclasses must implement this method.

abstract sample_choice(arch_param: torch.Tensor)[source]¶: Sample choice according arch parameters.

set_forward_args(arch_param: torch.nn.parameter.Parameter) → None[source]¶: Interface for modifying the arch_param using partial.

class mmrazor.models.mutables.DiffMutableOP(candidates: Dict[str, Dict], fix_threshold: float = 1.0, module_kwargs: Optional[Dict[str, Dict]] = None, alias: Optional[str] = None, init_cfg: Optional[Dict] = None)[source]¶

A type of MUTABLES for differentiable architecture search, such as DARTS. Search the best module by learnable parameters arch_param.

Parameters

candidates (dict[str, dict]) – the configs for the candidate operations.
fix_threshold (float) – The threshold that determines whether to fix the choice of current module as the op with the maximum probs. It happens when the maximum prob is fix_threshold or more higher then all the other probs. Default to 1.0.
module_kwargs (dict[str, dict], optional) – Module initialization named arguments. Defaults to None.
alias (str, optional) – alias of the MUTABLE.
init_cfg (dict, optional) – initialization configuration dict for BaseModule. OpenMMLab has implement 5 initializer including Constant, Xavier, Normal, Uniform, Kaiming, and Pretrained.

property choices: List[str]¶

all choices.

Type: list

dump_chosen() → mmrazor.utils.typing.DumpChosen[source]¶

Save the current state of the mutable as a dictionary.

DumpChosen has chosen and meta fields. chosen is necessary, fix_chosen will use the chosen . meta is used to store some non-essential information.

fix_chosen(chosen: Union[str, List[str]]) → None[source]¶

Fix mutable with choice. This operation would convert unfixed mode to fixed mode. The is_fixed will be set to True and only the selected operations can be retained.

Parameters: chosen (str) – the chosen key in MUTABLE. Defaults to None.

forward_all(x) → torch.Tensor[source]¶

Forward all choices. Used to calculate FLOPs.

Parameters: x (Any) – x could be a Torch.tensor or a tuple of Torch.tensor, containing input data for forward computation.
Returns: the result of forward all of the choice operation.
Return type: Tensor

forward_arch_param(x, arch_param: torch.nn.parameter.Parameter) → torch.Tensor[source]¶

Forward with architecture parameters.

Parameters

x (Any) – x could be a Torch.tensor or a tuple of Torch.tensor, containing input data for forward computation.
arch_param (str, optional) – architecture parameters for DiffMutableModule

Returns

the result of forward with arch_param.

Return type

Tensor

forward_fixed(x) → torch.Tensor[source]¶

Forward when the mutable is in fixed mode.

Parameters: x (Any) – x could be a Torch.tensor or a tuple of Torch.tensor, containing input data for forward computation.
Returns: the result of forward the fixed operation.
Return type: Tensor

sample_choice(arch_param: torch.Tensor) → str[source]¶: Sample choice based on arch_parameters.

class mmrazor.models.mutables.L1MutableChannelUnit(num_channels: int, choice_mode='number', divisor=1, min_value=1, min_ratio=0.9)[source]¶

Implementation of L1-norm pruning algorithm. It compute the l1-norm of modules and preferly prune the modules with less l1-norm.

Please refer to papre https://arxiv.org/pdf/1608.08710.pdf for more detail.

property current_choice: Union[int, float]¶: return current choice.

class mmrazor.models.mutables.MutableChannelContainer(num_channels: int, **kwargs)[source]¶

MutableChannelContainer inherits from BaseMutableChannel. However, it’s not a single BaseMutableChannel, but a container for BaseMutableChannel. The mask of MutableChannelContainer consists of all masks of stored MutableChannels.

MutableChannelContainer |

Important interfaces:

register_mutable: register/store BaseMutableChannel in the: MutableChannelContainer

property current_choice: torch.Tensor¶: Get current choices.

property current_mask: torch.Tensor¶: Return current mask.

register_mutable(mutable_channel: mmrazor.models.mutables.mutable_channel.base_mutable_channel.BaseMutableChannel, start: int, end: int)[source]¶: Register/Store BaseMutableChannel in the MutableChannelContainer in the range [start,end)

classmethod register_mutable_channel_to_module(module: mmrazor.models.architectures.dynamic_ops.mixins.dynamic_mixins.DynamicChannelMixin, mutable: mmrazor.models.mutables.mutable_channel.base_mutable_channel.BaseMutableChannel, is_to_output_channel=True, start=0, end=- 1)[source]¶: Register a BaseMutableChannel to a module with MutableChannelContainers.

class mmrazor.models.mutables.MutableChannelUnit(num_channels: int, **kwargs)[source]¶

config_template(with_init_args=False, with_channels=False) → Dict[source]¶

Return the config template of this unit. By default, the config template only includes a key ‘choice’.

Parameters

with_init_args (bool) – if the config includes args for initialization.
with_channels (bool) – if the config includes info about channels. the config with info about channels can used to parse channel units without tracer.

property current_choice¶: Choice of this unit.

fix_chosen(choice=None)[source]¶: Make the channels in this unit fixed.

classmethod init_from_cfg(model: torch.nn.modules.module.Module, config: Dict)[source]¶: init a Channel using a config which can be generated by self.config_template(), include init choice.

classmethod init_from_predefined_model(model: torch.nn.modules.module.Module)[source]¶: Initialize units using the model with pre-defined dynamicops and mutable-channels.

property is_mutable: bool¶: If the channel-unit is prunable.

property mutable_prefix: str¶: Mutable prefix.

abstract prepare_for_pruning(model)[source]¶

Post process after parse units.

For example, we need to register mutables to dynamic-ops.

abstract sample_choice()[source]¶: Randomly sample a valid choice and return.

class mmrazor.models.mutables.MutableValue(value_list: List[Union[int, float]], default_value: Optional[Any] = None, alias: Optional[str] = None, init_cfg: Optional[Dict] = None)[source]¶

Base class for mutable value.

A mutable value is actually a mutable that adds some functionality to a list containing objects of the same type.

Parameters

value_list (list) – List of value, each value must have the same type.
default_value (any, optional) – Default value, must be one in value_list. Default to None.
alias (str, optional) – alias of the MUTABLE.
init_cfg (dict, optional) – initialization configuration dict for BaseModule. OpenMMLab has implement 5 initializer including Constant, Xavier, Normal, Uniform, Kaiming, and Pretrained.

property choices: List[Any]¶: List of choices.

property current_choice: Union[int, float]¶: Current choice of mutable value.

dump_chosen() → mmrazor.utils.typing.DumpChosen[source]¶

Dump information of chosen.

Returns: Dumped information.
Return type: Dict[str, Any]

fix_chosen(chosen: Union[int, float]) → None[source]¶

Fix mutable value with subnet config.

Parameters: chosen (dict) – the information of chosen.

property mutable_prefix: str¶: Mutable prefix.

property num_choices: int¶

Number of all choices.

Returns: Number of choices.
Return type: int

class mmrazor.models.mutables.OneHotMutableOP(candidates: Dict[str, Dict], fix_threshold: float = 1.0, module_kwargs: Optional[Dict[str, Dict]] = None, alias: Optional[str] = None, init_cfg: Optional[Dict] = None)[source]¶

A type of MUTABLES for one-hot sample based architecture search, such as DSNAS. Search the best module by learnable parameters arch_param.

Parameters

candidates (dict[str, dict]) – the configs for the candidate operations.
module_kwargs (dict[str, dict], optional) – Module initialization named arguments. Defaults to None.
alias (str, optional) – alias of the MUTABLE.
init_cfg (dict, optional) – initialization configuration dict for BaseModule. OpenMMLab has implement 5 initializer including Constant, Xavier, Normal, Uniform, Kaiming, and Pretrained.

forward_arch_param(x: Any, arch_param: torch.nn.parameter.Parameter) → torch.Tensor[source]¶

Forward with architecture parameters.

Parameters

x (Any) – x could be a Torch.tensor or a tuple of Torch.tensor, containing input data for forward computation.
arch_param (str, optional) – architecture parameters for DiffMutableModule.

Returns

the result of forward with arch_param.

Return type

Tensor

sample_weights(arch_param: torch.nn.parameter.Parameter, probs: torch.Tensor, random_sample: bool = False) → torch.Tensor[source]¶

Use one-hot distributions to sample the arch weights based on the arch params.

Parameters

arch_param (nn.Parameter) – architecture parameters for DiffMutableModule.
probs (Tensor) – the probs of choice.
random_sample (bool) – Whether to random sample arch weights or not Defaults to False.

Returns

Sampled one-hot arch weights.

Return type

Tensor

class mmrazor.models.mutables.OneShotMutableChannel(num_channels: int, candidate_choices: List[Union[int, float]] = [], choice_mode='number', **kwargs)[source]¶

OneShotMutableChannel is a subclass of SquentialMutableChannel. The difference is that a OneShotMutableChannel limits the candidates of the choice.

Parameters

num_channels (int) – number of channels.
candidate_choices (List[Union[float, int]], optional) – A list of candidate width ratios. Each candidate indicates how many channels to be reserved. Defaults to [].
choice_mode (str, optional) – Mode of choices. Defaults to ‘number’.

property current_choice: Union[int, float]¶: Get current choice.

class mmrazor.models.mutables.OneShotMutableChannelUnit(num_channels: int, candidate_choices: List[Union[int, float]] = [0.5, 1.0], choice_mode='ratio', divisor=1, min_value=1, min_ratio=0.9)[source]¶

OneShotMutableChannelUnit is for single path supernet such as AutoSlim. In single path supernet, each module only has one choice invoked at the same time. A path is obtained by sampling all the available choices. It is the base class for one shot mutable channel.

Parameters

num_channels (_type_) – The raw number of channels.
candidate_choices (List[Union[int, float]], optional) – A list of candidate width ratios. Each candidate indicates how many channels to be reserved. Defaults to [0.5, 1.0](choice_mode=’ratio’).
choice_mode (str, optional) – Mode of candidates. One of “ratio” or “number”. Defaults to ‘ratio’.
divisor (int) – Used to make choice divisible.
min_value (int) – the minimal value used when make divisible.
min_ratio (float) – the minimal ratio used when make divisible.

config_template(with_init_args=False, with_channels=False) → Dict[source]¶: Config template of the OneShotMutableChannelUnit.

property current_choice: Union[int, float]¶: Get current choice.

property max_choice: Union[int, float]¶: Get Maximal choice.

property min_choice: Union[int, float]¶: Get Minimal choice.

prepare_for_pruning(model: torch.nn.modules.module.Module)[source]¶: Prepare for pruning.

sample_choice() → Union[int, float][source]¶: Sample a valid choice.

class mmrazor.models.mutables.OneShotMutableModule(module_kwargs: Optional[Dict[str, Dict]] = None, alias: Optional[str] = None, init_cfg: Optional[Dict] = None)[source]¶

Base class for one shot mutable module. A base type of MUTABLES for single path supernet such as Single Path One Shot.

All subclass should implement the following APIs and the other abstract method in MutableModule:

sample_choice()
forward_choice()

Note

forward_all() is called when calculating FLOPs.

forward(x: Any) → Any[source]¶

Calls either forward_fixed() or forward_choice() depending on whether is_fixed() is True and whether current_choice() is None.

Note

forward_fixed() is called in fixed mode. forward_all() is called in unfixed mode with

current_choice() is None.

forward_choice() is called in unfixed mode with: current_choice() is not None.

Parameters

x (Any) – input data for forward computation.
choice (CHOICE_TYPE, optional) – the chosen key in MUTABLE.

Returns

the result of forward

Return type

Any

abstract forward_choice(x, choice: str)[source]¶

Forward with the unfixed mutable and current_choice is not None.

All subclasses must implement this method.

abstract sample_choice() → str[source]¶

Sample random choice.

Returns: the chosen key in MUTABLE.
Return type: str

class mmrazor.models.mutables.OneShotMutableOP(candidates: Union[Dict[str, Dict], torch.nn.modules.container.ModuleDict], module_kwargs: Optional[Dict[str, Dict]] = None, alias: Optional[str] = None, init_cfg: Optional[Dict] = None)[source]¶

A type of MUTABLES for single path supernet, such as Single Path One Shot. In single path supernet, each choice block only has one choice invoked at the same time. A path is obtained by sampling all the choice blocks.

Parameters

candidates (dict[str, dict]) – the configs for the candidate operations.
module_kwargs (dict[str, dict], optional) – Module initialization named arguments. Defaults to None.
alias (str, optional) – alias of the MUTABLE.
init_cfg (dict, optional) – initialization configuration dict for BaseModule. OpenMMLab has implement 5 initializer including Constant, Xavier, Normal, Uniform, Kaiming, and Pretrained.

Examples

>>> import torch
>>> from mmrazor.models.mutables import OneShotMutableOP

>>> candidates = nn.ModuleDict({
...     'conv3x3': nn.Conv2d(32, 32, 3, 1, 1),
...     'conv5x5': nn.Conv2d(32, 32, 5, 1, 2),

>>> input = torch.randn(1, 32, 64, 64)
>>> op = OneShotMutableOP(candidates)

>>> op.choices
['conv3x3', 'conv5x5', 'conv7x7']
>>> op.num_choices
3
>>> op.is_fixed
False

>>> op.current_choice = 'conv3x3'
>>> unfix_output = op.forward(input)
>>> torch.all(unfixed_output == candidates['conv3x3'](input))
True

>>> op.fix_chosen('conv3x3')
>>> fix_output = op.forward(input)
>>> torch.all(fix_output == unfix_output)
True

>>> op.choices
['conv3x3']
>>> op.num_choices
1
>>> op.is_fixed
True

property choices: List[str]¶

all choices.

Type: list

dump_chosen() → mmrazor.utils.typing.DumpChosen[source]¶

Save the current state of the mutable as a dictionary.

DumpChosen has chosen and meta fields. chosen is necessary, fix_chosen will use the chosen . meta is used to store some non-essential information.

fix_chosen(chosen: str) → None[source]¶

Fix mutable with subnet config. This operation would convert unfixed mode to fixed mode. The is_fixed will be set to True and only the selected operations can be retained.

Parameters: chosen (str) – the chosen key in MUTABLE. Defaults to None.

forward_all(x) → torch.Tensor[source]¶

Forward all choices. Used to calculate FLOPs.

Parameters: x (Any) – x could be a Torch.tensor or a tuple of Torch.tensor, containing input data for forward computation.
Returns: the result of forward all of the choice operation.
Return type: Tensor

forward_choice(x, choice: str) → torch.Tensor[source]¶

Forward with the unfixed mutable and current choice is not None.

Parameters

x (Any) – x could be a Torch.tensor or a tuple of Torch.tensor, containing input data for forward computation.
choice (str) – the chosen key in OneShotMutableOP.

Returns

the result of forward the choice operation.

Return type

Tensor

forward_fixed(x: Any) → torch.Tensor[source]¶

Forward with the fixed mutable.

Parameters: x (Any) – x could be a Torch.tensor or a tuple of Torch.tensor, containing input data for forward computation.
Returns: the result of forward the fixed operation.
Return type: Tensor

sample_choice() → str[source]¶: uniform sampling.

class mmrazor.models.mutables.OneShotMutableValue(value_list: List[Any], default_value: Optional[Any] = None, alias: Optional[str] = None, init_cfg: Optional[Dict] = None)[source]¶

Class for one-shot mutable value.

one-shot mutable value provides sample_choice method and min_choice, max_choice properties on the top of mutable value.

Parameters

value_list (list) – List of value, each value must have the same type.
default_value (any, optional) – Default value, must be one in value_list. Default to None.
alias (str, optional) – alias of the MUTABLE.
init_cfg (dict, optional) – initialization configuration dict for BaseModule. OpenMMLab has implement 5 initializer including Constant, Xavier, Normal, Uniform, Kaiming, and Pretrained.

property max_choice: Any¶

Max choice of all choices.

Returns: Max choice.
Return type: Any

property min_choice: Any¶

Min choice of all choices.

Returns: Min choice.
Return type: Any

sample_choice() → Any[source]¶

Random sampling from choices.

Returns: Selected choice.
Return type: Any

class mmrazor.models.mutables.SequentialMutableChannelUnit(num_channels: int, choice_mode='number', divisor=1, min_value=1, min_ratio=0.9)[source]¶

SequentialMutableChannelUnit accepts a intger(number) or float(ratio) as the choice, which indicates how many of the channels are remained from left to right, like 11110000.

Parameters

num_channels (int) – number of channels.
choice_mode (str) – mode of choice, which is one of ‘number’ or ‘ratio’.
divisor (int) – Used to make choice divisible.
min_value (int) – the minimal value used when make divisible.
min_ratio (float) – the minimal ratio used when make divisible.

config_template(with_init_args=False, with_channels=False) → Dict[source]¶: Template of config.

property current_choice: Union[int, float]¶: return current choice.

fix_chosen(choice=None)[source]¶: fix chosen.

prepare_for_pruning(model: torch.nn.modules.module.Module)[source]¶: Prepare for pruning, including register mutable channels.

sample_choice() → Union[int, float][source]¶: Sample a choice in (0,1]

class mmrazor.models.mutables.SimpleMutableChannel(num_channels: int, **kwargs)[source]¶

SimpleMutableChannel is a simple BaseMutableChannel, it directly take a mask as a choice.

Parameters: num_channels (int) – number of channels.

property current_choice: torch.Tensor¶: Get current choice.

property current_mask: torch.Tensor¶: Get current mask.

expand_mutable_channel(expand_ratio: Union[int, float]) → mmrazor.models.mutables.derived_mutable.DerivedMutable [source]¶: Get a derived SimpleMutableChannel with expanded mask.

class mmrazor.models.mutables.SlimmableChannelUnit(num_channels: int, candidate_choices: List[Union[int, float]] = [], choice_mode='number', divisor=1, min_value=1, min_ratio=0.9)[source]¶

A type of MutableChannelUnit to train several subnets together.

Parameters

num_channels (int) – The raw number of channels.
candidate_choices (List[Union[int, float]], optional) – A list of candidate width ratios. Each candidate indicates how many channels to be reserved. Defaults to [0.5, 1.0](choice_mode=’ratio’).
choice_mode (str, optional) – Mode of candidates. One of ‘ratio’ or ‘number’. Defaults to ‘number’.
divisor (int, optional) – Used to make choice divisible.
min_value (int, optional) – The minimal value used when make divisible.
min_ratio (float, optional) – The minimal ratio used when make divisible.

alter_candidates_of_switchbn(candidates: List)[source]¶: Change candidates of SwitchableBatchNorm2d.

prepare_for_pruning(model: torch.nn.modules.module.Module)[source]¶: Prepare for pruning.

class mmrazor.models.mutables.SquentialMutableChannel(num_channels: int, choice_mode='number', **kwargs)[source]¶

SquentialMutableChannel defines a BaseMutableChannel which switch off channel mask from right to left sequentially, like ‘11111000’.

A choice of SquentialMutableChannel is an integer, which indicates how many channel are activated from left to right.

Parameters: num_channels (int) – number of channels.

property current_choice: Union[int, float]¶: Get current choice.

property current_mask: torch.Tensor¶: Return current mask.

fix_chosen(chosen=Ellipsis)[source]¶: Fix chosen.

property is_num_mode¶: Get if the choice is number mode.

mutators¶

ops¶

task_modules¶

utils¶

mmrazor.models.utils.add_prefix(inputs: Dict, prefix: str) → Dict[source]¶

Add prefix for dict.

Parameters

inputs (dict) – The input dict with str keys.
prefix (str) – The prefix to add.

Returns

The dict with keys updated with prefix.

Return type

dict

mmrazor.models.utils.get_module_device(module: torch.nn.modules.module.Module) → torch.device[source]¶

Get the device of a module.

Parameters: module (nn.Module) – A module contains the parameters.

mmrazor.models.utils.make_divisible(value: int, divisor: int, min_value: Optional[int] = None, min_ratio: float = 0.9) → int[source]¶

Make divisible function.

This function rounds the channel number down to the nearest value that can be divisible by the divisor.

Parameters

value (int) – The original channel number.
divisor (int) – The divisor to fully divide the channel number.
min_value (int, optional) – The minimum value of the output channel. Default: None, means that the minimum value equal to the divisor.
min_ratio (float) – The minimum ratio of the rounded channel number to the original channel number. Default: 0.9.

Returns

The modified output channel number

Return type

int

mmrazor.models.utils.parse_values(candidate_lists: List[list])[source]¶

Parse a list with format (min_range, max_range, step).

NOTE: this method is required when customizing search space in configs.

mmrazor.models.utils.pop_rewriter_function_record(rewriter_context, function_record_to_pop)[source]¶

Delete user-specific rewriters from RewriterContext._rewriter_manager.

We use the model which is rewritten by mmdeploy to build quantized models. However not all the functions rewritten by mmdeploy need to be rewritten in mmrazor. For example, mmdeploy rewrite mmcls.models.classifiers.ImageClassifier.forward and mmcls.models.classifiers.BaseClassifier.forward for deployment. But they can’t be rewritten by mmrazor as ptq and qat are done in mmrazor. So to ensure ptq and qat proceed normally, we have to remove these record from RewriterContext._rewriter_manager.

mmrazor.models.utils.set_requires_grad(nets: Union[torch.nn.modules.module.Module, List[torch.nn.modules.module.Module]], requires_grad: bool = False) → None[source]¶

Set requires_grad for all the networks.

Parameters

nets (nn.Module | list[nn.Module]) – A list of networks or a single network.
requires_grad (bool) – Whether the networks require gradients or not

mmrazor.registry¶

mmrazor.structures¶

delivery¶

graph¶

recorder¶

subnet¶

class mmrazor.structures.subnet.Candidates(initdata: Optional[Union[Dict, List[Dict], Dict[str, Dict], List[Dict[str, Dict]]]] = None)[source]¶

The data structure of sampled candidate. The format is Union[Dict[str, Dict], List[Dict[str, Dict]]]. .. rubric:: Examples

>>> candidates = Candidates()
>>> subnet_1 = {'1': 'choice1', '2': 'choice2'}
>>> candidates.append(subnet_1)
>>> candidates
[{"{'1': 'choice1', '2': 'choice2'}":
{'score': 0.0, 'flops': 0.0, 'params': 0.0, 'latency': 0.0}}]
>>> candidates.set_resources(0, 49.9, 'flops')
>>> candidates.set_score(0, 100.)
>>> candidates
[{"{'1': 'choice1', '2': 'choice2'}":
{'score': 100.0, 'flops': 49.9, 'params': 0.0, 'latency': 0.0}}]
>>> subnet_2 = {'choice_3': 'layer_3', 'choice_4': 'layer_4'}
>>> candidates.append(subnet_2)
>>> candidates
[{"{'1': 'choice1', '2': 'choice2'}":
{'score': 100.0, 'flops': 49.9, 'params': 0.0, 'latency': 0.0}},
{"{'choice_3': 'layer_3', 'choice_4':'layer_4'}":
{'score': 0.0, 'flops': 0.0, 'params': 0.0, 'latency': 0.0}}]
>>> candidates.subnets
[{'1': 'choice1', '2': 'choice2'},
{'choice_3': 'layer_3', 'choice_4': 'layer_4'}]
>>> candidates.resources('flops')
[49.9, 0.0]
>>> candidates.scores
[100.0, 0.0]

append(item: Union[Dict, List[Dict], Dict[str, Dict], List[Dict[str, Dict]]]) → None[source]¶: Append operation.

extend(other: Any) → None[source]¶: Extend operation.

insert(i: int, item: Union[Dict, List[Dict], Dict[str, Dict], List[Dict[str, Dict]]]) → None[source]¶: Insert operation.

resources(key_indicator: str = 'flops') → List[float][source]¶: The resources of candidates.

property scores: List[float]¶: The scores of candidates.

set_resource(i: int, resources: float, key_indicator: str = 'flops') → None[source]¶: Set resources to the specified subnet by index.

set_score(i: int, score: float) → None[source]¶: Set score to the specified subnet by index.

sort_by(key_indicator: str = 'score', reverse: bool = True) → None[source]¶

Sort by a specific indicator in descending order.

Parameters

key_indicator (str) – sort all candidates by key_indicator. Defaults to ‘score’.
reverse (bool) – sort all candidates in descending order.

property subnets: List[Dict]¶: The subnets of candidates.

update_resources(resources: list, start: int = 0) → None[source]¶: Update resources to the specified candidate.

mmrazor.structures.subnet.convert_fix_subnet(fix_subnet: Dict[str, mmrazor.utils.typing.DumpChosen])[source]¶: Convert the fixed subnet to avoid python typing error.

mmrazor.structures.subnet.export_fix_subnet(model: torch.nn.modules.module.Module, export_subnet_mode: str = 'mutable', slice_weight: bool = False, export_channel: bool = False) → Tuple[Dict[str, Any], Optional[Dict]][source]¶

Export subnet that can be loaded by load_fix_subnet(). Include subnet structure and subnet weight.

Parameters

model (nn.Module) – The target model to export.
export_subnet_mode (bool) – Subnet export method choice. Export by mutable.dump_chosen() when set to ‘mutable’ (NAS) Export by mutator.config_template() when set to ‘mutator’ (Prune)
slice_weight (bool) – Export subnet weight. Default to False.
export_channel (bool) – Whether to export the mutator’s channel. Often required when finetune is needed for the exported subnet. Default to False.

Returns

Exported subnet choice config. static_model (Optional[Dict]): Exported static model state_dict.

Valid when `slice_weight`=True.

Return type

fix_subnet (ValidFixMutable)

mmrazor.structures.subnet.load_fix_subnet(model: torch.nn.modules.module.Module, subnet_dict: Union[str, pathlib.Path, Dict[str, Any]], load_subnet_mode: str = 'mutable', prefix: str = '', extra_prefix: str = '') → None[source]¶: Load fix subnet.

tracer¶

mmrazor.utils¶

class mmrazor.utils.IndexDict[source]¶

IndexDict inherits from OrderedDict[Tuple[int, int], VT]. Each IndexDict object is a OrderDict object which using index(Tuple[int,int]) as key and Any as value.

The key type is Tuple[a: int,b: int]. It indicates a range in the [a,b).

IndexDict has three features: 1. ensure a key always is a index(Tuple[int,int]). 1. ensure the the indexes are sorted by ascending order. 2. ensure there is no overlap among indexes.

class mmrazor.utils.RuntimeInfo[source]¶: A tools to get runtime info in MessageHub.

mmrazor.utils.find_latest_checkpoint(path, suffix='pth')[source]¶

Find the latest checkpoint from the working directory.

Parameters

path (str) – The path to find checkpoints.
suffix (str) – File extension. Defaults to pth.

Returns

File path of the latest checkpoint.

Return type

latest_path(str | None)

References

1: https://github.com/microsoft/SoftTeacher /blob/main/ssod/utils/patch.py

mmrazor.utils.get_package_placeholder(string: str) → object[source]¶

Get placeholder instance which can avoid raising errors when down-stream dependency is not installed properly.

Parameters: string (str) – the dependency’s name, i.e. mmcls
Raises: ImportError – raise it when the dependency is not installed properly.
Returns: PlaceHolder instance.
Return type: object

mmrazor.utils.get_placeholder(string: str) → object[source]¶

Get placeholder instance which can avoid raising errors when down-stream dependency is not installed properly.

Parameters: string (str) – the dependency’s name, i.e. mmcls
Raises: ImportError – raise it when the dependency is not installed properly.
Returns: PlaceHolder instance.
Return type: object

mmrazor.utils.register_all_modules(init_default_scope: bool = True) → None[source]¶

Register all modules in mmrazor into the registries.

Parameters: init_default_scope (bool) – Whether initialize the mmrazor default scope. When init_default_scope=True, the global default scope will be set to mmrazor, and all registries will build modules from mmrazor’s registry node. To understand more about the registry, please refer to https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/registry.md Defaults to True.

mmrazor.utils.setup_multi_processes(cfg)[source]¶: Setup multi-processing environment variables.