mmrazor.models¶
algorithms¶
architectures¶
distillers¶
losses¶
mutables¶
- class mmrazor.models.mutables.BaseMutable(alias: Optional[str] = None, init_cfg: Optional[Dict] = None)[source]¶
Base Class for mutables. Mutable means a searchable module widely used in Neural Architecture Search(NAS).
It mainly consists of some optional operations, and achieving searchable function by handling choice with
MUTATOR
.All subclass should implement the following APIs:
fix_chosen()
dump_chosen()
current_choice.setter()
current_choice.getter()
- Parameters
alias (str, optional) – alias of the MUTABLE.
init_cfg (dict, optional) – initialization configuration dict for
BaseModule
. OpenMMLab has implement 5 initializer including Constant, Xavier, Normal, Uniform, Kaiming, and Pretrained.
- abstract property current_choice¶
Current choice will affect
forward()
and will be used inmmrazor.core.subnet.utils.export_fix_subnet()
or mutator.
- abstract dump_chosen() → mmrazor.utils.typing.DumpChosen[source]¶
Save the current state of the mutable as a dictionary.
DumpChosen
haschosen
andmeta
fields.chosen
is necessary,fix_chosen
will use thechosen
.meta
is used to store some non-essential information.
- abstract fix_chosen(chosen) → None[source]¶
Fix mutable with chosen. This function would fix the chosen of mutable. The
is_fixed
will be set to True and only the selected operations can be retained. All subclasses must implement this method.Note
This operation is irreversible.
- property is_fixed: bool¶
whether the mutable is fixed.
Note
- If a mutable is fixed, it is no longer a searchable module, just
a normal fixed module.
If a mutable is not fixed, it still is a searchable module.
- Type
bool
- class mmrazor.models.mutables.BaseMutableChannel(num_channels: int, **kwargs)[source]¶
BaseMutableChannel works as a channel mask for DynamicOps to select channels.
|---------------------------------------| |mutable_in_channel(BaseMutableChannel) | |---------------------------------------| | DynamicOp | |---------------------------------------| |mutable_out_channel(BaseMutableChannel)| |---------------------------------------|
All subclasses should implement the following APIs and the other abstract method in
BaseMutable
current_mask
- Parameters
num_channels (int) – number(dimension) of channels(mask).
- property activated_channels: int¶
Number of activated channels.
- abstract property current_mask: torch.Tensor¶
Return a mask indicating the channel selection.
- class mmrazor.models.mutables.DCFFChannelUnit(num_channels: int, candidate_choices: List[Union[int, float]] = [1.0], choice_mode: str = 'ratio', divisor: int = 1, min_value: int = 1, min_ratio: float = 0.9)[source]¶
DCFFChannelUnit
is for supernet DCFF and based on OneShotMutableChannelUnit. In DCFF supernet, each module only has one choice. The channel choice is fixed before training.- Parameters
num_channels (int) – The raw number of channels.
candidate_choices (List[Union[int, float]], optional) – A list of candidate width numbers or ratios. Each candidate indicates how many channels to be reserved. Defaults to [1.0](choice_mode=’number’).
choice_mode (str, optional) – Mode of candidates. One of “ratio” or “number”. Defaults to ‘ratio’.
divisor (int) – Used to make choice divisible.
min_value (int) – the minimal value used when make divisible.
min_ratio (float) – the minimal ratio used when make divisible.
- class mmrazor.models.mutables.DMCPChannelUnit(num_channels: int, choice_mode: str = 'number', divisor: int = 1, min_value: int = 1, min_ratio: float = 0.5)[source]¶
DMCPChannelUnit
is for supernet DMCP and based on OneShotMutableChannelUnit. In DMCP supernet, each module only has one choice. The channel choice is fixed before training.Note
In dmcpunit, a new attribute activated_tensor_channels is defined
in self.mutable_channel, which is specifically used to store the number of channels in the form of tensor. Defaults to None.
- Parameters
num_channels (int) – The raw number of channels.
choice_mode (str, optional) – Mode of candidates. One of “ratio” or “number”. Defaults to ‘ratio’.
divisor (int) – Used to make choice divisible.
min_value (int) – the minimal value used when make divisible.
min_ratio (float) – the minimal ratio used when make divisible.
- class mmrazor.models.mutables.DerivedMutable(choice_fn: Callable, mask_fn: Optional[Callable] = None, source_mutables: Optional[Iterable[mmrazor.models.mutables.base_mutable.BaseMutable]] = None, alias: Optional[str] = None, init_cfg: Optional[Dict] = None)[source]¶
Class for derived mutable.
A derived mutable is a mutable derived from other mutables that has current_choice and current_mask attributes (if any).
Note
A derived mutable does not have its own search space, so it is not legal to modify its current_choice or current_mask directly. And the only way to modify them is by modifying current_choice or current_mask in corresponding source mutables.
- Parameters
choice_fn (callable) – A closure that controls how to generate current_choice.
mask_fn (callable, optional) – A closure that controls how to generate current_mask. Defaults to None.
source_mutables (iterable, optional) – Specify source mutables for this derived mutable. If the argument is None, source mutables will be traced automatically by parsing mutables in closure variables. Defaults to None.
alias (str, optional) – alias of the MUTABLE. Defaults to None.
init_cfg (dict, optional) – initialization configuration dict for
BaseModule
. OpenMMLab has implement 5 initializer including Constant, Xavier, Normal, Uniform, Kaiming, and Pretrained. Defaults to None.
Examples
>>> from mmrazor.models.mutables import SquentialMutableChannel >>> mutable_channel = SquentialMutableChannel(num_channels=3) >>> # derive expand mutable >>> derived_mutable_channel = mutable_channel * 2 >>> # source mutables will be traced automatically >>> derived_mutable_channel.source_mutables {SquentialMutableChannel(name=unbind, num_channels=3, current_choice=3)} # noqa: E501 >>> # modify `current_choice` of `mutable_channel` >>> mutable_channel.current_choice = 2 >>> # `current_choice` and `current_mask` of derived mutable will be modified automatically # noqa: E501 >>> derived_mutable_channel DerivedMutable(current_choice=4, activated_channels=4, source_mutables={SquentialMutableChannel(name=unbind, num_channels=3, current_choice=2)}, is_fixed=False) # noqa: E501
- property current_choice¶
Current choice of derived mutable.
- property current_mask: torch.Tensor¶
Current mask of derived mutable.
- dump_chosen() → mmrazor.utils.typing.DumpChosen[source]¶
Dump information of chosen.
- Returns
Dumped information.
- Return type
Dict
- fix_chosen(chosen) → None[source]¶
Fix mutable with subnet config.
Warning
Fix derived mutable will have no actually effect.
- property is_fixed: bool¶
Whether the derived mutable is fixed.
Note
Depends on whether all source mutables are already fixed.
- static is_source_mutable(mutable: object) → bool[source]¶
Judge whether an object is source mutable(not derived mutable).
- Parameters
mutable (object) – An object.
- Returns
Indicate whether the object is source mutable or not.
- Return type
bool
- property num_choices: int¶
Number of all choices.
Note
Since derive mutable does not have its own search space, the number of choices will always be 1.
- Returns
Number of choices.
- Return type
int
- class mmrazor.models.mutables.DiffChoiceRoute(edges: torch.nn.modules.container.ModuleDict, num_chosen: int = 2, with_arch_param: bool = False, alias: Optional[str] = None, init_cfg: Optional[Dict] = None)[source]¶
A type of
MUTABLES
for Neural Architecture Search, which can select inputs from different edges in a differentiable or non-differentiable way. It is commonly used in DARTS.- Parameters
edges (nn.ModuleDict) – the key of edges is the name of different edges. The value of edges can be
nn.Module
orDiffMutableModule
.with_arch_param (bool) – whether forward with arch_param. When set to True, a differentiable way is adopted. When set to False, a non-differentiable way is adopted.
alias (str, optional) – alias of the DiffChoiceRoute.
init_cfg (dict, optional) – initialization configuration dict for
BaseModule
. OpenMMLab has implement 6 initializers including Constant, Xavier, Normal, Uniform, Kaiming, and Pretrained.
Examples
>>> import torch >>> import torch.nn as nn >>> edges_dict=nn.ModuleDict() >>> edges_dict.add_module('first_edge', nn.Conv2d(32, 32, 3, 1, 1)) >>> edges_dict.add_module('second_edge', nn.Conv2d(32, 32, 5, 1, 2)) >>> edges_dict.add_module('third_edge', nn.MaxPool2d(3, 1, 1)) >>> edges_dict.add_module('fourth_edge', nn.MaxPool2d(5, 1, 2)) >>> edges_dict.add_module('fifth_edge', nn.MaxPool2d(7, 1, 3)) >>> diff_choice_route_cfg = dict( ... type="DiffChoiceRoute", ... edges=edges_dict, ... with_arch_param=True, ... ) >>> arch_param Parameter containing: tensor([-6.1426e-04, 2.3596e-04, 1.4427e-03, 7.1668e-05, -8.9739e-04], requires_grad=True) >>> x = [torch.randn(4, 32, 64, 64) for _ in range(5)] >>> output=diffchoiceroute.forward_arch_param(x, arch_param) >>> output.shape torch.Size([4, 32, 64, 64])
- property choices: List[str]¶
all choices.
- Type
list
- dump_chosen() → mmrazor.utils.typing.DumpChosen[source]¶
Save the current state of the mutable as a dictionary.
DumpChosen
haschosen
andmeta
fields.chosen
is necessary,fix_chosen
will use thechosen
.meta
is used to store some non-essential information.
- fix_chosen(chosen: List[str]) → None[source]¶
Fix mutable with choice. This operation would convert to fixed mode. The
is_fixed
will be set to True and only the selected operations can be retained.- Parameters
chosen (list(str)) – the chosen key in
MUTABLE
.
- forward(x: Any, arch_param: Optional[torch.nn.parameter.Parameter] = None)[source]¶
Calls either
forward_fixed()
orforward_arch_param()
depending on whetheris_fixed()
isTrue
and whetherarch_param()
is None.To reduce the coupling between Mutable and Mutator, the arch_param is generated by the Mutator and is passed to the forward function as an argument.
Note
forward_fixed()
is called when in fixed mode.forward_arch_param()
is called when in unfixed mode.- Parameters
x (Any) – input data for forward computation.
arch_param (nn.Parameter, optional) – the architecture parameters for
DiffMutableModule
.
- Returns
the result of forward
- Return type
Any
- forward_all(x)[source]¶
Forward all choices.
- Parameters
x (Any) – x could be a Torch.tensor or a tuple of Torch.tensor, containing input data for forward computation.
- Returns
the result of forward all of the
choice
operation.- Return type
Tensor
- forward_arch_param(x, arch_param: torch.nn.parameter.Parameter) → torch.Tensor[source]¶
Forward with architecture parameters.
- Parameters
x (list[Any] | tuple[Any]]) – x could be a list or a tuple of Torch.tensor, containing input data for forward selection.
arch_param (nn.Parameter) – architecture parameters for for
DiffMutableModule
.
- Returns
the result of forward with
arch_param
.- Return type
Tensor
- forward_fixed(inputs: Union[List, Tuple]) → torch.Tensor[source]¶
Forward when the mutable is in fixed mode.
- Parameters
inputs (Union[List[Any], Tuple[Any]]) – inputs could be a list or a tuple of Torch.tensor, containing input data for forward computation.
- Returns
the result of forward the fixed operation.
- Return type
Tensor
- class mmrazor.models.mutables.DiffMutableModule(**kwargs)[source]¶
Base class for differentiable mutables.
- Parameters
module_kwargs (dict[str, dict], optional) – Module initialization named arguments. Defaults to None.
alias (str, optional) – alias of the MUTABLE.
init_cfg (dict, optional) – initialization configuration dict for
BaseModule
. OpenMMLab has implement 5 initializer including Constant, Xavier, Normal, Uniform, Kaiming, and Pretrained.
Note
forward_all()
is called when calculating FLOPs.- compute_arch_probs(arch_param: torch.nn.parameter.Parameter) → torch.Tensor[source]¶
compute chosen probs according to architecture params.
- forward(x: Any, arch_param: Optional[torch.nn.parameter.Parameter] = None)[source]¶
Calls either
forward_fixed()
orforward_arch_param()
depending on whetheris_fixed()
isTrue
and whetherarch_param()
is None.To reduce the coupling between Mutable and Mutator, the arch_param is generated by the Mutator and is passed to the forward function as an argument.
Note
forward_fixed()
is called when in fixed mode.forward_arch_param()
is called when in unfixed mode.- Parameters
x (Any) – input data for forward computation.
arch_param (nn.Parameter, optional) – the architecture parameters for
DiffMutableModule
.
- Returns
the result of forward
- Return type
Any
- class mmrazor.models.mutables.DiffMutableOP(candidates: Dict[str, Dict], fix_threshold: float = 1.0, module_kwargs: Optional[Dict[str, Dict]] = None, alias: Optional[str] = None, init_cfg: Optional[Dict] = None)[source]¶
A type of
MUTABLES
for differentiable architecture search, such as DARTS. Search the best module by learnable parameters arch_param.- Parameters
candidates (dict[str, dict]) – the configs for the candidate operations.
fix_threshold (float) – The threshold that determines whether to fix the choice of current module as the op with the maximum probs. It happens when the maximum prob is fix_threshold or more higher then all the other probs. Default to 1.0.
module_kwargs (dict[str, dict], optional) – Module initialization named arguments. Defaults to None.
alias (str, optional) – alias of the MUTABLE.
init_cfg (dict, optional) – initialization configuration dict for
BaseModule
. OpenMMLab has implement 5 initializer including Constant, Xavier, Normal, Uniform, Kaiming, and Pretrained.
- property choices: List[str]¶
all choices.
- Type
list
- dump_chosen() → mmrazor.utils.typing.DumpChosen[source]¶
Save the current state of the mutable as a dictionary.
DumpChosen
haschosen
andmeta
fields.chosen
is necessary,fix_chosen
will use thechosen
.meta
is used to store some non-essential information.
- fix_chosen(chosen: Union[str, List[str]]) → None[source]¶
Fix mutable with choice. This operation would convert unfixed mode to fixed mode. The
is_fixed
will be set to True and only the selected operations can be retained.- Parameters
chosen (str) – the chosen key in
MUTABLE
. Defaults to None.
- forward_all(x) → torch.Tensor[source]¶
Forward all choices. Used to calculate FLOPs.
- Parameters
x (Any) – x could be a Torch.tensor or a tuple of Torch.tensor, containing input data for forward computation.
- Returns
the result of forward all of the
choice
operation.- Return type
Tensor
- forward_arch_param(x, arch_param: torch.nn.parameter.Parameter) → torch.Tensor[source]¶
Forward with architecture parameters.
- Parameters
x (Any) – x could be a Torch.tensor or a tuple of Torch.tensor, containing input data for forward computation.
arch_param (str, optional) – architecture parameters for DiffMutableModule
- Returns
the result of forward with
arch_param
.- Return type
Tensor
- class mmrazor.models.mutables.L1MutableChannelUnit(num_channels: int, choice_mode='number', divisor=1, min_value=1, min_ratio=0.9)[source]¶
Implementation of L1-norm pruning algorithm. It compute the l1-norm of modules and preferly prune the modules with less l1-norm.
Please refer to papre https://arxiv.org/pdf/1608.08710.pdf for more detail.
- property current_choice: Union[int, float]¶
return current choice.
- class mmrazor.models.mutables.MutableChannelContainer(num_channels: int, **kwargs)[source]¶
MutableChannelContainer inherits from BaseMutableChannel. However, it’s not a single BaseMutableChannel, but a container for BaseMutableChannel. The mask of MutableChannelContainer consists of all masks of stored MutableChannels.
MutableChannelContainer |- Important interfaces:
- register_mutable: register/store BaseMutableChannel in the
MutableChannelContainer
- property current_choice: torch.Tensor¶
Get current choices.
- property current_mask: torch.Tensor¶
Return current mask.
- register_mutable(mutable_channel: mmrazor.models.mutables.mutable_channel.base_mutable_channel.BaseMutableChannel, start: int, end: int)[source]¶
Register/Store BaseMutableChannel in the MutableChannelContainer in the range [start,end)
- classmethod register_mutable_channel_to_module(module: mmrazor.models.architectures.dynamic_ops.mixins.dynamic_mixins.DynamicChannelMixin, mutable: mmrazor.models.mutables.mutable_channel.base_mutable_channel.BaseMutableChannel, is_to_output_channel=True, start=0, end=- 1)[source]¶
Register a BaseMutableChannel to a module with MutableChannelContainers.
- class mmrazor.models.mutables.MutableChannelUnit(num_channels: int, **kwargs)[source]¶
- config_template(with_init_args=False, with_channels=False) → Dict[source]¶
Return the config template of this unit. By default, the config template only includes a key ‘choice’.
- Parameters
with_init_args (bool) – if the config includes args for initialization.
with_channels (bool) – if the config includes info about channels. the config with info about channels can used to parse channel units without tracer.
- property current_choice¶
Choice of this unit.
- classmethod init_from_cfg(model: torch.nn.modules.module.Module, config: Dict)[source]¶
init a Channel using a config which can be generated by self.config_template(), include init choice.
- classmethod init_from_predefined_model(model: torch.nn.modules.module.Module)[source]¶
Initialize units using the model with pre-defined dynamicops and mutable-channels.
- property is_mutable: bool¶
If the channel-unit is prunable.
- property mutable_prefix: str¶
Mutable prefix.
- class mmrazor.models.mutables.MutableValue(value_list: List[Union[int, float]], default_value: Optional[Any] = None, alias: Optional[str] = None, init_cfg: Optional[Dict] = None)[source]¶
Base class for mutable value.
A mutable value is actually a mutable that adds some functionality to a list containing objects of the same type.
- Parameters
value_list (list) – List of value, each value must have the same type.
default_value (any, optional) – Default value, must be one in value_list. Default to None.
alias (str, optional) – alias of the MUTABLE.
init_cfg (dict, optional) – initialization configuration dict for
BaseModule
. OpenMMLab has implement 5 initializer including Constant, Xavier, Normal, Uniform, Kaiming, and Pretrained.
- property choices: List[Any]¶
List of choices.
- property current_choice: Union[int, float]¶
Current choice of mutable value.
- dump_chosen() → mmrazor.utils.typing.DumpChosen[source]¶
Dump information of chosen.
- Returns
Dumped information.
- Return type
Dict[str, Any]
- fix_chosen(chosen: Union[int, float]) → None[source]¶
Fix mutable value with subnet config.
- Parameters
chosen (dict) – the information of chosen.
- property mutable_prefix: str¶
Mutable prefix.
- property num_choices: int¶
Number of all choices.
- Returns
Number of choices.
- Return type
int
- class mmrazor.models.mutables.OneHotMutableOP(candidates: Dict[str, Dict], fix_threshold: float = 1.0, module_kwargs: Optional[Dict[str, Dict]] = None, alias: Optional[str] = None, init_cfg: Optional[Dict] = None)[source]¶
A type of
MUTABLES
for one-hot sample based architecture search, such as DSNAS. Search the best module by learnable parameters arch_param.- Parameters
candidates (dict[str, dict]) – the configs for the candidate operations.
module_kwargs (dict[str, dict], optional) – Module initialization named arguments. Defaults to None.
alias (str, optional) – alias of the MUTABLE.
init_cfg (dict, optional) – initialization configuration dict for
BaseModule
. OpenMMLab has implement 5 initializer including Constant, Xavier, Normal, Uniform, Kaiming, and Pretrained.
- forward_arch_param(x: Any, arch_param: torch.nn.parameter.Parameter) → torch.Tensor[source]¶
Forward with architecture parameters.
- Parameters
x (Any) – x could be a Torch.tensor or a tuple of Torch.tensor, containing input data for forward computation.
arch_param (str, optional) – architecture parameters for DiffMutableModule.
- Returns
the result of forward with
arch_param
.- Return type
Tensor
- sample_weights(arch_param: torch.nn.parameter.Parameter, probs: torch.Tensor, random_sample: bool = False) → torch.Tensor[source]¶
Use one-hot distributions to sample the arch weights based on the arch params.
- Parameters
arch_param (nn.Parameter) – architecture parameters for DiffMutableModule.
probs (Tensor) – the probs of choice.
random_sample (bool) – Whether to random sample arch weights or not Defaults to False.
- Returns
Sampled one-hot arch weights.
- Return type
Tensor
- class mmrazor.models.mutables.OneShotMutableChannel(num_channels: int, candidate_choices: List[Union[int, float]] = [], choice_mode='number', **kwargs)[source]¶
OneShotMutableChannel is a subclass of SquentialMutableChannel. The difference is that a OneShotMutableChannel limits the candidates of the choice.
- Parameters
num_channels (int) – number of channels.
candidate_choices (List[Union[float, int]], optional) – A list of candidate width ratios. Each candidate indicates how many channels to be reserved. Defaults to [].
choice_mode (str, optional) – Mode of choices. Defaults to ‘number’.
- property current_choice: Union[int, float]¶
Get current choice.
- class mmrazor.models.mutables.OneShotMutableChannelUnit(num_channels: int, candidate_choices: List[Union[int, float]] = [0.5, 1.0], choice_mode='ratio', divisor=1, min_value=1, min_ratio=0.9)[source]¶
OneShotMutableChannelUnit is for single path supernet such as AutoSlim. In single path supernet, each module only has one choice invoked at the same time. A path is obtained by sampling all the available choices. It is the base class for one shot mutable channel.
- Parameters
num_channels (_type_) – The raw number of channels.
candidate_choices (List[Union[int, float]], optional) – A list of candidate width ratios. Each candidate indicates how many channels to be reserved. Defaults to [0.5, 1.0](choice_mode=’ratio’).
choice_mode (str, optional) – Mode of candidates. One of “ratio” or “number”. Defaults to ‘ratio’.
divisor (int) – Used to make choice divisible.
min_value (int) – the minimal value used when make divisible.
min_ratio (float) – the minimal ratio used when make divisible.
- config_template(with_init_args=False, with_channels=False) → Dict[source]¶
Config template of the OneShotMutableChannelUnit.
- property current_choice: Union[int, float]¶
Get current choice.
- property max_choice: Union[int, float]¶
Get Maximal choice.
- property min_choice: Union[int, float]¶
Get Minimal choice.
- class mmrazor.models.mutables.OneShotMutableModule(module_kwargs: Optional[Dict[str, Dict]] = None, alias: Optional[str] = None, init_cfg: Optional[Dict] = None)[source]¶
Base class for one shot mutable module. A base type of
MUTABLES
for single path supernet such as Single Path One Shot.All subclass should implement the following APIs and the other abstract method in
MutableModule
:sample_choice()
forward_choice()
Note
forward_all()
is called when calculating FLOPs.- forward(x: Any) → Any[source]¶
Calls either
forward_fixed()
orforward_choice()
depending on whetheris_fixed()
isTrue
and whethercurrent_choice()
is None.Note
forward_fixed()
is called in fixed mode.forward_all()
is called in unfixed mode withcurrent_choice()
is None.forward_choice()
is called in unfixed mode withcurrent_choice()
is not None.
- Parameters
x (Any) – input data for forward computation.
choice (CHOICE_TYPE, optional) – the chosen key in
MUTABLE
.
- Returns
the result of forward
- Return type
Any
- class mmrazor.models.mutables.OneShotMutableOP(candidates: Union[Dict[str, Dict], torch.nn.modules.container.ModuleDict], module_kwargs: Optional[Dict[str, Dict]] = None, alias: Optional[str] = None, init_cfg: Optional[Dict] = None)[source]¶
A type of
MUTABLES
for single path supernet, such as Single Path One Shot. In single path supernet, each choice block only has one choice invoked at the same time. A path is obtained by sampling all the choice blocks.- Parameters
candidates (dict[str, dict]) – the configs for the candidate operations.
module_kwargs (dict[str, dict], optional) – Module initialization named arguments. Defaults to None.
alias (str, optional) – alias of the MUTABLE.
init_cfg (dict, optional) – initialization configuration dict for
BaseModule
. OpenMMLab has implement 5 initializer including Constant, Xavier, Normal, Uniform, Kaiming, and Pretrained.
Examples
>>> import torch >>> from mmrazor.models.mutables import OneShotMutableOP
>>> candidates = nn.ModuleDict({ ... 'conv3x3': nn.Conv2d(32, 32, 3, 1, 1), ... 'conv5x5': nn.Conv2d(32, 32, 5, 1, 2),
>>> input = torch.randn(1, 32, 64, 64) >>> op = OneShotMutableOP(candidates)
>>> op.choices ['conv3x3', 'conv5x5', 'conv7x7'] >>> op.num_choices 3 >>> op.is_fixed False
>>> op.current_choice = 'conv3x3' >>> unfix_output = op.forward(input) >>> torch.all(unfixed_output == candidates['conv3x3'](input)) True
>>> op.fix_chosen('conv3x3') >>> fix_output = op.forward(input) >>> torch.all(fix_output == unfix_output) True
>>> op.choices ['conv3x3'] >>> op.num_choices 1 >>> op.is_fixed True
- property choices: List[str]¶
all choices.
- Type
list
- dump_chosen() → mmrazor.utils.typing.DumpChosen[source]¶
Save the current state of the mutable as a dictionary.
DumpChosen
haschosen
andmeta
fields.chosen
is necessary,fix_chosen
will use thechosen
.meta
is used to store some non-essential information.
- fix_chosen(chosen: str) → None[source]¶
Fix mutable with subnet config. This operation would convert unfixed mode to fixed mode. The
is_fixed
will be set to True and only the selected operations can be retained.- Parameters
chosen (str) – the chosen key in
MUTABLE
. Defaults to None.
- forward_all(x) → torch.Tensor[source]¶
Forward all choices. Used to calculate FLOPs.
- Parameters
x (Any) – x could be a Torch.tensor or a tuple of Torch.tensor, containing input data for forward computation.
- Returns
the result of forward all of the
choice
operation.- Return type
Tensor
- forward_choice(x, choice: str) → torch.Tensor[source]¶
Forward with the unfixed mutable and current choice is not None.
- Parameters
x (Any) – x could be a Torch.tensor or a tuple of Torch.tensor, containing input data for forward computation.
choice (str) – the chosen key in
OneShotMutableOP
.
- Returns
the result of forward the
choice
operation.- Return type
Tensor
- class mmrazor.models.mutables.OneShotMutableValue(value_list: List[Any], default_value: Optional[Any] = None, alias: Optional[str] = None, init_cfg: Optional[Dict] = None)[source]¶
Class for one-shot mutable value.
one-shot mutable value provides sample_choice method and min_choice, max_choice properties on the top of mutable value.
- Parameters
value_list (list) – List of value, each value must have the same type.
default_value (any, optional) – Default value, must be one in value_list. Default to None.
alias (str, optional) – alias of the MUTABLE.
init_cfg (dict, optional) – initialization configuration dict for
BaseModule
. OpenMMLab has implement 5 initializer including Constant, Xavier, Normal, Uniform, Kaiming, and Pretrained.
- property max_choice: Any¶
Max choice of all choices.
- Returns
Max choice.
- Return type
Any
- property min_choice: Any¶
Min choice of all choices.
- Returns
Min choice.
- Return type
Any
- class mmrazor.models.mutables.SequentialMutableChannelUnit(num_channels: int, choice_mode='number', divisor=1, min_value=1, min_ratio=0.9)[source]¶
SequentialMutableChannelUnit accepts a intger(number) or float(ratio) as the choice, which indicates how many of the channels are remained from left to right, like 11110000.
- Parameters
num_channels (int) – number of channels.
choice_mode (str) – mode of choice, which is one of ‘number’ or ‘ratio’.
divisor (int) – Used to make choice divisible.
min_value (int) – the minimal value used when make divisible.
min_ratio (float) – the minimal ratio used when make divisible.
- property current_choice: Union[int, float]¶
return current choice.
- class mmrazor.models.mutables.SimpleMutableChannel(num_channels: int, **kwargs)[source]¶
SimpleMutableChannel is a simple BaseMutableChannel, it directly take a mask as a choice.
- Parameters
num_channels (int) – number of channels.
- property current_choice: torch.Tensor¶
Get current choice.
- property current_mask: torch.Tensor¶
Get current mask.
- expand_mutable_channel(expand_ratio: Union[int, float]) → mmrazor.models.mutables.derived_mutable.DerivedMutable[source]¶
Get a derived SimpleMutableChannel with expanded mask.
- class mmrazor.models.mutables.SlimmableChannelUnit(num_channels: int, candidate_choices: List[Union[int, float]] = [], choice_mode='number', divisor=1, min_value=1, min_ratio=0.9)[source]¶
A type of
MutableChannelUnit
to train several subnets together.- Parameters
num_channels (int) – The raw number of channels.
candidate_choices (List[Union[int, float]], optional) – A list of candidate width ratios. Each candidate indicates how many channels to be reserved. Defaults to [0.5, 1.0](choice_mode=’ratio’).
choice_mode (str, optional) – Mode of candidates. One of ‘ratio’ or ‘number’. Defaults to ‘number’.
divisor (int, optional) – Used to make choice divisible.
min_value (int, optional) – The minimal value used when make divisible.
min_ratio (float, optional) – The minimal ratio used when make divisible.
- class mmrazor.models.mutables.SquentialMutableChannel(num_channels: int, choice_mode='number', **kwargs)[source]¶
SquentialMutableChannel defines a BaseMutableChannel which switch off channel mask from right to left sequentially, like ‘11111000’.
A choice of SquentialMutableChannel is an integer, which indicates how many channel are activated from left to right.
- Parameters
num_channels (int) – number of channels.
- property current_choice: Union[int, float]¶
Get current choice.
- property current_mask: torch.Tensor¶
Return current mask.
- property is_num_mode¶
Get if the choice is number mode.
mutators¶
ops¶
task_modules¶
utils¶
- mmrazor.models.utils.add_prefix(inputs: Dict, prefix: str) → Dict[source]¶
Add prefix for dict.
- Parameters
inputs (dict) – The input dict with str keys.
prefix (str) – The prefix to add.
- Returns
The dict with keys updated with
prefix
.- Return type
dict
- mmrazor.models.utils.get_module_device(module: torch.nn.modules.module.Module) → torch.device[source]¶
Get the device of a module.
- Parameters
module (nn.Module) – A module contains the parameters.
- mmrazor.models.utils.make_divisible(value: int, divisor: int, min_value: Optional[int] = None, min_ratio: float = 0.9) → int[source]¶
Make divisible function.
This function rounds the channel number down to the nearest value that can be divisible by the divisor.
- Parameters
value (int) – The original channel number.
divisor (int) – The divisor to fully divide the channel number.
min_value (int, optional) – The minimum value of the output channel. Default: None, means that the minimum value equal to the divisor.
min_ratio (float) – The minimum ratio of the rounded channel number to the original channel number. Default: 0.9.
- Returns
The modified output channel number
- Return type
int
- mmrazor.models.utils.parse_values(candidate_lists: List[list])[source]¶
Parse a list with format (min_range, max_range, step).
NOTE: this method is required when customizing search space in configs.
- mmrazor.models.utils.pop_rewriter_function_record(rewriter_context, function_record_to_pop)[source]¶
Delete user-specific rewriters from RewriterContext._rewriter_manager.
We use the model which is rewritten by mmdeploy to build quantized models. However not all the functions rewritten by mmdeploy need to be rewritten in mmrazor. For example, mmdeploy rewrite mmcls.models.classifiers.ImageClassifier.forward and mmcls.models.classifiers.BaseClassifier.forward for deployment. But they can’t be rewritten by mmrazor as ptq and qat are done in mmrazor. So to ensure ptq and qat proceed normally, we have to remove these record from RewriterContext._rewriter_manager.
- mmrazor.models.utils.set_requires_grad(nets: Union[torch.nn.modules.module.Module, List[torch.nn.modules.module.Module]], requires_grad: bool = False) → None[source]¶
Set requires_grad for all the networks.
- Parameters
nets (nn.Module | list[nn.Module]) – A list of networks or a single network.
requires_grad (bool) – Whether the networks require gradients or not
mmrazor.registry¶
mmrazor.structures¶
delivery¶
graph¶
recorder¶
subnet¶
- class mmrazor.structures.subnet.Candidates(initdata: Optional[Union[Dict, List[Dict], Dict[str, Dict], List[Dict[str, Dict]]]] = None)[source]¶
The data structure of sampled candidate. The format is Union[Dict[str, Dict], List[Dict[str, Dict]]]. .. rubric:: Examples
>>> candidates = Candidates() >>> subnet_1 = {'1': 'choice1', '2': 'choice2'} >>> candidates.append(subnet_1) >>> candidates [{"{'1': 'choice1', '2': 'choice2'}": {'score': 0.0, 'flops': 0.0, 'params': 0.0, 'latency': 0.0}}] >>> candidates.set_resources(0, 49.9, 'flops') >>> candidates.set_score(0, 100.) >>> candidates [{"{'1': 'choice1', '2': 'choice2'}": {'score': 100.0, 'flops': 49.9, 'params': 0.0, 'latency': 0.0}}] >>> subnet_2 = {'choice_3': 'layer_3', 'choice_4': 'layer_4'} >>> candidates.append(subnet_2) >>> candidates [{"{'1': 'choice1', '2': 'choice2'}": {'score': 100.0, 'flops': 49.9, 'params': 0.0, 'latency': 0.0}}, {"{'choice_3': 'layer_3', 'choice_4':'layer_4'}": {'score': 0.0, 'flops': 0.0, 'params': 0.0, 'latency': 0.0}}] >>> candidates.subnets [{'1': 'choice1', '2': 'choice2'}, {'choice_3': 'layer_3', 'choice_4': 'layer_4'}] >>> candidates.resources('flops') [49.9, 0.0] >>> candidates.scores [100.0, 0.0]
- append(item: Union[Dict, List[Dict], Dict[str, Dict], List[Dict[str, Dict]]]) → None[source]¶
Append operation.
- insert(i: int, item: Union[Dict, List[Dict], Dict[str, Dict], List[Dict[str, Dict]]]) → None[source]¶
Insert operation.
- property scores: List[float]¶
The scores of candidates.
- set_resource(i: int, resources: float, key_indicator: str = 'flops') → None[source]¶
Set resources to the specified subnet by index.
- sort_by(key_indicator: str = 'score', reverse: bool = True) → None[source]¶
Sort by a specific indicator in descending order.
- Parameters
key_indicator (str) – sort all candidates by key_indicator. Defaults to ‘score’.
reverse (bool) – sort all candidates in descending order.
- property subnets: List[Dict]¶
The subnets of candidates.
- mmrazor.structures.subnet.convert_fix_subnet(fix_subnet: Dict[str, mmrazor.utils.typing.DumpChosen])[source]¶
Convert the fixed subnet to avoid python typing error.
- mmrazor.structures.subnet.export_fix_subnet(model: torch.nn.modules.module.Module, export_subnet_mode: str = 'mutable', slice_weight: bool = False, export_channel: bool = False) → Tuple[Dict[str, Any], Optional[Dict]][source]¶
Export subnet that can be loaded by
load_fix_subnet()
. Include subnet structure and subnet weight.- Parameters
model (nn.Module) – The target model to export.
export_subnet_mode (bool) – Subnet export method choice. Export by mutable.dump_chosen() when set to ‘mutable’ (NAS) Export by mutator.config_template() when set to ‘mutator’ (Prune)
slice_weight (bool) – Export subnet weight. Default to False.
export_channel (bool) – Whether to export the mutator’s channel. Often required when finetune is needed for the exported subnet. Default to False.
- Returns
Exported subnet choice config. static_model (Optional[Dict]): Exported static model state_dict.
Valid when `slice_weight`=True.
- Return type
fix_subnet (ValidFixMutable)
tracer¶
mmrazor.utils¶
- class mmrazor.utils.IndexDict[source]¶
IndexDict inherits from OrderedDict[Tuple[int, int], VT]. Each IndexDict object is a OrderDict object which using index(Tuple[int,int]) as key and Any as value.
The key type is Tuple[a: int,b: int]. It indicates a range in the [a,b).
IndexDict has three features: 1. ensure a key always is a index(Tuple[int,int]). 1. ensure the the indexes are sorted by ascending order. 2. ensure there is no overlap among indexes.
- mmrazor.utils.find_latest_checkpoint(path, suffix='pth')[source]¶
Find the latest checkpoint from the working directory.
- Parameters
path (str) – The path to find checkpoints.
suffix (str) – File extension. Defaults to pth.
- Returns
File path of the latest checkpoint.
- Return type
latest_path(str | None)
References
- 1
https://github.com/microsoft/SoftTeacher /blob/main/ssod/utils/patch.py
- mmrazor.utils.get_package_placeholder(string: str) → object[source]¶
Get placeholder instance which can avoid raising errors when down-stream dependency is not installed properly.
- Parameters
string (str) – the dependency’s name, i.e. mmcls
- Raises
ImportError – raise it when the dependency is not installed properly.
- Returns
PlaceHolder instance.
- Return type
object
- mmrazor.utils.get_placeholder(string: str) → object[source]¶
Get placeholder instance which can avoid raising errors when down-stream dependency is not installed properly.
- Parameters
string (str) – the dependency’s name, i.e. mmcls
- Raises
ImportError – raise it when the dependency is not installed properly.
- Returns
PlaceHolder instance.
- Return type
object
- mmrazor.utils.register_all_modules(init_default_scope: bool = True) → None[source]¶
Register all modules in mmrazor into the registries.
- Parameters
init_default_scope (bool) – Whether initialize the mmrazor default scope. When init_default_scope=True, the global default scope will be set to mmrazor, and all registries will build modules from mmrazor’s registry node. To understand more about the registry, please refer to https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/registry.md Defaults to True.