SOL’s High Level Intermediate Representation (HLIR) is used to represent the
computation graph. HLIR mainly consists of the classes
sol::compiler::Layer, sol::compiler::LayerInput,
sol::compiler::LayerOutput and sol::compiler::Operation. Each Layer
requires at least one LayerOutput, no inputs and can have one Operation
assigned.
To add new Operations to SOL, you need to inherit the class
sol::compiler::Operation and implement the necessary function calls. You
can’t use class variables but instead need to use editJson("key") = value
to guarantee that the options get correctly stored within the SOL cache. Then
this new Operation needs to be registered to SOL using:
class MyFancyOp final : sol::compiler::Operation {
...
};
SOL_REGISTER_LAYERS(
Operation::registerOperation<MyFancyOp>();
)
Further, you need a function that can be called from the parser to add your new Operations. The following shows a minimalistic example with one input and one output.
extern "C" {
sol::compiler::LayerOutput* myNewCoolOp(LayerOutput* in, const int option) {
auto op = new(_fl) MyFancyOp(in->editNetwork(), option);
auto l = new(_fl) Layer(op);
auto li = l->addInput(IOType::Input, in);
auto lo = l->addOutput(IOType::Output, li);
return lo;
}
}
In your Python code this then needs to be connected like the following example. Please look at the framework’s SDK description how the exact syntax for the framework parser hooks look like, as this differs between these:
lib = ctypes.cdll.load('.../libsol-backend-...so')
def parser_hook(in, option):
return sol.hlir.Tensor(lib.myNewCoolLayer(sol.hlir.ptr(in), option))
SOL runs different processing stages and applies different optimizations. Please ensure that your processing steps can be run multiple times.
Once transformations are applied directly after parsing the model and are only applied once. These are mainly meant to correct problems from parsing the AI model, i.e., to higher the implementation of TF’s RNNs, or correct weird structure of TorchVision densenets that are incompatible to SOL’s HLIR.
The optimizer transformations are general transformations to simplify and
optimize the computation graph independent of the underlying device, i.e., to
simplify computations like: A * 1 = A. These can be added using:
sol::compiler::Optimizer::addTransformation(int, std::function<void(sol::compiler::Network*)>). The int is a priority, the
lower the earlier it gets executed. Be advised you can’t use the same priority
twice!
The device transformations are more specific. With
sol::compiler::Device::addTransformation(int, std::function<void(sol::compiler::Device*)>)
transformations can be added to ALL device types. By overriding
sol::compiler::Device::transformPostAutotune() or
sol::compiler::Device::transformPostOptimize() device type specific
optimizations can be added. Of course you can check the
sol::compiler::Device::deviceType() to apply transformations only to
specific devices, but it’s better to implement these within the backends, see
below.
When looking for optimal Backends for each layer, SOL executed the
transformations that can be added with
sol::compiler::AutoTuner::addTransformation(int, std::function<void(sol::compiler::Layer*)>). Further, the AutoTuner calls
sol::compiler::Operation::autotune() once the Algo gets assigned to the
Layer. This can be used to update/modify the hyper parameters of the Operation,
i.e., if the data layouts have changed.
To determine which Backend is optimal to use, SOL probes each Backend to get an
sol::compiler::Algo object and an estimated execution time (either
Heuristic, or measured). The best Algo will be used. The Algo object supports
the editDims(IOType) API, which allows to modify the Dims of the
LayerInputs and LayerOutputs. If modified, SOL will automatically add Reorder
layers if necessary to automatically ensure correct data formats for the layer.
This is, i.e., useful if the layer only supports NCHW data layouts, but the
network provides NHWC as layer input.
When SOL has determined a Backend to be the optimal choice, it runs the
sol::compiler::Backend::transformAlgo() function, allowing to transform
the layer to the user’s need. I.e., to modify the hyperparameters, split the
layers, or replace them with an equivalent but more optimal optimization. BE
AWARE that if you create new Layers in this step you need add an Algo object to
EACH new layer using sol::compiler::Layer::setAlgo().
The following is the execution order of all SOL transformations. Be aware that the Optimizer and Device transformations get executed multiple times, to ensure that whatever transformations Backends might apply, the generic optimizations get applied onto these.
sol::compiler::Optimizer::transformations()sol::compiler::AutoTuner::transformations()sol::compiler::Operation::autotune()sol::compiler::Optimizer::transformations()sol::compiler::Device::transformations()sol::compiler::Backend::transformAlgo()sol::compiler::Optimizer::transformations()sol::compiler::Device::transformations()sol::compiler::Device::transformPostAutotune()sol::compiler::Optimizer::transformations()sol::compiler::Device::transformations()sol::compiler::Device::transformPostOptimize()