Tutorial

This tutorial goes through a simple example generating a C file from an input block specification. For this example the running sources are found in the tests/inputs folder of the ZOTI-Gen project root.

The target file which we are trying to obtain in this tutorial is a C file representing the sequential implementation of a moving-average algorithm applied on a sequence of numbers. It consists in a composite pattern described functionally as

\[mav_1 (x) = \mathtt{ShiftMap}(\mathtt{MapReduce}(\mathtt{mulacc}))(x)\]

Specification Files

We define the code blocks structure for the implemented system in the files main.yaml representing module main and genspec_leafs.yaml representing module genspec_leafs.

main.yaml

module: main
version: '0.1.0'
description: Source for tutorial
top: main
---
block:
- name: main
  type: {module: Generic, name: Infinite}
  prototype: |
      int {{ name }}(void) {
        int input[10];
        int output[10];
        int COEF[4] = {1, 2, 2, 1};
        {{placeholder['code']}}
      } 
  label:
  - {name: input, usage: "{{ label[p].name }}"}
  - {name: output, usage: "{{ label[p].name }}"}
  param:
    schedule: ["readio", "exec", "writeio"]
    myformat: "\"%d\"" # TEST
  instance:
  - placeholder: readio
    block: {module: Generic.IO, name: ReadArray}
    directive: ["expand"]
    bind:
    - value_to_param: {value: "\"%d\"", child: format}
    - value_to_param: {value: "10", child: size}
    - label_to_label: {parent: input, child: arg, usage: "{{ label[p].name }}"}
  - placeholder: writeio
    block: {module: Generic.IO, name: PrintArray}
    directive: ["expand"]
    bind:
    - value_to_param: {value: "10", child: size}
    - param_to_param: {child: format, parent: myformat}
    - label_to_label: {child: arg, parent: output, usage: "{{ label[p].name }}"}
  - placeholder: exec
    block: {module: main, name: mav_1}
    directive: ["expand"]
    bind: 
    - label_to_label: {child: in1, parent: input, usage: "{{ label[p].name }}"}
    - label_to_label: {child: out1, parent: output, usage: "{{ label[p].name }}"}
    usage: "int _it;\nint _range;\n{{placeholder['code']}}"

- name: mav_1
  type: {module: Skeleton.Compute, name: ShiftFarm}
  label:
  - {name: in1,    usage: "{{ label[p].name }}"}
  - {name: out1,   usage: "{{ label[p].name }}"}
  - {name: COEF,   usage: "{{ label[p].name }}"}
  - {name: _it,    usage: "{{ label[p].name }}"} 
  - {name: _range, usage: "{{ label[p].name }}"}
  param:
    iterate_over: [in1]
  instance:
  - placeholder: f
    block: {module: main, name: fred_1}
    directive: ["expand"]
    bind:
    - label_to_label:
        {parent: in1, child: in1, usage: "{{ label[p].name }}+{{label._it.name}}"}
    - label_to_label: {child: in2, parent: COEF, usage: "{{ label[p].name }}"}
    - label_to_label: {child: size1, parent: _range, usage: "{{ label[p].name }}"}
    - label_to_label:
        {parent: out1, child: out1, usage: "{{ label[p].name }}[{{label._it.name}}]"}
    usage: "int _it0;\nint _acc;\n{{placeholder['code']}}"

- name: fred_1
  type: {module: Skeleton.Compute, name: FarmRed_Acc}
  label:
  - {name: in1,   usage: "{{ label[p].name }}"}
  - {name: in2,   usage: "{{ label[p].name }}"}
  - {name: size1, usage: "{{ label[p].name }}"}
  - {name: out1,  usage: "{{ label[p].name }}"}
  - {name: _acc,  usage: "{{ label[p].name }}"}
  - {name: _it,   usage: "{{ label[p].name }}"}
  param:
    iterate_over:
      size1: { range: "name"}
      in2:
  instance:
  - placeholder: f
    block: {module: genspec_leafs, name: mulacc}
    # directive: ["expand"]
    usage: "{{name}}({{label.in1.name}}, {{label.in2.name}}, {{label.acc.name}}, {{label.out.name}})"
    bind:
    - label_to_label:
        parent: in1
        child: in1
        usage: "({{ label[p].name }})[{{label._it.name}}]"
    - label_to_label:
        parent: in2
        child: in2
        usage: "{{ label[p].name }}[{{label._it.name}}]"
    - label_to_label: {child: acc, parent: _acc, usage: "{{ label[p].name }}"}
    - label_to_label: {child: out, parent: _acc, usage: "{{ label[p].name }}"}

Notice the definition of mav_1 which is an implementation of component Skeleton.Compute.ShiftFarm (i.e., the shift-map pattern) and instantiates block fred_1 which is an implementation of Skeleton.Compute.FarmRed_Acc (i.e., the map-reduce pattern with an accumulator). The latter instantiates mulacc defined below as a regular block, containing the “native” kernel code in its code entry. Also note that the top block main implements a Generic.Infinite template (i.e., an infinite loop) scheduling three placeholders as of its param/schedule: readio (for reading a stream of input numbers), exec (expanding the mav_1 block) and writeio (for printing the results to the terminal).

genspec_leafs.yaml

module: genspec_leafs
top: mulacc
---
block:
- name: mulacc
  label:
  - {name: in1, usage: "{{ label[p].name }}"}
  - {name: in2, usage: "{{ label[p].name }}"}
  - {name: acc, usage: "{{ label[p].name }}"}
  - {name: out, usage: "{{ label[p].name }}"}
  prototype: "void {{name}}(in1, in2, acc, &out) {\n {{ placeholder['code'] }} \n};"
  code:
    out = acc + in1 * in2;

Notice that the input specification is as verbose as it needs to be to facilitate the parsing of data. This is because these files are not meant to be written by humans, much rather generated by upstream tools[1]. Also, the convenient information in the usage and prototype entries is also meant to be provided by an upstream pre-processor with the help of a type system, but this is outside the scope of ZOTI-Gen.

Bindings and Names

After resolving the structure here is how the block components look like. Pay attention to the label bindings, as well as their resolved names, as compared to their initial description.

Library Components

The library components are grouped into two modules (or packages): Generic for harboring generic templates, and Skeleton for defining templates for specialized collective operations, all targeting a sequential C implementation.

Generic

The generic components are spread throughout several files showcasing some of the template design tips described in Template Libraries.

Generic/__init__.py

from dataclasses import dataclass, field
from zoti_gen import with_schema, Block


@with_schema(Block.Schema)
@dataclass
class Infinite(Block):
    """ The base C implementation for an infinite loop. """

    code: str = field(
        default="\
while (1) { \n\
  {% for inst in param.schedule %} \n\
  {{ placeholder[inst] }} \n\
  {% endfor %} \n\
}"
    )

    def check(self):
        assert "schedule" in self.param
        assert isinstance(self.param["schedule"], list)

Generic/IO.py

from dataclasses import dataclass, field
from zoti_gen import read_at, with_schema, Block, Requirement, Template, C_DELIMITERS

file_args = {"module": __name__,
             "filename": "templates.c",
             "delimiters": C_DELIMITERS}


@with_schema(Block.Schema)
@dataclass
class ReadArray(Block):
    """Generic implementation for monitoring a system. Gets outside
    information from the console.
    """

    requirement: Requirement = field(
        default=Requirement({"include": ["stdio.h"]}))

    code: str = field(
        default=Template(
            read_at(**file_args, name="ReadArray.C"),
            parent="code")
    )

    def check(self):
        assert "format" in self.param
        assert "size" in self.param
        assert "arg" in self.label


@with_schema(Block.Schema)
@dataclass
class PrintArray(Block):
    """Generic implementation for monitoring a system. Prints an array to
    the console.
    """

    requirement: Requirement = field(
        default=Requirement({"include": ["stdio.h"]}))

    code: str = field(
        default=Template(
            read_at(**file_args, name="PrintArray.C"),
            parent="code")
    )

    def check(self):
        assert "format" in self.param
        assert "size" in self.param
        assert "arg" in self.label

Generic/templates.c

// Template: ReadArray.C
{int __io_it;
  for (__io_it = 0; __io_it < {{ param.size }}; __io_it++){
    scanf({{ param.format }}, &{{ label.arg.name }}[__io_it]);
  }
}
// End: ReadArray.C


// Template: PrintArray.C
{int __io_it;
  for (__io_it = 0; __io_it < {{ param.size }}; __io_it++){
    printf({{ param.format }}, {{ label.arg.name }}[__io_it]);
  }
}
// End: PrintArray.C

Specialized Collective Operations

The Skeleton module showcases more advanced template design using the ZOTI-Gen Jinja extensions.

Skeleton/Compute.py

from dataclasses import dataclass, field
from zoti_gen import read_at, with_schema, Block, Requirement, Template, C_DELIMITERS


file_args = {"module": __name__,
             "filename": "templates.c",
             "delimiters": C_DELIMITERS}


@with_schema(Block.Schema)
@dataclass
class ShiftFarm(Block):
    """
    Shift-farm specialized skeleton
    """

    requirement: Requirement = field(
        default=Requirement({"include": ["cutils.h"]}))

    code: str = field(
        default=Template(
            read_at(**file_args, name="ShiftFarm.C"),
            parent="code")
    )

    def check(self):
        # internal iterator (resolved name)
        assert "_it" in self.label

        # auto-bound to 'size1' from placeholder? TODO
        assert "_range" in self.label

        # called template
        assert "f" in [i.placeholder for i in self.instance]


@with_schema(Block.Schema)
@dataclass
class FarmRed_Acc(Block):
    """
    Farm-reduce skeleton with initial element, fused map-reduce kernel function
    and programmable (exposed) size param.
    """

    requirement: Requirement = field(
        default=Requirement({"include": ["cutils.h"]}))

    code: str = field(
        default=Template(
            read_at(**file_args, name="FarmRed_Acc.C"),
            parent="code")
    )

    def check(self):
        # internal iterator (resolved name)
        assert "_it" in self.label

        # auto-bound to 'size1' from placeholder? TODO
        assert "_acc" in self.label

        # label IDs over which '_it' iterates
        assert "iterate_over" in self.param

        # called fused farm-reduce kernel
        assert "f" in [i.placeholder for i in self.instance]

Notice that both ShiftFarm and FarmRed_Acc specify a requirement upon a certain target library header cutils.h. This is where functions used in their template such as min are supposed to be defined.

Skeleton/templates.c

// Template: ShiftFarm.C
{% set iterate_over = param.iterate_over|setdefaults({"offset": "0", "range": "name" }) %}

{% macro itrange() -%}
{% set ranges = [] %}
{% for k,p in iterate_over.items() -%}
{% do ranges.append(label[k]|find(p.range)) %}
{%- endfor %}min({{ ranges|join(", ") }})
{%- endmacro %}

for ({{ label._it.name }} = 0; {{ label._it.name }} < {{ itrange() }} ; {{ label._it.name }}++){
  {{ label._range.name }} = {{ itrange() }} - {{ label._it.name }};
  {{ placeholder["f"] }}
}
// End: ShiftFarm.C


// Template: FarmRed_Acc.C
{% set iterate_over = param.iterate_over|setdefaults({"offset": "0", "range": "name" }) %}

{% macro itrange() -%}
{% set ranges = [] %}
{% for k,p in iterate_over.items() -%}
{% do ranges.append(label[k]|find(p.range)) %}
{%- endfor %}min({{ ranges|join(", ") }})
{%- endmacro %}

for ({{ label._it.name }} = 0; {{ label._it.name }} < {{ itrange() }} ; {{ label._it.name }}++){
  {{ placeholder["f"] }}
}
{{ label.out1.name }} = {{ label._acc.name }};
// End: FarmRed_Acc.C

The template definition shows an example of using local Jinja macro, as well as the in-house zoti_gen.render.JinjaExtensions.setdefaults() filter.

Output Artifacts

Finally, the generated artifacts can be seen in this section. In this case the artifacts are presented as two separate files:

• a JSON file representing the include dependencies, useful for a post-processor, e.g. to generate the preamble of the .c file.

• a .c file containing the bulk of the code (without the preamble).

Dependencies

{"include": ["stdio.h", "cutils.h"]}

Notice that the generated code reflects the directive flags passed during block instantiation: all blocks are expanded as inline text except mulacc which is not using the "expand" directive. This is because it contains plain non-template native code which needs to be isolated as a separate function.

Output C code

void mulacc(in1, in2, acc, &out) {
  // vvv mulacc
  out = acc + in1 * in2;
  // ^^^ mulacc
};

int main(void) {
  int input[10];
  int output[10];
  int COEF[4] = {1, 2, 2, 1};
  // vvv main
  while (1) {

    // vvv ReadArray

    {
      int __io_it;
      for (__io_it = 0; __io_it < 10; __io_it++) {
        scanf("%d", &input[__io_it]);
      }
    }
    //
    // ^^^ ReadArray

    int _it;
    int _range;
    // vvv mav_1

    for (_it = 0; _it < min(input); _it++) {
      _range = min(input) - _it;
      int _it0;
      int _acc;
      // vvv fred_1

      for (_it0 = 0; _it0 < min(_range, COEF); _it0++) {
        mulacc((input + _it)[_it0], COEF[_it0], _acc, _acc)
      }
      output[_it] = _acc;
      //
      // ^^^ fred_1
    }
    //
    // ^^^ mav_1

    // vvv PrintArray

    {
      int __io_it;
      for (__io_it = 0; __io_it < 10; __io_it++) {
        printf("%d", output[__io_it]);
      }
    }
    //
    // ^^^ PrintArray
  }
  // ^^^ main
}

To obtain these files we ran the ZOTI-Gen CLI tool using the following configuration, assuming that all source files, libraries as well as the CLI tool are accessible from the current folder.

zoticonf.json

[zoti.gen]

lib = ["."]
deps = "deps.json"
input = ["main.yaml", "genspec_leafs.yaml"]
output = "out.c"
begin_block = "// vvv {comp.name}\n" 
end_block = "\n// ^^^ {comp.name}" 

---

[1]

For specifying the same input in a less verbose and slightly more feature-rich syntax, one could consider a more human-readable frontend, such as ZOTI-YAML.