bootstrap-vz/docs/guidelines.rst

Development guidelines
======================
The following guidelines should serve as general advice when
developing providers or plugins for bootstrap-vz. Keep in mind that
these guidelines are not rules , they are advice on how to better add
value to the bootstrap-vz codebase.


+ **The manifest should always fully describe the resulting image. The
  outcome of a bootstrapping process should never depend on settings
  specified elsewhere.**

  This allows others to easily reproduce any
  setup other people are running and makes it possible to share
  manifests. `The official debian EC2 images <https:/aws.amazon.com/marketplace/seller-
  profile?id=890be55d-32d8-4bc8-9042-2b4fd83064d5>`_
  for example can be reproduced using the manifests available
  in the manifest directory of bootstrap-vz.

+ **The bootstrapper should always be able to run fully unattended.**
  
  For end users, this guideline minimizes the risk of errors. Any
  required input would also be in direct conflict with the previous
  guideline that the manifest should always fully describe the resulting
  image.

  Additionally developers may have to run the bootstrap
  process multiple times though, any prompts in the middle of that
  process may significantly slow down the development speed.

+ **The bootstrapper should only need as much setup as the manifest
  requires.**

  Having to shuffle specific paths on the host into place
  (e.g. ``/target`` has to be created manually) to get the bootstrapper
  running is going to increase the rate of errors made by users.
  Aim for minimal setup.

  Exceptions are of course things such as the path to
  the VirtualBox Guest Additions ISO or tools like ``parted`` that
  need to be installed on the host.

+ **Roll complexity into which tasks are added to the tasklist.**

  If a ``run()`` function checks whether it should do any work or simply be
  skipped, consider doing that check in ``resolve_tasks()`` instead and
  avoid adding that task alltogether. This allows people looking at the
  tasklist in the logfile to determine what work has been performed. If
  a task says it will modify a file but then bails , a developer may get
  confused when looking at that file after bootstrapping. He could
  conclude that the file has either been overwritten or that the
  search & replace does not work correctly.

+ **Control flow should be directed from the task graph.**

  Avoid creating complicated ``run()`` functions. If necessary, split up
  a function into two semantically separate tasks.

  This allows other tasks to interleave with the control-flow and add extended
  functionality (e.g. because volume creation and mounting are two
  separate tasks, `the prebootstrapped plugin
  <http://andsens.github.io/bootstrap-vz/plugins.html#prebootstrapped>`_
  can replace the volume creation task with a task of its own that
  creates a volume from a snapshot instead, but still reuse the mount task).

+ **Task classes should be treated as decorated run() functions, they 
  should not have any state**

  Thats what the BootstrapInformation object is for.

+ **Only add stuff to the BootstrapInformation object when really necessary.**

  This is mainly to avoid clutter.

+ **Use a json-schema to check for allowed settings**
  The json-schema may be verbose but it keeps the bulk of check work outside the
  python code, which is a big plus when it comes to readability. This of
  course only applies bas long as the checks are simple. You can of
  course fall back to doing the check in python when that solution is
  considerably less complex.

+ **When invoking external programs, use long options whenever possible**

  This makes the commands a lot easier to understand, since
  the option names usually hint at what they do.

+ **When invoking external programs, don't use full paths, rely on ``$PATH``**

  This increases robustness when executable locations change.
  Example: Use ``log_call(['wget', ...])`` instead of ``log_call(['/usr/bin/wget', ...])``.


Commandline switches
--------------------
As a developer, there are commandline switches available which can
make your life a lot easier.

+ ``--debug``: Enables debug output in the console. This includes output
  from all commands that are invoked during bootstrapping.
+ ``--pause-on-error``: Pauses the execution when an exception occurs
  before rolling back. This allows you to debug by inspecting the volume
  at the time the error occured.
+ ``--dry-run``: Prevents the ``run()`` function from being called on all
  tasks. This is useful if you want to see whether the task order is
  correct.


Logfile
-------
Every run creates a new logfile in the ``logs/`` directory. The filename
for each run consists of a timestamp (``%Y%m%d%H%M%S``) and the basename
of the manifest used. The log also contains debugging statements
regardless of whether the ``--debug`` switch was used.


Inner workings
--------------

Tasks
~~~~~
At its core bootstrap-vz is based on tasks that perform units of work.
By keeping those tasks small and with a solid structure built around
them a high degree of flexibility can be achieved. To ensure that
tasks are executed in the right order, each task is placed in a
dependency graph where directed edges dictate precedence. Each task is
a simple class that defines its predecessor tasks and successor tasks
via attributes. Here is an example:

::

    class MapPartitions(Task):
    	description = 'Mapping volume partitions'
    	phase = phases.volume_preparation
    	predecessors = [PartitionVolume]
    	successors = [filesystem.Format]
    
    	@classmethod
    	def run(cls, info):
    		info.volume.partition_map.map(info.volume)

In this case the attributes define that the task at hand should run
after the ``PartitionVolume`` task — i.e. after volume has been
partitioned (``predecessors``) — but before formatting each
partition (``successors``).
It is also placed in the ``volume_preparation`` phase.
Phases are ordered and group tasks together. All tasks in a phase are
run before proceeding with the tasks in the next phase. They are a way
of avoiding the need to list 50 different tasks as predecessors and
successors.

The final task list that will be executed is computed by enumerating
all tasks in the package, placing them in the graph and
`sorting them topoligcally <http://en.wikipedia.org/wiki/Topological_sort>`_.
Subsequently the list returned is filtered to contain only the tasks the
provider and the plugins added to the taskset.


System abstractions
~~~~~~~~~~~~~~~~~~~
There are several abstractions in bootstrap-vz that make it possible
to generalize things like volume creation, partitioning, mounting and
package installation. As a rule these abstractions are located in the
``base/`` folder, where the manifest parsing and task ordering algorithm
are placed as well.


Coding style
------------
bootstrap-vz is coded to comply closely with the PEP8 style
guidelines. There however a few exceptions:

+ Max line length is 110 chars, not 80.
+ Multiple assignments may be aligned with spaces so that the = match
  vertically.
+ Ignore ``E101``: Indent with tabs and align with spaces
+ Ignore ``E221 & E241``: Alignment of assignments
+ Ignore ``E501``: The max line length is not 80 characters
+ Ignore ``W191``: Indent with tabs not spaces

The codebase can be checked for any violations quite easily with:
::
    find . -name '*.py' | /usr/bin/grep -v minify_json | xargs pep8 --ignore=E101,E221,E241,E501,W191

Note: ``common/minify_json.py`` is ignored since it is foreign code.