Combine documentation from github wiki with sphinx

This is the first step in the effort of combining all documentation about
bootstrap-vz into sphinx while still being able to access it from github
(github can parse rst as well)

bootstrapvz/plugins/admin_user/README.rst

@@ -0,0 +1,12 @@
+Admin user
+----------
+
+This plugin creates a user with passwordless sudo privileges. It also
+disables the SSH root login. If the EC2 init scripts are installed, the
+script for fetching the SSH authorized keys will be adjust to match the
+username specified.
+
+Settings
+~~~~~~~~
+
+-  ``username``: The username of the account to create. *``required``*

bootstrapvz/plugins/apt_proxy/README.rst

@@ -0,0 +1,21 @@
+APT Proxy
+---------
+
+This plugin creates a proxy configuration file for APT, so you could
+enjoy the benefits of using cached packages instead of downloading them
+from the mirror every time. You could just install ``apt-cacher-ng`` on
+the host machine and then add ``"address": "127.0.0.1"`` and
+``"port": 3142`` to the manifest file.
+
+Settings
+~~~~~~~~
+
+-  ``address``: The IP or host of the proxy server.
+    *``required``*
+-  ``port``: The port (integer) of the proxy server.
+    *``required``*
+-  ``persistent``: Whether the proxy configuration file should remain on
+   the machine or not.
+   Valid values: true, false
+   Default: ``false``
+   *``optional``*

bootstrapvz/plugins/cloud_init/README.rst

@@ -0,0 +1,23 @@
+cloud-init
+----------
+
+This plugin installs and configures
+`cloud-init <https://packages.debian.org/wheezy-backports/cloud-init>`__
+on the system. Depending on the release it installs it from either
+backports or the main repository.
+
+cloud-init is only compatible with Debian wheezy and upwards.
+
+Settings
+~~~~~~~~
+
+-  ``username``: The username of the account to create.
+   *``required``*
+-  ``disable_modules``: A list of strings specifying which cloud-init
+   modules should be disabled.
+   *``optional``*
+-  ``metadata_sources``: A string that sets the
+   `datasources <http://cloudinit.readthedocs.org/en/latest/topics/datasources.html>`__
+   that cloud-init should try fetching metadata from. The source is
+   automatically set when using the ec2 provider.
+   *``optional``*

bootstrapvz/plugins/docker_daemon/README.rst

@@ -0,0 +1,18 @@
+Docker daemon
+-------------
+
+Install `docker <http://www.docker.io/>`__ daemon in the image. Uses
+init scripts for the official repository.
+
+This plugin can only be used if the distribution being bootstrapped is
+at least ``wheezy``, as Docker needs a kernel version ``3.8`` or higher,
+which is available at the ``wheezy-backports`` repository. There's also
+an architecture requirement, as it runs only on ``amd64``.
+
+Settings
+~~~~~~~~
+
+-  ``version``: Selects the docker version to install. To select the
+   latest version simply omit this setting.
+   Default: ``latest``
+   *``optional``*

bootstrapvz/plugins/image_commands/README.rst

@@ -0,0 +1,19 @@
+Image commands
+--------------
+
+The image commands plugin allows you to run arbitrary commands during
+the bootstrap process. The commands are run at an indeterminate point
+*after* packages have been installed, but *before* the volume has been
+unmounted.
+
+Settings
+~~~~~~~~
+
+-  ``commands``: A list of lists containing strings. Each top-level item
+   is a single command, while the strings inside each list comprise
+   parts of a command. This allows for proper shell argument escaping
+   (to circumvent this, simply put the entire command in a single
+   string). In addition to the manifest variables ``{root}`` is also
+   available. It points at the root of the image volume.
+   *``required``*
+   *``manifest vars``*

bootstrapvz/plugins/minimize_size/README.rst

@@ -0,0 +1,37 @@
+minimize size
+-------------
+
+This plugin can be used to reduce the size of the resulting image. Often
+virtual volumes are much smaller than their reported size until any data
+is written to them. During the bootstrapping process temporary data like
+the aptitude cache is written to the volume only to be removed again.
+
+The minimize size plugin employs three different strategies to keep a
+low volume footprint:
+
+-  Mount folders from the host into key locations of the image volume to
+   avoid any unneccesary disk writes.
+-  Use `zerofree <http://intgat.tigress.co.uk/rmy/uml/index.html>`__ to
+   deallocate unused sectors on the volume. On an unpartitioned volume
+   this will be done for the entire volume, while it will only happen on
+   the root partition for partitioned volumes.
+-  Use
+   `vmware-vdiskmanager <https://www.vmware.com/support/ws45/doc/disks_vdiskmanager_eg_ws.html>`__
+   to shrink the real volume size (only applicable when using vmdk
+   backing). The tool is part of the `VMWare
+   Workstation <https://my.vmware.com/web/vmware/info/slug/desktop_end_user_computing/vmware_workstation/10_0>`__
+   package.
+
+Settings
+~~~~~~~~
+
+-  ``zerofree``: Specifies if it should mark unallocated blocks as
+   zeroes, so the volume could be better shrunk after this.
+   Valid values: true, false
+   Default: false
+   *``optional``*
+-  ``shrink``: Whether the volume should be shrunk. This setting works
+   best in conjunction with the zerofree tool.
+   Valid values: true, false
+   Default: false
+   *``optional``*

bootstrapvz/plugins/ntp/README.rst

@@ -0,0 +1,12 @@
+NTP
+---
+
+This plugins installs the Network Time Protocol daemon and optionally
+defines which time servers it should use.
+
+Settings
+~~~~~~~~
+
+-  ``servers``: A list of strings specifying which servers should be
+   used to synchronize the machine clock.
+   *``optional``*

bootstrapvz/plugins/opennebula/README.md

@@ -1,13 +0,0 @@
-# Open Nebula provider
-
-This provider adds OpenNebula contextualization to the virtual image (see http://opennebula.org/documentation:rel4.2:cong).
-
-It set ups the network and ssh keys. TO do so you should configure your virtual machine context with something like:
-
-    ETH0_DNS            $NETWORK[DNS, NETWORK_ID=2]
-    ETH0_GATEWAY        $NETWORK[GATEWAY, NETWORK_ID=2]
-    ETH0_IP             $NIC[IP, NETWORK_ID=2]
-    ETH0_MASK	        $NETWORK[MASK, NETWORK_ID=2]
-    ETH0_NETWORK        $NETWORK[NETWORK, NETWORK_ID=2]
-    FILES               path_to_my_ssh_public_key.pub
-

bootstrapvz/plugins/opennebula/README.rst

@@ -0,0 +1,27 @@
+Open Nebula
+-----------
+
+This plugin adds `OpenNebula
+contextualization <http://opennebula.org/documentation:rel4.2:cong>`__
+to the image, which sets up the network configuration and SSH keys.
+
+The virtual machine context should be configured as follows:
+
+::
+
+    ETH0_DNS      $NETWORK[DNS, NETWORK_ID=2]
+    ETH0_GATEWAY  $NETWORK[GATEWAY, NETWORK_ID=2]
+    ETH0_IP       $NIC[IP, NETWORK_ID=2]
+    ETH0_MASK     $NETWORK[MASK, NETWORK_ID=2]
+    ETH0_NETWORK  $NETWORK[NETWORK, NETWORK_ID=2]
+    FILES         path_to_my_ssh_public_key.pub
+
+The plugin will install all *.pub* files in the root authorized\_keys
+file. When using the ec2 provider, the USER\_EC2\_DATA will be executed
+if present.
+
+Settings
+~~~~~~~~
+
+This plugin has no settings. To enable it add ``"opennebula":{}`` to the
+plugin section of the manifest.

bootstrapvz/plugins/pip_install/README.rst

@@ -0,0 +1,14 @@
+Pip install
+-----------
+
+Install packages from the Python Package Index via pip.
+
+Installs ``build-essential`` and ``python-dev`` debian packages, so
+Python extension modules can be built.
+
+Settings
+~~~~~~~~
+
+-  ``packages``: Python packages to install, a list of strings. The list
+   can contain anything that ``pip install`` would accept as an
+   argument, for example ``awscli==1.3.13``.

bootstrapvz/plugins/prebootstrapped/README.rst

@@ -0,0 +1,26 @@
+prebootstrapped
+---------------
+
+When developing for bootstrap-vz, testing can be quite tedious since the
+bootstrapping process can take a while. The prebootstrapped plugin
+solves that problem by creating a snapshot of your volume right after
+all the software has been installed. The next time bootstrap-vz is run,
+the plugin replaces all volume preparation and bootstrapping tasks and
+recreates the volume from the snapshot instead.
+
+The plugin assumes that the users knows what he is doing (e.g. it
+doesn't check whether bootstrap-vz is being run with a partitioned
+volume configuration, while the snapshot is unpartitioned).
+
+When no snapshot or image is specified the plugin creates one and
+outputs its ID/path. Specifying an ID/path enables the second mode of
+operation which recreates the volume from the specified snapshot instead
+of creating it from scratch.
+
+Settings
+~~~~~~~~
+
+-  ``snapshot``: ID of the EBS snapshot to use. This setting only works
+   with EBS backed EC2 configurations.
+-  ``image``: Path to the loopbackvolume snapshot. This setting works
+   with all configurable volume backings except EBS.

bootstrapvz/plugins/puppet/README.rst

@@ -0,0 +1,24 @@
+Puppet
+------
+
+Installs `puppet <http://puppetlabs.com/>`__ and optionally applies a
+manifest inside the chroot. You can also have it copy your puppet
+configuration into the image so it is readily available once the image
+is booted.
+
+Keep in mind that when applying a manifest, the system is in a chrooted
+environment. This can prevent daemons from running properly (e.g.
+listening to ports), they will also need to be shut down gracefully
+(which bootstrap-vz cannot do) before unmounting the volume. It is
+advisable to avoid starting any daemons inside the chroot at all.
+
+Settings
+~~~~~~~~
+
+-  ``manifest``: Path to the puppet manifest that should be applied.
+   *``optional``*
+-  ``assets``: Path to puppet assets. The contents will be copied into
+   ``/etc/puppet`` on the image. Any existing files will be overwritten.
+   *``optional``*
+-  ``enable_agent``: Whether the puppet agent daemon should be enabled.
+   *``optional``*

bootstrapvz/plugins/root_password/README.rst

@@ -0,0 +1,11 @@
+root password
+-------------
+
+Sets the root password. This plugin removes the task that disables the
+SSH password authentication.
+
+Settings
+~~~~~~~~
+
+-  ``password``: The password for the root user.
+   *``required``*

bootstrapvz/plugins/salt/README.rst

@@ -0,0 +1,26 @@
+Salt
+----
+
+Install `salt <http://www.saltstack.com/>`__ minion in the image. Uses
+`salt-bootstrap <https://github.com/saltstack/salt-bootstrap>`__ script
+to install.
+
+Settings
+~~~~~~~~
+
+-  ``install_source``: Source to install salt codebase from. ``stable``
+   for current stable, ``daily`` for installing the daily build, and
+   ``git`` to install from git repository.
+   *``required``*
+-  ``version``: Only needed if you are installing from ``git``.
+   \ ``develop`` to install current development head, or provide any tag
+   name or commit hash from `salt
+   repo <https://github.com/saltstack/salt>`__
+   *``optional``*
+-  ``master``: Salt master FQDN or IP
+   *``optional``*
+-  ``grains``: Set `salt
+   grains <http://docs.saltstack.com/en/latest/topics/targeting/grains.html>`__
+   for this minion. Accepts a map with grain name as key and the grain
+   data as value.
+   *``optional``*

bootstrapvz/plugins/unattended_upgrades/README.rst

@@ -0,0 +1,18 @@
+Unattended upgrades
+-------------------
+
+Enables the `unattended update/upgrade
+feature <https://packages.debian.org/wheezy/unattended-upgrades>`__ in
+aptitude. Enable it to have your system automatically download and
+install security updates automatically with a set interval.
+
+Settings
+~~~~~~~~
+
+-  ``update_interval``: Days between running ``apt-get update``.
+   *``required``*
+-  ``download_interval``: Days between running
+   ``apt-get upgrade --download-only``
+   *``required``*
+-  ``upgrade_interval``: Days between installing any security upgrades.
+   *``required``*

bootstrapvz/plugins/vagrant/README.rst

@@ -0,0 +1,12 @@
+Vagrant
+-------
+
+Vagrant is a tool to quickly create virtualized environments. It uses
+"boxes" to make downloading and sharing those environments easier. A box
+is a tarball containing a virtual volumes accompanied by an `OVF
+specification <http://en.wikipedia.org/wiki/Open_Virtualization_Format>`__
+of the virtual machine.
+
+This plugin creates a vagrant box that is ready to be shared or
+deployed. At the moment it is only compatible with the VirtualBox
+provider and doesn't requires any additional settings.

bootstrapvz/providers/azure/README.rst

@@ -1,22 +1,28 @@
-Azure provider
-===========
+Azure
+=====
 
-This provider generates raw images for Microsoft Azure computing platform.
+This provider generates raw images for Microsoft Azure computing
+platform.
 
 Setup
-=====
+-----
 
 qemu-img >= 1.7.0 required to convert raw image to vhd fixed size disk.
 This release is available in wheezy-backports.
 
 *wget* must be installed on local computer.
 
-
-Manifest must use the *raw* format, provider will automatically transform the disk to a vhd disk format.
+Manifest must use the *raw* format, provider will automatically
+transform the disk to a vhd disk format.
 
 Do not create swap space on the OS disk:
 
-The Windows Azure Linux Agent can automatically configure swap space using the local resource disk that is attached to the VM after provisioning on Azure. Modify the following parameters in /etc/waagent.conf appropriately:
+The Windows Azure Linux Agent can automatically configure swap space
+using the local resource disk that is attached to the VM after
+provisioning on Azure. Modify the following parameters in
+/etc/waagent.conf appropriately:
+
+::
 
     ResourceDisk.Format=y
     ResourceDisk.Filesystem=ext4
@@ -24,7 +30,10 @@ The Windows Azure Linux Agent can automatically configure swap space using the l
     ResourceDisk.EnableSwap=y
     ResourceDisk.SwapSizeMB=2048    ## NOTE: set this to whatever you need it to be.
 
-You can specify a waagent.conf file to replace the default one in the manifest in the azure/waagent section of the provider:
+You can specify a waagent.conf file to replace the default one in the
+manifest in the azure/waagent section of the provider:
+
+::
 
     "system" : { 
         "waagent" : {
@@ -33,4 +42,5 @@ You can specify a waagent.conf file to replace the default one in the manifest i
         }
     }, ...
 
-Waagent versions are available at: https://github.com/Azure/WALinuxAgent/releases
+Waagent versions are available at:
+https://github.com/Azure/WALinuxAgent/releases

bootstrapvz/providers/ec2/README.rst

@@ -0,0 +1,71 @@
+EC2
+===
+
+The `EC2 <http://aws.amazon.com/ec2/>`__ provider automatically creates
+a volume for bootstrapping (be it EBS or S3), makes a snapshot of it
+once it is done and registers it as an AMI. EBS volume backing only
+works on an EC2 host while S3 backed volumes *should* work locally (at
+this time however they do not, a fix is in the works).
+
+Unless `the cloud-init plugin <dev_Plugins#cloud_init>`__ is used,
+special startup scripts will be installed that automatically fetch the
+configured authorized\_key from the instance metadata and save or run
+any userdata supplied (if the userdata begins with ``#!`` it will be
+run).
+
+Credentials
+-----------
+
+The AWS credentials can be configured in two ways: Via the manifest or
+through environment variables. To bootstrap S3 backed instances you will
+need a user certificate and a private key in addition to the access key
+and secret key, which are needed for bootstraping EBS backed instances.
+
+The settings describes below should be placed in the ``credentials`` key
+under the ``provider`` section.
+
+-  **``access-key``**: AWS access-key.
+   May also be supplied via the environment variable
+   ``$AWS_ACCESS_KEY``
+   *``required for EBS & S3 backing``*
+-  **``secret-key``**: AWS secret-key.
+   May also be supplied via the environment variable
+   ``$AWS_SECRET_KEY``
+   *``required for EBS & S3 backing``*
+-  **``certificate``**: Path to the AWS user certificate. Used for
+   uploading the image to an S3 bucket.
+   May also be supplied via the environment variable
+   ``$AWS_CERTIFICATE``
+   *``required for S3 backing``*
+-  **``private-key``**: Path to the AWS private key. Used for uploading
+   the image to an S3 bucket.
+   May also be supplied via the environment variable
+   ``$AWS_PRIVATE_KEY``
+   *``required for S3 backing``*
+-  **``user-id``**: AWS user ID. Used for uploading the image to an S3
+   bucket.
+   May also be supplied via the environment variable ``$AWS_USER_ID``
+   *``required for S3 backing``*
+
+Example:
+
+.. code:: yaml
+
+    ---
+    provider:
+      name: ec2
+      virtualization: hvm
+      enhanced_networking: simple
+      credentials:
+        access-key: AFAKEACCESSKEYFORAWS
+        secret-key: thes3cr3tkeyf0ryourawsaccount/FS4d8Qdva
+
+Dependencies
+------------
+
+To communicate with the AWS API `boto <https://github.com/boto/boto>`__
+is required (version 2.14.0 or higher) you can install boto with
+``pip install boto`` (on wheezy, the packaged version is too low). S3
+images are chopped up and uploaded using
+`euca2ools <https://github.com/eucalyptus/euca2ools>`__ (install with
+``apt-get install euca2ools``).

bootstrapvz/providers/gce/README.rst

@@ -0,0 +1,9 @@
+Google Compute Engine
+---------------------
+
+The `GCE <https://cloud.google.com/products/compute-engine/>`__ provider
+can creates image as expected by GCE - i.e. raw disk image in \*.tar.gz
+file. It can upload created images to Google Storage Engine (to URI
+provided in manifest by ``gcs_destination``) and can register image to
+be used by Google Compute Engine to project provided in manifest by
+``gce_project``. Both of those functionalities are not fully tested yet.

bootstrapvz/providers/kvm/README.rst

@@ -0,0 +1,8 @@
+KVM
+---
+
+The `KVM <http://www.linux-kvm.org/page/Main_Page>`__ provider creates
+virtual images for Linux Kernel-based Virtual Machines. It supports the
+installation of `virtio kernel
+modules <http://www.linux-kvm.org/page/Virtio>`__ (paravirtualized
+drivers for IO operations).

bootstrapvz/providers/virtualbox/README.rst

@@ -0,0 +1,12 @@
+VirtualBox
+----------
+
+The `VirtualBox <https://www.virtualbox.org/>`__ provider can bootstrap
+to both .vdi and .vmdk images (raw images are also supported but do not
+run in VirtualBox). It's advisable to always use vmdk images for
+interoperability (e.g.
+`OVF <http://en.wikipedia.org/wiki/Open_Virtualization_Format>`__ files
+*should* support vdi files, but since they have no identifier URL not
+even VirtualBox itself can import them). VirtualBox Guest Additions can
+be installed automatically if the ISO is `provided in the
+manifest <dev_Manifest#bootstrapper>`__.

docs/conf.py

@@ -14,6 +14,8 @@
 
 import sys
 import os
+import glob
+import os.path
 
 # If extensions (or modules to document with autodoc) are in another directory,
 # add these directories to sys.path here. If the directory is relative to the
@@ -261,3 +263,21 @@ texinfo_documents = [('index', 'bootstrap-vz', u'bootstrap-vz Documentation',
 
 # If true, do not generate a @detailmenu in the "Top" node's menu.
 #texinfo_no_detailmenu = False
+
+
+# -- Link to rst files scattered throughout the project -------------------
+
+for readme_path in glob.glob('../bootstrapvz/providers/*/README.rst'):
+	provider_name = os.path.basename(os.path.dirname(readme_path))
+	include_path = os.path.join('providers', provider_name + '.rst')
+	path_to_readme = os.path.join('../../bootstrapvz/providers', provider_name, 'README.rst')
+	with open(include_path, 'w') as include:
+		include.write('.. include:: ' + path_to_readme)
+
+
+for readme_path in glob.glob('../bootstrapvz/plugins/*/README.rst'):
+	plugin_name = os.path.basename(os.path.dirname(readme_path))
+	include_path = os.path.join('plugins', plugin_name + '.rst')
+	path_to_readme = os.path.join('../../bootstrapvz/plugins', plugin_name, 'README.rst')
+	with open(include_path, 'w') as include:
+		include.write('.. include:: ' + path_to_readme)

docs/plugins/.gitignore

@@ -0,0 +1,3 @@
+*
+!index.rst
+!.gitignore

docs/plugins/index.rst

@@ -1,3 +1,19 @@
 
 Plugins
 =======
+
+Plugins are a key feature of bootstrap-vz. Despite their small size
+(most plugins do not exceed 100 source lines of code) they can modify
+the behavior of bootstrapped systems to a great extent.
+
+Below you will find documentation for all plugins available for
+bootstrap-vz. If you cannot find what you are looking for, consider
+`developing it yourself <http://bootstrap-vz.readthedocs.org>`__ and
+contribute to this list!
+
+
+.. toctree::
+   :maxdepth: 1
+   :glob:
+
+   *

docs/providers/.gitignore

@@ -0,0 +1,3 @@
+*
+!index.rst
+!.gitignore

docs/providers/index.rst

@@ -1,3 +1,19 @@
 
 Providers
 =========
+
+Plugins are a key feature of bootstrap-vz. Despite their small size
+(most plugins do not exceed 100 source lines of code) they can modify
+the behavior of bootstrapped systems to a great extent.
+
+Below you will find documentation for all plugins available for
+bootstrap-vz. If you cannot find what you are looking for, consider
+`developing it yourself <http://bootstrap-vz.readthedocs.org>`__ and
+contribute to this list!
+
+
+.. toctree::
+   :maxdepth: 1
+   :glob:
+
+   *