Docs for integration testing

docs/testing/index.rst

@@ -7,5 +7,6 @@ Testing
 
    unit_tests
    integration_tests
+   integration_test_providers
 
 .. include:: ../../tests/README.rst

docs/testing/integration_test_providers.rst

@@ -0,0 +1 @@
+.. include:: ../../tests/integration/providers/README.rst

tests/README.rst

@@ -2,6 +2,5 @@ The testing framework consists of two parts:
 The unit tests and the integration tests.
 
 The `unit tests <unit>`__ are responsible for testing individual
-parts of bootstrap-vz,
+parts of bootstrap-vz, while the `integration tests <integration>`__ test
-while the `integration tests <integration>`__ test entire manifests by
+entire manifests by bootstrapping and booting them.
-bootstrapping and booting them.

tests/integration/README.rst

@@ -1,5 +1,73 @@
 Integration tests
 =================
+`Integration tests`__ test
+bootstrap-vz in its entirety.
+This testing includes building images from manifests and
+creating/booting said images.
+
+__ http://en.wikipedia.org/wiki/Integration_testing
+
+Since hardcoding manifests for each test, bootstrapping them and booting the
+resulting images is too much code for a single test, a testing harness has
+been developed that reduces each test to it's bare essentials:
+
+* Combine available `manifest partials <manifest-partials>`__ into a single manifest
+* Boot an instance from a manifest
+* Run tests on the booted instance
+
+In order for the integration testing harness to be able to bootstrap it must
+know about your `build-servers <../../bootstrapvz/remote#build-servers-yml>`__.
+Depending on the manifest that is bootstrapped, the harness chooses
+a fitting build-server, connects to it and starts the bootstrapping process.
 
 When running integration tests, the framework will look for ``build-servers.yml``
 at the root of the repo and raise an error if it is not found.
+
+
+Manifest combinations
+---------------------
+The tests mainly focus on varying key parts of an image
+(e.g. partitioning, Debian release, bootloader, ec2 backing, ec2 virtualization method)
+that have been problem areas.
+Essentially the tests are the cartesian product of these key parts.
+
+
+Aborting a test
+---------------
+You can press ``Ctrl+C`` at any time during the testing to abort -
+the harness will automatically clean up any temporary resources and shut down
+running instances. Pressing ``Ctrl+C`` a second time stops the cleanup and quits
+immediately.
+
+
+Manifest partials
+-----------------
+Instead of creating manifests from scratch for each single test, reusable parts
+are factored out into partials in the manifest folder.
+This allows code like this:
+
+.. code:: python
+
+	partials = {'vdi': '{provider: {name: virtualbox}, volume: {backing: vdi}}',
+	            'vmdk': '{provider: {name: virtualbox}, volume: {backing: vmdk}}',
+	            }
+
+	def test_unpartitioned_extlinux_oldstable():
+		std_partials = ['base', 'stable64', 'extlinux', 'unpartitioned', 'root_password']
+		custom_partials = [partials['vmdk']]
+		manifest_data = merge_manifest_data(std_partials, custom_partials)
+
+The code above produces a manifest for Debian stable 64-bit unpartitioned
+virtualbox VMDK image.
+``root_password`` is a special partial in that the actual password is
+randomly generated on load.
+
+
+Missing parts
+-------------
+The integration testing harness is in no way complete.
+
+* It still has no support for providers other than virtualbox and EC2.
+* Creating an SSH connection to a booted instance is cumbersome and does not
+  happen in any of the tests - this would be particularly useful when manifests
+  are to be tested beyond whether they boot up.

tests/integration/providers/README.rst

@@ -0,0 +1,2 @@
+Integration test providers
+==========================