Puppet module update (#365)

* #347 - Fix for debconf validator pointing to wrong file.

* reported in https://github.com/andsens/bootstrap-vz/issues/347

  flake8: commands succeeded
  congratulations :)

* # development commit

* trying to fix the packages install with a forced apt-update prior to
installing packages. should fix any issues prior to

* # dev commit

* Trying to parse the commands with a chrotted log_check_call

* # dev commit

* "TypeError: sequence item 2: expected string, list found" , trying to
fix this

* # dev commit - changed the way local packages get installed

This commit changes the way local deb packages get installed.

* rationale: a local deb package mostly includes a deb to configure apt,
e.g.: Puppet apt package. Therefore, after a local dep pkg install , apt
should update itself to be able to install packages that come with the
repo's configured from the local deb package. This assumes you install
LOCAL packages (deb files) first (as a rule of thumb) and AFTER the
REMOTE packages (you include other packages by name)

* reverted command plugin tasks code

* # Dev commit - puppet module update

Goal is to install a puppet 4 agent on the bootstrapped image

* limitations: Only for debian Jessie, installs only puppet4

* # Feature improvement - Puppet module

* a new way to install and configure puppet on a debian jessie
* only tested on debian jessie, for now it only installs the PC1 agent
package.
* future endeavors include all mentioned todo's:
* TODO: plugin must be able to install on any debian release
* TODO: plugin must be able to offer choice of distro package or
apt.puppetlabs.com package
* TODO: plugin must be able to select release version package (vanilla
or PC1)
* TODO: plugin must be able to set up a puppet SERVER , puppetDB
optional
* TODO: plugin have proper linting
* TODO: write test cases

* # Derp commit

* added project files in GitIgnore

* # Dev commit

* added release detection and installs appropriate package.

* # derp commit

* removed unnecessary class parameter.

* # derp dev commit

* add predecessor to ensure repo package gets installed BEFORE the
agent.

* # derp fixes commit

* i must use info.manifest.release
* a little bit refactoring

* # derp commit

* fixed correct predecessor

* # feature & derp dev commit

* added feature to install puppet modules with the plugin

* # Version upgrade commit

New puppet plugin module update

* Plugin can select proper debian release package to install
* Plugin can install modules directly from forge.puppetlabs.com
* Updated documentation with working example 
* Limitations TL;DR: agent software only, PC1 package only, Absolute
pathing, wheezy/jessie only, assumes production environment for modules.
* Several ugly derp bugs squashed.

* * PR fix commit

* Modified puppet readme file as recommended
* added small description in changelog
https://github.com/andsens/bootstrap-vz/pull/365#discussion_r101378921
https://github.com/andsens/bootstrap-vz/pull/365#discussion_r101379039

* # PR fix commit

* Cleaned debugging leftovers
https://github.com/andsens/bootstrap-vz/pull/365#discussion_r101379293

* # PR fix commit

* Removed todo's from code, creating issues on the project issue
tracker.
https://github.com/andsens/bootstrap-vz/pull/365#discussion_r101381742

* # PR fix commit

* Moved EnableAgent task to the bottom as requested
https://github.com/andsens/bootstrap-vz/pull/365#discussion_r101382775

* # PR fix commit

* Removed gitignore file, I have absolutely no idea how that got there.

* # PR fix commit

* Fixed several TOX issues, it's all green now

* #MR-365 - dev commit 

* first try at installing agent with sources/key injection
* New tasks in puppet module: AddPuppetlabsPC1SourcesList;
InstallPuppetlabsPC1ReleaseKey
* it does NOT do any checks. it assumes the url is correct.

* #MR-365 - derp commit

* removed some unused code in tasks

* #365 - tox fix commit

* fixed several tox warnings

* #365 - tox fix commit

* missed one line...

* #365 - tox fix commit

* Noticed the nature of the tox warnings of 'undefined name', fixed.

* # 365 - Task order fix commit

* Some tasks had some invalid predecessors defined, removed.
* when running a test manifest, it fails due to 'NO_PUBKEY
7F438280EF8D349F', trying a hacky way to address this

* #365 - Puppet module update - install with source-key combo

* It now properly installs puppet agent package with sources/keys
instead of package
* Tox is green

* Delete .project

* Delete .pydevproject

* #365 - Puppet module update 

General puppet module update.

* Fixed several issues discussed in the pull request.
* Tox is all green

* General puppet module update.

* This update allows for the installation of the puppetlabs.com agent VERSION 4 from the official apt.puppetlabs.com repo
* Allows for the installation of puppet modules. Useful for masterless setups.
* Puppet modules you declare in your bootstrap-vz manifest are installed with the --force flag.

Limitations and gotcha's:

* Only works for Debian Wheezy & Jessie.
* The Puppetlabs.com agents are only released for i386 and amd64 architectures.
* If you require the puppet 3.x agent, you should not use this module but add "puppet-agent" into the packages list
* You need to add your own puppet.conf file to fit your needs, this can be easily included in the assets directive.
* Assets path is absolute.

* General puppet module update.

* This update allows for the installation of the puppetlabs.com agent VERSION 4 from the official apt.puppetlabs.com repo
* Allows for the installation of puppet modules. Useful for masterless setups.
* Puppet modules you declare in your bootstrap-vz manifest are installed with the --force flag.

Limitations and gotcha's:

* Only works for Debian Wheezy & Jessie.
* The Puppetlabs.com agents are only released for i386 and amd64 architectures.
* If you require the puppet 3.x agent, you should not use this module but add "puppet-agent" into the packages list
* You need to add your own puppet.conf file to fit your needs, this can be easily included in the assets directive.
* Assets path is absolute.

* #365 - Changed the way trusted keys are fetched.

* @andsens is right, keyrings are better added by injecting them into
the image, removing the need to install package 'ca-certificates'

* * Added new feature: puppet module version can now be included in the
manifest.
* modified manifest-schema to reflect proper module installation
* Updated documentation.
* Example manifest added to demonstrate changes
* @andsens is right, keyrings are better added by injecting them into
the image, removing the need to install package 'ca-certificates'

* Fix several requests

* Example manifest moved to a more suitable location, documentation
changed as well.
* A bit more efficient programming.
* reverted file_copy to it's proper state

CHANGELOG.rst

@@ -1,6 +1,18 @@
 Changelog
 =========
 
+2017-02-20
+----------
+Hugo Antoniio Sepulveda Manriquez:
+    * Updated puppet plugin module:
+		* Installs Puppetlabs 4 PC1 agent software from apt.puppetlabs.com
+		* Enables you to install modules from forge.puppetlabs.com in the image
+		
+	* Important limitations
+		* Only works for Wheezy and Jessie for now.
+		* If you need puppet 3, just add 'puppet' packages provider list.
+		* modules: When installing from forge, it assumes 'install --force'
+		* modules: When installing from forge, It assumes master version on forge 	  
 
 2016-06-04
 ----------

bootstrapvz/plugins/puppet/README.rst

@@ -1,16 +1,25 @@
 Puppet
 ------
 
-Installs `puppet <http://puppetlabs.com/>`__ and optionally applies a
+Installs `puppet version 4 <http://puppetlabs.com/>` PC1 From the site 
+repository `<http://apt.puppetlabs.com/>` and optionally applies a
 manifest inside the chroot. You can also have it copy your puppet
 configuration into the image so it is readily available once the image
 is booted.
 
-Keep in mind that when applying a manifest, the system is in a chrooted
-environment. This can prevent daemons from running properly (e.g.
-listening to ports), they will also need to be shut down gracefully
-(which bootstrap-vz cannot do) before unmounting the volume. It is
-advisable to avoid starting any daemons inside the chroot at all.
+Rationale and use case
+~~~~~~~~~~~~~~~~~~~~~~
+
+You want to use this plugin when you wish to create an image and to be able to
+manage that image with Puppet. You have a Puppet 4 setup in mind and thus you 
+want the image to contain the puppet agent software from the puppetlabs repo. 
+You want it to almost contain everything you need to get it up and running 
+This plugin does just that!
+While you're at it, throw in some modules from the forge as well!
+Want to include your own modules? Include them as assets!
+ 
+For now this plugin is only compatible with Debian versions Wheezy and Jessie.
+These are also the distributions supported by puppetlabs.
 
 Settings
 ~~~~~~~~
@@ -18,7 +27,61 @@ Settings
 -  ``manifest``: Path to the puppet manifest that should be applied.
    ``optional``
 -  ``assets``: Path to puppet assets. The contents will be copied into
-   ``/etc/puppet`` on the image. Any existing files will be overwritten.
+   ``/etc/puppetlabs`` on the image. Any existing files will be overwritten.
    ``optional``
+-  ``install_modules``: A list of modules you wish to install available from 
+   `<https://forge.puppetlabs.com/>` inside the chroot. It will assume a FORCED
+   install of the modules.
+   This list is a list of tuples. Every tuple must at least contain the module 
+   name. A version is optional, when no version is given, it will take the 
+   latest version available from the forge. 
+   Format: [module_name (required), version (optional)]
 -  ``enable_agent``: Whether the puppet agent daemon should be enabled. 
-   ``optional``
+   ``optional - not recommended``. disabled by default. UNTESTED
+   
+An example bootstrap-vz manifest is included in the ``KVM`` folder of the 
+manifests examples directory.
+      
+Limitations
+~~~~~~~~~~~
+(Help is always welcome, feel free to chip in!)
+General:
+
+- This plugin only installs the PC1 package for now, needs to be extended to 
+  be able to install the package of choice
+- The puppetlabs agent is only available to i386 and amd64 architectures. ARM 
+  is not available from repository at this time. If you need to have the agent 
+  on ARM, you need to build the agent yourself, and install them through the 
+  ``packages`` section of the bootstrap-vz manifest
+
+Manifests:
+
+- Running puppet manifests is not recommended and untested, see below
+
+Assets:
+
+- The assets path must be ABSOLUTE to your manifest file.  
+
+install_modules:
+
+- It assumes installing the given list of tuples of modules with the following 
+  command: 
+  "... install --force $module_name (--version $version_number)"
+  The module name is mandatory, the version is optional. When no version is 
+  given, it  will pick the master version of the
+  module from `<https://forge.puppetlabs.com/>`
+- It assumes the modules are installed into the "production" environment. 
+  Installing into another environment e.g. develop, is currently not 
+  implemented.
+- You cannot include local modules this way, to include you homebrewn modules,
+  You need to inject them through the assets directive.
+
+UNTESTED:
+
+- Enabling the agent and applying the manifest inside the chrooted environment.
+	Keep in mind that when applying a manifest when enabling the agent option,
+	the system is in a chrooted environment. This can prevent daemons from 
+	running	properly (e.g. listening to ports), they will also need to be shut 
+	down gracefully (which bootstrap-vz cannot do) before unmounting the 
+	volume. It is advisable to avoid starting any daemons inside the chroot at 
+	all.

bootstrapvz/plugins/puppet/__init__.py

@@ -7,12 +7,17 @@ def validate_manifest(data, validator, error):
 
 
 def resolve_tasks(taskset, manifest):
-    taskset.add(tasks.AddPackages)
+    taskset.add(tasks.CheckRequestedDebianRelease)
+    taskset.add(tasks.AddPuppetlabsPC1SourcesList)
+    taskset.add(tasks.InstallPuppetlabsPC1ReleaseKey)
+    taskset.add(tasks.InstallPuppetAgent)
     if 'assets' in manifest.plugins['puppet']:
         taskset.add(tasks.CheckAssetsPath)
         taskset.add(tasks.CopyPuppetAssets)
     if 'manifest' in manifest.plugins['puppet']:
         taskset.add(tasks.CheckManifestPath)
         taskset.add(tasks.ApplyPuppetManifest)
+    if 'install_modules' in manifest.plugins['puppet']:
+        taskset.add(tasks.InstallModules)
     if manifest.plugins['puppet'].get('enable_agent', False):
         taskset.add(tasks.EnableAgent)

bootstrapvz/plugins/puppet/manifest-schema.yml

@@ -12,6 +12,19 @@ properties:
           assets: {$ref: '#/definitions/absolute_path'}
           enable_agent: {type: boolean}
           manifest: {$ref: '#/definitions/absolute_path'}
+          install_modules:
+            type: array
+            minItems: 1              
+            items:
+              type: array
+              items:
+                name: 
+                  anyOf: 
+                  - pattern: ^[^/]+(/[^/]+)?$ 
+                version: 
+                  type: number
+              required: [name]               
+              minItems: 1
         minProperties: 1
         additionalProperties: false
 definitions:

bootstrapvz/plugins/puppet/tasks.py

@@ -1,12 +1,32 @@
+import os
 from bootstrapvz.base import Task
 from bootstrapvz.common import phases
-from bootstrapvz.common.tools import sed_i
-import os
+from bootstrapvz.common.exceptions import TaskError
+from bootstrapvz.common.releases import jessie, wheezy
+from bootstrapvz.common.tools import sed_i, log_check_call, rel_path
+from __builtin__ import str
+
+
+ASSETS_DIR_JESSIE = rel_path(__file__, 'assets/gpg-keyrings-PC1/jessie')
+ASSETS_DIR_WHEEZY = rel_path(__file__, 'assets/gpg-keyrings-PC1/wheezy')
+
+
+class CheckRequestedDebianRelease(Task):
+    description = 'Checking whether there is a release available for {info.manifest.release}'
+    phase = phases.validation
+
+    @classmethod
+    def run(cls, info):
+        from bootstrapvz.common.exceptions import TaskError
+        if not info.manifest.release == (jessie or wheezy):
+            msg = 'Debian {info.manifest.release} is not (yet) available in the Puppetlabs.com APT repository.'
+            raise TaskError(msg)
 
 
 class CheckAssetsPath(Task):
     description = 'Checking whether the assets path exist'
     phase = phases.validation
+    predecessors = [CheckRequestedDebianRelease]
 
     @classmethod
     def run(cls, info):
@@ -21,12 +41,12 @@ class CheckAssetsPath(Task):
 
 
 class CheckManifestPath(Task):
-    description = 'Checking whether the manifest path exist'
+    description = 'Checking whether the manifest file path exist inside the assets'
     phase = phases.validation
+    predecessors = [CheckAssetsPath]
 
     @classmethod
     def run(cls, info):
-        from bootstrapvz.common.exceptions import TaskError
         manifest = info.manifest.plugins['puppet']['manifest']
         if not os.path.exists(manifest):
             msg = 'The manifest file {manifest} does not exist.'.format(manifest=manifest)
@@ -36,28 +56,77 @@ class CheckManifestPath(Task):
             raise TaskError(msg)
 
 
-class AddPackages(Task):
-    description = 'Add puppet package'
+class InstallPuppetlabsPC1ReleaseKey(Task):
+    description = 'Install puppetlabs PC1 Release key into the keyring'
+    phase = phases.package_installation
+
+    @classmethod
+    def run(cls, info):
+        from shutil import copy
+        if (info.manifest.release == jessie):
+            key_path = os.path.join(ASSETS_DIR_JESSIE, 'puppetlabs-pc1-keyring.gpg')
+        if (info.manifest.release == wheezy):
+            key_path = os.path.join(ASSETS_DIR_WHEEZY, 'puppetlabs-pc1-keyring.gpg')
+        destination = os.path.join(info.root, 'etc/apt/trusted.gpg.d/puppetlabs-pc1-keyring.gpg')
+        copy(key_path, destination)
+
+
+class AddPuppetlabsPC1SourcesList(Task):
+    description = 'Adding Puppetlabs APT repo to the list of sources.'
     phase = phases.preparation
 
     @classmethod
     def run(cls, info):
-        info.packages.add('puppet')
+        if (info.manifest.release == jessie):
+            info.source_lists.add('puppetlabs', 'deb http://apt.puppetlabs.com jessie PC1')
+        if (info.manifest.release == wheezy):
+            info.source_lists.add('puppetlabs', 'deb http://apt.puppetlabs.com wheezy PC1')
 
 
-class CopyPuppetAssets(Task):
-    description = 'Copying puppet assets'
+class InstallPuppetAgent(Task):
+    description = 'Install puppet-agent from https://apt.puppetlabs.com for {system.release}'
     phase = phases.system_modification
 
+    @classmethod
+    def run(cls, info):
+        log_check_call(['chroot', info.root, 'apt-get', 'install', '--assume-yes', 'puppet-agent'])
+
+
+class InstallModules(Task):
+    description = 'Installing Puppet modules'
+    phase = phases.system_modification
+    predecessors = [InstallPuppetAgent]
+
+    @classmethod
+    def run(cls, info):
+        for module in info.manifest.plugins['puppet']['install_modules']:
+            command = ['chroot', info.root, '/opt/puppetlabs/bin/puppet', 'module', 'install', '--force']
+            if (len(module) == 1):
+                [module_name] = module
+                command.append(str(module_name))
+            if (len(module) == 2):
+                [module_name, module_version] = module
+                command.append(str(module_name))
+                command.append('--version')
+                command.append(str(module_version))
+            log_check_call(command)
+
+
+class CopyPuppetAssets(Task):
+    description = 'Copying declared custom puppet assets.'
+    phase = phases.system_modification
+    predecessors = [InstallModules]
+
     @classmethod
     def run(cls, info):
         from bootstrapvz.common.tools import copy_tree
-        copy_tree(info.manifest.plugins['puppet']['assets'], os.path.join(info.root, 'etc/puppet'))
+        copy_tree(info.manifest.plugins['puppet']['assets'], os.path.join(info.root, 'etc/puppetlabs/'))
 
 
 class ApplyPuppetManifest(Task):
-    description = 'Applying puppet manifest'
-    phase = phases.user_modification
+    description = 'Applying puppet manifest.'
+    phase = phases.system_modification
+    predecessors = [CopyPuppetAssets]
 
     @classmethod
     def run(cls, info):
@@ -65,19 +134,14 @@ class ApplyPuppetManifest(Task):
             hostname = handle.read().strip()
         with open(os.path.join(info.root, 'etc/hosts'), 'a') as handle:
             handle.write('127.0.0.1\t{hostname}\n'.format(hostname=hostname))
-
         from shutil import copy
         pp_manifest = info.manifest.plugins['puppet']['manifest']
         manifest_rel_dst = os.path.join('tmp', os.path.basename(pp_manifest))
         manifest_dst = os.path.join(info.root, manifest_rel_dst)
         copy(pp_manifest, manifest_dst)
-
         manifest_path = os.path.join('/', manifest_rel_dst)
-        from bootstrapvz.common.tools import log_check_call
-        log_check_call(['chroot', info.root,
-                        'puppet', 'apply', manifest_path])
+        log_check_call(['chroot', info.root, 'puppet', 'apply', '--verbose', '--debug', manifest_path])
         os.remove(manifest_dst)
-
         hosts_path = os.path.join(info.root, 'etc/hosts')
         sed_i(hosts_path, '127.0.0.1\s*{hostname}\n?'.format(hostname=hostname), '')
 

manifests/examples/kvm/jessie-puppet.yaml

@@ -0,0 +1,45 @@
+---
+name: debian-{system.release}-{system.architecture}-{%Y}{%m}{%d}
+provider:
+  name: kvm
+  virtio_modules:
+  - virtio_pci
+  - virtio_blk
+bootstrapper:
+  workspace: /target
+system:
+  release: jessie
+  architecture: amd64
+  bootloader: grub
+  charmap: UTF-8
+  locale: en_US
+  timezone: UTC
+volume:
+  backing: raw
+  partitions:
+    type: msdos
+    root:
+      filesystem: ext4
+      size: 10GiB
+packages: 
+  install_standard: true
+  mirror: http://httpredir.debian.org/debian
+  install:
+    # required to be pre-installed for proper puppet functioning of 
+    #  puppetlabs-apt, it is also the primary puppet module
+    - lsb-release
+plugins:
+  # It is advisable to avoid running things as root, use a sudo account instead
+  admin_user:
+    username: administrator 
+    password: something
+  # puppet plugin
+  puppet:
+    # The assets path MUST be ABSOLUTE on your project. 
+    assets: /your/absolute/path/to/etc/puppetlabs
+    install_modules:
+      - [puppetlabs-accounts]
+      - [puppetlabs-apt]
+      - [puppetlabs-concat, 3.0.0]
+      - [puppetlabs-stdlib]
+      - [puppetlabs-apache, 1.11.0]