Commit graph

236 commits

Author SHA1 Message Date
Jonh Wendell
a5cd6e077d ec2_launch: Allow to deregister the AMI after launching image
If all you want is to test an image or product and to achieve this
you need to generate several images a day, you will end up with
lots of AMI's and snapshots that have no use in the end of the day.

This commit adds the new boolean manifest option "deregister_ami"
that, if True, deletes the recently created AMI and snapshot. So, the
final result will be only the running instance, nothing else.
2015-04-06 14:23:36 -03:00
Jonh Wendell
50d61c735d ec2_launch: Store the instance object directly in the info dictionary
Instead of storing just its ID.

This gives quick access to the recently created instance, which
allows us to simplify the code that needs to fetch the instance object
every time it was necessary.
2015-04-06 14:21:13 -03:00
Jonh Wendell
b934808cce ec2_launch: Simplify the logic for getting the IP address
Use the function waituntil() available in the integration-test branch
to simplify the code that retrieves the IP of the instance.

A 'TODO' note was also added to remember us to merge this function
once it gets merged.
2015-04-06 11:47:15 -03:00
Jonh Wendell
af68be15c6 New plugin: ec2_launch
It adds the ability to automatically launch an EC2 instance
after AMI registration.

It has no mandatory configuration, only optional ones, like
instance type, security groups, etc. They should be documented in
later documentation patches.
2015-03-30 15:36:33 -03:00
Jonh Wendell
8ee0af1a3b Execute entries with a single string as shell commands
Documentation says about that, but it's not implemented like that.

If the command to be executed is an array with just one string, it's
likely the user passed the entire command inside the string.

Plus, they benefit from shell expansion of wildcards like '*'.
2015-03-27 11:23:58 -03:00
Jonh Wendell
1290694f9a Add the manifest "include-source-type" key for packages object
It controls whether to include the 'deb-src' lines in image's
source.list.

Currently they are always included. This patch changes this
behavior by not including them by default; the user must set
this new config to true in order to include them.

This saves a bit of bandwidth in default installations. Also,
the use of src packages is not so usual in ordinary installations.
2015-03-03 19:39:58 -03:00
Anders Ingemann
d046c8e1f7 Merge pull request #194 from Rory-Finnegan/ansible
Added an Ansible plugin, which runs a playbook on the chroot
2015-02-22 16:56:33 +01:00
Anders Ingemann
31d61e778b Fix some more pep8 errors 2015-02-22 16:49:52 +01:00
Anders Ingemann
f48d392df8 Fix pep8 error 2015-02-22 16:44:58 +01:00
Rory Finnegan
c372757104 Added an Ansible plugin, which runs a playbook on the chroot before before build completion.
NOTE: I'm not doing any validation on the opt_flags param and I don't recommend using for more then adding a -vvvv.
Also, I'm purposely excluding the vault flags (which also pretty commonly used) because you shouldn't be baking
private keys and certs into your images. Instead, just avoid running the vault specific code or use the
opt_flags if absolutely necessary.
2015-02-11 21:20:59 +00:00
Jonh Wendell
ea21dcfcec Don't boot quietly on extlinux (aws)
So that we are able to see the System Log through EC2 console
or command line tools.
2015-02-05 10:45:31 -02:00
Anders Ingemann
c2b37c7329 Merge pull request #192 from wrigri/disk-resize
Disable resize on disks larger than 2TB
2015-01-31 18:26:15 +00:00
Rick Wright
0d494fb49e Disable resize on disks larger than 2TB
Change-Id: I9764fe2a06cf47e8c0daf38df41c288c280bd6f7
2015-01-29 16:22:30 -08:00
Jonh Wendell
8418090018 Recover file permissions after shrink a vmdk image
After vmware-vdiskmanager runs, the image is left with a 600
permission. This commit fixes it by preserving the file permission
before the shrink operation.
2015-01-27 21:12:39 -02:00
Brian Mattern
4bd71a2dbe Blacklist floppy module to speed up boot by several seconds.
Add UpdateInitramfs task which is needed for this to take effect.
Enable both tasks for GCE.

Console output before
=====================
[    1.877142] sd 0:0:1:0: [sda] Attached SCSI disk
[    1.880163] sd 0:0:1:0: Attached scsi generic sg0 type 0
[    2.684132] tsc: Refined TSC clocksource calibration: 2500.000 MHz
[    4.824081] floppy0: no floppy controllers found
[    5.103671] work still pending
Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
Begin: Running /scripts/local-premount ... done.
[    5.313107] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
...
[    7.751955] alg: No test for crc32 (crc32-pclmul)
[   10.728078] floppy0: no floppy controllers found
[   11.006680] work still pending
[....] Activating swap... done
[   11.258954] EXT4-fs (sda1): re-mounted. Opts: (null)

Console output after
====================
[    1.829785] sd 0:0:1:0: [sda] Attached SCSI disk
[    1.832806] sd 0:0:1:0: Attached scsi generic sg0 type 0
Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
Begin: Running /scripts/local-premount ... done.
[    1.969862] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
...
[    2.878920] alg: No test for crc32 (crc32-pclmul)
[....] Activating swap... done
[    2.986642] EXT4-fs (sda1): re-mounted. Opts: (null)

Delint.

Delint
2015-01-24 13:17:42 -08:00
Anders Ingemann
255f0624b4 Merge pull request #175 from andrewbogott/file_copy_plugin
Add the file_copy plugin
2014-12-15 15:33:32 +06:00
Andrew Bogott
81cb82a335 Fix a slightly broken plugins/puppet/manifest-schema.yml 2014-12-15 09:24:14 +00:00
Andrew Bogott
d34aeab6e0 Add the file_copy plugin
File_copy supports creation of dirs and copying of files
from the host system into the image.

bug 168
2014-12-15 09:20:02 +00:00
Noah Fontes
4093693c2e Add support for enhanced networking on EC2.
This change adds a provider option, enhanced_networking,
which installs the Intel virtual networking driver for
SR-IOV using DKMS. It also modifies the EC2 AMI registration
to include support for SR-IOV.
2014-11-29 13:46:57 -08:00
Dan Merino
be5da54b46 Use DOCKER_LOCKEDMEMORY to set ulimit -l before starting Docker. Very important when applications inside the containers need to lock to large amounts of memory. 2014-11-27 16:28:56 -06:00
Dan Lorenc
718ef7ed26 Fix log_check_call in docker_daemon. 2014-11-24 11:24:13 -08:00
Dan Lorenc
170ece3691 Add opts parameter to docker_daemon. 2014-11-21 17:35:57 -08:00
Anders Ingemann
2e1c419ea0 Merge pull request #156 from jbergler/development
Allow mixed kernel/userspace architecture.
2014-11-22 02:26:27 +06:00
Johan Euphrosine
e4663f4fbe docker_daemon: add comments 2014-11-21 11:22:11 -08:00
Johan Euphrosine
141e6399f3 fix ident, raise TaskError 2014-11-21 10:51:03 -08:00
jbergler
57909eb9bc Allow mixed kernel/userspace architecture.
This is to allow building an image for a 64bit machine but with 32bit userspace.

Probably not a common usecase but ideal if you need to address more higher quantities of memory but cant migrate to a full 64bit userspace due to something like ruby eating twice as much memory.
2014-11-20 17:23:28 +00:00
Johan Euphrosine
99786539c7 plugins/docker_daemon: flake8 2014-11-19 16:25:27 -08:00
Johan Euphrosine
e53e727c9c plugins/docker_daemon: only add PullDockerImages tasks when pull_images is set 2014-11-19 16:21:38 -08:00
Johan Euphrosine
6fdf0cc403 plugins/docker_daemon: align with space 2014-11-19 16:16:26 -08:00
Johan Euphrosine
31ba98821c plugins/docker_daemon: logcheckcall, inline pull and indent 2014-11-19 16:14:51 -08:00
Johan Euphrosine
feb4d093c9 bootstrapvz/plugins/docker_daemon: spaces to tabs 2014-11-19 11:53:23 -08:00
Johan Euphrosine
962532065c bootstrapvz/plugins/docker_daemon: flake8 2014-11-19 11:49:26 -08:00
Johan Euphrosine
f3a9a1b1ec docker_daemon: add pull_images option 2014-10-31 05:32:15 -07:00
Tim Smith
7944db886f Enable auto-rootdisk growth on gce-backports.
This commit adds the cloud-initramfs-growroot package to the
installation list for GCE images with backports enabled, and updates
the gce tasklist to add the /etc/init.d/expand-root script
(with provider-appropriate touch-ups) to the image.
2014-10-13 10:02:15 -07:00
Tim Smith
3410118250 Fix typo in AdjustExpandRootScript
This commit fixes a typo in common.tasks.initd.AdjustExpandRootScript (a
missing double-quote in the search regex) that rendered it ineffective.
2014-10-09 21:05:48 -07:00
Dave Bailey
bad378e28b fix test for DOCKER_NOFILE 2014-09-22 17:11:26 +00:00
Dave Bailey
107577cb16 allow custom ulimit -n for Docker 2014-09-17 18:12:49 +00:00
Tiago Ilieve
0f5de13bae Add version option to docker_daemon plugin
As requested on #147, there is now an option to specify the Docker
version to be installed when using the `docker_daemon` plugin. The
version string is validated against a pattern extracted from the
Docker's CHANGELOG. If the version is not present, it will just download
the latest available.

The download method was also changed from `urllib` to `wget`, so we can
see its progress if needed.

This closes #147.
2014-09-13 15:08:52 -03:00
Rick Wright
96a1683c26 Fix task ordering to better support customizations
This ensures that preferences are written before apt update and also ensures
that in GCE the backports are added to the sources before the provider-specific
SetPackageRepositories happens.

Change-Id: I3c85f922c49c2a6fbd3c0f2bad1072eff0d098c8
2014-09-08 11:45:55 -07:00
Tiago Ilieve
89a74a33c8 Fix linux-headers package version detection
The `uname -r` command returns the version of the running kernel running
on the host machine, as the chroot environment doesn't load a new one.
This prevents the proper version of the `linux-headers-*` package from
being added when the target has a different kernel version or
architecure.

This closes #121.
2014-09-02 19:53:10 -03:00
Filipe Brandenburger
1fe10207b8 Check the value of the --color argument
Make sure it's either `auto' (the default), `always' or `never'. If it
does not match any of the values, raise a Docopt exception that causes
it to print usage and exit.

Tested:
- $ sudo ./bootstrap-vz --color=invalid manifests/gce.manifest.yml
  Value of --color must be one of auto, always or never.
  Usage: bootstrap-vz [options] MANIFEST

Signed-off-by: Filipe Brandenburger <filbranden@google.com>
2014-08-24 10:52:08 -07:00
Filipe Brandenburger
84cf497c66 Support --color option to indicate whether to use colors in the terminal
Mimic the behavior of the --color=<auto|always|never> found in tools
like `ls' and `grep'. Default to `auto' which checks whether stderr is a
tty to define whether colors are used.

Tested:
- Ran ./boostrap-vz --color=never and --color=always, confirmed
  colorization worked as expected.
- Ran ./boostrap-vz --color=auto ${manifest_file} 2>bootstrap.log,
  confirmed colors were not added to boostrap.log by default, repeated
  the test with --color=always and confirmed escape sequences were
  output.

Signed-off-by: Filipe Brandenburger <filbranden@google.com>
2014-08-22 11:46:19 -07:00
Max Illfelder
3ff1bfb7d9 Removed a check to verify file type
The file command is not installed by bootstrap-vz by default.

Change-Id: I59d5f04145b7db517c28bfec1d4d758be468b398
2014-08-14 18:10:54 -07:00
Jimmy Kaplowitz
89e1a701eb Disable SSH password auth on GCE
This change was unintentional but occurred as part of GCE's transition
from build-debian-cloud to bootstrap-vz.

Might be replaced later with a similar change that applies to all
bootstrap-vz providers, based on the opinion of the debian-cloud list.

Change-Id: I72a694c49f32df06252d9cc01b1d5c7cfc015347
2014-08-14 18:10:54 -07:00
Jimmy Kaplowitz
3bfe9dddf9 Reinstate hostname hook for GCE
This DHCP exit hook to shorten the system hostname on GCE was previously
installed by build-debian-cloud and bootstrap-vz, but seems to have been
inadvertently removed in commit c81045cc6e
as part of a broad cross-cloud cleanup. Again, this was caught by our
validation tests, and we might be done with the fixes at this point.

In this commit, I'm reinstating the hook with a name change and an
explanatory comment, to reduce the risk of this vanishing accidentally
in the future.

Change-Id: I4e7268f8b9ab3b2a7fc8b510898c6fbdd685aa53
2014-08-05 17:56:33 -07:00
Anders Ingemann
9f46bdc519 Merge pull request #131 from jkaplowitz/cloudsdk
Install Cloud SDK, which includes gcutil and gsutil.
2014-08-05 17:12:51 -06:00
Jimmy Kaplowitz
a352f4d576 Installing Cloud SDK which includes gcutil and gsutil. We still don't
have a package for it so laying out the tarball for now.

Change-Id: If66f0f1c074e6077e1ca57375cac9c4832bbd7fc
2014-08-01 20:05:23 -07:00
Anders Ingemann
7ad565cbe9 Merge pull request #126 from jkaplowitz/development
Install ca-certificates on GCE
2014-07-28 21:39:56 +02:00
Jimmy Kaplowitz
e8f04d0baf Install ca-certificates on GCE
Needed to fetch GCE startup scripts over HTTPS, among other reasons.

Change-Id: I89e3afb44f786539b5e3275b5f1f3b7201ab44fd
2014-07-25 09:49:52 -07:00
Jimmy Kaplowitz
c412c4cdcf Fix list of tasks and their ordering
- GCE provider wasn't including the GCE SetHostname task, without which
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=604883 was
  preventing the hostname from getting set after reboot.
- During the GCE build, one of the GCE cleaning tasks was trying to run
  an apt-get update after the build-time resolv.conf file was removed.
  Fix this ordering by moving the network.Remove* tasks to the
  system_cleaning phase as they should have been all along, and adding
  an appropriate ordering rule for the GCE cleaning task.
- Add the fallback http.debian.net mirror after, not before, our mirror.
- The puppet plugin's ApplyPuppetManifest task specified that it should
  run before the network.Remove* tasks within the system_modification
  phase. Now that those tasks have been moved to a later phase
  (system_cleaning), remove this dependency. I have no puppet manifest
  to test this change, but am including it in hopes of avoiding a
  breakage there. Hopefully someone who uses puppet can test this or at
  least confirm that it's correct.

Change-Id: Ieca97f288f456bab119989f4cbc4c3993a755830
2014-07-24 20:20:54 -07:00