Unix

Containerization, Virtualization, “Microservice Architectures”

• Microservices, Wikipedia

• Microservices: a definition of this new architecture, my Martin Fowler

• Operating-system-level virtualization (a.k.a., “containers”), Wikipedia

• Linux Containers

  • Container Overview, from CoreOS

• When to use containers (and when not to), by Steven Vaughan-Nichols, May 15, 2017

• The Twelve-Factor App (”The twelve-factor methodology can be applied to apps written in any programming language, and which use any combination of backing services (database, queue, memory cache, etc).”)

Docker

Caution

Docker and all the related entities in the Docker ecosystem are changing constantly (and at a fast pace) so many of the resources below can become outdated or obsolete in a matter of months. Docker Releases occur regularly, so pay attention to dates and version numbers in the resources below when considering implementing something, but also look for useful tips and tricks that may still be relevant over time. Also, you may want to Subscribe to Docker Weekly Newsletter to maintain your own situational awareness of updates and high-quality articles, blog posts, and videos that can help you keep up.

Docker intro

• Docker (“Build, ship, and run any application, anywhere.”)

• Docker: Build, Ship and Run Any App, Anywhere, by Martijn Dwars, Wiebe van Geest, Rik Nijessen and Rick Wieman

• Docker Reference Architecture: Designing Scalable, Portable Docker Container Networks, by Mark Church, Docker web site, October 18, 2016

• Docker Command Line

• Jeff Lindsay on Best Practices for Working with Containers and Docker

• GitHub docker/docs/sources/articles/dockerfile_best-practices.md (Best practices for writing Dockerfiles)

• Cooking with Docker: useful tips for beginners, by Blair Hudson

• Docker traps (and how to avoid them), by Mr. Blue Coat

• What is Docker?

• Introduction to Docker, Solomon Hykes (dotCloud founder and CTO of Docker), Twitter University

• 5 Reasons to Start Using Docker, by David Bolton, June 1, 2015

• GitHub prakhar1989/docker-curriculum (“A comprehensive tutorial on getting started with Docker! http://prakhar.me/docker-curriculum/”)

• Play with Docker classroom, Docker web site

Tips and Tricks for More Advanced Docker Use

• ctop - manage and monitor your Docker containers, by Tobias Trelle, December 17, 2018

• Avoiding Permission Issues With Docker-Created Files, by vsupalov

• How To Add A “Try In Play-With-Docker” Button To Your Github Project, by Pattern Recognizer, October 8, 2018

• GitHub rifelpet/docker-demo (“Deep Dive on everyday Docker functionality”)

• What’s eating my disk? Docker System Commands explained, by Nils De Moor, April 12, 2017

• Docker Container Anti Patterns, by Arun Gupta, October 30, 2016

• What’s New In Docker 1.12, by Emmet O’Grady, August 2, 2016

• Docker Tips And Tricks: Some tips about running and building with docker, by Luís Armando Bianchin, February 15, 2016

• Flexible Docker entrypoint scripts, by camptocamp SA, March 22, 2016

• Deploying public keys in Docker containers, by camptocamp SA, March 22, 2016

• When and why should I use apt-get update?, stackexchange, May 19, 2014

• Best practices for writing Dockerfiles, Docker.com web site

• Create a base image, Docker.com web site

• Building good docker images, by Jonathan Berghnoff, October 3, 2014

• Where are my containers? Dockerized service discovery with Consul, by Jose Luis Ordiales, January 23, 2015

• Automatic container registration with Consul and Registrator, by Jose Luis Ordiales, February 3, 2015

• Trapping signals in Docker containers, by Grigoriy Chudnov, April 7, 2015

• Docker Inspect Template Magic, by Adrian Mouat

• Wrangling Grafana and InfluxDB into a Docker image, by Si Beaumont, February 17, 2016

• An Example of Docker Multi Host Clustering Part 1, by Joshua Davis, January 20, 2016

• An Example of Docker Multi-Host Networking with Hadoop - Part 2, by Joshua Davis, February 10, 2016

• An Example of Docker Multi-Host Networking With Hadoop Part 3 - Validating your Swarm Cluster, by Joshua Davis, February 18, 2016

• See also: dockercleanup

Books and guides (and a mindmap!)

• Best practices for writing Dockerfiles, Docker documentation

• The Docker Ecosystem (mindmap by Krishnan Subramanian)

• The Docker Book, by James Turnbull

  • Sample Chapter

  • Code examples

• O’Reilly Docker cookbook, by Sebastien Goasguen

  • GitHub how2dock/docbook (Sample code and Vagrant files for O’Reilly Docker cookbook)

• Welcome to the Docker User Guide

  • Understanding Docker

  • Network Configuration

  • Docker Compose

    • See also: How to use Docker Compose to run complex multi container apps on your Raspberry Pi and the RaspberryPi section of Small form-factor hardware systems.

• How to Use Docker on OS X: The Missing Guide, by Chris Jones

Why Docker?

• Sailing Past Dependency Hell With Docker, by Alex Johnson, July 2, 2015

Architecture

• Docker vs. VMs? Combining Both for Cloud Portability Nirvana, by Thorsten von Eicken, September 2, 2014

• Understand the architecture (v1.10), Docker web site

• Higher Order Infrastructure - Microservices on the Docker Swarm, by Nicola Paolucci, GOTO 2016

Docker in Production

• Why Docker is Not Yet Succeeding Widely in Production, by Simon Hørup Eskildsen, July 2015

• The Realities of Docker in Production, by Ben Ball, March 31, 2015

• Docker in Production, by Jérôme Petazzoni, September 22, 2014

• Docker deployments, by Paul Showalter & Karl Matthias, New Relic, September 10, 2014

  • GitHub 6si/shipwright (The right way to build, tag and ship Docker containers.)

  • GitHub newrelic/centurion (A mass deployment tool for Docker fleets)

  • GitHub newrelic/check_docker (A Go Nagios check for Docker)

  • GitHub newrelic/go_nagios (Go lang package for writing Nagios checks)

Docker and Development

• How To Cook Microservices (with Ruby spices) is a continuously updated collection of

  notes, insights and ideas about building software platforms empowered by microservices architecture with Ruby language

• Docker for Developers, by Jérôme Petazzoni, November 25, 2014

• Tips for running Docker in development, by Philip Kallberg, May 27, 2015

• GitHub wsargent/docker-cheat-sheet (Docker Cheat Sheet)

• Test, Develop, Build, Stage with Docker, by Simone Di Maulo, May 2, 2015

• A Docker Dev Environment in 24 Hours! (Part 1 of 2), by John Fiedler, October 31, 2013

• A Docker Dev Environment in 24 Hours! (Part 2 of 2), by John Fiedler, November 5, 2013

  • GitHub relateiq/docker_public (Instant RelateIQ Development Environment - code from parts 1 and 2)

• Top 10 Open-Source Docker Developer Tools, by Lucas Carlson, March 5, 2014

• Executable Images - How to Dockerize Your Development Machine, by Quinten Krijger, August 29, 2015

• Development Environments with Vagrant, Docker, and Supervisord, by Tyler H.T. Cipriani, May 25, 2014

Docker and Security

Docker security be like pic.twitter.com/EZDUK9iRVx

— the grugq (@thegrugq) May 31, 2016

• Docker Security, Docker web site

• Why A Privileged Container in Docker Is a Bad Idea, by Trend Micro, December 20, 2019

• Docker Ramps Up Container Security, by Jack M. Germain, May 13, 2016

• Understanding and Hardening Linux Containers (PDF), NCC Group whitepaper by Aaron Grattafiori, April 20, 2016

  • Docker and High Security Microservices: A Summary of Aaron Grattafiori’s DockerCon 2016 Talk, by Daniel Bryant on August 14, 2016

• Containing and Attack with Linux Containers (PDF), Shmoocon 2016 presentation by Jay Beale, InGuardians, January 17, 2016

• Using Linux Containers to Jail Programs, by Jay Beale, Raleigh B-Sides, October 9, 2015

• GitHub docker/docker-bench-security (The Docker Bench for Security is a script that checks for all the automatable tests included in the CIS Docker 1.6 Benchmark. https://dockerbench.com)

• Introducing a *Super* Privileged Container Concept, by rhatdan, November 6, 2014

• Are Docker containers really secure?, by Daniel J Walsh, July 22, 2014

• Bringing new security features to Docker, by Daniel J Walsh, September 3, 2014

• Someone said that 30% of the images on the Docker Registry contain vulnerabilities, by jpetazzo, May 27, 2015

• Exploring Docker Volumes for Phases of Development, by Alan Kent, May 31, 2015

• Understanding Docker Security and Best Practices, Docker blog, May 5, 2015

• Container Security: Just The Good Parts, by Trevor Jay, April 29, 2015

• Docker SELinux Experimentation with Reduced Pain, by zwischenzugs, April 29, 2015

• The sad state of sysadmin in the age of containers, by Erich Shubert [pointing out some extremely poor security practices by many in DevOps]

  • reddit.com/r/programming comments thread

• Docker security gets thumbs-up despite containers’ rapid rise, by Toby Wolpe, ZDNet News, January 12, 2015

• Docker security in the future, by Daniel J Walsh, March 19, 2015

• Docker Security: Best Practices for your Vessel and Containers, by Michael Boelen, January 22, 2015

• GitHub GDSSecurity/Docker-Secure-Deployment-Guidelines (Deployment checklist for securely deploying Docker)

• CoreOS vs. Project Atomic: A Review, Major Hayden, May 13, 2014

• Analysis of Docker Security, by Than Bui

• Storage Concepts in Docker: Persistent Storage, by Mark Lamourine, October 10, 2014 [Has a good discussion of SELinux labeling of directories for use by Docker containers]

• Docker’s just a bit dodgy, but ready for rollout says Gartner, by Simon Sharwood

• Snappy Security

• Launch secure LXC containers on Fedora 20 using SELinux and sVirt, by Major Hayden

• See also the SElinux, Apparmor, GRsecurity section

Orchestration

• Docker Orchestration workshop - Jérôme Petazzoni, Feb 21, 2016

  • Part 1 (1:21:09)

  • Part 2 (0:59:37)

  • Parts 3 and 4 (3:20:54)

  • GitHub jpetazzo/orchestration-workshop (Slides and examples for the workshop)

• Docker Orchestration & Metrics, Tiffany Jernigan, Seattle Docker Meetup, December 2016

  • Slides from Tiffany’s presentation

• Docker 1.12 Swarm Mode Deep Dive Part 2: Orchestration, YouTube by Andrea Luzzardi, July 28, 2016

• Create a swarm cluster with Docker 1.12 swarm mode, by Luc, July 5, 2016

• ContainerPilot and the Autopilot Pattern, by Tim Gross, Container Summit Austin, July 19, 2016

• 3 Node Swarm Cluster in 30 seconds (Docker 1.12), by John Zaccone, July 29, 2016

• Kubernetes + How did they ever come up with that kooky ‘Kubernetes’ name? Here’s the inside story, by Dan Richman, November 17, 2016 + Introduction to Microservices, Docker, and Kubernetes, YouTube video by James Quigley, November 7, 2017 + Deploy to Kubernetes, Docker for Mac documentation + kubectl Cheat Sheet, Kubernetes reference documentation + The Open Policy Agent Rego Playground

• Apollo

  • Capgemini Apollo: An Open Source Microservice and Big Data Platform, by Daniel Bryant, June 6, 2015

  • GitHub Capgemini/Apollo, An open-source platform for cloud native applications based on Apache Mesos and Docker. http://capgemini.github.io/devops/apollo/”)

  • How Apollo Uses Weave and Weave Scope, by Graham Taylor, June 30, 2015

• Atlassian Orchestration with Docker: multi-host support for the win!, by Nicola Paolucci, December 16, 2015

• DockerCon EU: Keynote on Orchestration (Docker Machine, Swarm, Compose)

• Docker 101: Dockerizing Your Infrastructure, by Stanley Lewis

• Docker and Maestro for fun, development and profit, by Maxime Petazzoni

• Docker Containers and Kubernetes with Brian Dorsey, YouTube, December 23, 2014

• CoreOS Meetup: etcd and Kubernetes, YouTube, March 19, 2015

• Docker orchestration with maestro-ng, by heisel

• GitHub signalfuse/maestro-ng (Orchestration of Docker-based, multi-host environments http://www.signalfuse.com)

  • GitHub signalfx/maestro-base (Base Docker image for Maestro-enabled components http://www.signalfuse.com)

  • GitHub WIZARD-CXY/docker-cassandra (Docker image for Cassandra (Maestro orchestration))

  • GitHub iantruslove/docker-elasticsearch (Docker image for ElasticSearch (Maestro orchestration))

  • GitHub signalfx/docker-zookeeper (Docker image for ZooKeeper (Maestro orchestration) http://www.signalfuse.com)

  • GitHub signalfx/docker-kafka (Docker image for Kafka (Maestro orchestration) http://www.signalfuse.com)

• (See also A production ready Docker workflow. Part 3: Orchestration tools)

• Getting Started with CoreOS and Docker using Vagrant, by Luke Bond

• Deploying Docker Containers on a Vagrant CoreOS Cluster with fleet, by Luke Bond

• Vessel automates the setup & use of dockerized development environments

Using a private registry

• Docker Registry, Docker web site

• Deploying a registry server, Docker web site

• Running Secured Docker Registry 2.0, by Jaroslav Holub, April 28, 2015

• How To Set Up a Private Docker Registry on Ubuntu 14.04, by Nik van der Ploeg, October 15, 2014

• How to Secure Your Private Docker Registry, by Alex Welch, January 2, 2015 (goes with previous post)

• Setup Your Own Docker Registry on CoreOS, Vultr, May 7, 2015

• Setting Up Docker Private Registry, by beingasysadmin, January 14, 2015

• Docker: How to Use Your Own Private Registry, Twitter University, November 12, 2013

Configuration management and service discovery

• Demystifying Service Discovery under Docker Engine 1.12.0, by ajeetraina, July 27, 2016

• Consul (“Service discovery and configuration made easy. Distributed, highly available, and datacenter aware.”)

• Consul Part 1: Service discovery, the easy way, YouTube video by OpsForce, July 17, 2016

• Consul Part 2: Service health and templates, YouTube video by OpsForce, August 3, 2016

• Configuration management with Consul, by Michael de Jong, October 26, 2015

• Configuration management from Git to Consul, by Ryan Breen, May 8, 2015

  • GitHub Cimpress-MCP/git2consul (“Mirrors the contents of a git repository into Consul KVs.”)

  • GitHub Cimpress-MCP/fsconsul (“Write Consul K/Vs to the filesystem.”)

  • GitHUb Cimpress-MCP/gosecret (“A Go library for encrypting and decrypting slices of byte arrays.”)

• Distributed Configuration Management and Dark Launching Using Consul, by Bill Monkman, November 26, 2014 (Has examples of creating configuration management kv store)

• GitHub kelseyhightower/confd (“Manage local application configuration files using templates and data from etcd or consul”)

• An Introduction to Using Consul, a Service Discovery System, on Ubuntu 14.04 (Part 1 of 3), by Justin Ellingwood, August 15, 2014

• How to Configure Consul in a Production Environment on Ubuntu 14.04 (Part 2 of 3), by Justin Ellingwood, August 15, 2014

• How To Secure Consul with TLS Encryption on Ubuntu 14.04 (Part 3 of 3), by Justin Ellingwood, August 15, 2014

• Understanding Modern Service Discovery with Docker, by Jeff Lindsay

• Consul Service Discovery with Docker, by Jeff Lindsay

• Automatic Docker Service Announcement with Registrator, by Jeff Lindsay

• A High Available Docker Container Platform Using CoreOS And Consul, by Mark van Holsteijn

• GitHub democracyworks/consul-coreos (“Bootstraps a Consul cluster on CoreOS using fleet and etcd”)

• Simulating service discovery with Docker and etcd, by Aaditya Talwai

• GitHub Cimpress-MCP/git2consul (“Mirrors the contents of a git repository into Consul KVs.”)

• GitHub ianbytchek/docker-coreos-ansible-toolbox (“Using Ansible with CoreOS? Use CoreOS Ansible toolbox!”)

Networking and Docker containers

• Docker native networking

  • Docker Stacks and Attachable networks, by Alex Ellis, February 17, 2017

  • Docker 1.12 Swarm Mode Deep Dive Part 1: Topology, YouTube by Andrea Luzzardi, July 28, 2016

  • Splendors and Miseries of Docker Network, by Aleksandr Tarasov, November 16, 2015

  • Add Docker Machine to Swarm cluster after creation, stackoverflow thread, January 4, 2016

  • Docker Networking takes a step in the right direction, Docker blog, April 30, 2015

• Weave

  • GitHub weaveworks/weave (“Simple, resilient multi-host Docker networking http://weave.works”)

  • Multi-host Docker deployment with Swarm and Compose using Weave 0.11, by errordeveloper, May 27, 2015

  • Networking Docker Containers with Weave on CoreOS, Weave web site

  • Using Weave Scope Standalone to Visualize and Monitor Docker Containers, Weave web site

  • Using Docker Machine and Swarm with Weave 0.10, by errordeveloper, May 6, 2015

  • How to set up networking between Docker containers, by Dan Nanni, March 20, 2015

  • Using Docker Machine with Weave 0.10, by Ilya Dmitrichenko, April 22, 2015

    • GitHub infrabricks/powerstrip-demo (This is a TLS powerstrip weave demo, installed with docker-machine!)

  • Elasticsearch, Weave and Docker, by errordeveloper, January 20, 2015

    • GitHub errordeveloper/weave-demos/hello-apps/elasticsearch-js/scripts/run_elasticsearch_2_clusters.sh

    • Curator: Tending your time-series indices, by Aaron Mildenstein, January 20, 2014

      • GitHub elastic/curator (Curator: Tending your Elasticsearch indices”)

    • Logstash how to remove old logs, by Andriy Podanenko

  • Deploying and migrating an Elasticsearch-Logstash-Kibana stack using Docker Part 1, by Ryan Wallner, January 12, 2016

  • Deploying and migrating an Elasticsearch-Logstash-Kibana stack using Docker Part 2, by Ryan Wallner, January 12, 2016

  • I just created a Cassandra cluster that spans 3 different network domains, by using 2 simple shell commands. How cool is that?, by Yaron Rosenbaum, October 8, 2014

  • Running a Weave Network on CoreOS, by errordeveloper, October 28, 2014

  • How to Network Docker Containers with Weave, by Benjamin Ball, September 14, 2014

  • Getting Started with Weave and Docker on CoreOS (GitHub fintanr/weavegs)

  • Docker, Weave, Raspberry Pi and a bit of Networked Cloud Computing!, by Alexander Grendel, December 9, 2014

• Pipework

  • GitHub jpetazzo/pipework (Software-Defined Networking tools for LXC (LinuX Containers))

  • Advanced Docker Networking with Pipework, by Sam Leathers

• Flannel

  • CoreOS + Layer Meetup 10/6/14: Brian “Redbeard” Harrington - Warming Up With Flannel, YouTube, October 7, 2014

  • Docker Networking - CoreOS Flannel, Sreenivas Makam, January 18, 2015

• OpenVPN

  • How To Run OpenVPN in a Docker Container on Ubuntu 14.04, by Kyle Manna, February 2, 2015

  • Start an OpenVPN Container as a systemd managed service under CoreOS, by c-garcia, March 15, 2015

  • OpenVPN in a container, by Ed Vielmetti, November 20, 2015

  • Securing wifi traffic with OpenVPN and Docker, by Christopher Bunn, April 03, 2015

  • Docker + Joyent + OpenVPN = Bliss, by Jérôme Petazzoni, September 10, 2013

    • GitHub jpetazzo/dockvpn (“Recipe to build an OpenVPN image for Docker”)

• Integrating Proxy With Docker Swarm (Tour Around Docker 1.12 Series), by Viktor Farcic, August 1, 2016 [Good networking diagrams]

• Docker networking overview, by Filip Verloy, February 17, 2016

• Docker Swarm and experimental multihost networking with docker-machine and boot2docker, by IIkka Anttonen, July 15, 2015

• Three Solutions to Bi-directional Linking Problem in Docker Compose, by Abdelrahman Hosny, July 1, 2015

• Docker Networking Meetup - Intro to Weave/Flannel, by Dhananjay ‘DJ’ Sampath, January 23, 2015

• 5 ways Docker is fixing its networking woes, by Serdar Yegulalp, October 20, 2014

• Docker networking - IP per container, one /24 per pod(worker), by Vicente De Luca, April 30, 2015

• See also: Network Configuration

Persistent storage in Docker containers

• Docker: Storage Patterns for Persistence, by Karim Vaes, February 11, 2016

• Comprehensive Overview of Storage Scalability in Docker, by Jeremy Eder, September 30, 2014

• Storage Concepts in Docker: Shared Storage and the VOLUME directive, by Mark Lamourine, October 7, 2014

• Storage Concepts in Docker: Persistent Storage, by Mark Lamourine, October 10, 2014 [Has a good discussion of SELinux labeling of directories for use by Docker containers]

• Share disks through NFS on a CoreOS cluster?, stackexchange post, November 25, 2014

• Enabling and Mounting NFS on CoreOS, by Scott Lowe, February 20, 2015

• Exploring Docker Volumes for Phases of Development, by Alan Kent, May 31, 2015

• Quickly build arbitrary size Hadoop Cluster based on Docker, by KiwenLau, May 29, 2015

• See also: Hadoop section of Databases and database tools

Persistent services within Docker containers

• Automatically start containers, Docker web site

• Introducing dumb-init, an init system for Docker containers, by Chris K., Jan 6, 2016

• Docker All The Things: Nginx And Supervisor, by Matthew McKeen, December 14, 2013

• Running node and nginx in docker with supervisor, stackoverflow post, December 2, 2014

• Creating a Docker Container to run PHP, NGINX and Hip Hop VM (HHVM), July 15, 2014

• Roll your own Docker registry with Docker, Compose, Supervisor, and Nginx, by Philipp Wintermantel, March 18, 2014

Clustering Docker containers

• Using Docker Stack And Compose YAML Files To Deploy Swarm Services, by Viktor Farcic, January 23, 2017

• Running a Small Docker Swarm Cluster, Scott Lowe, March 6, 2015

• Docker Machine, Compose & Swarm, by MediaGlasses

• Clustering Using Docker Swarm 0.2.0, by Arun Gupta

• A quick overview of Docker Swarm, by Olivier Robert, February 2, 2016

Logging/monitoring activity of containers

• Collecting All Docker Logs with Fluentd, by Kiyoto Tamura, July 7, 2015

• Automating Docker Logging: ElasticSearch, Logstash, Kibana, and Logspout, by Nathan LeClaire, April 27, 2015

• Running High Performance and Fault Tolerant Elasticsearch Clusters on Docker, sematext group, November 24, 2015

• Scalable Docker Monitoring with Fluentd, Elasticsearch and Kibana 4, by manu, November 21, 2014

• Performance tuning ELK stack, by Josh Reichardt, March 30, 2015

• syslog logging driver for Docker, by Mark Wolfe, May 3, 2015

• Real-time monitoring of Hadoop clusters, by Attila Kanto, October 7, 2014

  • Apache Hadoop 2.6.0 on Docker, by Janos Matyas, September 15, 2014

Cleaning up (or keeping Docker images as small as possible)

• Reducing the size of Docker Images, by Richard Woudenberg, March 18, 2015

• Making Debian Docker Images Smaller, by Dave Beckett, April 18, 2015

• How I shrunk a Docker image by 98.8% – featuring fanotify, by Jean-Tiare LE BIGOT, April 25, 2015

• Low on disk space, cleaning up old Docker containers, by johnarce, May 20, 2014

• Squashing Docker Images, by Jason Wilder, August 19, 2014

• Optimizing Docker Images, by Brian DeHamer, July 28, 2014

• Create The Smallest Possible Docker Container, by Adriaan de Jonge, July 4, 2014

• How To Create The Smallest Possible Docker Container Of Any Image, by Mark van Holsteijn, June 30, 2015

Docker and Continuous Integration

• Docker Jenkins, by Yuri Kushch, February 17, 2017

• Easy CD-CI with Jenkins, Docker Swarm and Docker secrets, by KrazY CoCoon Lab, March 6, 2017

• Fully automated development environment with docker-compose, by Andrew Orsich, February 22, 2017

• Saving money with Jenkins, Digital Ocean and Docker, by Damien Coraboeuf, May 5, 2016

• Docker, Amazon EC2 and continuous builds using Jenkins, by Jan Haderka, February 10, 2016

• Continuous Integration, Delivery or Deployment with Jenkins, Docker and Ansible by Viktor Farcic, February 11, 2015

• Deployment Pipeline using Docker, Jenkins, Java and Couchbase, by Arun Gupta, September 9, 2016

• Scaling to Infinity with Docker Swarm, Docker Compose and Consul, by Viktor Farcic, July 2, 2015

  • (Part 1/4) - A Taste of What is to Come

  • (Part 2/4) - Manually Deploying Services

  • (Part 3/4) - Blue-Green Deployment, Automation and Self-Healing Procedure

  • (Part 4/4) - Scaling Individual Services

• Maxfield F. Stewart blog series, Riot Games Engineering blog

  • Part I: Thinking Inside the Container

  • Part II: Putting Jenkins in a Docker Container

  • Part III: Docker & Jenkins: Data That Persists

  • Part IV: Jenkins, Docker, Proxies, and Compose

  • Part V: Taking Control of Your Docker Image

  • Part VI: Building with Jenkins Inside an Ephemeral Docker Container

  • Part VII: Tutorial: Building with Jenkins Inside an Ephemeral Docker Container

  • Part VIII: DockerCon Talk and the Story So Far

  • GitHub maxfields2000/dockerjenkins_tutorial (“A repository for items learned in my Getting Started with Jenkins and Docker tutorial series”)

• Delivering eBay’s CI Solution with Apache Mesos – Part I, by The eBay PaaS Team, April 4, 2014

• Delivering eBay’s CI Solution with Apache Mesos – Part II, by The eBay PaaS Team, May 12, 2014

• DockerCon Video: Delivering eBay’s CI Solution with Apache Mesos & Docker, by Victor Coisne, June 23, 2014

• Disaster-proofing slaves with Docker Swarm and the CloudBees Jenkins Platform, by Tracy Kennedy, June 19, 2015

• Continuous Integration and Delivery with Docker, by Jaroslav Holub, May 28, 2015

• Jenkins, Docker Hub (Official Jenkins Docker Image)

• Running Jenkins in Docker Containers, by Peter Sellers, February 11, 2015

• Import Jenkins Configuration to a dockerized jenkins, by Allan Espinoza, January 25, 2015

• Docker in Docker with Jenkins and Supervisord, by Johan Haleby, March 14, 2015

• Jenkins in a Docker container, by by Thomas Einwaller, September 1, 2014

Miscellaneous Docker related articles and blog posts

• Consul Template for transparent load balancing of containers, by Jose Luis Ordiales, April 1, 2015

• Orchestrating your containers with CoreOS, an introduction, by Jose Luis Ordiales, July 12, 2015

• 3 hours to Docker fundamentals: Jumpstart your Docker knowledge, by Aater Suleman, October 13, 2014

• Docker Registry richhaase/bigtop-hadoop

• Automated docker ambassadors with CoreOS + registrator + ambassadord

• Understanding user file ownership in docker: how to avoid changing permissions of linked volumes, stackoverflow

• GitHub signalfx/docker-java (Java Docker API Client)

• A production ready Docker workflow (part 1), by Luis Elizondo

• A production ready Docker workflow. Part 2: The Storage Problem, by Luis Elizondo

• A production ready Docker workflow. Part 3: Orchestration tools, by Luis Elizondo

• Your very own server with Docker, by Erwyn

• A Not Very Short Introduction to Docker, by Anders Janmyr

• Deploying NGINX and NGINX Plus with Docker, by Rick Nelson

• Make Docker Machine Do Anything With Our Experimental Extensions, by kcoleman, September 26, 2015

• Consul + Consul-Template with Docker-Compose, by Neil Ni, September 14, 2015

• Virtualenv the Docker way, by Justyna Ilczuk, Nov 16, 2014

• GitHub daniel-illi/docker-haproxy-letsencrypt (“Dockerized HAProxy with Let’s Encrypt integration”)

• GitHub BradJonesLLC/docker-nginx-letsencrypt (“Nginx container with Let’s Encrypt auto-renew “)

Operating systems for running Docker

CoreOS

• CoreOS

  • Container Overview, CoreOS web site

  • CoreOS Clustering, CoreOS web site

  • CoreOS Cluster Architectures, CoreOS web site

  • CoreOS Overview and Current Status, Slideshare by Sreenivas Makam, April 17, 2016

  • etcd, CoreOS web site

  • Running CoreOS on Vagrant, CoreOS web site

  • Booting CoreOS via PXE, CoreOS web site

    • GitHub kelseyhightower/coreos-ipxe-server (“CoreOS iPXE server”)

  • Customizing Docker, CoreOS web site

  • Deploying a Service Using fleet CoreOS web site

  • Installing CoreOS to Disk, CoreOS web site

    • GitHub nyarla/coreos-live-iso/makeiso.sh

  • Ignition: Better Machine Configuration (CoreOS Fest 2015), May 27, 2015 (has some details of boot sequence that are hard to learn)

  • Lessons Learned From Building Platforms on Top of CoreOS (CoreOS Fest 2015), May 27, 2015

    • 10 Lessons Learned Using CoreOS, by Gabriel Monroy (slides for CoreOS Fest 2015 talk)

• Pay attention to these bug reports (and what is said in them)

  • Network settings should be set in oem cloud-config.yml (coreos/bugs #11, April 28, 2014)

  • custom iptables - boot order, (coreos/bugs #58, June 26, 2014)

  • docker-tcp.socket fails: Socket service docker.service already active, refusing. (coreos/coreos-vagrant bug #172, September 30, 2014)

  • How does docker-tcp.socket actually enable Docker’s remote API on CoreOS?, superuser post, January 5, 2015

• How can I customize bashrc, bash_profile or profile on a CoreOS installation?, Stackoverflow post by Richard, Jun 2 2015

• GitHub https://github.com/coreos/coreos-overlay/tree/master/app-shells/bash/files/

• CoreOS: Orchestrating the Fleet, YouTube video by Brian Waldon, July 8, 2014

• Digital Ocean tutorial series: Getting Started with CoreOS

  • How To Troubleshoot Common Issues with your CoreOS Servers, (part 8 of 9), September 18, 2015

  • How To Secure Your CoreOS Cluster with TLS/SSL and Firewall Rules, (part 9 of 9), December 7, 2015

OpenNode OS

• OpenNode OS (“Lightweight bare-metal cloud OS combining Linux Containers and KVM full virtualization options into payload optimized solution.”)

• NodeFabric Host Image (“NodeFabric delivers hyperconverged database and storage solution for highly available, self-healing and load-balanced cloud services”)

RancherOS

• RancherOS (A minimalist distribution of Linux designed from the ground up to run Docker containers.)

• Announcing RancherOS

Project Atomic

• Project Atomic (Trusted Distributions, Atomic Updates)

Ubuntu “Snappy”

• Snappy Ubuntu Core

  • SNAPPY UBUNTU CORE image for Raspberry Pi

• It’s a Snap!

• Snappy Security

ArchLinux

• ArchLinux

Configuration management and automated provisioning

• Collins (“Infrastructure management for engineers”)

• GitHub coreos/etcd (A highly-available key value store for shared configuration and service discovery)

  • Brandon Philips Explains etcd, by Phil Whelan

  • CoreOS: etcd 2.0, by Brandon Philips

• The Marriage of Ansible and Docker, by Zuletzt geändert von Unbekannter Benutzer, June 5, 2015

• docker, ansible and vagrant, by Wojtek Oledzki, January 1, 2015

• Ubuntu Cloud-Init Technology, YouTube video by ubuntucloud, December 8, 2010

• The Assimilation Project

• Ansible and Docker, by Ash Wilson

• OpenStack

  • Building HA Clusters with Ansible and Openstack, by Remy van Elst, July 25, 2014

    • GitHub RaymiiOrg/ansible (ansible/openstack-example)

  • Automating Openstack with cloud init run a script on VM’s first boot, by Remy van Elst, March 11, 2015

  • Provisioning IaaS Clouds with Dynamic Ansible Inventories and OpenStack Metadata, by Lukas Pustina

  • GitHub ewindisch/dockenstack (OpenStack Devstack on Docker)

  • OpenStack: Docker wiki section from OpenStack

  • OpenStack is overkill for Docker: New tooling is necessary for effectively managing Docker at scale, by Matt Asay, August 10, 2015

  • Up and Running with Docker Machine and OpenStack, by Spencer Smith, Solinea blog, July 8, 2015

  • OpenStackClient and OpenStack Python SDK, youtube video by Dean Troyer, October 27, 2015

  • Life Without DevStack: OpenStack Development With OSA, by Miguel Grinberg

    • (Demonstration video)

• Agile Configuration Management – Intermezzo by Marcus Philip, Diabol

• Top 5 Open Source Linux Server Provisioning Software, by NIXCRAFT

Virtualbox

• Virtual Box Headless Cheatsheet: Headless Virtual Machine Install/Import and setup, by Tim Arneaud, October 9, 2012

Packer

• Packer

  • Command Line: Build

  • Debugging Packer Builds

• GitHub boxcutter/ubuntu (“Virtual machine templates for Ubuntu”)

• GitHub tylert/packer-build (“Packer Automated VM Image and Vagrant Box Builds”)

• Packer: In 10 minutes, from zero to bootable VirtualBox Ubuntu 12.04, by @kappataumu, September 8, 2013

Vagrant

• Vagrant

• How To Use Vagrant To Create Small Virtual Test Lab on a Linux / OS X / MS-Windows

• How to Create and Share a Vagrant Base Box, by George Fekete, July 17, 2014

• Vagrantfile Explained: Setting Up and Provisioning with Shell, by George Fekete, July 19, 2014

• Multi-Machine environment documentation from Hashicorp

• Using Vagrant and Ansible, Ansible documentation

• Change Insecure Key To My Own Key On Vagrant, by ermaker, November 18, 2015

• GitHub AAFC-MBB/vagrant-specify7 (“Package to launch a Specify7 instance in a Vagrant VM”)

• IP lookup for Vagrant private networking, by wamonite, July 12, 2014

• Elegant virtualization with Vagrant, by zenonharley, July 21, 2015

Ansible

• Ansible (web site)

  • Playbooks Best Practices

  • GitHub ansible/ansible-examples (“A few starter examples of ansible playbooks, to show features and how they work together. See http://galaxy.ansible.com for example roles from the Ansible community for deploying many popular applications.”)

  • docker - manage docker containers

• Ansible ‘Collections’ and modular delivery

  • The Future of Ansible Content Delivery, by Dylan Silva, July 23, 2019

  • Make your Ansible Playbooks flexible, maintainable, and scalable, by Jeff Geerling, September 28, 2018

  • Patching or using a forked version of an Ansible Galaxy role, by Jeff Geerling, September 11, 2017

  • Release and Deploy Ansible Collection with GitHub Actions, by Jose Angel Munoz, August 9, 2020

• Testing

  • How to test Ansible playbook/role using Molecules with Docker, by sumitksuman, December 24, 2019

  • Continuous Testing with Molecule, Ansible, and GitHub Actions, YouTube video by Jeff Geerling, October 13, 2020

  • Rapidly Build & Test Ansible Roles with Molecule + Docker, YouTube video by Percy Grunwald, February 15, 2019

• Alternate “Best Practices” (possibly conflicting, but helpful to consider none the less)

  • Laying out roles, inventories and playbooks, by Michel Blanc, July 2, 2015

  • Best practices to build great Ansible playbooks, by Maxime Thoonsen, October 12, 2015

  • Ansible (Real Life) Good Practices, by Raphael Campardou, March 19, 2014 (has pre-commit Git hook for ansbile-vault)

  • Lessons from using Ansible exclusively for 2 years, by Corban Raun, March 24, 2015

  • 6 practices for super smooth Ansible experience, by Maxim Chernyak, June 18, 2014

  • GitHub enginyoyen/ansible-best-practises (“A project structure that outlines some best practices of how to use ansible”)

  • More Tips and Tricks, slideshare by bcoca, October 11, 2016 https://www.slideshare.net/bcoca/more-tips-n-tricks

• Episode #43 - 19 Minutes With Ansible (Part 1/4), Justin Weissig, sysadmincasts.com, January 13, 2015

• Episode #45 - Learning Ansible with Vagrant (Part 2/4), Justin Weissig, sysadmincasts.com, March 19, 2015

  • GitHub jweissig/episode-45 (“Episode #45 - Learning Ansible with Vagrant”)

• Episode #46 - Configuration Management with Ansible (Part 3/4), Justin Weissig, sysadmincasts.com, March 26, 2015

• Episode #47 - Zero-downtime Deployment with Ansible (Part 4/4), Justin Weissig, sysadmincasts.com, April 2, 2015

  • GitHub jweissig/episode-47 (“Episode #47 - Zero-downtime Deployments with Ansible (Part 4/4)”)

• Graduating Past Playbooks: How to Use Ansible When Your Infrastructure Grows Up, by Rob McQueen

  • GitHub nylas/ansible-flask-example (“Example using ansible-test and wrapper roles to implement a simple flask webapp”)

• The Fedora Project ansible playbook/files/etc repository for fedora infrastructure

• How Twitter Uses Ansible, YouTube video by Ansible, May 21, 2014

• GitHub ePages-de/mac-dev-setup (“Automated provisioning of your Apple Mac (Java) development machine using Ansible”)

• Advanced Ansible concepts, gotchas, things to keep in mind…

  • Security hardening for openstack-ansible, Openstack web site

    • Automated Security Hardening with OpenStack-Ansible, by Major Hayden, Openstack Austin Summit, May 1, 2016

    • GitHub openstack/openstack-ansible-security (“Security Role for OpenStack-Ansible http://openstack.org”)

  • Debugging

    • Debugging Ansible Tasks, by Greg Hurrell, August 7, 2015

    • Debug Ansible Playbooks Like A Pro, by Daniel Marks, June 27, 2017

  • Templating

    • Jinja2 for better Ansible playbooks and templates, by Daniel Schneller, August 25, 2014

    • Ansible: “default” and “bool” filters, by dddpaul-github, November 30, 2015

    • Ansible loop through group vars in template, Stackoverflow post, November 18, 2014

    • Ansible loop over variables, Stackoverflow post, October 28, 2014

    • Jinja2 Template Inheritence

    • [jinja2] Help with blocks, Reddit /r/Ansible post by by FlowLabel

    • Primer on Jinja Templating, by Real Python

  • Dynamic Inventory

    • Dynamic Inventory, Ansible documentation

    • Adapting inventory for Ansible, by Jan-Piet Mens

    • Creating custom dynamic inventories for Ansible, by Jeff Geerling, June 11, 2015

    • Writing a Custom Ansible Dynamic Inventory Script, by Adam Johnson, December 4, 2016

    • Using DNS as an Ansible dynamic inventory, by Remie Bolte, January 1, 2016

  • Facts vs. Variables

    • Fact Caching and gathering, Ansible documentation

    • Fastest way to gather facts to fact cache, Stackoverflow post, September 1, 2015

    • Ansible Custom Facts, serverascode.com

  • Ansible Plugins

    • Ansible module development in Python - 101, by Yves Fauser, Ansible Munich Meetup - going into 2016, February 23, 2016

    • Ansible: Modules and Action Plugins, by Nicholas Grisey Demengel, January 20, 2015

    • An action plugin for Ansible to handle SSH host keys and DNS SSHFP records, by Jan-Piet Mens, November 3, 2012

    • v2 callback plugin migration (thread), Google Groups

  • Front-ends for Ansible

    • Ansible Tower

    • DevOps Automation – Ansible+Semaphore is Indispensable!, by Thaddeus, code-complete.com

      • GitHub ansible-semaphore/semaphore (“Open Source Alternative to Ansible Tower https://ansible-semaphore.github.io/semaphore”)

    • Building an Automated Config Management Server using Ansible+Flask+Redis, by deepakmdas (beingsysadmin), April 21, 2015

    • rundeck (“Go fast. Be secure.”)

    • stackstorm (“Event-Driven Automation”)

      • GitHub StackStorm/st2 (“StackStorm (aka “IFTTT for Ops”) is event-driven automation commonly used for auto-remediation, security responses, facilitated troubleshooting, complex deployments, and more. Includes rules engine, workflow,1800+ integrations (see /st2contrib), native ChatOps and so forth.”

      • New In StackStorm: Ansible Integration, by Eugen C., June 5, 2015

  • Handling multi-stage or multi-deployment environments

    • Multistage environments with Ansible, by Ross Tuck, May 15, 2014

    • Multi-stage provisioning, by Victor Volle, Ansible Munich Meetup - going into 2016, February 23, 2016

  • Ansible Tips and Tricks on ReadTheDocs

  • How to Use Ansible Roles to Abstract your Infrastructure Environment, by Justin Ellingwood, February 11, 2014

  • Jinja2 for better Ansible playbooks and templates, by Daniel Schneller, August 25, 2014

  • Ansible - some random useful things, by David Goodwin, August 4, 2014

  • Tagging, ThinkAnsible, June 4, 2014

  • Scalable and Understandable Provisioning with Ansible and Vagrant, by Julien Ponge, October 15, 2013

  • Alejandro Guirao Rodríguez - Extending and embedding Ansible with Python, YouTube video from EuroPython 2015

  • etcd + ansible = crazy delicious, by UnicornClouds

  • How I Fully Automated OS X Provisioning With Ansible, by Daniel Jaouen

  • Ansible tips, by Deni Bertović, October 13, 2014

  • GitHub dellis23/ansible-toolkit (“Ansible toolkit hopes to solve [some Ansible playbook] problems by providing some simple visibility tools.”)

  • GitHub ks888/ansible-playbook-debugger (“A Debugger for Ansible Playbook”)

  • Hacking ansible, slideshare, October 15, 2014 (“a quick presentation on ansible internals and a focus on the ease of expansion through the plugin”)

  • ansible-exec: ansible-playbook wrapper for executing playbooks, by Hagai Kariti, August 26, 2014

  • Using virtualenv Python in local Ansible, by Matt Behrens, April 5, 2014

  • Ansible: A Simple Rollback Strategy for Roles and Playbooks, by Valentino Gagliardi, June 25, 2014

  • Proposal for fixing playbooks with dynamic include problems, Ansible Development Google Group post

  • See also Continuous Integration and DevOps.

Storing Secrets for Development and Configuration

• Case studies in secrets leaks + Avoiding Default Passwords & Secrets Breaches Using Open Source Katherine Carpenter & Dave Dittrich, ToorCon XX, November 26, 2018 + How to build and deploy a beautiful personal portfolio site with AWS S3, Route53, and CloudFront?, by Nicholas Vincent-Hill, freeCodeCamp, April 10, 2019 (Please DO NOT put aws-keys.json in your project directory!!!)

• Managing Secrets In Docker Swarm Clusters, by Viktor Farcic, February 23, 2017

• Docker Compose v3.1 file format now supports Docker 1.13.1 Secret Management, by ajeetraina, February 15 2017

• Secrets at Scale: Automated Bootstrapping of Secrets and Identity in the Cloud, by Ian Haken, USENIX, January 30, 2017

• Ask HN: In a microservice architecture, how do you handle managing secrets?, HackerNews post, January 9, 2016

• Variable separation using Ansible

  • Variable File Separation, Ansible documentation

  • How to open source provisioning script yet keep secrets secret? Is a two repository approach recommended?, self.ansible post by sovietmudkipz

• Ansible Vault

  • Safely storing Ansible playbook secrets, On Web Security blog, June 23, 2015

  • Best Practices: Variables and Vaults, Ansible web site

  • Ansible: Using Vault, video tutorial by ServersForHackers, Feb 16, 2015

  • Managing Secrets with Ansible Vault – The Missing Guide (Part 1 of 2), by Dan Tehranian, July 24, 2015

  • Managing Secrets with Ansible Vault – The Missing Guide (Part 2 of 2), by Dan Tehranian, July 24, 2015

  • Ansible: How to encrypt some variables in an inventory file in a separate vault file?, Stackoverflow post by Adam Matan, May 13 2015

  • GitHub Gist tristanfisher/Ansible-Vault how-to.md (“A short tutorial on how to use Vault in your Ansible workflow. Ansible-vault allows you to more safely store sensitive information in a source code repository or on disk.”)

  • Encrypting Login Credentials in Ansible Vault, by Dan Tehranian, August 17, 2015

• Hashicorp Vault

  • Introduction to Vault, YouTube video by Seth Vargo, December 24, 2015

  • Vault: A tool for managing secrets, by Mitchell Hashimoto, April 28, 2015

  • 2 Key-Value Stores compared - Keywhiz (Square) & Vault (Hashicorp), YouTube video, June 12, 2015

  • GitHub hashicorp/vault (“A tool for managing secrets. http://vaultproject.io”)

  • Storing Secrets at Scale with HashiCorp’s Vault: Q&A with Armon Dadgar, by Daniel Bryant, September 9, 2015

  • GitHub jhaals/ansible-vault (“ansible lookup plugin for secrets stored in Vault by HashiCorp”)

  • Managing all your secrets with Vault - Review and Walkthrough, by Martin Rusev, January 29, 2016

  • Secrets management in the Autopilot Pattern, by Tim Gross, January 26, 2017

  • Chickens & Eggs: Managing secrets in AWS with Hashicorp Vault, Slideshare by Jeff Horwitz, December 2, 2016

• Consul

  • Crypt: Store and retrieve encrypted configuration parameters from etcd or consul

Terraform

• GitHub hashicorp/terraform (“Terraform is a tool for building, changing, and combining infrastructure safely and efficiently. https://www.terraform.io/”)

• A Comprehensive Guide to Terraform, blog post series by Yevgeniy Brikman, September 26, 2016

• How to Manage a Home Network with Infrastructure as Code, by Paul Tyng, February 6, 2020

• Intermediate variables (OR: add interpolation support to input variables) #4084, mrwilby opened this issue on November 26, 2015

• Seth Vargo on Hashicorp Terraform, YouTube video by Seth Vargo, February 17, 2015

• Terraform Deploying Consul Cluster in One Command, Vimeo video from HashiCorp

• How To Use Terraform with DigitalOcean, by Mitchell Anicas, September 25, 2017

• Getting Started with Terraform for Digitalocean, by Eric Wright, January 9, 2017

• How to manage Terraform state, Yevgeniy Brikman, October 3, 2016

• How to create reusable infrastructure with Terraform modules, by Yevgeniy Brikman, October 5, 2016

• Terraform Infrastructure Design Patterns, by Bart Spaans, September 14, 2015

• Setting up AWS EC2 Assume Role with Terraform, by Amit Saha, February 27, 2018

• Interpolation Syntax, Terraform documenation

• Using Vagrant, Docker, & Terraform to streamline your development & demo environments, YouTube video by Nikhil Vaze, October 18, 2015

• Orchestrating Docker with Terraform and Consul by Mitchell Hashimoto, YouTube video, January 7, 2015

• Guide to automating a multi-tiered application securely on AWS with Docker and Terraform, by Greg Osuri

• ansible stuffs - how to securely get SSH fingerprints/public keys from newly created AWS instances, by Bill Cawthra, September 8, 2016

• Fun with Terraform Template Rendering, by Aurynn Shaw, February 23, 2017

• Best way to get an interpolated value into a Terraform lookup, by MichaelF, December 9, 2017

Pulumi

• Pulumi

• Pulumi (One source file to rule them all), Wilson Mar blog

Nomad

• Nomad, by Armon Dadgar, September 28, 2015

• Introduction to Nomad, nomadproject.io web site

• GitHub hashicorp/nomad (“A Distributed, Highly Available, Datacenter-Aware Scheduler https://www.nomadproject.io/”)

Otto

• Otto: Development and Deployment Made Easy ottoproject.io web site

• First look at HashiCorp’s Otto, YouTube video by Mat Schaffer, September 29, 2015

• Mitchell Hashimoto - Introducing Nomad and Otto, YouTube video, October 27, 2015

• Otto: Getting Started, ottoproject.io web site

• GitHub hashicorp/otto (“Development and deployment made easy. https://ottoproject.io”)

Automating Deployment on Cloud Services

• Amazon Web Services (AWS)

  • Best Practices for Managing AWS Access Keys, Amazon web site

  • AWS Identity and Access Management: User Guide, Amazon web site

  • Amazon Security Groups – 5 Important Best Practices for Your To-Do List, by Eyal Posener, Stratoscale, November 2, 2016

  • Becoming a Command Line Expert with the AWS CLI, by James Saryerwinnie, November 14, 2013

  • 15 Essential Amazon AWS EC2 CLI Command Examples, by Ramesh Natarajan, April 20, 2016

  • Securing Local AWS Credentials, by Ryan McGeehan, November 23, 2016

  • IAM Best Practices, Amazon web site

  • How to Rotate Access Keys for IAM Users, by Ben Brauer, October 2, 2013

  • A Convenient AWS CLI Key Rotation Script for IAM Users, by Ville Walveranta, April 19, 2017

  • AWS CLI Key Rotation Script for IAM Users revisited, by Ville Walveranta, October 28, 2017

  • AWS CLI using Roles instead of User Credentials, by pumpdev

  • Terraform AWS Provider

  • Generate Temporary AWS Credentials, by Martijn van Dongen, January 9, 2018

  • Setting up AWS EC2 Assume Role with Terraform, by Amit Saha, February 27, 2018

  • Tags in AWS | AWS Resource Tagging and Management using AWS CLI, by Yogesh Mehta, January 13, 2018

  • Storing AWS CLI Credentials in 1Password, by Kenneth Falck, March 18, 2018

  • GitHub boto/boto3 (“Boto3 - The AWS SDK for Python”)

  • SAWS: A Supercharged AWS Command Line Interface (CLI)

  • AWS Documentation » Catalog » Code Samples for Python » Python Code Samples for Amazon EC2 » get_console_output.py

• Google Cloud Platform (GCP)

  • Cloud Identity and Access Management (IAM), Google Cloud docs

  • Install and configure Google Cloud SDK using Homebrew, by Pete Houston,February 23, 2020

Automated distributed system deployment options using Ansible

• ansible-dims-playbooks (ansible-dims-playbooks Documenations)

• Mantl (Mantl Documentation)_

• DC/OS

• Intel Trusted Analytics Platform

• Debops

• The Fedora Project ansible playbook/files/etc repository for fedora infrastructure

Using systemd and upstart for services

• Demystifying systemd: A Practical Guide, by Ben Breard and Lennart Poettering, Red Hat Summit, April 14-17, 2014 (PDF file)

• systemd for Administrators, eBook by psankar

• Systemd Essentials: Working with Services, Units, and the Journal, Justin Ellingwood, DigitalOcean, April 20, 2015

• How To Use Systemctl to Manage Systemd Services and Units, Justin Ellingwood, DigitalOcean, February 1, 2015

• Understanding Systemd Units and Unit Files, Justin Ellingwood, DigitalOcean, February 17, 2015

• How to debug Systemd problems, Fedora Project web site

• Auditing systemd: solving failed units with systemctl, by Michael Boelen, November 24, 2014

• systemd.service — Service unit configuration, Freedesktop.org man page

• Investigating systemd errors, ArchLinux web page

• SystemdForUpstartUsers, Ubuntu web site

• Gist damncabbage/unicorn.conf (“I have the most terrible Upstart config for Unicorn. Support start, stop, restart and reload (rolling restart).”)

• How can I tell upstart to restart a service when a different service is restarted?, by Marius Gedminas, AskUbuntu post, February 13, 2013

• Logging

  • Reading the System Log, CoreOS web site

  • How To Use Journalctl to View and Manipulate Systemd Logs, Justin Ellingwood, DigitalOcean, February 5, 2015

  • GitHub systemd/journal2gelf (“Ships new systemd journal entries to a remote destination in Graylog Extended Log Format (GELF)”)

  • Sending CoreOS Logs to Loggly, by Garland Kan, October 27, 2015

  • Platform logging, Deis

  • Responsive infrastructure - How to setup rsyslog, elasticsearch and kibana on CoreOS and AWS (2), by Simon Dittlman, April 3, 2015

  • A UDP syslog client in Python — for Windows and UNIX, by Christian Stigen Larsen, December 3, 2013

Nginx Reverse Proxy

• Securing HTTP Traffic to Upstream Servers, NGINX guides

• Pitfalls and Common Mistakes, NGINX guides

• Strong SSL Security on nginx, by Remy van Elst, April 27, 2019

• The most important steps to take to make an nginx server more secure, Dreamhost blog, March 27, 2019

• Nginx proxy_pass: examples for how does nginx proxy_pass map the request, by Srain, September 16, 2013

Small form-factor hardware systems

• Raspberry Pi

  • Raspberry Pi Wiki Hub

  • RPi SD cards

  • How to Flash an SD Card for Raspberry Pi, by Johnny Winters

    • GitHub hypriot/flash (“Command line script to flash SD card images for the Raspberry Pi”)

  • PiBakery (“The easiest way to setup a Raspberry Pi”)

  • Pi-hole (“Network-wide ad blocking via your own Linux hardware”)

  • Docker Pirates ARMed with explosive stuff (hypriot blog)

    • Getting started with Docker on your Raspberry Pi, hypriot blog, October 13, 2015

    • Let Docker Swarm all over your Raspberry Pi Cluster, by Stefan, July 3, 2015

    • Heavily ARMed after major upgrade: Raspberry Pi with Docker 1.5.0, March 3, 2015

    • How to use Docker Compose to run complex multi container apps on your Raspberry Pi, April 6, 2015

  • Using a Raspberry Pi as a PXE boot server

    • Raspberry Pi as a PXE, TFTP, NFS, proxy DHCP server, by Adam Niedzwiedzki, February 7, 2014

    • Network booting machines with a PXE server running in a Docker container, by Jérôme Petazzoni, December 7, 2013

    • Raspberry Pi as PXE Server, by Cody Bunch, August 19, 2014

    • How To: Setup a PXE Boot Server on Debian Part 2, May 9, 2011

  • Raspberry Pi 3 B+ Review and Performance Comparison, by Jeff Geerling, April 5, 2018

  • Raspbian Image with Docker 1.5.0 Released for Raspberry Pi Boards, by cnxsoft

  • Docker on Raspberry Pi in 4 Simple Steps, by resin.io

  • Visualize your Raspberry Pi containers with Portainer or UI for Docker, by Stefan, October 31, 2016

  • Cloudy with a chance of Raspberries, by Matt Williams

  • Swarming Raspberry Pi - Part 1, by Matt Williams

  • Let’s build a PicoCluster for Docker Swarm, Hypriot blog, Mar 23, 2016

    • 5 Node PicoCluster (Raspberry PI), PicoCluster web site

    • DIY 5 Node Cluster of Raspberry Pi 3s, by Nick Smith, April 2016

  • OpenVPN with 2 factor authentication on the Raspberry Pi, by Mark, Coder36 blog, 15 December 2014

• UP

  • UP Board

• Beaglebone Black

  • Using Docker on Beaglebone Black, by Mark Fink

  • Snappy Ubuntu for Devices - The Year of the Linux Countertop!

• CuBox-i

  • CuBox-i Mini Computer for XBMC/Kodi player, Android TV Box and Linux

  • ArchLinux is one of the available operating systems from Solid Run

• Clustering small form-factor devices

  • #Building ARM cluster Part 1: Collecting , wiring and powering devices, by Mateusz Kaczanowski

  • #Building ARM cluster Part 2: Create and write system image with goback!, by Mateusz Kaczanowski

  • #Building ARM cluster Part 3: Docker, fleet, etcd. Distribute containers!, by Mateusz Kaczanowski

Caching or Mirroring Packages

• Caching debian/etc (apt) repositories on your local server with nginx and dsniff, by Yeupou, January 28, 2014

• Caching Apt packages with Nginx - and without Squid, by Andrew Simpson, August 21, 2013

• Setting up an ‘Apt-Cache’ Server Using ‘Apt-Cacher-NG’ in Ubuntu 14.04 Server, by Babin Lonston, August 29, 2014

• Squid result codes, from squid-cache wiki

Unix System Administration/Monitoring Tools

• GitHub MichielDerhaeg/build-linux (“A short tutorial about building Linux based operating systems.”)

• Big Brother Monitoring System

• Nagios

• Simple Event Correlator (SEC)

• DeadCat.net collection of scripts, tools, and documentation related to Big Brother Monitoring Software

Miscellaneous Distributed System Construction

• GitHub uw-dims/ansible-dims-playbooks (“DIMS Ansible playbooks”)

• GitHub trustedanalytics (“Components for Trusted Analytics Platform”)

• DebOps

• The Open Home Lab Stack, by Mighty Womble, October 1, 2017

• Boot my (secure)->(gov) cloud, by Nicki Watt, August 10, 2015

Incident tracking/trouble ticketing systems

• Call Center, Bug Tracking and Project Management Tools for Linux

• Slashdot thread on Automated Support Email Systems

LiveCD distributions

• Knoppix Security Tools Distribution (STD)

• Ubuntu Linux

• The AVG Rescue CD is quite useful for helping disinfect computers for family and friends. (Hint: Pre-position a CD at their house so when they call you for help, you don’t have to talk them through downloading and burning an ISO image to CD-R!)

• Penguin Sleuthkit (a remaster of Knoppix)

• Helix Incident Response and Forensics LiveCD

• The Auditor security tools Live CD

• Fedora 8 on a USB stick, by Harold Hoyer

• Pen Drive Linux

• Juju ODS Demo – The Home Version and Ubuntu Cloud Live CD

• Creating bootable USB drives

  • Create a bootable CentOS USB drive with a Mac (OS X) for a PC, by James Purdy, May 23, 2014

  • Create a bootable USB stick on macOSX, Ubuntu Tutorial

  • UNetbootin

  • Etcher (Mac OS X - “Burn images to SD cards & USB drives, safely and easily.”)

Secure Alternative Operating Systems

• TAILS (The Amnesiac Incognito Live System)

• QubesOS + The Operating System That Can Protect You Even if You Get Hacked, by Micah Lee, April 10, 2014

Scripting in bash

• Basic grammar rules of Bash, BashHackersWiki

• Commandlinefu.com

  • Find Duplicate Files (based on size first, then MD5 hash)

  • Recursively remove all empty directories

  • GitHub google/styleguide (“Style guides for Google-originated open-source projects”)

    • Shell Style Guide

• Command line option parsing

  • GitHub kward/shflags (Automatically exported from https://code.google.com/p/shflags)

  • Easy Bash Scripting With Shflags, by Steve Francia, July 8, 2011

  • Using getopts in bash shell script to get long and short command line options, Stackoverflow post

• Advanced Bash scripting

  • How “Exit Traps” Can Make Your Bash Scripts Way More Robust And Reliable, by Aaron Maxwell

  • I set variables in a loop that’s in a pipeline. Why do they disappear after the loop terminates? Or, why can’t I pipe data to read?, BasFAQ/024

  • The Ultimate Bash Array Tutorial with 15 Examples, by Sasikala on June 3, 2010

  • BashGuide/Arrays

• Debugging Bash scripts

  • Debug your shell scripts with bashdb, by Ben Martin, November 24, 2008

  • Debugging a script, Bash Hackers Wiki

  • Why does my shell script choke on whitespace or other special characters?, StackExchange post by Gilles, May 24 2014

Email and Secure Email Delivery

• About SPF and DKIM

• SPF, DKIM and DMARC brief explanation and best practices, by Emanuele “Lele” Calò, April 15, 2014

• How To Install and Configure DKIM with Postfix on Debian Wheezy, DigitalOcean web site, February 28, 2014

• DNSSEC, SPF, DKIM and DMARC explained, by ScorpionSting, May 23, 2017

• Debian Mail Server, Part II: SPF and DKIM

• Protecting Parked Domains Best Common Practices (PDF), M³AAWG

• GitHub ksylvan/docker-mail-server (“Ansible playbooks to deploy a full featured mail server stack using Docker.”)

• Enhance security for forged spam (DMARC), G Suite Admin Help

• Debugging + Postfix Debugging Howto + postfix + opendkim: “Sender” and “Message-Id” fields not included in signature, ServerFault, March 15, 2013 + postfix + opendkim not signing correctly. how to debug this?, ServerFault, April 15, 2013 + Turn on debug mode in Postfix

Postfix

• Postfix Configuration Parameters, postfix web site

• Postfix Standard Configuration Examples, postfix web site

• Basic settings in the Postfix main.cf file, Rackspace web site, December 29, 2015

• Setting up a mail server using Postfix in 5 minutes, by Rudd-O

• Question about DNSmasq and MX records, FreeBSD forums post by danaeckel, March 12, 2013

SPF

• SPF Record Testing Tools

• Setting an SPF record for all subdomains of my domain, Stackexchange post by Belmin Fernandez, December 26, 2011

DKIM

• opendkim + opendkim.conf

• Common Errors Causing DKIM Verification Failures, by Jim Fenton, October 26, 2009

• Extensions to DomainKeys Identified Mail (DKIM) for Failure Reporting, RFC6651

• One DKIM Signature for multiple Mail-Domains, by Anton Dollmaier

DMARC

• DMARC web page

• DMARC Wikipedia

Unix Sytem Administration Tasks/Tips/Tricks

• Regular expressions

  • Regular expressions 101 (https://regex101.com)

  • GitHub gskinner/regexr (https://regexr.com)

• Simply exchange files with WOOF

• FAI - Fully Automatic Installation of Debian, Ubuntu, CentOS, RHEL, SUSE, …

• Stacki

• Ubuntu (Debian) Linux

  • AptGet/Howto, Ubuntu web site

  • How do I create a completely unattended install of Ubuntu?, StackExchange, April 16, 2012

  • Building a Fully Automated Ubuntu Installation Process, by Scott Lowe, May 20, 2015

    • System Automation – Part 1 – PXE and Preseed, by Brian Carpio, April 4, 2012

  • 25 Useful Basic Commands of APT-GET and APT-CACHE for Package Management, by Ravi Saive, January 3, 2015

Unix Administration Courses/Tools

• A curated list of amazingly awesome open source sysadmin resources inspired by Awesome PHP, by Francisco Augusto (kahun)

• Server World [online server administration help]

• Unix Toolbox, by Colin Barschel

• IT Tutorials and Interview Questions

• CIS 399: Introduction to System Administration course at the University of Oregon

• Troubleshooters.com (Linux troubleshooting web site)

• Network and System Administration Resources, by Mark Burgess, University College Oslo

• System administration (SO 136 A)

• CIS 410/510, Introduction to System Administration, by Steve VanDevender, University of Oregon

• The Perl Oasis archive

• A Perl Tutorial: Super-Basics

• Raymond (a platform-independent package manager for use in a heterogeneous environment)

• SunWorld On-Line emagazine

• Useless Use of ‘cat’ Award

Tar

• Comparison of several ‘tar’ implementations vis-a-vis compatibility

• All tar files are not created equal: UNIX IN THE ENTERPRISE, July 24, 2003

Network and Distributed File Systems

• GlusterFS

• Create your own high-performance NAS using GlusterFS, by Michael Reed

SElinux, Apparmor, GRsecurity

• Linux Kernel Security (SELinux vs AppArmor vs Grsecurity), by Vivek Gite, May 27, 2009

• The National Security Agency (NSA) Secure Enhanced Linux project

• Stop Disabling SELinux!, by Jeff Sheltren

Linux Kernel

• grsecurity (Linux kernel hardening patches)

• Linux Headquarters

• Kernel-HOWTO

• Journal File Systems, by Juan I. Santos Florido

• LinuxPlanet - Tutorials - How to Compile the Linux Kernel

• Linux kernel capabilities FAQ

• Linux capabilities 101, Linux Audit blog

• Upgrading the Linux Kernel on Red Hat Linux systems (RPM style upgrade, not from source code)

• IEEE 1394 (FireWire) for Linux

• Large File Support in Linux

• Wacky uses for RAID, /dev/ram, and ramfs, by Mark Nielsen

Dell Edge Server Stuff

• Problem with PERC H710 Raid Controller

• Building the LSI 9285-8e Driver for Openfiler 2.99, windowsmasher blog

• Dell Driver Details: Linux SLES10 SP4 Driver 00.00.05.39-1 for PERC H310 / H710 / H710P / H810 Controllers.