Quickly warm up EBS volumes created from snapshot

When you restore EBS volumes from snapshot, they initially run slow, because Amazon handles copying the blocks from the S3-backed snapshots in the backend. You'll want to warm up your EBS volumes after restoring them from snapshot.

Amazon recommends using either the dd utilty, or the fio utility, to warm up your volumes. See http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-initialize.html.

The dd utility is single-threaded and RAID volumes consist of multiple disks, so introducing a multi-threaded application, and then running the same multi-threaded application on each disk in an attached RAID volume, will speed up the EBS volume warm-up

Run as root (yes, bad habits):

apt-get update apt-get -y install fio

for i in /dev/xvd{f..m} ; do echo
nohup fio --filename=$i --rw=randread --bs=128k --iodepth=32 --ioengine=libaio --direct=1 --name=volume-initialize-`basename $i` \& ; done | 

The result? On an r4 class EC2 instance, I'm reading 100MBps from disk, where if I just ran dd against the RAID array (/dev/md1), I'd be lucky to get a sustained read speed of 4MBps.

Jenkins: Combine Github OAuth and Role Based Strategy Plugins

I don't know much about Jenkins, the CI platform. I just recently set my first server up from scratch, and since then my interactions have mostly been "create jobs" and "search Google" when I want Jenkins to do something it doesn't.

Seems that 100% of the time, my Google search results in "install this plugin." And when the plugin works, I record it in the README.md file I keep, to document the construction of my Jenkins server. The important word, here, is "my". It's MY Jenkins server. While I've allowed a few other people to access it, I have been relying this whole time on the internal Jenkins user database.

Recently, our lead dev (my customer) said, "create these jobs on your Jenkins server and let my people run them." Since we use Github, the easiest way to do this is to create an API key in our Github organization and configure my Jenkins server to authenticate users using OAuth. The Github Authentication Plugin" handles OAuth nicely.

Well, it would be that easy, except that giving everyone on the dev team access to my Jenkins server enables them to destroy cloud resources. We can't have that.

Limiting users' access to specific jobs on the server is a matter of installing the "Role-Based Authorization Strategy," and setting up project roles.

Basically, we're talking about installing two Jenkins plugins and configuring them to work together. Here's how I did it:

Setup

Before you start installing plugins, you will need to create a Github Organization. You can do this by visiting https://github.com/organizations/new. Once there, setting up an organization is straightforward. You give your organization a name, set up an e-mail address, and choose the correct type (free, or paid).

Your organization is now available at https://github.com/your-org-name. In this example, my organization name is "gswallow-oauth-test," so my personal website is https://github.com/gswallow-oauth-test/. On your organization's home page, you will see a "settings" tab. Click that, and then you will see a link called "OAuth Applications."

Once you click on the "OAuth Applications" link, you can click on a button that says "Register a new application." Click that. Next, you will be prompted for several pieces of information:

Application Name: my-jenkins-server Homepage URL: https://my-jenkins-server Application description: My Jenkins Server Authorization Callback URL: https://my-jenkins-server/securityRealm/finishLogin

The first three settings can be custom tailored however you want, but the Callback URL must be accessible by your users. See the plugin docs, for more details.

Once you submit your changes, you will be given an ID and a secret. These are important when setting up the Github Authentication Plugin in the next step.

There are two ways to install Jenkins plugins. The first, is to use the "Manage Plugins" page in Jenkins' Web UI. The second is through the command line. I'll cover that here to be helpful. Let's assume you're actually logged into your Jenkins server. The steps necessary to install plugins are to get the Jenkins CLI jar, and execute it:

wget http://localhost:8080/jnlpJars/jenkins-cli.jar
java -jar jenkins-cli.jar -s http://localhost:8080/ install-plugin github-oauth
java -jar jenkins-cli.jar -s http://localhost:8080/ install-plugin role-strategy
sudo service jenkins restart

Authentication

Note that I installed both the github-oauth and role-strategy plugins at the same time.

Once Jenkins restarts, you should be able to use its web interface and configure your system's security. Click on "Manage Jenkins," then choose the "Configure Global Security" link:

jenkins-github-auth.png

Here, I switched from using Jenkins's own user database to using the Github Authentication Plugin. Note that you will need to supply your OAuth client's ID and key, here.

Now, adding users' access to your Jenkins server is a matter of inviting them to your Github Organization. To invite a user, simply go to your org's site (https://github.com/your-org-name and click on the "Invite Someone" button. Your invitee will get an e-mail, where he or she may join your organization.

Next, it's time to set up access controls over who can do what.

Authorization

Your Github users are now able to log into your Jenkins server, but do you really want to give them carte blanche access to the server, itself? Maybe; maybe not. In my case, not.

Again, we'll click on the "Manage Jenkins" link from our Jenkins server's home page, and choose the "Configure Global Security" link. This time, we'll change the authorization strategy to the role-based strategy:

jenkins-auth-strategy.png

At this point, you must create roles. Again, click on the "Manage Jenkins" link, but this time, click on the "Manage and Assign roles" link. The first thing we'll do is manage roles. There are a few types of roles: global, and project roles, which we're interested in, and slave roles, which we're not.

jenkins-roles.png

Create a new global role, called "Users," or something to imply that the role is not for Administrators. There are a number of permissions you can assign to this role, but I found that I only need to assign a few permissions globally. These are:

  • Overall/Read
  • Job/Read
  • View/Read

Project roles are where I allow users to actually run jobs. A project role has a name, and it takes a pattern (regex) to specify which jobs it allows access to. For instance, all of my product management people will be able to run jobs starting with "pm-". In the "Product Management Jobs" role, I allowed access to the following actions:

  • Job/Build
  • Job/Read
  • Workspace/Read

Of course, your security requirements may vary.

Next, we must assign the roles to Github users. Click on "Manage Jenkins," then "Manage and Assign Roles," and choose the "Assign Roles" linkj:

jenkins-assign.png

Note that you can assign roles to Github teams. If you have invited two or more people to your Github organization, you'll most certainly want to add them to teams. The important thing here is the syntax you use for a Github team. It's "organization""team". e.g. our engineering team is indigobioEngineering. I believe these team names are case sensitive.

To give your team members access to the server itself, assign the team to your restricted global role. To give them access to actually run certain jobs, assign the team to the appropriate project role.

Grilled cheese and tomato soup

Be forewarned. I cheat a lot in this recipe. Who has time to cook tomato soup from scratch? But it goes from zero to done in 10 minutes.

Ingredients:

  • prepared cook surfaces
  • Texas toast
  • butter or margarine
  • non-stick spray
  • sliced fresh tomato
  • Kraft singles
  • white american cheese
  • croutons
  • Parmesan cheese

Most important: cookware. You need two preheated pans. First, preheat a saucepan and coat it with veggie oil spray. Second, preheat a frying pan and coat it with non-stick spray, too.

Butter one side of your Texas toast. Add a slice of tomato to the sauce pan and the buttered side of your Texas toast to the frying pan. Toast the buttered bread sides, and flip the tomato so that it heats up. Remove everything to a plate.

Butter the other side of each slice of toast and make sure the uncooked side is the outside of your sandwich (top and bottom). Between the toasted sides (insides), add a slice of Kraft cheese and a slice of white american cheese, then add the heated tomato, then repeat the cheeses on top in opposite order.

When your grilled cheese sandwiches are ready to go (cold butter on the outside), add your tomato soup to the sauce pan. Heat it and start cooking your sandwiches liked a regular grilled cheese. You may need to cover the frying pan with a lid just to melt the cheese at the end of cooking. Shouldn't take more than 20 seconds, covered..

When the grilled cheese is done and the soup is hot, serve, slicing the grilled cheese in half and topping the tomato soup with croutons and Parmesan cheese to thicken.

GitHub – gswallow

Greg Swallow

Indianapolis

I push bits around for a small company in Indianapolis, IN, Good ol'e US of ...