till's blog Vagrant: ShellProvisioner vs. Chef (20.6.2012, 09:51)

In my last blog entry, I demo'd how to get started with Vagrant and the ShellProvisioner.

To further illustrate how amazingly simple it is to get started on some Ruby, I'll convert the shell script from my last blog post to a little recipe for chef. Same objective, we install a PEAR package — but it could be anything really.

Follow me.

Shell

This is the shell script from before:

#!/bin/sh

apt-get update

apt-get install -y php5 php5-cli php-pear
hash -r

pear upgrade-all
pear install -f HTTP_Request2

Ruby

Create a cookbooks directory and create structure for your first cookbook in it:

$ mkdir -p my-cookbooks/first/recipes/

Create a default.rb file with the following content:

# my-cookbooks/first/recipes/default.rb
execute "apt-get update"

packages = ["php5", "php5-cli", "php-pear"]

packages.each|p| do
  package p
end

execute "pear upgrade-all"
execute "pear install -f HTTP_Request2"

The recipe is later referred to as first or first::default (name of the recipe directory, name of the .rb).

It's so simple it hurts. ;)

Step by step

  1. I run apt-get update using Chef's execute resource.
  2. I create an array of the packages (Arrays are ordered in Ruby, hashes are not (up until Ruby 1.9.x). Order is important here.)
  3. I run pear upgrade-all using the execute resource.
  4. I run pear install using the execute resource.

Vagrantfile

The Vagrantfile looks slightly different when you provision with chef-solo:

Vagrant::Config.run do |config|

  config.vm.define :web do |web_config|
    web_config.vm.box       = "lucid64"
    web_config.vm.host_name = "web"

    web_config.vm.provision :chef_solo do |chef|
      chef.cookbooks_path = "PATH-TO-YOUR-COOKBOOKS"
      chef.add_recipe "first"
      chef.log_level = :debug
    end
  end

end

The important bit: the path to the location of your cookbooks — could be an /absolute/path or ./../relative/path.

Because a Vagrantfile is essentially Ruby code, anything goes here.

Further reading

Getting started with Chef and Ruby can be intimidating or even frustrating at times. Google "chef cookbooks" and you know what I mean.

These links are what you need:

Fin

Save, and enjoy — vagrant up.

Link
till's blog Vagrant sans Ruby (5.6.2012, 12:44)

Development, testing, staging and production — this is how most people devide up different environments for application development.

Maintenance and setup of these environments is often not a trivial goal to achieve. Having worked with a couple different code bases and setups over the last decade, I often noticed things like environment specific hacks (if ($env == 'testing') { ... }) in application code and service configurations and a lot of manual labour all around. It's still very common that code deployed to staging does not work — but it worked for you right?

And it's also too common that members of a team do not know how something works because whoever set it up is currently out sick or on vacation.

My opinion is that the best setup currently available is something like: chef(-solo) on the server and Vagrant on the desktop. Something like because aside from chef, there is also puppet, cfengine and a couple others. Leaving specific projects aside, it really just boils down to automation (within reason).

Automation in my opinion is not just the easiest but the only viable way to setup development, testing, staging and production environments. Without some automation in place it's required that all team members know how all of it works when maybe that is not yet important and the end goal is that all environments actually resemble each other.

Enter Vagrant

Vagrant is a very nifty toolkit to bootstrap Virtualbox images. Bootstrapping means installing your application stack into one or multiple virtual machines in order to resemble production a lot better.

So a lot of times when I rave about how useful Vagrant and chef are, I get the crazy eye from PHP developers:

You want me to learn Ruby to setup my local development environment?

My response:

  1. Do not fear the Ruby.
  2. You don't have to. (Well, not a whole lot!)

Real talk

First off, of course you need Virtualbox — get a download from their website or use your package manager.

Then, it's not possible to avoid Ruby a 100% — after all Vagrant is written in Ruby.

When you're on a Linux or a Mac, this is usually enough (you may need sudo unless you RVM):

$ sudo gem install vagrant

For a sudo-less install, use RVM:

$ sudo gem install rvm
...
$ rvm install
...
$ rvm use 1.8.7
...

When set up, this is what you do:

$ gem install vagrant
...

RVM allows me to run multiple versions of ruby side by side and also leverage local gem installs — per Ruby version. Think of it as a nifty way to run multiple PHP versions and have a PEAR install per version — similar projects exist for PHP today:

To learn more about RVM, visit their website.

Once vagrant is installed, we can continue!

ShellProvisioner for fun and profit

So, in case you are more comfortable writing some shell script for the time being — Vagrant got you covered!

First off, create a new Vagrantfile for your project:

$ mkdir -p Documents/workspaces/blog-example/
$ cd Documents/workspaces/blog-example
$ vagrant init lucid64
A `Vagrantfile` has been placed in this directory. You are now
ready to `vagrant up` your first virtual environment! Please read
the comments in the Vagrantfile as well as documentation on
`vagrantup.com` for more information on using Vagrant.

lucid64 (Ubuntu Lucid 10.04, 64bit) is the name of one of my local box files for Vagrant. In case you haven't got a box yet, head over to vagrantbox.es. There are a couple images you (aka box files) you can download. To get the lucid64 box, use the following command:

$ vagrant box add lucid64 http://files.vagrantup.com/lucid64.box

Once you made it past vagrant init, you should have a Vagrantfile with a lot of stuff in there.

While it's not important for my tutorial, I recommend you review it some (other time). The created Vagrantfile contains examples for all the provisioners (puppet, chef and shell) and a couple other configuration options, etc..

Let's skip over this and get to the ShellProvisioner.

This is all you need

First off, let's create a shell script

Truncated by Planet-PEAR, read more at the original (another 2033 bytes)

Link
Christians Tagebuch phorkie 0.2.0: Elasticsearch (25.5.2012, 20:41)

I've just released phorkie 0.2.0 which brings search functionality via Elasticsearch.

The nice thing about Elasticsearch is that it allows boosting of certain fields (paste title is higher ranked than the file content) and has a bunch of search query features:

  • Exclusion: +foo -bar
  • Logical OR: foo OR bar
  • Quoting: "foo bar"
  • Partial words: foo*
  • Restriction to certain fields: content:Hello

All of that was just there without me having to do any programming, whereas with MySQL this would have taken weeks to implement (query parsing and sql-mapping).

phorkie will work fine without an Elasticsearch service, but you won't see the search field in that case.

PEAR updates

While hacking on phorkie, I took the time to do fixes for libraries that I use here:

  • Services_Libravatar (federated, open source gravatar) nearly got a complete rewrite and is unit-tested now. Melissa merged my changes and released versions 0.2.0 and 0.2.1.
  • Date_HumanDiff (lib to display relative dates in a readable form like "2 hours ago") supports translations (en, de and el already there!) and custom times ("2 dozen seconds ago"). It's available in version 0.2.0.

Link
Christians Tagebuch Introducing phorkie: git based pastebin (18.4.2012, 20:22)

At work, we've been looking for a pastebin application with a special feature set:

  • Self-hosted, so we can paste code and data without having to care about sensitive customer data becoming public accidentially
  • written in PHP, since we're a PHP shop
  • editability so other people can do fixes for us
  • title support for pastes so we can find it later
  • navigation among pastes so we don't have to remember the number later
  • search because navigation doesn't help with 1k+ pastes

We did not find anything suitable after researching for some days. Since I needed a new spare time project, I began working on phorkie.

After working on it for a month, I've just released the first version 0.1.0 and you're welcome to try it.

phorkie

phorkie is a self-hosted pastebin software written in PHP. Pastes are editable, may have multiple files and are stored in git repositories.

It turned out to be a clone of github's gist tool, but with less features - for now.

Features

  • Every paste is a git repository
    • Repositories can be cloned
    • Clone url can be displayed
  • Multiple files in one paste
  • Pastes can be edited
    • Add and delete files
    • Replace file with upload
    • History in sidebar
    • Easy access to old revisions
  • Syntax highlighting with GeSHi
  • rST rendering with rst2html (python docutils).
  • image upload and display
  • external tool support (xmllint, php syntax check)

Missing tools

For the history/log display, I wanted to display the timestamps in a human readable manner, e.g. "5 minutes ago" instead of 2012-04-18 22:36.

Looking for existing libraries, I found plenty for Python and Javascript, but not a single one for PHP. After asking on stackoverflow without getting a usable answer, I wrote my own and proposed it to PEAR: Date_HumanDiff.

Screenshots

new paste paste list history tools

Also see the wiki.

Code

The code is in a git repository on sourceforge but also mirrored on github for your convenience.

Reviews

Link
Codelog Quickest way to split up multiple pear packages into different git repositories (11.4.2012, 20:09)

I was doing it the hard way before, but now:

$ git clone git@github.com:pear/Validate.git

# Go hit https://github.com/organizations/pear/repositories/new to make a new repo

$ git clone git@github.com:pear/Validate_CA.git

$ cd Validate_CA

$ git pull git@github.com:pear/Validate.git

$ git rm -r *

$ pear list ../Validate/package_CA.xml | grep ".* /" | cut -f 1 -d " " | xargs git reset HEAD

$ pear list ../Validate/package_CA.xml | grep ".* /" | cut -f 1 -d " " | xargs git checkout --

$ git commit -m "Splitting off to own package"

$ git push -u origin master

$ cd ..

$ cd Validate

$ pear list package_CA.xml | grep ".* /" | cut -f 1 -d " " | xargs git rm

$ git rm package_CA.xml

$ git reset HEAD LICENSE

$ git checkout -- LICENSE

$ git commit -m "Splitting off to own package"

$ git push

Link
Chuck Burgess DocBlox Is Unmasked ... It Is Really phpDocumentor 2 ! (16.3.2012, 13:10)

Link
till's blog Deploying PHP applications: PEAR and composer resources for chef (24.2.2012, 16:51)

This is something experimental I have been working on for our chef deployments. So the objective was/is to find a sane way to install PEAR packages and install dependencies with composer.

execute in chef recipes

In chef recipes, almost everything is a resource. In case you're just getting started with Chef, a list of current resources is available on the Opscode Wiki. It's a link I put in my browser bar since I frequently work on chef recipes.

Some examples for resources are:

  • package (to install software)
  • cron (setup a crontab)
  • directory (create directories)
  • template (install customized configuration files)
  • user and group (to create users and groups)
  • mdadm (to setup a RAID)

The above list are examples — so there is more. But if there isn't a designated resource, you can always use an execute block.

An example for an execute block is the following:

execute "discover a pear channel" do
  command "pear channel-discover easybib.github.com/pear"
end

This works pretty well, but it is also not very robust.

Fail hard

By default whenever a command fails, chef fails hard.

To illustrate what I'm talking about, let's test and execute the command from our execute block multiple times on the shell to see its exit status ($?):

till:~/ $ pear channel-discover easybib.github.com/pear
Adding Channel "easybib.github.com/pear" succeeded
Discovery of channel "easybib.github.com/pear" succeeded
till:~/ $ echo $?
0
till:~/ $ pear channel-discover easybib.github.com/pear
Channel "easybib.github.com/pear" is already initialized
till:~/ $ echo $?
1

So whenever a command returns not 0, chef will bail.

One solution is to brute-force your way through these things with ignore_failure true in your execute block. But that's usually not a great idea either because it hides other issues from you (and me) when we need to debug this later on.

For example, if this PEAR channel is unavailable during your next chef-run, it would be much, much harder to find the root cause as of why the install commands failed.

Another solution is using the not_if or only_if options with execute:

execute "discover a pear channel" do
  command "pear channel-discover easybib.github.com/pear"
  not_if do
    `pear channel-info easybib.github.com/pear`
  end
end

If the command wrapped in not_if succeeds (success is exit status), we would skip the execute block.

Optimize?

Since I discovered not_if and only_if, it allows me write recipes which work in most cases. More robust code, which allows me to re-execute recipes on already running instances. So for example when I update a recipe or configuration file which is distributed through a recipe I can re-run the entire recipe and it will not fail but instead complete successfully.

One problem remains with this approach I end up doing the same checks again and again.

... more after the jump.

Link
Christians Tagebuch WebFinger library for PHP released (24.2.2012, 09:12)

I'm implementing OpenID for SemanticScuttle, your self-hosted social bookmark manager. To log in with OpenID, you need to know your OpenID URL, which many people do not know, and don't want to know. Most know their email address, and thanks to WebFinger, this is all you have to know!

WebFinger enables applications to discover information about people by just their e-mail address - for example their OpenID URL!

I didn't find a single standalone WebFinger library for PHP, so I asked on StackOverflow, but did not get any responses. Failed to stand on the shoulders of giants, I went the hard way and implemented it all myself: Net_WebFinger, based on XML_XRD.

Implementation

WebFinger weaves RFC 6415: Web Host Metadata with LRDD which both use XRD files.

Thus the first step was to build a clean XRD library for PHP, with an intuitive API and 100% unit test coverage. I proposed the XML_XRD package on 2012-02-01, called for votes 8 days later. It was accepted with 11 votes. Extensive documentation does also exist now.

After the foundation was laid, I proposed the Net_WebFinger package. It was accepted as new PEAR this night, and just some minutes ago it got its first official release and a lot of documenation.

Usage

So, discovery is easy now! First, install the PEAR package:

$ pear install net_webfinger-alpha

Now the PHP code:

finger('user@example.org');
if ($react->openid !== null) {
    echo 'OpenID provider found: ' . $react->openid . "\n";
}

//list all other links:
foreach ($react as $link) {
    echo 'Link: ' . $link->rel . ' to ' . $link->href . "\n";
}
?>

WebFinger CLI

Net_WebFinger ships with a command line client that you can use to try it out. Find it with

$ pear list-files net_webfinger|grep cli
doc  /usr/share/php/docs/Net_WebFinger/examples/webfinger-cli.php

Yahoo and Google already support WebFinger. Distributed social networks like status.net (that powers identi.ca) and Diaspora use WebFinger to distribute public encryption keys, OStatus and Salmon URLs. You can try one of those user addresses, too.

$ php /usr/share/php/docs/Net_WebFinger/examples/webfinger-cli.php klimpong@gmail.com
Discovering klimpong@gmail.com
Information secure? false
OpenID provider: http://www.google.com/profiles/klimpong
Link: http://portablecontacts.net/spec/1.0: http://www-opensocial.googleusercontent.com/api/people/
Link: http://portablecontacts.net/spec/1.0#me: http://www-opensocial.googleusercontent.com/api/people/102024993121974049099/
Link: http://webfinger.net/rel/profile-page: http://www.google.com/profiles/klimpong
Link: http://microformats.org/profile/hcard: http://www.google.com/profiles/klimpong
Link: http://gmpg.org/xfn/11: http://www.google.com/profiles/klimpong
Link: http://specs.openid.net/auth/2.0/provider: http://www.google.com/profiles/klimpong
Link: describedby: http://www.google.com/profiles/klimpong
Link: describedby: http://www.google.com/s2/webfinger/?q=acct%3Aklimpong%40gmail.com&fmt=foaf
Link: http://schemas.google.com/g/2010#updates-from: https://www.googleapis.com/buzz/v1/activities/102024993121974049099/@public
$ php /usr/share/php/docs/Net_WebFinger/examples/webfinger-cli.php singpolyma@identi.ca
Discovering singpolyma@

Truncated by Planet-PEAR, read more at the original (another 1043 bytes)

Link
PEAR What would you do with 5 million lines of code? (24.1.2012, 14:05)

Since October 2011, 5 million lines of the PEAR codebase has shifted to github.

Hand in hand with this shift has been the tireless work of Daniel C – someone who brazenly said “I will fix the failing packages!” in the tail end of last year.

Coupling his efforts with a call to arms, we’ve now seen an evaluation of the Known Good packages against PHP 5.4, and massive input by the community. The net result is as follows:

  • Releases of Text_LanguageDetect, HTTP2, Net_Growl, Image_QRCode, Tree, HTML_BBCodeParser, Net_IMAP, Net_DNSBL, Services_Amazon, Image_Barcode2, Validate, Console_Color2, Services_ExchangeRates, Validate_DK, PEAR_PackageFileManager_Frontend, Text_Highlighter, PHP_Shell, Date, Image_Text, PEAR_Frontend_Gtk2, PHP_DocBlockGenerator, & Validate_AR through Dec/January
  • All test infrastructure upgrading to PHP 5.4 release candidates
  • All database driven test suites executing properly, catching a variety of simple bugs
  • Just shy of 900 commit emails to the pear-cvs list for Dec/Jan – many containing multiple commits & fixes
  • Hitting a point of “near zero” patches to be applied to unmaintained packages
  • Applying no less than 30+ patches contributed by the community across all of PEAR
  • Increasingly, the PEAR QA team is delivering PHP 5.3+ friendly forks of existing packages

I’d like to thank Daniel C for his efforts to date, as well as the contributors who may have previously lurked or found themselves distracted by other concerns.
Dec/Jan has been a great and vigorous period for the project – I heartily look forward to a great 2012.

Link
Christians Tagebuch Suhosin: Include filename .phar is not allowed (21.12.2011, 18:02)

When trying to run the PEAR installer go-pear.phar on one of our servers, it exited without any notice and without seeming to do anything.

After some searching I found that the syslog contained the error message:

Dec 15 12:59:47 scms suhosin[13658]:
  ALERT - Include filename ('phar://go-pear.phar/index.php')
  is an URL that is not allowed
  (attacker 'REMOTE_ADDR not set', file '/root/go-pear.phar', line 1236)

Dec 15 12:59:47 scms suhosin[13658]:
  ALERT - script tried to disable memory_limit by setting it to a negative value
   -1 bytes which is not allowed
  (attacker 'REMOTE_ADDR not set', file 'unknown')

So I knew it was Suhosin, the "advanced protection module for PHP5". It's enabled by default on Debian and Ubuntu.

Instead of simply disabling it, I allowed the phar stream wrapper with Suhosin:

$ emacs /etc/php5/cli/conf.d/suhosin.ini
.. add the following line:
suhosin.executor.include.whitelist = phar

Link
Links RSS 0.92   RDF 1.
Atom Feed  
PHP5 powered   PEAR
Link the Planet <a href="http://www.planet-pear.org/">Planet PEAR</a>