Archive for the 'recipe' category

Return to blog homepage

Older Entries

I've been working with the Basecamp API to plugin our IRC bot that we use for time tracking and I'm astounded to learn that escaping single and/or double quotes for XPath queries in PHP does not have a well documented, best practices solution.

In fact, it seems as though this is not peculiar to PHP. I took a look around and found this excellent article by "Kushal" (s/he doesn't have his full name on his blog):

http://kushalm.com/the-perils-of-xpath-expressions-specifically-escaping-quotes

I've produced a PHP solution for the general escaping/concatenation problem:

https://gist.github.com/1155973

Anyone got a better/more elegant solution? I'm always a big fan of code golf :)

Filed under: recipe | View Comments

I recently deployed a job on which the timeline was so tight that my ability to type quickly was what made the difference between delivering on time or not.

Everything was rushed, the budget was tight, it was one of those real seat of the pants deals and there was far too little testing done.

Just before I cut the site live (ie. minutes before I had to put this into production and have hundreds of people using it) I thought "You know, I bet something will go wrong somewhere here and I'll need complete logs of all the SERVER, POST and GET requests to piece it back together".

Boy am I glad I had that thought! Because it did. Nothing catastrophic, mind you, but there was a tricky integration with an external system that, as it turned out, periodically died without any errors or exceptions and just started returning bogus values.

The only problem is that, in the 5 minutes before the site was supposed to go live, I didn't really have much time to thoughtfully prepare a logging system to record all this stuff and, in my haste, I settled for:

catchall_logging.txt

When I came to actually process this information into something usable I realised what a dumb format for a log file this was and began kicking myself.

I had a look around for a snippet to parse the output of PHPs print_r but couldn't find anything so I thought I'd post my solution here just in case anyone else ever makes the same mistake - might save them a couple of hours (I've added comments in the code below):

logfile_antipattern.txt

Filed under: recipe | View Comments

Overview

Disclaimer and warning: I suggest you read this entire blog post first before running anything and do some testing locally if you can, or at least read the contents of each script, before running them. I accept absolutely no responsibility for you locking yourself out of your server using these scripts. They worked for me on FreeBSD 6.2 and FreeBSD 6.3. At the very least you should do these on a remote machine that is not currently in production.

I had a whole bunch of firewall rules running on my gateway for various servers. It became cumbersome administrating them all centrally, so I decided that it would be much better to have all my servers run their own firewall rules.

My servers are in a different city, so I had to do this all remotely. Unfortunately playing around with ipfw remotely is pretty dicey because you generally have the ability to completely lock yourself out of your server.

The Code

So, without further ado, here is my set of scripts for safely doing ipfw stuff remotely:

http://www.workingsoftware.com.au/downloads/ipfw_safe.tar.gz

That tarball contains the following files:

  • add_rules_safely: the script you call to add rules safely
  • add_rules_then_flush: used by add_rules_safely
  • ipfw_flush: called by add_rules_then_flush if it's not killed
  • kldload_ipfw_safely: the script you call to initialise ipfw on a server
  • kldload_ipfw_then_allow: called by kldload_ipfw_safely
  • rules_to_add: a list of firewalls you want to add

Usage

When you're running this on a server which doesn't have ipfw compiled in the kernel, you can either compile it into the kernel, or you can load it into the kernel dynamically by doing (as root):

./kldload_ipfw_safely

Once you're happy with your rules, you can load ipfw automatically in /boot/loader.conf (or statically compile it into the kernel), but until then it's best to leave it out so that if you really mess up you can always just power cycle the machine and ipfw won't be there anymore.

Once you've got ipfw running, you should be able to do (as root):

ipfw list

which will display:

00010 allow ip from any to any
65535 deny ip from any to any

Now, having done that, you can test out the script to add rules safely. By default when you first download the tarball above, and kldload ipfw safely using the command above, you'll have a rule 00010 which allows any to any. The rules_to_add file as you downloaded it will delete this rule, locking you out of your server. The "sleep" value in add_rules_then_flush as you downloaded it is 10, so if you were to do:

./add_rules_safely

with the files downloaded in that tarball "out of the box" so to speak, you would be locked out of your server for 10 seconds, then the rules will be flushed and you will be able to get back in.

Now, the general procedure for adding more rules will be to first of all edit the sleep interval in add_rules_then_flush to something that will allow you to kill that script if your rules did not block you out, then put some more meaningful rules in your rules_to_add file.

What this means is that you'll be able to add your rules safely. If all goes well, and you still have access to your box, then you'll be able to kill the script before it flushes your rules.

If you experience ipFAILw, and you get locked out of your server, then after the sleep interval is over, your rules will get flushed and you can log back in.

Comments Archive

This post originally appeared on an older version of the Working Software website which implemented it's own comment mechanism. This new version of our website/blog uses Disqus for comments (see below) however I wanted to preserve the comments made on the previous post here:

ORLY?

jsoftw (2008-08-27)

Filed under: recipe | View Comments

Overview

I wanted to do an anonymous read only CVS repository over SSH for my RocketSled framework. I already had a FreeBSD jail setup for my CVS repository, but I didn't want to just hand out an SSH login on it, and I also didn't want to have two repositories (one readonly and one read/write).

So the goal is to have an easy/safe way for one public CVS user to be able to execute only CVS commands and have read only access to one or more projects in my repository, whilst still allowing me to hand out logins to trusted developers for read/write access to that and other projects in the same repository.

Creating a Public CVS User

For starters, I created a script called cvs-shell:

cvs_shell.txt

and put the following content in it:

cvs_shell_contents.txt

The umask above means that, later when our development team and the public user share a lock directory with the "set gid" bit set on it, they will not prevent each other from checking out files. I then added a passwordless user bentley in the group bentley and made that script bentley's shell. Now when that user logs in, the only command they have access to is the cvs server command. You can try logging in now

ssh bentley@rs.workingsoftware.com.au

It's worth noting here that there are other ways of restricting SSH to only a single command, however this involves having to download and install a private key and I figured that the above was easier. That being said I'd love to know if there's some massive security flaw with doing things that way. From what I can find around the place there doesn't seem to be any problem with it. Please add a comment if you have any advice!!

Creating the Read Only Repository

I had an existing repository, so I copied it onto my new CVS server:

copy_cvsroot.txt

then as root on rs.workingsoftware.com.au:

chown_cvsroot.txt

Before we can create a read only repository we have to create a spot for CVS to write lock files outside of the repository itself:

cvs_lock_dir_change.txt

Now uncomment the the LockDir line and change it to /var/lock/anoncvs and save the file, then check the changes in with:

cvs_lock_dir_change_commit.txt

After I did that I was able to go onto a different server and do:

anonymous_checkout_command.txt

I tested things out by making a change to the source code and then attempting to commit that change, and I was unable to. Success!

Mixing and Matching

There were two things wrong: firstly, I didn't want to give public access to every project in my repository and secondly, I wanted to allow write access to some of my developers.

This was pretty easy, just a matter of juggling the permissions a bit. First, I created a group cvsdev and added all my developer accounts to it (not the bentley account though). I then granted permission to that group on the LockDir:

lockdir_permissions.txt

I then set permissions on the cvsroot which would allow read only access to the bentley account, and read/write access to users in the cvsdev group, and did the same on the CVSROOT folder inside the repository:

cvsroot_permissions.txt

Note that above, for directories have "4" in front of them, which means that the "set gid" bit is set on those directories. This means that files that are created in those directories will inherit the group cvsdev. You should then change the umask for each of your developer accounts to 002, so that new files created in your repository are writable by all members of your development team.

Now for any project that you want to provide anonymous access to, you do the following (using RocketSled as an example):

make_public.txt

Users in the cvsdev group will be able to write to the public repository, but the bentley user account will only be able to read from it. Then for any project that you wish to exclude from public read access, but then allow read/write access to your development team:

make_private.txt

Because this is all permissions based, even if someone were to h4x0r my bentley@rs.workingsoftware.com.au account, they still wouldn't be able to read my private project files, and since I've got this running on a FreeBSD jail without sendmail or anything installed I doubt there's much fun they could really have. You never know though, I may be eating my words within days.

So that's it! You can now access my public RocketSled repository:

anonymous_checkout_command_rsv04.txt

and my development team can checkout using their accounts and commit changes to both the public and private projects.

Incidentally, I'd wait til release v0-5 of RocketSled (which should be within the next month or so) before starting to play around with it. If you want to find out more about RocketSled you can go to:

http://rocketsled.workingsoftware.com.au/ (ed: this link is now broken, refer to the RocketSled page)

We're working on it :)

Comment Archive

This post originally appeared on an older version of the Working Software website which implemented it's own comment mechanism. This new version of our website/blog uses Disqus for comments (see below) however I wanted to preserve the comments made on the previous post here:

In the scenario where I give read/write access to the devs and make the project public. After a developer commits a file he becomes the owner of the file and the readonly user can no longer see it. How did you manage do overcome this?

Anton Avramov (2009-10-10)

Filed under: recipe | View Comments

This article assumes you have already experimented with using the vim debugger as described in this fine article. Big respect to Sueng Woo Shin for writing the debugger script itself. If you've already experimented with it and think you want to use it more then read on...

Using a debugger wins hands down over the good old echo/die approach but if you want to use the aforementioned debugger (which you do because you want to use vim right? right!) then you may need to get used to a few little quirks and maybe even tweak it a little bit so feels a bit....righter.

So with that in mind here's a few little things that I've done and realised that have made it start to feel that little bit righter to me ...

How to debug a script which is invoked via the PHP CLI

To debug a page being served up via apache you point your browser to:

http://example.com/index.php?XDEBUG_SESSION_START=1

in order to set a cookie your browser and then the presence of this cookie indicates that you're keen to debug some stuff.

If the code you want to debug is being run via the PHP CLI then the above won't work so instead you need to set an environment variable before you run invoke the PHP CLI to run your script. This works for tcsh:

setenv "XDEBUG_CONFIG=vim=1"

Then run your script via the CLI (eg, php -f my_script.php) and as long as you have Vim open and have pressed F5 (like you normally do when you want to debug in vim) then you should be able to debug your script.

What's going on with regards to the files that get opened when a debugging session starts up?

Well regardless of if you like it or not the debugger always opens the "first" file in the call stack, if you like. So if you are debugging a handler that gets invoked from say http://workingsoftware.com.au/myapp/index.php?h=FindProduct then it always opens index.php. This is quite annoying cos most of the time you don't really want to debug index.php...in the above example you probably want to debug find_product.class.php, which is the file where the application code actually resides (whereas index.php is part of a framework and you're usually interested in debugging the application, not the framework). (ed: this is referring to usage of the RocketSled PHP framework but will be similar for any framework where you have one script which dispatches control to a Controller or Handler class or script)

There's no real way around this that I can see. So this is what I do if say I want to debug a file called find_product.class.php which is a Handler class (or Controller in MVC terminology) invoked by hitting the URL http://workingsoftware.com.au/myapp/index.php?h=FindProduct:

  1. cd myapp/packages/myapp/handlers/
  2. open find_product.class.php at the command line by doing: vim find_product.class.php
  3. hit F5
  4. quickly (within the 5 second limit....see below - I have made this limit longer) go to the URL in your browser (ie, http://workingsoftware.com.au/myapp/index.php?h=FindProduct)
  5. the debugger will open up index.php and in the process close your find_product.class.php leaving you with index.php and the other debugger specific windows. Bit annoying. Since you dont want to debug index.php close this file by doing :q
  6. now you're left with just the debugger specific windows (actually viewports) in vim. Now open up the file you want to actually debug, find_product.class.php, by doing :sp find_product.class.php
  7. Now go ahead and do your debugging. Once the debugger session finishes you'll be left with the find_product.class.php file open in vim.

The good news is that if you set any breakpoints in your find_product.class.php file while you were debugging, as long as you don't close vim itself (doesn't matter if you close the find_product.class.php file and reopen it, as long as vim stays running) then you can hit F5 again to start another debugging session and repeat the above steps and once you've gotten back past the final step you will see that the breakpoints you set in the last debugging session have been remembered in the new one.

What are all these bloody windows?

I have found that using the debugger was made quite hokey due to the fact that the screen space got crowded out with a lot of superflous stuff. I can't see, at this stage, why you really need the TRACE_WINDOW or HELP_WINDOW to be there in every debugging session. The WATCH_WINDOW and the STACK_WINDOW are the only real useful ones.

So I have edited the debugger.py file to do away with these windows. You can download my edited version here

http://www.workingsoftware.com.au/downloads/debugger.py
NB: All the original code has been more or less left in there, I've just commented the various bits out. You can run a diff on the original to see exactly what the changes are.

Now when you start up a debugging session you'll see the file(s) you're debugging, the WATCH_WINDOW and the STACK_WINDOW all stacked on top each other like you'd get if you did :sp normally.

The fine peeps who developed this debugger have mapped the F1 key so that it resizes the debugger windows. This is cool but I found you had to press F1 three times to get the window to resize the way I wanted it. So I edited debugger.py again to make it just toggle between the equivalent of Ctrl+W= and Ctrl+W_ . So when you're debugging if you are in the find_product.class.php file's window and you press F1 it will blow that window up to take up the majority of the screen. Press it again and it will go back to an even split between find_product.class.php, WATCH_WINDOW and STACK_WINDOW. Go into the WATCH_WINDOW window and press F1 and it will make the WATCH_WINDOW blow up, hit again and back to even split. Same goes for STACK_WINDOW obviously.

Controlling the debugger - keyboard commands

I realise this is probably stating the obvious for a lot of people and that it is also covered in the debugger script documentation but I'll include it anyway for completeness... Also if you're using the aforementioned edited version of debugger.py you will no longer have the HELP_WINDOW in your debugging session so you'll have to remember the commands with your brain... this might help:

  • Command

    What it does

  • F1

    toggle resize

  • F2

    step into

  • F3

    step over

  • F4

    step out

  • F5

    run (to next breakpoint or end of script)

  • F6

    end debugging session

  • F11

    dump context to WATCH_WINDOW. this will dump the values of all the variables in scope into the WATCH_WINDOW

  • F12

    if you have the cursor positioned over a variable in the file that you are debugging then hitting F12 will dump the value of that variable out into the watch window. NB: I have listed 2 caveats associated with F12 below

  • ,e

    If you do ,e and then type in the name of some variable then you will see that variable's value. If its an array you'll see the array contents, if its an object you'll see the object's contents. if it's empty or out of scope you'll see (null).

  • :B

    sets a breakpoint

  • :Bp

    unsets a breakpoint

F12 caveats

  • If you hit F12 and the cursor is positioned on an empty line then it will cause vim to go into INSERT mode. Quite annoying but good to know.
  • If you hit F12 and the cursor is in the WATCH_WINDOW or the STACK_WINDOW then it will print "no commands none" and end yourde bugging session. Again annoying but good to know.

A couple of handy commands for managing breakpoints

So you can use :B and :Bp to toggle a break point on/off. But some other useful stuff that's not immediately obvious is:

  • Command

    What it does

  • :sign place

    lists all breakpoints, showing which line each one is on

  • :sign unplace *

    clears all breakpoints

  • ???

    go to next breakpoint (haven't figured this one out yet ... anyone?)

Five seconds doesn't seem like long enough..

The debugger.py script is hard coded to wait 5 seconds for a connection after you've pressed F5. This didn't seem like long enough to me so I changed it to wait 15 seconds.

This change is in the edited debugger.py which you can download here:

http://www.workingsoftware.com.au/downloads/debugger.py

Just search for 'serv.settimeout(15);' and change 15 to however many seconds you think is reasonable if you want to change it.

So they're a few little things I found handy. If you find more let us know ;-)

Comment Archive

This post originally appeared on an older version of the Working Software website which implemented it's own comment mechanism. This new version of our website/blog uses Disqus for comments (see below) however I wanted to preserve the comments made on the previous post here:

hi just to add another usefull thing :Bp would set up a breakpoint with an expression according to context you might wanna check, http://slackdna.blogspot.com/search/label/vim-dbgpclient

Zeft (2009-03-11)

Filed under: recipe | View Comments

Usually, you can keep the design of your interface pretty well separated from the logic behind it. One example where this falls down, however, is in the case where the interface uses image inputs.

The image input type allows the use of an image for a button. They're not really all that popular due to accessibility issues, but this is not an ideal world and sometimes you have to work to a design brief that includes image buttons.

So the catch with these is that Browsers send the name attribute of the button to the server differently. In Firefox, for example, you get three $_POST variables in PHP:

  • button_name
  • button_name_x
  • button_name_y

The x and y are the positions of the button on the page and are (were) used in server side image maps. However Internet Explorer will send only the latter two values:

  • button_name_x
  • button_name_y

So the safest way to check if a particular button was clicked in your PHP script (or other server side scripting language) is to look for the parameter with the _x or _y appended (in .NET this becomes .x or .y):

post_argument_check.txt

The use of both _x and _y here is redundant but I thought I'd just include it for the sake of completeness.

The problem I have with this is that it ties the way the form is being processed to the way the form appears. So my workaround for this is to always use a function to check for the presence of button parameters:

button_clicked_function.txt

That way you can check which button the user clicked and it won't matter if a later design change adds in image buttons.

Filed under: recipe | View Comments

I recently had to take a string with some delimited values as placeholders and replace them with variables from an array. This is useful for things like sending out email newsletters or SMS messages with people's names or other unique information in each message.

An example of a string with placeholders in it, using the string '%%' as a delimeter is:

Hello %%NAME%%, your %%CAR%% is ready to pickup from %%STORE%%

Now take an array with some key/value pairs:

key_value_pairs.txt

The following code will perform the replacement based on the filter:

replace_code.txt

The interesting parts of the above code are, in the pattern:

pattern_1.txt

the "?" makes this a "non-greedy" match, meaning that it won't match everything in between the first and last occurence of $delim. Using "?" in this way allows you to make individual matches non-greedy. You can also use the /U modifier at the end of the expression to make all matches non greedy:

pattern_2.txt

The next interesting bit is the fact that the second argument to preg_replace_callback can be an array with an object and a method. This allows you to use preg_replace_callback in your OO based PHP application without breaking out into procedural code (cos where would you put it!?).

Filed under: recipe | View Comments

I'm working on a job for a client where legacy database data are being used to generate an XML document for processing with an XSLT stylesheet.

The data are encoded HTML entities in the database. So when I created my DOMDocument, I got the following warnings:

Warning: DOMDocument::loadXML() [function.DOMDocument-loadXML]: Entity 'middot' not defined in Entity, line: 963 in /usr/local/www/data-dist/sheds/includes/SDEHSFunctions.php on line 414

Instead of passing in '·' in the XML string to the constructor of the DOMDocument object, I needed to either declare all entities in the XML doctype (bothersome) or I needed to convert these text entities into numeric ones (eg. '·' becomes '·').

I took a look around and found this handy function:

http://php.net/get_html_translation_table

I did a print_r on the translation table returned and found that it returns an array where the key is the actual character represented and the element is the textual HTML entity. So here's a quick function to get the character coded equivalent:

html_entity_convert.txt

Filed under: recipe | View Comments

Subscribe

Subscribe via RSS

Building software in the real world - the Working Software blog

We write about our experiences, ideas and interests in business, software and the business of software. We also sometimes write about our own products (in order to promote them).