Wednesday, January 27, 2016

How To: Filter an XML file using Powershell

Once again I've been using Powershell to work around a problem. A few thousand users have been serialised into an XML file that needs to be imported into an external system. The problem is that there are many users that need to be removed from this file and the system that generated the file in the first place isn't able to filter them out before doing the export, don't ask.

Fortunately Powershell makes this laborious task fairly straight forward. Using just a few lines of code, I've loaded in the export file along with a file of users that should be removed; then filter through the user objects in the XML and write the results out to a new file.

$usernames = Get-Content excludedUsers.txt
[xml]$xmlFile = Get-Content ".\users-export.xml"
$xmlFile.Application.Users.User | where-object {$usernames -notcontains $_.username} | % {$_.ParentNode.RemoveChild($_)}
$xmlFile.OuterXml | Out-File .\users-export-filtered.xml

Wednesday, December 30, 2015

D plugin for Intellij IDEA

I've been programming in D for over a year now and for the most part I've used Sublime Text as my editor of choice. This has been fine for experimenting with little bits of code but a proper IDE with good auto completion, code inspections and other such features can really boost productivity.

For the last five years I've been using Intellij IDEA and have grown accustomed to it's ease of use so using it for D projects would be really handy. Fortunately Kingsley Hendrickse (kingsleyh) wrote a D Programming Language plugin for Intellij (there is another plugin for dlang in the JetBrains plugin repo but this is the one to use). Kingsleyh's plugin makes use of some D tools that were written by Brian Schott (Hackerpilot):

DCD (D Completion Daemon, auto-complete program for the D programming language)
D-Scanner (tool for analyzing D source code)
dfmt (formatter for D source code)
dfix (Tool for automatically upgrading D source code)

Brian Schott did a talk about these projects at this years DConf so it's worth checking out the video on youtube. As they are stand-alone programs they need to be installed on your system manually. There are not pre-compiled binaries available currently so to get them installed you have to clone the repositories, build the projects, then put them somewhere on the system PATH. This can be done with the following steps:
git clone git@github.com:Hackerpilot/DCD.git
git clone git@github.com:Hackerpilot/Dscanner.git
git clone git@github.com:Hackerpilot/dfmt.git

cd DCD
git submodule update --init --recursive
make
sudo mv bin/dcd-client /usr/local/bin/
sudo mv bin/dcd-server /usr/local/bin/

cd ../Dscanner
git submodule update --init --recursive
make
sudo mv bin/dscanner /usr/local/bin/

cd ../dfmt
git submodule update --init --recursive
make
sudo mv bin/dfmt /usr/local/bin/
Then it's just a case of installing the plugin from the plugin repo and configuring it to use the tools. This is done in:

File > Settings > Other Settings > D Tools

Simply clicking the 'Auto Find' buttons should do the job.


Now that's all done you're good to go with dlang on Intellij.

Monday, July 27, 2015

Updating all guids in a file using Powershell

I'm getting to really like powershell. Today I needed to generate a whole lot of guids in multiple build files and was able to do what I needed with a single line:

(Get-Content 'filename') | ForEach-Object { $_ -replace "[a-fA-F0-9]{8}-([a-fA-F0-9]{4}-){3}[a-fA-F0-9]{12}", [guid]::NewGuid() } | Set-Content 'filename'

Friday, May 15, 2015

Conways Game Of Life

Like many students studying software engineering, when I was at university one of my assignments was to write an implementation of John Conway's Game of Life. At the time I found the assignment fairly tricky but out of the whole three years in University it was probably my favorite piece of work.


Saturday, May 09, 2015

AES File Encryptor for Windows

This week I spent some time working on a powershell script which will use AES Encryption to encrypt a file prior to it being transffered to a remote destination. The receiver will be able to unencrypt the file using a provided key.


I decided it would be a good idea to use the script as a basis for a Windows application that provides the same functionality. Using Visual Studio 2013 I quickly put together a simple user interface using WPF (Windows Presentation Foundation) and wired up the controls. It took hardly any time to get the whole thing working and with the inbuilt Git integration it's been pushed up to Visual Studio Online for future reference.

Wednesday, March 04, 2015

How To: Migrate from svn to git

Source control is a vital tool to any developer, if you're not using some form of source control when doing development then you're missing out. Creating a zip file with a version number in it does not count. Amazingly I still come across people doing this.

Most of the projects I work on are using git but there are still a few projects that are kept in subversion. There's nothing wrong with svn, it's perfectly fine but git does make it convenient to work on different features locally.

In the past I've moved projects from svn over to git by simply doing svn export then adding the exported code base to git. The drawback here is that you lose all the commit history and tags/branches.

Fortunately it's possible to migrate a svn repository over to git whilst keeping all of the history and some of the developers at Atlassian have made a handy tool to make it even easier.

Naturally you'll need svn and git but you will also need git-svn, Java, and 'svn-migration-scripts.jar' (available from BitBucket).

I did the process on a Fedora Linux system (you should use a caps specific system) so the commands used on this page could differ. For full details of the process you can refer to Atlassian's Migrating Overview page.

To start off make sure you have git-svn:
sudo yum install git-svn
Then if you havn't already downloaded it, get the svn-migration-scripts.jar:
wget https://bitbucket.org/atlassian/svn-migration-scripts/downloads/svn-migration-scripts.jar
The first part of the migration is to map all the users that have done a commit to the svn repo, to their corresponding git user. To do this use the migration-scripts tool then manually edit the generated 'authors.txt' file:
java -jar svn-migration-scripts.jar authors https://mydomain.com/svn/repos/myproject/ > authors.txt
For every user that has done a commit to the project in the svn repo, map them to a name/email combination that is used by git. eg:
BGATES = Bill Gates bill.gates@microsoft.com
Once that's done it time to get git-svn to get all the commits from the svn repo:
git svn clone --stdlayout --username=SVN_USERNAME_HERE --authors-file=authors.txt https://mydomain.com/svn/repos/myproject/ myproject
Change in to the newly created directory that the code has been cloned to. It's worth noting that git can be configured to a particular user both globally and for a particular repository so you may want to set your name/email for the repository before continuing:
git config user.name "name"
git config user.email "name@domain.com"
Now to convert all the svn tags to tags in git (assuming you have the standard svn layout of tags, branches, and trunk):
java -Dfile.encoding=utf-8 -jar ../svn-migration-scripts.jar clean-git --force
That's just about it. Assuming you already have an ssh key and a remote git server to push to then you can add the remote is the usual way and push the code. Note that by default git won't push the tags up to the server so use '--tags' or '--all' (not both):
git push -u origin master --all

Tuesday, January 27, 2015

Simple Java Ray Tracer

This evening I decided to go through a few old folders on my laptop and delete anything I don't need anymore. One of the items was a zip file of some source code for a ray tracer written in Java. I downloaded it a few years ago from another blog that's no longer online. I believe it was originally written in 2008 by Barak Cohen and Gur Dotan.


As the authors blog is no longer available I couldn't bring myself to deleting it and have put a copy on GitHub in case any other people out there want to take a look. I've rejigged the project structure a little and removed all the Eclipse related files in favour of a Gradle configuration.

Friday, January 23, 2015

Vendor specific meta tags

The company I work for use IE almost exclusively. Every member of staff has been on IE9 for some time and are now on IE11 which means that I'm in the fortunate position of not needing worry too much about implementing CSS and Javascript workarounds for old browsers. What I am seeing more of though is vendor specific meta data.

Although the company use IE on the desktop (I still use Chrome & Firefox for development), we do also test projects on Android and iOS. All three platforms we support have their own tags for icons and the like.

It would be good see some standards introduced so that developers can add this type of information in a way that can be used by multiple browsers.

Thursday, January 22, 2015

How To: Deploy a Java war file to OpenShift

I recently put together a responsive web application for someone that wanted a convenient way to keep track of the types of food they've had each day with the ability to perform some statistical analysis on the data recorded. The application uses Spring, Hibernate, JSF, Primefaces, and Twitter Bootstrap. They've opted to have it hosted in the cloud using Red Hat's OpenShift which gave me an opportunity to use the platform for the first time.

Openshift provide a range of options for Java. My app needed Java 7 and Tomcat 7 (there currently doesn't seem to be an option for Java 8) so I selected the appropriate option and set about getting my app deployed.
$ git clone
$ cd nutritionapp/
A directory structure and pom file is already setup for those wanting to commit source code to be built by Openshift prior to deployment. This wasn't what I wanted as I've already compiled a war file and just want a quick way to deploy it. For this option, Red Hat have also put a 'webapps' folder in the root of the repository so I needed to do was delete the 'pom.xml' and the 'src' folder, add the war then push.
$ git rm pom.xml
$ git rm -r src/
$ git add webapps/nutritionapp-1.1.war
$ git commit -am "deploy v0.1 to cloud"
$ git push
I then ssh'd into OpenShift using the url provided in the web interface and did 'tail -f app-root/logs/jbossews.log' to see what was happening with deployment. There was a problem during start up but the log entry wasn't too helpful. Luckily I found a post here which helped out, seems that in OpenShift the 'web.xml' in the app needs to have the Log4jConfigListener removed.

    org.springframework.web.util.Log4jConfigListener
So after a quick edit of the 'web.xml' one last commit was needed.
$ git commit -am "removing Log4jConfigListener"
$ git push
That was it, the whole process was pretty simple.


Monday, December 22, 2014

European Road Trip - xmas 2014

It's been a little over 2 years since our last road trip. This This time it's a little shorter but also spread over a longer time making the whole experience a little less intense.


We set off from Leeds on Friday night and stayed in a hotel on route to the channel tunnel then again in Luxembourg on Saturday evening, leaving a 5 hour drive to Munich on Sunday. We'll stay here for a couple of days before going south to Innsbruck for Christmas.

Monday, December 01, 2014

How To: Get started with D

As much as I like working with Java, it's nice to use a language that can be compiled to native code. Occasionally I dabble with C or C++ but the ecosystem is about as old as the languages. I want to work with a language that doesn't just have clear syntax but decent tools to handle builds and project dependencies as well as making it easy to compile on multiple platforms (Windows, Mac and Linux is a must).

D meets these needs (partly due to DUB). It's been around for over a decade but it's the second version (first released in 2007) that is currently in use.

The D compiler can be downloaded from dlang.org and DUB can also be downloaded from the sub-domain code.dlang.org. Once they are both installed and available on your PATH open up a command prompt and use DUB to initialise a project.


DUB will create your project structure and a simple manifest file 'dub.json' which will be familiar to anyone that uses NPM or Gradle.


For this simple example I've added 'vibe-d' to the project dependencies. Then in the source file I've replaced the contents of the 'app.d' file with the simple example given on the Vibe-d website.


To compile the project simply run the 'dub' command in the root of the project. DUB will then download the vibe-d library, compile and link the project, the run the compiled executable.


That's it! We have a simple server running that can be opened in the browser.


Obviously this example is pretty trivial but I've been playing with D in my spare time for some months now (working on a GTK app) and have found the transition from Java to D to be relatively painless. So go give it a try.

Saturday, October 25, 2014

How To: List all compile dependencies using Gradle

Gradle makes it easy to add steps to the build process. Today I found a nice little task to list all project dependencies:


The only problem with this is when a project has some local dependencies that are included using compile files('lib/some.jar'). A dependency included in this way has no group, name, or version as it is not in a repository. This was causing a dependency in my project to show up as "null : unspecified : null". To get around this I made a variation of the task above which will list the filename of each jar file that is being included:


It comes in handy when you want your build server to provide a little more information for debugging problems. Simply add the task to your projects build.gradle file (anywhere below the dependencies block).

Monday, October 06, 2014

What happened to nine?

Microsoft may not acknowledge the number nine but one thing they are doing with their upcomming version of Windows is accepting that they were wrong about the start screen. This is something the GNOME Foundation should take notice of. At least Microsoft have listened to the backlash and realised that people still need a traditional desktop.

The idea of forcing a user interface designed for touch-screen devices onto users that need a productive desktop machine was always doomed to fail. Who cares if the average consumer is now buying a tablet instead of a desktop, as long there are jobs there will be a need for computers that can be productive in the workplace.


I do feel that the team at Microsoft deserve some respect for the work they've done in merging the start screen from Windows 8 with the menu from previous releases but most of all for backing down and listening to their users.

Thursday, August 28, 2014

Things to remember when creating an HTML email template

Every web developer knows about the kind of problems that can be encountered when supporting multiple browsers, especially older versions of IE. These days things are getting better though, even IE is able to do a half decent job. The world of email clients is another story all together though. The pain of supporting multiple browsers is nothing compared to the nightmare of trying to get your well designed email to render correctly in even a small selection of the hundreds of email clients out there.

Having spent the last few months doing a major overhaul of an application at work I decided it would be a good idea to redesign the email templates being sent from the system to be more in line with it's new layout.

As the emails are only received by members of staff during work I'm in the fortunate position of only needing to support two email clients. However, rather unfortunately the two clients we need to support a rather hideous; Microsoft Outlook and the built-in client used in BlackBerry devices.

There are two types of problems you'll encounter when designing email templates. Firstly email clients tend to lag far behind browsers in terms of HTML and CSS support and secondly there's a lot of weird quirky behaviour in some email clients.

For some strange reason Microsoft uses the same HTML engine used in Word to display HTML emails in Outlook, meaning that Outlook has minimal support for CSS properties. BlackBerry has full support for XHTML 1.0 and partial support for HTML 4 & CSS 2.1. So forget about using anything recent, email templates need to be done the old fashioned way, with tables.

To keep things simple I stuck to using a fairly basic layout using XHTML strict and validating it periodically using the W3C Validator to make sure everything was correct, not that it would make much of a difference, as some email clients (including Outlook 2007 and 2010) strip or ignore your doctype entirely.

Another section that is likely to be striped out is the head along with any style tags. For this reason it's best to use in-line styling on your markup. MailChimp provide a handy CSS Inliner Tool which can help do this once the design is complete.

Thursday, August 14, 2014

The worst javascript kludge ever

Due to limitations in a third party application, today I've had to create the most hideous workaround I've ever done. Essentially I needed to create a new instance of IE rather than just creating a new browser window. Using ActiveX I first attempted to create an IE instance directly.


When looking at the Task Manager I could see that the new window was still a child of the same parent process so I needed to go even further. I used ActiveX to get a shell then run iexplore.exe passing in the url I want to open:


It's for an internal site which should only be accessed by IE and does in fact do exactly what is needed but it still makes me shudder.

Friday, May 16, 2014

WTFilter now on the Atlassian Marketplace

Over the last week I've been updating the WTFilter plugin for Confluence. The last release was back in 2012 and it was unfinished. I figured that it's about time I finished it off and did a proper build.

It now filters comments as well as pages and blog posts both during creation and edits. It's also been updated for the latest versions of Confluence, anything since 5.0.

The new build, version 1.2.0, can now be downloaded from the Atlassian Marketplace.

Sunday, March 16, 2014

Elder Scrolls Online

Whilst checking my inbox earlier I found that I'd been sent a beta invite for Elder Scrolls Online. So I downloaded the Mac client and started the 22GB net install. Three and a half hours later...





I've only played about an hour but so far it's been impressive, except for the fact that the game crashed on me after completing the first mages guild quest. I'm not usually into MMO games but for this one I may make an exception.

Friday, February 28, 2014

How To: Use Bootstrap 3 in your Java Web App

Last year I built a Java webapp for work using JSF, Primefaces, Spring, PrettyFaces, the regular kind of stuff that is generally used when developing an app to run on Tomcat Server. I chose not to use the Primefaces styling, wanting to create my own styles making use of the LESS files that are used to generate Bootstraps CSS. It's worth noting that Primefaces does have various themes including a Bootstrap theme but it isn't quite the same. You can remove the Primefaces styling by placing the following in your web.xml:


The app originally used Bootstrap 2.3.2 and much has changed since then. I've spent the whole of today going through the process of upgrading to Bootstrap 3.1.1. For anyone wanting to update from version 2 there is a convenient page on the Bootstrap website: Migrating to v3.x. Apart from the obvious changes to classnames were are a few steps required to get the new version working properly.

The best way to make use of Twitter Bootstrap in a JSF web application is to use bootstraps LESS files. These should be added the same way as any other resources. When you come to build your application the 'lesscss-maven-plugin' can compile all the LESS files into a minified css file, which can be used in your template using the following:


When upgrading to Bootstrap 3 I found that the plugin was no longer able to compile Bootstraps LESS files as the used features that were not supported in the version of LESS that was used by the plugin. Fortunately this can be resolved in the configuration block for the plugin by manually providing a path to a more recent LESS version.


Add the fonts to '/webapp/resources/fonts' then edit 'glyphicons.less', change the @font-face block to match the following:
@font-face {
font-family: 'Glyphicons Halflings';
src: url("#{resource['fonts/glyphicons-halflings-regular.eot']}");
src: url("#{resource['fonts/glyphicons-halflings-regular.eot']}?#iefix") format('embedded-opentype'),
url("#{resource['fonts/glyphicons-halflings-regular.woff']}") format('woff'),
url("#{resource['fonts/glyphicons-halflings-regular.ttf']}") format('truetype'),
url("#{resource['fonts/glyphicons-halflings-regular.svg']}#glyphicons-halflingsregular") format('svg');
}
Using the steps provided here you should be all set to start theming you app using Bootstrap. The great thing about using the LESS files rather than just adding the compiled CSS to the app is that you can use the same mixins that Bootstrap provides as well as easy access to the global values used in variables.less.

Monday, December 16, 2013

Setting up Arch Linux on Raspberry Pi

I've had a Raspberry Pi for almost a year now and despite my intention to use it for development it's ended up being used to run Raspbmc. I couldn't let it go a year without it being used for something more interesting so I spent some time this evening getting it configured for use.

I gave RISC OS a go first, thinking it'd be fun to use something completely alien to me but the novelty wore off within the first few minutes of using it. So I decided to scrap that idea and install Arch instead.

I've not used Arch since the move to systemd so it seemed like a good opportunity document the install process as my previous post about installing Arch is a little out of date.

To start with you need an SD Card with a recent Arch image. Here is my bash history from preparing the memory card:
$ diskutil list
$ diskutil unmountDisk /dev/disk1
$ sudo dd bs=4m if=archlinux-hf-2013-07-22.img of=/dev/disk1
$ diskutil eject /dev/disk1
Now is also a good time to change the root password. With that done, set the timezone and locale info. Start by uncommenting the line for your locale (in my case en_GB) from '/etc/locale.gen'. Now generate the locale settings:
# locale-gen
You'll also need to run the floowing two commands with the appropriate language and timezone:
# localectl set-locale LANG="en_GB.UTF-8"
# timedatectl set-timezone Europe/London
I'm using a Belkin wireless-N adapter with my Pi so I needed to setup my wifi using the following:
# install -m640 /etc/netctl/examples/wireless-wpa /etc/netctl/home-wifi
Then edit the '/etc/netctl/home-wifi' file with the correct SSID and password. Once done enable it:
# netctl enable home-wifi
You should now have internet access so do an update pacman -Syu then install vim:
# pacman -S vim
To get rid of some unsightly black borders on my monitor (overscan) I needed to edit the boot config:
# vim /boot/config.txt
Uncomment the line 'disable_overscan=1'. I also chose to uncomment 'hdmi_drive=2' as I only ever intend to force audio over HDMI.

I find the default console font a little small on a 1080p monitor so I also decided to create a /etc/vconsole.conf file with the following:
FONT=ter-218n
FONT_MAP=8859-2
While on the subject of fonts, it may be worth installing these if you need some nice truetype fonts:
# pacman -S ttf-dejavu ttf-droid ttf-linux-libertine ttf-ubuntu-font-family
Before creating a user account install the sudo package and configure it so that all users in the wheel group can use sudo:
# pacman -S sudo
# EDITOR=vim visudo
Uncomment the line '%wheel ALL=(ALL) ALL'.

I'm using a 32GB SD Card for the installation so I had to resize the root partition. This can be done within Arch by using fdisk to delete the extended partition then recreate a new extended partition using the remaining space and another logical partition. Once the changes are written you'll need to reboot then run resize2fs. See Growing the RPi root partition for full details.

Most Linux users stick to using bash but I want my user account to default to Zsh, so have installed the package prior to creating the user account:
# pacman -S zsh
# useradd -m -g users -G audio,lp,optical,storage,scanner,video,wheel,games,power -s /usr/bin/zsh USERNAME
Set a password for the new user then exit and login as the newly created user.

To have a little colour in your zsh prompt add something like the following to the end of your .zshrc file:
PS1="%{%F{cyan}%}%n%{%F{red}%}@%{%F{blue}%}%m %{%F{yellow}%}%~ %{$%f%}%% "
As I'm going to be doing development on the Pi I also installed Java7, Python, NodeJS, Ruby, and a C compiler:
# pacman -S python python-pip ruby clang nodejs jdk7-openjdk openjdk7-src maven