/var/blog/messages

Blanket bomb a C source file with function entry/exit trace

2011-07-05T13:04:00.000+01:00

'Blanket bomb' a file with trace for function entry / exit, from vim:

:%s/^{/{\r\ \ \ \ printf(">>>%s:%d\\n", __FUNCTION__,__LINE__);\r/
:%s/$\ *$$return.*\n$/\1printf("<<<%s:%d\\n", __FUNCTION__,__LINE__);\r\1\2/

or just using sed from the command line:

sed 's/^{/{\n\ \ \ \ printf(">>>%s:%d\\n", __FUNCTION__,__LINE__);\n/' Filename.c > Filename.c
sed 's/$\ *$$return.*\n$/\1printf("<<<%s:%d\\n", __FUNCTION__,__LINE__);\n\1\2/' Filename.c > Filename.c

How to use a 64bit machine to build & install a 32bit Linux kernel for another PC

2011-06-19T12:13:00.036+01:00

I've often had the problem of having to build a Linux kernel for a slow machine, while having a much faster machine available. In my case, the slow machine is an Asus EEE-PC running 32bit Fedora, and my faster machine is an Intel i7 running 64bit Fedora.

In order for the faster 64bit machine to build a kernel and install it on the slower 32bit, you'll need to bear in mind three issues:

* How to build a 32bit kernel on a 64bit machine

* How to give access to the slower machine for installing the kernel

* How to actually install the kernel on the slower machine.

How to build a 32bit kernel on a 64bit machine

Assuming you already have kernel source and a suitable .config available on the faster machine, simply run your usual kernel make, with the addition of ARCH=x86 and with menuconfig as a target. In my case, this would be:

fast$: make ARCH=x86 -j16 O=/home/mark/Source/build-linux-2.6-32bit/ menuconfig

Then deselect the top option "[ ] 64-bit kernel" before exiting and saving. Now you can build the kernel with:

fast$: make ARCH=x86 -j16 O=/home/mark/Source/build-linux-2.6-32bit/ 

Note that the '-j' option gives the number of simultaneous jobs the make can handle, which is usually 2x the number of processors/cores in your system. I have 8 cores, so I use them all by specifying '-j16'.

The O= specifies the build directory where the build objects are placed. The .config you wish to use should go in here. There are many reasons why it is a good idea to have separate build and source dirs.

How to give access to the slower machine for installing the kernel

You could also use a Samba share for this, but I prefer using NFS. Note that this isn't the most secure method, using NFS in this way, but it works.

On the remote, slower, machine with NFS installed, export your root (/) and boot (/boot) directories by editing the /etc/exports file or using the system-config-nfs program. My /etc/exports file looks like this:

/boot                      *(rw,sync,no_root_squash)

/             *(rw,sync,no_root_squash)

Then restart the NFS services nfs. nfslock and rpcbind (portmap on older systems):

slow$: sudo service rpcbind restart

slow$: sudo service nfslock restart 

slow$: sudo service nfs restart

Now it should be possible to remotely mount these exports on the faster machine, replace <remote-ip> by the name or IP address of the remote PC:

fast$: mkdir ~/remote-root

fast$: mkdir ~/remote-boot 

fast$: sudo mount -t nfs <remote-ip>:/ ~/remote-root

fast$: sudo mount -t nfs <remote-ip>:/boot ~/remote-boot

Installing the kernel on the slower machine

Almost there, we just need to install the kernel image on the slower machine and reboot to run it!

For this to work seamlessly, a small patch is needed to the /sbin/installkernel script (on my Fedora machine, at least):

--- /sbin/installkernel
+++ /sbin/installkernel
@@ -32,7 +32,10 @@
cfgLoader=1
fi
-LINK_PATH=/boot
+if [ -z "$LINK_PATH" ]; then
+    LINK_PATH=/boot
+fi
+
RELATIVE_PATH=`echo "$INSTALL_PATH/" | sed "s|^$LINK_PATH/||"`
KERNEL_VERSION=$1
BOOTIMAGE=$2

Now we can export some environment variables to tell the kernel make install scripts where we would like our kernel and modules to be installed (Note the difference between remote-root and remote-boot!), while logged in as the superuser:

fast#: export INSTALL_PATH=/home/mark/remote-boot/

fast#: export LINK_PATH=/home/mark/remote-boot/

fast#: export INSTALL_MOD_PATH=/home/mark/remote-root/

For a Samba export, just replace the exported variables above with their equivalents (usually ~/.gvfs/root\ on\ remote/ or similar for Fedora).

Now we can simply run the install make targets on our fast machine, which installs to our slower, 32bit target PC:

fast#: make ARCH=x86 O=/home/mark/Source/build-linux-2.6-32bit/ modules_install install

Then edit your grub.conf (or equivalent, as I haven't yet found out why /sbin/new-kernel-pkg doesn't update this automagically on the remote machine).

Reboot, select the new kernel from the GRUB menu (or equivalent), and happy hacking!

Keeping it all in mind

2009-01-05T11:43:00.002Z

Back in the 50's, the psychologist George A. Miller wrote a paper entitled 'The Magical Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing Information'. While quite a dry and involved read, it's findings (and other corroborating evidence) are of great significance to anyone creating a software design, architecture, process, checklist, API or code listing. In fact, anything where a large amount of items or concepts can be grouped or categorised may benefit from its findings, namely that:

There is a limit to the number of items we can keep in short-term memory - 7 plus or minus 2.

While not applicable to basic lists - where the reader doesn't simultaneously need to be aware of every item in the list - it is a powerful heuristic when applied to, for example, a software architecture layer. In this case a person trying to understand the architecture would benefit from awareness of all parts of the architecture at the level being studied, and in doing so finds the architecture more easily discoverable and transparent than if the layer were more finely partitioned.

Another great example comes in the form of a process description or checklist, especially where safety is critical. Here it is important not to miss any steps or checks, and the chance of doing so is reduced if the number of steps are kept to a manageable amount.

It's difficult to list everything to which this heuristic can be applied - there are always opportunities available if you can spot them. With practice, it can become second nature - like many other design heuristics (The appendix of 'The Art of Systems Architecting' by Maier and Rechtin is a great source for such things).

If you want some more examples, just take a look at some of the other posts in this blog... :o), but also next time you're wading through a mass of code, design information or an architecture description and finding it difficult then bear in mind how it is organised, is the author trying to make you 'bite off more than you can chew'?

This applies even more so if you are designing or developing. Your own software (especially open source) could benefit greatly if it's more easily discoverable, understandable and therefore maintainable by others who may not be so familiar with its workings.

Peer code reviews

2008-12-17T11:29:00.005Z

Peer code reviews can have a devastating effect on software bugs, greatly reducing their number and scope. However, not all code reviews are the same and some are more effective than others. There is no one-shoe-fits-all guide to code reviewing, but there are ways of getting the right code review process for your situation.

At the bottom end of the effectiveness scale is a peer looking at a screen resplendent with the review code. This is better than no review, but still has drawbacks:

The review procedure will vary according to the reviewer and their state of mind, some reviewers will be more picky about certain issues or even miss things others would find easily.
Any problems found and commented on in the review may not be fixed - due to forgetfulness, other 'more important' pressing issues or thinking that they are trivial (to the author under review). Anyhow, the problems are probably not followed up afterwards to make sure they're resolved.
A lack of reviewing guidelines can lead to some comments being taken a bit too personally.
Most people when reviewing will only look through the code once from top to bottom, and pick out any issues of any type as they go along (syntax issues, loop counting and memory allocations for example). This is much less efficient than looking for one type of issue at a time, and scanning the code once for each type - if you don't believe me, try it!

In order to mitigate missing as many bugs as possible during a code review, the generally agreed best practice contains the following guidelines:

Use a checklist, some examples can be found in the links below.
Review for one type of issue at a time.
Improve the checklist to look for bugs which slip through the reviews as time goes on.
use code printouts on which you can scribble comments - I've not come across any software tool that is as effective as paper.
Log any issues not resolved during the review process, for example in the form of a bug database entry - even it does seem trivial now, experience has taught me that it might be the the most important thing in the world next month! It should also be possible to make the reviewer aware of any previous issues found with the code under review.
Have your process geared towards reviewing little and often.
Be balanced - make sure to note what the code does successfully, as well as it's issues and offer praise where it is due.

More thorough details and further reading can be found here:

Easy SCM

2008-11-20T16:50:00.009Z

Software Configuration Management (SCM) is an important part of software development - without robust SCM, no software development project can achieve a high quality result.
To this end, there are a few simple characteristics that indicate good SCM - Personally, I haven't encountered a successful project whose SCM plan does not have these characteristics:

Be a snail - leave a trail! Changes are never versioned without a documented reason.
Documented reasons are always either defect fixes, or software enhancements. Both defects and enhancements can be identified by an ID without the need to log detailed explanations for each change - all that is required is for the change to cross reference the defect or enhancement (from a bug database or project plan, for example).
Each change is made for a logical reason, and one reason only. Changes aren't combined or split over several different versions.
It is easy to 'do some archaeology' by quickly running older configurations. e.g. There exists a store of automated nightly builds with corresponding configuration IDs available.
The way of working is cultural, not 'enforced' - although it should be difficult to 'absent-mindedly' not follow the process. i.e. You can't make a change without a defect or enhancement ID to log against it.
The SCM system is transparent and visible, so others with an interest can view progress (although not necessarily take part in the activities - that can be damaging to progress!).
Each change represents no more than two man weeks work.

So, how does your project rate against these?

The IEEE software standards

2008-10-31T23:06:00.006Z

The IEEE software standards is a very useful set of documents, even if you're not in an organisation that is into particularly formal software development methods. The standards do not necessarily have to be implemented to use the information they contain.
For example, they can be a good starting point if you're ever asked to plan or produce documents such as a software requirements specification (SRS), software configuration management plan (SCM plan), or a software architecture document (although for this I've found HP have available a much more usable document, "A Template for Documenting Software and Firmware Architectures").

The only downside is that they are not free, but many organizations have IEEE membership and have the standards available for employees.

Here's the list of IEEE standards and guides I've found to be of use, in no particular order, vaguely grouped for readability:

Project Management

1490 - Adoption of PMI Standard - A Guide to the Project Management Body of Knowledge (PMBOK) - A guide to project management knowledge and practices in general widespread use.
1045 - Standard for Software Productivity Metrics - A bit dated, but the practices and concepts it gives for measuring productivity are well described.
1058 - Standard for Software Project Management Plans - Worth a look if you're unsure as to what you'll need to put in a project management plan, but a little too specific to the IEEE way of doing things.
ISO/IEC 12207.0 - Software Life Cycle Processes - Attempts to classify all the processes that contribute to software, and puts them into a framework.
ISO/IEC 12207.1 - Software Life Cycle Processes - Life Cycle Data - Attempts to classify all the processes that contribute to software, and puts them into a framework.
982.1 - Standard Dictionary of Measures to Produce Reliable Software - A bit ancient (1988), but tries to give a description of measures that can be made on software and software projects that indicate the quality of the software, with all the mathematical rigour that involves.
1220 - Standard for Application and Management of the Systems Engineering Process - Similar to 1074, again use CMMI instead.

Software Engineering

Guide to the Software Engineering body of Knowledge (SWEBOK) - Available here. A guide to software engineering knowledge and practices in general widespread use.
610 - Glossary of Software Engineering Terminology - A little dated, and leans towards IEEE understanding (as opposed to widespread understanding) of some terms, but can still be useful for reference, and is referenced by most other IEEE standards.
828 - Standard for Software Configuration Management Plans - If you need to produce an SCM plan and have nowhere to start, this will show you the way. Also discusses useful activities involved in managing and adhering to a SCM plan.
1002 - Taxonomy for Software Engineering Standards - Also quite ancient. A taxonomy is a method for classification, and this describes how a set of standards can be chosen to cover all necessary areas of software engineering.
1028 - Standard for Software Reviews - Gives criteria and practices for reviewing software - be it for development, acquisition or operation.
1061 - Standard for a Software Quality Metrics Methodology - Aimed at those measuring or assessing the quality of software, in a formal manner.
1074 - Standard for Developing Software Life Cycle Processes - Attempts to define a way of creating a good sw process. Not half as useful as CMMI.
1471 - Recommended Practice for Architectural Description of Software-intensive Systems - A version of Kruchen 4+1 for software architecture.
1042 - Guide to Software Configuration Management - Practices for performing SCM, and managing SC items within a project.

Quality

730.1 - Guide for Software Quality Assurance Planning - Great if you need to write and manage a Software Quality Assurance Plan, and have no idea of where to start - this lists and discusses the contents of such a document and good practices involved in managing it.
730 - Standard for Software Quality Assurance Plans - A lot more detailed than 730.1, and gives the format and content requirements a SQA plan should meet to conform to the IEEE standard.
830 - Recommended Practice for Software Requirements Specifications - Describes what should be contained in a good (albeit formal) SRS, and gives several example outlines of SRS documents.
1062 - Recommended Practice for Software Acquisition - Obtaining and using the right software, that's right for your needs is not an easy task. This gives some useful practices on performing this task.
1063 - Standard for Software User Documentation - Good practices for putting the relevant information into your user documentation.
1219 - Standard for Software Maintenance - This standard describes an iterative process for managing and executing software maintenance activities.
1228 - Standard for Software Safety Plans - Establishes criteria for the content of a software safety plan.
1233 - Guide for Developing System Requirements Specifications - A guide to obtaining and managing requirements in an SRS.

Testing

829 - Standard for Software Test Documentation - Gives a description of what should be in software test documentation (cases, logs, plans etc), and why. Gives the form and content of test documents, but does not say which documents are needed in particular situations.
1008 - Standard for Software Unit Testing - A standard for planning, building and executing unit tests.
1012 - Standard for Software Verification and Validation Plans - Gives a standard for V&V plans, describing what inputs, outputs and criteria are recommended for a project's V&V activities and should be recorded in a plan.
1044 - Guide to Classification of Software Anomalies - How to write and manage bug reports, very useful as in my experience, even some very experience software engineers have trouble in taking the time to write useful bug reports.
1059 - Guide for Software Verification and Validation Plans - Gives a process for using and managing V&V plans.

Design

1016.1 - Guide to Software Design Descriptions - Concentrates on documenting and using 'views' into a design, much like the Krutchen 4+1 paper.
1016 - Recommended Practice for Software Design Descriptions - How to go about writing a SDD, within the project life cycle.
1209 - Recommended Practice for the Evaluation and Selection of CASE Tools - CASE tools are notoriously difficult to choose and use successfully (see an earlier post), this tries to guide you around this particular minefield.
1320.1 - Standard for Functional Modelling Language - syntax and semantics for IDEF0 - A formal system/process modelling technique. Quite heavy, I've found ETVX to be much more usable.
1320.2 - Standard for Conceptual Modelling Language Syntax and Semantics for IDEF1X97 (IDEFobject) - Again, a formal system/process modelling technique. Quite heavy, I've found ETVX to be much more usable.
1348 - Recommended Practice for the Adoption of CASE Tools - Once you've got a CASE tool, the fun doesn't stop there. Carries on from 1209.

Can you sell a software process?

2008-09-13T09:51:00.002+01:00

Most software groups (and I'm sure that this doesn't apply to just software groups) with any sort of history have a record of trying out prescribed process improvement after process improvement, many of them ending in some sort of failure - generally meaning that the process didn't meet the expectation or goals it was introduced with.

This can be especially true of process improvements that are built around an expensive software tool. I've had my fair share of colleagues lost to some sales type or evangelist touting the latest and greatest in software silver bullets. The use of the word 'evangelist' is telling, and conjures up images of an overzealous individual pushing unfounded ideas at you - it gives an indication of the mysic aura that a 'process guru' can give themselves by which one can be blinded.

CASE tools aren't that useful when they constrain you too much to one process, and some companies that push tools as well as processes rely on a self-created cult-like following whipped up from the guru of the day's scribblings. When these words of wisdom are unfounded, unproven and (even better) not understandable, they can be taken by savvy marketers to sell the tool.

A prescribed process (from a heavy one to a very light or 'agile' one) may fit perfectly into the company, work environment, or culture it grew up in but moving it into another environment, more often than not, can transform it into a useless or even dangerous beast.

In the translation, you can lose the meaning of important concepts, miss out on understanding vital assumptions and all the other pitfalls associated with one person attempting to describe a complex system to another. No matter how many reams of written documents or concise manifestos one writes, some things just get missed - even when the two communicator's work cultures are very similar.

What would be much more useful and less prone to failure is an expert in processes to mentor a group over a period of time (even indefinitely), who can prescribe and tailor a process to meet their current needs, to put in place mechanisms to measure the success and progress of the group and ultimately advance the group's software capability - a process specialist or change agent.

A software CASE tool is no substitute for an experienced mentor!

Ethical policies of professional IT bodies

2008-08-23T10:15:00.002+01:00

I consider myself a professional software engineer. To be recognised as such in the wider community involves, amongst other things, subscribing to a professional ethical policy. Personally, I feel an ethical policy must involve not working on anything that will result in harm or even death to others (i.e. no 'defence' work). So which professional IT bodies would be suitable for me to join, with this in mind?

The BCS - avoids the issue by not defining an ethical policy - only has a code of conduct which

'... governs your personal conduct as an individual member of the BCS and not the nature of business or ethics of the relevant authority'.

This is a bit of a cop-out, and avoids the difficult questions altogether.

The IEEE - does have a code of ethics - which does, on first glance seem to fit the bill:

1. to accept responsibility in making decisions consistent with the safety, health and welfare of the public, and to disclose promptly factors that might endanger the public or the environment;
....
9. to avoid injuring others, their property, reputation, or employment by false or malicious action;

But I'm also aware that the IEEE has many high profile members who work in the defence industry and also produces specifications created and used in the defence industry.

The ACM - Much better, uses the term 'human rights', and takes the most care of these three bodies to make clear their ethical stance and to give direction:

This principle concerning the quality of life of all people affirms an obligation to protect fundamental human rights and to respect the diversity of all cultures. An essential aim of computing professionals is to minimize negative consequences of computing systems, including threats to health and safety. When designing or implementing systems, computing professionals must attempt to ensure that the products of their efforts will be used in socially responsible ways, will meet social needs, and will avoid harmful effects to health and welfare.

In addition to a safe social environment, human well-being includes a safe natural environment. Therefore, computing professionals who design and develop systems must be alert to, and make others aware of, any potential damage to the local or global environment.

There's also an interesting discussion of ethics in this paper here, and a very thorough discussion of ICT bodies and ethics, with recommendations here.

I'm still undecided - I believe I need to spend a lot more time researching this topic, as the standard policy for most IT organisations appear to allow members working in defence - even if it's only discernible by reading between the lines.

A Linux Literary Trilogy

2008-07-29T10:33:00.009+01:00

When I started delving into the world of Linux development, I was not only befuddled by the strange code layout and conventions, I also found the culture and ethos of Linux very confusing. There were three books that were invaluable in pulling my understanding out of this quagmire, which I'll mention briefly:

The Linux Programmer's Toolbox by John Fusco. Allows your Linux usefulness to go from 0-60 in six seconds. Not totally exhaustive on all tools Linux, but it's brilliant for giving you an up to date map of the Linux development environment. Not only that, but it can give you a greater understanding of any development environment which uses make or GCC. I really can't recommend this book highly enough - it's so well written and laid out that I use it regularly as a reference manual. Not only does it cover many of the useful Linux tools (and shows you how to look for the rest), it covers how the kernel works, gnu make systems, debugging and has a nice comprehensive guide to using Vim and Emacs effectively (although, sadly, it doesn't say which is best - but I think you know the answer to that).

The Art of Unix Programming by Eric S. Raymond (Link is to the full book text). Once you have all the tools, you'll want to know how to use them. Not only does this book give you the why behind the what of Linux - explaining the design and implementation mechanisms that have shaped it, it gives an excellent narrative on the history and context in which these mechanisms evolved from someone who was there at the time. Make sure you absorb his 17 basics of the Unix philosophy, don't trust yourself to touch a line of code until you do! I'll repeat them here, just in case you miss them:

Rule of Modularity: Write simple parts connected by clean interfaces.
Rule of Clarity: Clarity is better than cleverness.
Rule of Composition: Design programs to be connected to other programs.
Rule of Separation: Separate policy from mechanism; separate interfaces from engines.
Rule of Simplicity: Design for simplicity; add complexity only where you must.
Rule of Parsimony: Write a big program only when it is clear by demonstration that nothing else will do.
Rule of Transparency: Design for visibility to make inspection and debugging easier.
Rule of Robustness: Robustness is the child of transparency and simplicity.
Rule of Representation: Fold knowledge into data so program logic can be stupid and robust.
Rule of Least Surprise: In interface design, always do the least surprising thing.
Rule of Silence: When a program has nothing surprising to say, it should say nothing.
Rule of Repair: When you must fail, fail noisily and as soon as possible.
Rule of Economy: Programmer time is expensive; conserve it in preference to machine time.
Rule of Generation: Avoid hand-hacking; write programs to write programs when you can.
Rule of Optimization: Prototype before polishing. Get it working before you optimize it.
Rule of Diversity: Distrust all claims for “one true way”.
Rule of Extensibility: Design for the future, because it will be here sooner than you think.

The Art of Happiness: A Handbook for Living, by the Dalai Lama. Not a technology book, but it has to be said some of the greatest challenges I've had in my working environment is not the code or the technology, but the people.
This book gives a good grounding in principles giving you a greater knowledge of oneself and others - based around the idea that compassion for others is the main source of happiness.
Useful for those times when you need to take a deep breath and stand back....

Using the correct case

2008-06-12T18:36:00.007+01:00

I think use cases are great, but unfortunately the term 'Use Case' has one meaning if you use them as a working tool, and more often than not, another meaning if used as a buzz-word in conversation.

It's a commonly used term by developers, but personally, I often cringe when someone in a management or marketing role uses it to identify a product feature or even just a single scenario - a clear case of use case misuse!

Use cases are a bit more technical than just an idea of how something may be used - the term was first coined by Iva in the 80's, and has since been expanded on greatly by the likes of Alistair Cockburn and his peers. They are a brilliant tool for gathering requirements and consist of nothing more than simple text descriptions.

However, even though a finished use case should be a shining beacon of simplicity and clarity, getting to that point can be devilishly complex and requires more than a little experience in writing use cases - the heuristic 'practice makes perfect' certainly holds true here. The trouble is, once you begin utilising use cases it quickly becomes apparent how useful they are in many more areas of software development than just requirements gathering.

A short list of areas where use cases can add value :

Requirements analysis - Is the use case describing what you want the system to do?
Requirement traceability - Justifying the inclusion of a particular piece of design or code.
Software design - Use cases can lead straight into the design phase, e.g. by using sequence diagrams for each important use case thread.
Planning and tracking - A set of use cases breaks the software system down into more manageable chunks, which can be planned and progress measured against.
Test design and writing - A use case is also a ready-made test case.
Release management - When deciding which features to include or wait for inclusion, use cases provide a mechanism for linking features to user goals.
Change management - Especially for iterative development, use cases can, for example, aid by scoping change requests to work estimates.

I won't go into more detail here, as plenty of insightful information is available elsewhere on these topics (See Alistair Cockburn's work, for example). Suffice to say that you can drive almost the whole software process using a use-case centric approach - not that you should, of course, and the circumstances to which you are fitting a process must always be considered carefully.

Once you realise the power of well-written use cases and understand the areas in which you will use them, it seems a good idea to spend more time considering their details and form - unfortunately there is no one-shoe-fits-all approach to this as the structure of a use case is very dependent on the domain and environment in which it is being used.

One example of this is for use cases geared towards embedded systems, where I have found the assumption of everything happening in 'zero time' to be very useful - you end a use case scenario whenever you have to wait for something. This approach would be unwieldy when writing a set of high-level use cases for a user interface, on the other hand.

One can spend many hours getting a relatively small use case or set of use cases together into a usable form that may appear to be a pedantic waste of time, but to an experienced use case author is time well spent. Many lessons are learnt and dead-ends reached when going through the process of writing and using use cases 'in anger', that time spent early on considering these lessons proves very beneficial in the long run.