Nail-Tinted Glasses

Natural Languages

A big part of my job is thinking about how we develop and test our software - how can we increase effectiveness, decrease friction, better align our understanding of the software we deliver with the needs of the customers we deliver it to, ensuring that our solutions organization, our partners, and our pre-sales teams can successfully deliver demonstrations and solutions.

And never, not even once, have I found myself thinking "If only I had an assertion style that resembles natural languages."

Who the hell are you selling that to?

PsExec: unknown user name

Today's fun: trying to push a command to pull a file.

So, sitting on a.thisdomain.com, trying to ask b.thatdomain.com to copy a file available on a.

psexec \\b.thatdomain.com -e -u thatdomain\userB -p password cmd /c copy /Y \\a.thisdomain.com\source.file target.file

And this was exploding...

Copyright (C) 2001-2008 Mark Russinovich Sysinternals - www.sysinternals.com

Logon failure: unknown user name or bad password.
cmd exited on b.thatdomain.com with error code 1.

ANSWER: the -e switch was getting in the way. The docs say "Does not load the specified account's profile." I got a little bit of a hint that was happening: although experiment demonstrated that I was in the right (remote) directory, the set command wasn't showing the USERNAME, etc environment variables that I had expected to see.

I'm not convinced that documentation really describes what is going on, which seems superficially to be "once you've established that you have established access to the machine, operate as GUEST".

Time keeps on ticking ticking ticking

via Bad Astronomy

The official clock will be ticking tonight, counting down to the New Year. Just before midnight it will read 23 hours 59 minutes 59 seconds. But instead of clicking over to 00 hours 00 minutes 00 seconds on January 1, it will first read 11 hours 59 minutes 60 seconds on December 31!

OK, I admit that I missed this edge case.

Good problems to have

... a release engineer who keeps starving out the pipeline.

We're trying to define the requirements for our new release process, that he's going to implement for us. And the more crap we bounce back and forth, the better it gets - we haven't even gotten to start playing with it yet, and already the questions we are asking are improving dramatically. Lots of "wouldn't it be cool if X verbed Y?" "Wow, that shouldn't even be hard - we've got all the pieces, at least enough to put together the prototype".

This thing is going to be insanely awesome, and he'll be headlining a bunch of conferences showing our results, or our brains are going to be mounted in glass jars decorating the desk of the Chairman of the Human Physiology department at UT.

Floating point numbers aren't.

So what, specifically, are the values that cause this display problem? Of the 9.214*10^18 different floating point numbers (floating point on wikipedia) that Excel 2007 can store, there are 6 floating point numbers (using binary representation) between 65534.99999999995 and 65535, and 6 between 65535.99999999995 and 65536 that cause this problem. You can't actually enter these numbers into Excel directly (since Excel will round to 15 digits on entry), but any calculation returning one of those results will display this issue if the results of the calculation are displayed in a cell. All other calculation results are not affected.

David Gainer

Mixed Content

Unsecured

Secured:

Include paths in C++

An innocent question yesterday lead to a burst of research on #include paths in C++.

Ring Buffer Logging

Brian Marick blogs an entry on logging techniques, linking to his own paper on
Ring Buffers and some work by Spolsky.

Testing away the evil spirits

An interesting analogy from Phlip:

200 years ago, doctors believed evil spirits caused disease. Then some doctors started washing their hands before surgery - instead of only afterwards. They reported higher a success rate, and advertized this in terms of a "germ theory". But other doctors, because they did not "believe in" the germ theory, refused to wash their hands, even though washing couldn't possibly hurt the patient.

Some believe TDD is the best way to design. But there's no conceivable reason not to write tests before writing the code to pass the tests, even if you design some other way. But "believing in" the design theory is irrelevant.

Time Warp

60 seconds in a minute.
3600 seconds in an hour.
86400 seconds in a day.

Oops.

One of our testers turned up a bug, demonstrating that the last of those is not always true - weird things happen in the translation from seconds to calendar days when you are spanning daylight savings.

This, I decided, is the excuse I've been waiting for to pick up a copy of Calendrical Calculations by Reingold and Dershowitz.

Optimization

An entertaining essay on code optimization by Newcomer.

What's a feature

William Pietri, @ extreme programming

"If a feature doesn't affect a human, then it doesn't matter, and it's good to keep in mind what humans a feature is for."

ShellLord

Ooh, there are all sorts of interesting goodies hiding behind

SET /?

SET /P is one I've been missing for a while.

Programmer Tests

An interesting observation by Phlip

A "Unit Test" is a test whose failure implicates one (1) and only one (1) unit.

A "Programmer Test" only implicates the most recent edit.

Circle vs Ellipse

A thought watered by the ExtremeProgramming mailing list: inheritance, at least in the languages with which I am most familiar, is common behavior, rather than common properties. If your behavior model for ellipse includes "don't be a circle", then neither of these classes can inherit from the other. However, you may be able to get away with "don't act like a circle", if that act doesn't change the state of the object.

It may also help to discriminate between a circle (behavior) and the locus of the circle (properties). Changing the object may mean replacing the locus - you keep your common behavior there, and the rest is seperable. But this is just the mutable/immutable demarkation arising again; nothing new here.

Karma points

I scored a karma point today.

The software was acting up in the test lab today, and the PFY was called in to trouble shoot. His investigations lead him to the log file, where he found a report of a winsock code: 11001.

As he fired up his browser to begin a search, he noticed that the report also included a URL pointing to Appendix C: Winsock Error Reference where the error definition could be found.

Measure First

Busted again.

J2EE performance testing, looking at concurrency issues. We have a particular process that looks great at one user, but turned into a horrid mess at about four users. Very worrisome.

Some of what we were measuring was noise, introduced by a coding error in the test itself. The original version was reporting each result as it came through; the updated version, which caches the results and reports once at the end of the test has a much smoother curve. But still not good enough.

We happened to have an introspection tool running on that system, so we turned our attention to it. The tool indicated a specific hotspot, and examining the code showed that, indeed, a call needed to be refactored out of a loop. To keep things simple, we simply short circuited the call altogether, and discovered that performance did not improve significantly - the tool was now blaming another method.

The methods getting blamed did all have something in common, which was particularly worrisome since the common point is in fact the center of our library, called from everywhere. We weren't going to be getting around an architectural oversight there without much painful work.

The problems get worse... much worse, as thread counts go up. Memory is plentiful, as is CPU, so it has to be contention on some resource. There are no synchronized methods in the test rig, or in the code path under test. Something in the core? In the EJB layer itself....

OH SHIT.

It's the introspection tool. It's trying to record times and invocation counts for each call. Multiple threads, but a single bundle of statistics - so the recording of that information has to be synchronized. Because our invocation counts are so high in these tests, there is contention for the lock all the time. Yeesh.

We've actually had the answer in front of us for several days - the functions being blamed were those with the highest invocation counts; thread dumps which we had read as telling us that we were blocked in those methods had actually been telling us that we were blocked in the measuring of those methods.

Once you know the answer, you discover that all of the evidence has been pointing that way all along....

Lessons to be learned: be certain that you understand the implications of the technology you introduce into your test lab.

On Comments

"The rule I give my students is that comments are only allow on entire classes
or methods (not in the middle of code).

Under this rule, if you do anything tricky enough to require a reference or explanation, you can comment it only by making that code into a separate method (usually, private)."

-- Steven Gordon

Premature Optimization

The latest geek adventure. The browser, rather than happily rendering our web page, was instead complaining that it was taking too long to load (a valid complaint, it had been chewing for a couple of minutes already).

So I snapped off a copy of the html, and went to work....

Update: now we know what IE was looking at.

Buffer Overruns for Dummies

http://www.cse.ogi.edu/DISC/projects/immunix/StackGuard/profit.html

Programming by example

Brian Marick, on TestDrivenDevelopment

We (some of us) are engaged in a struggle to show that this:

You should drive programming with requirements that
unambiguously (though abstractly) describe what the product should do in
all situations.

can be replaced by this:

You should drive programming with a sequence of examples.

Test Eventually Development

I'm currently the only TestInfected developer on this project; I think it shows in the code, which is very tightly coupled to the environment it runs in. In some cases, I've been able to take advantage of Java's NamingContexts to splice in a mock environment.

It's getting harder - foreign libraries are being introduced, without an insulating layer of interfaces.

The solution I'm exploring right now is mocking my test target.

Compile your designs

The seed of an idea (which I doubt is new, but don't have time to research just yet)....

Anybody else find themselves playing telephone on a regular basis? The problem starts out as "the user submits an order, and we save it in the database", which everybody agrees is fairly straightforward, and much later you discover that the database is assuming one character set, where the UI is assuming another. The transfer happens via XML, which needs to be encoded then data passed to a procedure, which requires a different set of escapes. Meanwhile, QA is trying to implement a testharness, based on submissions of URL-encoded key/value pairs....

I want to be able to represent this, in a way that exposes as early as possible any incompatible assumptions.

Maybe puzzle pieces, so that you can only put them together if the pins and holes line up (what are those things called, anyway?).

Or perhaps a type library, which can be combined with a compiler of some strongly typed language, so that it screams when you try to pass ASCII 8 to a routine expecting ISO-8859-1.

The goal of automation

William Pietri, on an Extreme Programming Mailing list:

You know how word processors do that red squiggle thing when you
spell a word wrong? I want it so that my IDE does that every
time I make a mistake. But not just a spelling mistake. Every
time I type, I want my IDE to spin off a parallel universe, ship
the product to several hundred thousand users, carefully collect
and tabulate their opinions, and see if what I type is a net
improvement. If not, then the word turns red, with a helpful
little popup, "Warning! Will cause confusion in some
grandmothers, resulting in lost emails to grandchildren.
Increase font size by 2 points."

Function Points

How can you automatically assess the quality of your source code?
Guess.

What's my name

Brian Marick points out that this blog didn't actually contain the name of its author. You should be able to find it by the MoveableType notice.

Background - my name had been attached to each entry, thanks to the default template, but was ditched as I cleaned up the clutter (calendar date headers and time stamps went at the same time).

What sort of bug is that?

In my case it was a missing requirement; like much of Mark's accessibility work, it simply hadn't occured to me until somebody pointed it out. In an ideal setting, I think would be caught in requirements review, or in usability testing.

A more dangerous concern would be that it had been in the requirements, along with another that asserted each article was associated with an author. When the second requirement was dropped, the first was taken with it.

If you were using TDD in this situation, and addressed the second requirement first, you wouldn't have written a driving test for this, so when would you catch it? Same place everybody else does, I suppose.

Yeah, yeah - at some point, my blog will have an assortment of automated tests.

Beep

Laurent describes a usability horror show, then asks the wrong question: "Could a tester have seen this one coming ?"

I say wrong question, because as I grok testing, this sort of question is outside the tester's domain. Getting this sort of thing right ought to be happening upstream of testing.

Or, put differently, even if an organization did see this one coming, they might reasonbly leave the described behavior as is. If library closings are sufficiently infrequent, the utility of an improved feature wouldn't offset the cost of producing it.

Now, the interface design team ought to have a heavy shelf of usability references dropped on their heads - because preventing the user from getting work done is a Bad Thing.

I think it's reasonable, or at least practical, to take advantage of the fact that your testing team is the center of expertise on the user experience. But if you are going to be leaning on that, again they need to be farther upstream in the process. By the time product reaches testing, the questions about desired behaviors should be answered - testing is about assessing the product, not specifying it.

JBuilder: Cups and Balls

The latest amusements at work have all revolved around J2EE, and the environment that we use for developing the same. I really like being able to test my code, and in particular test it in isolation - without having to wait for a server to start, without dependencies on the database, and so forth. UnitTests, in the XP sense of the word.

At the end of the week, I was refactoring - extracting some classes into their own files. Having pulled the first one out, I reran the test - just out of habit. And the test failed - the compiler complained of duplicated classes. Ah, says I, I need to rebuild. Same error. Hmm. Perhaps it's confused. Rebuild the package? Same error.

OK, reality check. Drop to the command line, build the package with Ant. No problems. Back to JBuilder, and the conflict isn't resolved. Take a look at the dates in my classes directory. They are correct. Blow those files away and rebuild. Nope.

A hint from a colleague - rename the class directory. Hey presto! A bit of digging (suggested by colleague number two) shows that JBuilder cached a bunch of class names, and tucked them away in a subdirectory of classes. Who knew?

Code Coverage

Brian Marick: Customers don't care if the coverage tool is happy.

Adventures in Software Archaeology

package ntg ;
public class Foo
{
   public void put( Key k, Float f )
   {
      put( k, (Object)f ) ;
   }
   public void put( Key k, Object o )
   {
      // do something sensible with k and o
   }
   // remainder of the class elided
}

Why does put( Key k, Float f ) exist today?

Testing or QA

Testing finds problems in product.

QA finds problems in process.

Testers can work through a test plan, and tell you how many bugs they found in a particular build.

QA can predict, from the trend in the bug counts, when a product will finally reach the quality target.

Testers measure that product meets its requirements.

QA ensures that those same requirements include the behaviors which make testing more cost effective.

Testers tend to acquire knowledge of the user domain or the production domain.

QA tends to acquire knowledge of the development domain.

YAWS vs. Apache

Joe Armstrong's report of testing the performance of web servers YAWS and Apache under heavy load.

I like that the experiment is simple, and the description of it reflects this. Very clean. Very neat.

STS-107

"The equipment is not operating as expected, and therefore there is a danger that it can operate with even wider deviations in this unexpected and not thoroughly understood way." Richard Feynman

OK, everybody - time to break out your copies of Inviting Disaster

Compressing Spam

Kendall Willets writes of spam filtering with gzip.

This is one of the source tests I'd like to set up - though I probably picked up the idea from Ron Jeffries: track how compressed the source code becomes. The idea being that, if the code compresses a lot, you've a lot of redundancy.

Where there's smoke

Using Redundancies to Find Errors, by Xie and Engler. Found via Lambda the Ultimate.

My rough translation: turn on your warnings, then eliminate them.

At the office, I keep hoping to run a study of our source code through time, tracking different metrics in the hopes that something interesting will shake out. The source control system permits checkout by date, so it ought to be a fairly simple matter. Add it to the list of ThingsToDo.

Working with SQLPlus

Oh goody! Thanks to Thomas Eibner, a bunch of my questions on how to automate interaction with the database has been answered.

John Robbins - Debugging Applications

ISBN:0-7356-0886-5

Chapter 1: How to test code
Chapter 2: How to use ASSERT
Chapter 3: How to write a debugger.

The transition isn't quite that abrupt, but it sure felt that way the first time I read the book. The techniques described here have proven to be quite useful, and the chapters on assembler were a pretty good introduction for me.

The book originally shipped with a CD, though my copy didn't come with one. Buy the book even without the disc, but don't be afraid to ask for a price break.

The Latest in Automation Technology

So how do you automate the testing of your software?

I spent considerable time, in my youth, working with various automation technologies because - let's be honest - I'm too lazy to check my changes by hand. What tools does a former autotest expert reach for to verify his own software?

stdout and diff.