When is a unit test fragile?

Overview

Recently I had the misfortune to discover that several unit tests for a piece of code I'd put in place had been butchered, when I raised concerns with the person in question I received a tirrade of hatred.  The sad thing is I didn't disagree with everything he was saying just the vile and aggressive way in which he tried to get his point across.

His argument was that he'd changed code that had no links with the code being tested, an incorrect assumption, mind you this particular guy does assume that all the code he writes works so if something isn't working it's the other code that his uses which is very frustrating!

What he had changed was the way in which a table was constructed in a database so that instead of the id being generated it was now provided by the system under test, the unit test that failed was that the date added to the database matched the date of a file.  The trouble is this test does require data to be inserted in the table whose structure has now been altered albeit only slightly.  The real issue is that an assertion was made on the exact SQL insert text being sent to the database, the solution should have been to make the assertion less strict, however the "solution" put in place was to rip out any mock objects and any assertions being tested for in the unit test and replace the mock connection with an actual database created each time the test is run.

I didn't appreciate the unprofessional commit comment either:

"fixed broken unit test and changed it from using EasyMock to work out if the date is added correctly ( which is crap and fragile as evidenced by it breaking!) to using an actual database and then reading the values back and asserting that he values are correct."

Anyway not wanting to be as arrogant as this guy I decided to investigate if he's actually right, is the test fragile?

Is the test fragile?

I discovered a nice little site that explains everything to do with testing, following the flow chart we end up at "Probably Behaviour Sensitivity" since the tests compile, don't error and some code has changed.  Now since the table construction has changed the cause is identified as
"the functionality they use to set up the pre-test state of the SUT has been modified"

Reading further in this article you can see we're approaching the "Cause: Overspecified Software"
 section, however I see the database being the separation between the SUT and an integral module therefore think the reason does still fall under the heading of behavioural sensitivity but it is sort of a grey area, so here's my reasoning which I invite people to comment on since hearing other opinions is the only way I can improve my understanding.

Test: dataWrittenMatchesFileDates

A "DatabaseWriteHandler" requires a DBConnection, DatabaseReadHandler and HierarchyBuilder thus:

As in any good object these composites are interfaces that can be freely injected into the DatabaseWriteHandler at construction time.

The injected items can be mocks with expected behaviour ie. expected calls to be made on them.  Since "Statements" are created by making a call on the DBConnection it makes sense that DBConnection should return, when asked, a new mock PreparedStatement.

So we have this arrangement

So we get a connection object to create statements that we use to write to a database.  The database itself has a structured language to communicate with it, but not only 'it' but 'any' conformant database, we should therefore be able to replace DBConnection with details for any database but leave the rest of the system untouched.  By using mock statements we're able to do this.

Since the isolation point between our system and the database is:

1. At the point statements are requested.
2. The point statments are "sent" to the database

It seems logical to ensure that what our code is sending to the database makes sense, therefore it seems logical to ensure our code sends "valid" SQL since this is the common language we're using.  We ensure the writer makes valid inserts by using:

expect( mockStatements.execute( "INSERT INTO TABLE(COLUMNS) VALUES(THE VALUES EXPECTED)" ) );

We can now test the system to ensure that it's trying to add the data we expect without the need for an actual database in the test.

THE BAD WAY

Imagine now that rather than using mocks we're using an actual database.  The DBConnection, DatabaseReadHandler are no longer mock objects and they're all talking to an actual database, there are therefore no expects on the calls being made.

Scenario #1

The database creation fails for an unknown reason, this causes the dataWrittenMatchesFileDates tests to fail when no code has been touched!

Scenario #2 - and the worst of the two

The production system changes the database that it's using and the SQL that this new database accepts is more strict than the derby database used for the tests, or the type for TIMESTAMP changes e.t.c. The tests all pass because they're using a derby database however the data isn't really added in production and the first we know of it is when customers data is broken.  Now we see why this system is bad!

Or even the very case when this shortsighted change was made, because there's no longer any checks on the data sent to the database  if the dates come back correctly then this test will pass whereas any of the other data being written might be garbage and we wouldn't know until a customer discovered it.

Conclusion

As I said at the beginning of this article I am not sure this is the right approach, in fact I am pretty sure the units under test are too large but this is the place we're at and until we at least get this in place we can't progress further forwards.

Just as there is no spoon, there is no development and testing!

I've been a developer for a long-ish time now, well over a decade at least, and something has always surprised me a little and I've just accepted it as the way it should be, but no more!

The thing that's surprised me so much is that there are two departments development and testing.  In one department programs are written and automated tests are written to test them every time a build is run, in the other department programs are used and then tests are written to check the things that have been found will automatically be picked up the next time a build is run.

Now correct me if I am wrong but isn't there a massive intersect in these primary functions?

With cross-over like this comes great oppurtunity to collaboarate, share and generally inject a bit more enthusiasm and dare I say it competition because we all want to be the best that we can be :)

Don't take my word for it 

I just read a blog and in there it states

"...your responsibility as the God of Programming is also to be the God of Testing..."

The blog does state this is how good developers deal with bad code, so does it mean that those developers that don't also try to be good testers aren't good?  Interesting thought in a world where so many people assume anyone in testing is looking for a way to become a developer!

Windows 10 - ongoing verdict

Ok first things first I actually really like Windows 10, I didn't want to since I've always considered Microshaft to be pretty damn evil and rally to the call of GNU and Linux.
So far I like it though even from the very simple upgrade with the nice soothing messages that glow occasionally to let you know what's going on.
These are minor issues I've had in using it only for a few hours, no doubt I'll add to it as I use it more.  The good thing is that all my CUDA projects and Netbeans java stuff just work :)

1. I hibernated the laptop at home, when I got to work I was unable to login because windows simply reported "you are offline please use the last login used on this device to login" I obviously tried this and the little spinning blobs appeared next to the word welcome only for me to be faced with the same "you are offline..." message about 15 seconds later.  Since I'd not been in the office until that day I didn't have any of the wifi details configured and it's only through sheer luck that I had a connected cable that was within reach.
2. The scroll is annoying firstly when using the gesture scroll of my touchpad the vertical scroll is in the wrong direction i.e. Scroll top to bottom on the touchpad would scroll the text up the screen in the same way as if I'd been using a touch screen.  Then for some reason the scroll simply stopped working altogether.
3. System freezing, I've been experiencing moments of around 4-15 seconds of the mouse not moving and no windows responding I think this is occurring because something is going on in the background, I'd like to know what's going on in the background.
4. The email viewer lets me open an email so it appears in the view pane but without using the mouse to select the view pane there's no simple way to scroll the mail using the keyboard, in fact the default is not to allow scrolling controlled by caret.
5. The default of having everything scalled by 150% is ludicrous, I paid extra for a full hd screen so of course I want to be able to use it to view my source code rather than have windows scale all the text so it looks massive, if I wanted to use a 640x480 display from the 90's I'd use my ancient netbook!

I do really like Windows 10 though and I am one of the developers that is being won back, I even used edge and despite how much I wanted to hate it I really didn't.  There were annoyances but it was much better than ieX [X being any version of that pile]