Programming rules/suggestions to live by

1.  whenever I see nested loops I weep for the performance of the code
2. beware anyone that says they have a quality MR/PR but can't explain what it is that their code does - 9 times out of 10 it has been written for them by AI and 9 times out of 10 it's wrong for the codebase for some reason.
3. write code with the intention that it is going to be read a lot, if it's easy to understand I can almost guarantee it'll get used unaltered forever so leave your mark.  If it's written in such a way that no one understands it then be prepared for it to be re-written or removed by the next eager developer keen to leave their mark.
4. code that's not thoroughly tested isn't a solution to a problem, it's a suggestion of a solution - test, test, test, then give it to someone that loves destroying things and let them test
5. user experience is everything, I am sure your multimillion dollar company has given you the best kit to work with, guess what, your customers might not have budget for that so make sure you see what it looks like on a less spectacular screen and less powerful set up, getting it right for the many is worth more than pandering to the few spoilt little rich kids.
6. you don't know more than the just in time compiler, stop trying to second guess what it will do and optimise your code in such a way that makes it unreadable in the hope that performance will be better, most of the time it won't be and if you can't write it to be both understandable and  performant then chances are your fundamental algorithm needs further thought.
7. understand your problem domain, this is so fundamental that I can't believe it needs to be said but oh so many times the wrong problem is addressed.  Rory Sutherland has plenty of examples of this (https://www.youtube.com/results?search_query=Rory+Sutherland) for example spend millions or even billions making a train journey about 10 mins shorter rather than spend less money and make the train journey pleasurable, the issue wasn't the time taken it was the fact the time spent was detested so address that fact.
8.  just because some code is old doesn't mean it needs to be replaced, also because some code is old doesn't mean it shouldn't be replaced.  Old code can and does still do the job perfectly adequately, don't waste your time fixing a problem that doesn't exist.  Likewise as new innovations are made around code and libraries that some old code uses can lead to the old code being sub-optimal or insecure so definitely needs replacing you need to be able to recognise both scenarios
9. the term best practice is terrible, it implies that whatever practice is being used is the pinnacle ie best, and nothing can surpass it - this is bollocks - applying the term best practice to anything causes people to stop questioning it and simply dogmatically applying it and as with any practice time moves on and what was the best solution yesterday may be the worst solution today.  I really think this term needs to be consigned to the rubbish bin.
10. you're not a guru, you're not a genius, you're not a god to developer kind.  You will forever be learning as a developer, not just from those further along in their careers but also from the most junior of developers you can imagine, the difference is most probably that a junior won't realise they're educating you as they ask what may at first seem like naive questions.

A thought about AI

 Artificial intelligence is designed to be very good at estimating what response should come next.

if you ask AI to guess what your next move in a game of tic-tac-toe will be then it will pick the next move that would get you closer to winning, but what if you want to lose, what if for some reason you really don't want to win the game then AI will always be wrong.

The context the AI has been trained on is "I want to win" and that's because its training data was awash with winning moves because the "normal" context is to win the game rather than lose.  Why would anyone want to lose a game, who cares, it could be anything, maybe you want your opponent to feel happy or maybe if you win you have to buy the beers and you're broke, whatever the reason doesn't matter you just do.

Now let me get to the actual point, for the AI to "win" when asked a question means that the answer it gives will get it one step closer to what it's training data tells it is the correct answer.  Imagine what that means if your at a Nazi convention or pro-life rally, in order for the AI to "win" when you ask should Jews be killed it would have to say "yes" the AI doesn't understand the abhorrent nature of the response all it knows is that in this crowd this answer is a "win".  Most people realise this answer is shocking, however there are some people that exist on the fringe of these groups, they may be questioning if it's right or wrong that another human being should be destroyed simply because of their religion and when they hear views like this from AI they may decide that it's genuinely OK to think like this and their own reasoning will be affected by it.  Now consider just who's in charge of the training data for most of the AI that people use, it isn't democratised it's not governed and it isn't the wisdom of crowds, it is controlled by very few very rich people so imagine what happens when you ask AI if rich people should pay more tax for example, the views within that training data can be tailored to fit the agenda of the owners - you can see where this is going, and it's going there fast!  There have already been incidents of grok spouting some next level right wing BS and when they were caught out it was blamed on a rouge developer.  Really so a single developer can influence responses that much and it gets through all the layers of testing despite being incorrect - don't make me laugh - the responses get through because that's what the boss wants! 

The above example is the extreme but plausible outcome, especially given Elon Musks reported use of recreational drugs, but there is another issue with this.

Sometimes the least popular opinion in the crowd is the right one, the general view in the past was the Earth was flat, that changed, once upon a time we believed that all celestial bodies orbited the Earth, now popular opinion is rapidly becoming more sinister and with AI amplifying this decay it's simply happening faster who knows, maybe AI will convince us that a human has no rights unless they're rich and we'll have a very few kings of the world with most of humanity dancing to their twisted tune as we hand more of our own independent thought over to the glory of AI

An idea re-visited

 A long time ago after attending a rather good talk all about java refle.x, I don't think the framework exists anymore but the idea resonated with me.

Essentially the idea was that a program could consist of any number of units that would operate concurrently, with the notable distinction between concurrency and parrallelism.

This prompted me to make the observation that protocol is king, not the language.  The reason for my sweeping statement was the polyglot nature of the refle.x framework, any of those units could be written in whatever language they chose to be.

Now I look back whilst also looking forward, I can see there are many languages and there are always many more coming up in the ranks.  I recently looked at Go, it's a nice language, but the trouble is they are all nice and when doing certain things they all look pretty, my issue is none of them achieve something radical or profound, it's just repackaging the same ol' same ol'.

I don't mean this to sound derogatory, but I am yet to find something that can be achieved in one language that cannot be achieved in another with altered syntax.

So I started to look at transpilers and realised that essentially all languages can be transpiled down to some very simple language.  This again got me thinking about that article from 2015 the polyglot of refle.x was achieved by the fact that it had a strict messaging but what's to stop all languages being transpiled to some base language.  If that were possible then language really does become irrelevant as all languages can be converted to each other.

What's more, I now think the pursuit of new languages has actually served to stagnate innovation as necessity being the route of invention I believe that people have been creating new languages in the belief that this is what they need when all they need is to have a good transpiler, they could even define their own language as long as it had rules to transpile it to the base language it really wouldn't matter.  This way a lot of the languages invented over the last 10 years would probably not have been needed and attention could've been turned to providing some innovative new paradigm rather than yet another way to write instructions that are, at some level at least, semantically the same as it's predecessor.

With this in mind I think it's time to actually put my money where my mouth is and run through a few examples whereby the language is flip-flopped without changing the intent of the actual program.

Sorting revisited with C++17

Let's start with the original Bubble Sort

I posted this back in 2014, I have no idea if it's good code or not
but it did serve its purpose which was to demonstrate how the bubble
sort works in a simple enough fashion. 

char** bsort(const unsigned int count, char** values) {
  unsigned int i, j, moveCount = 0, swapCount = 0;
  char* tmp;
  printf("sorting a list of %i values\n", count);
  for(j=count; j > 0; j--) {
    for(i = 0; i < j - 1; i++) {
      printf("\n%i:\tcomparing %s to %s ", moveCount++, values[i], values[i+1]);
      if(strcmp(values[i], values[i + 1]) > 0) {
        swap((const char**) values, (const unsigned int) i);
        ++swapCount;
      }
    }
  }
  printf("\nsorting %i items required %i swaps",
    count, swapCount);
  return values;
}

/**
 * Swap the string at values[idx] with values[idx+1]
 */
void swap(const char** values, const unsigned int idx) {
  const char* tmp = (const char*)values[idx];
  values[idx] = values[idx + 1];
  values[idx + 1] = tmp;
  printf("swap made");
}

 

So I revisited this code and made it a little more C++17, again I don't
know if this is the best way to write it, but it does seem clearer to read

std::vector<int> bubble(std::vector<int> list) {
    int moveCount = 0, swapCount = 0;
    std::cout << "sorting a list of " << list.size() << "values" << std::endl;
    for (int j = list.size() - 1; j > 0; j--) {
        for (int i = 0; i < j; i++) {
            std::cout << moveCount++ << ":\tcomparing " << list[i] <<
                " to " << list[i + 1] << std::endl;
            if (list[i] > list[i + 1]) {
                std::swap(list[i], list[i + 1]);
                swapCount++;
            }
        }
    }
    std::cout << "\nsorting " << list.size() << " items required " <<
        swapCount << " swaps" << std::endl;
    return list;
}

int main(const int argc, char** argv) {
    std::vector<int> sortlist = { 9, 8, 7, 6, 5, 4, 3, 2, 1, 0 };
    for (auto i: bubble(sortlist)) {
        std::cout << i << " ";
    }
}

Protocol is king

In the beginning computers were incredibly expensive so large corporations had them and you'd book time with them, the programs lived on these machines and the output was generally printed reports.

Then general workstations became cheap enough and workstations to do simple tasks and generally interface with the big iron the company had became normal, enter the mainframe.  The programs lived mostly on the big iron with a simple bit on the workstation.

Workstations became consumer available, woohoo, programs lived every where and worked well on their little island of a machine.

Then the internet came and broke the rules, the islands were now connected, this is a good thing.  Programs continue to live everywhere but the data is moved about and we start to see protocol's being well defined and utilised to standardise the flow between machines.

The it was possible to carry around a pretty decent computer in your pocket, that could connect to the internet.  All of a sudden the rules are now really changing there's a vast amount of data being created and stored somewhere (in the cloud, a term that I hate) but the little pocket devices are quite beefy enough to crunch everything so we still have powerful machines that want to use the data so we have protocols to transfer from the pocket to the machine via the internet.

Now the game has once again been changed, in order to make use of the huge amounts of data we must now process it in a different way, plus users expect to be able to see results on their pocket devices so now we have cloud computing and microservices taking hold, again all good things.

The language that's used to describe the programs that these programs use no longer has to be the same across all platforms involved in this well connected equation, and indeed why would it, why would a language suitable for describing the gui on a phone be every bit as useful as describing the mapreduce functions required to process BigData (another term I hate).

So here's the important bit all of these things can be written in whatever language they like, to run on whatever platform is appropriate the important bit is the protocol allowing communication between them, I love vert.x for this polyglot is amazing so let's use it and stop having petty arguments about whose language is best erlang, C#, java, scala, Haskell, lisp who cares it's all about the data and communications so once again I say the protocol is king!

When is a unit test fragile?

Overview

Recently I had the misfortune to discover that several unit tests for a piece of code I'd put in place had been butchered, when I raised concerns with the person in question I received a tirrade of hatred.  The sad thing is I didn't disagree with everything he was saying just the vile and aggressive way in which he tried to get his point across.

His argument was that he'd changed code that had no links with the code being tested, an incorrect assumption, mind you this particular guy does assume that all the code he writes works so if something isn't working it's the other code that his uses which is very frustrating!

What he had changed was the way in which a table was constructed in a database so that instead of the id being generated it was now provided by the system under test, the unit test that failed was that the date added to the database matched the date of a file.  The trouble is this test does require data to be inserted in the table whose structure has now been altered albeit only slightly.  The real issue is that an assertion was made on the exact SQL insert text being sent to the database, the solution should have been to make the assertion less strict, however the "solution" put in place was to rip out any mock objects and any assertions being tested for in the unit test and replace the mock connection with an actual database created each time the test is run.

I didn't appreciate the unprofessional commit comment either:

"fixed broken unit test and changed it from using EasyMock to work out if the date is added correctly ( which is crap and fragile as evidenced by it breaking!) to using an actual database and then reading the values back and asserting that he values are correct."

Anyway not wanting to be as arrogant as this guy I decided to investigate if he's actually right, is the test fragile?

Is the test fragile?

I discovered a nice little site that explains everything to do with testing, following the flow chart we end up at "Probably Behaviour Sensitivity" since the tests compile, don't error and some code has changed.  Now since the table construction has changed the cause is identified as
"the functionality they use to set up the pre-test state of the SUT has been modified"

Reading further in this article you can see we're approaching the "Cause: Overspecified Software"
 section, however I see the database being the separation between the SUT and an integral module therefore think the reason does still fall under the heading of behavioural sensitivity but it is sort of a grey area, so here's my reasoning which I invite people to comment on since hearing other opinions is the only way I can improve my understanding.

Test: dataWrittenMatchesFileDates

A "DatabaseWriteHandler" requires a DBConnection, DatabaseReadHandler and HierarchyBuilder thus:

As in any good object these composites are interfaces that can be freely injected into the DatabaseWriteHandler at construction time.

The injected items can be mocks with expected behaviour ie. expected calls to be made on them.  Since "Statements" are created by making a call on the DBConnection it makes sense that DBConnection should return, when asked, a new mock PreparedStatement.

So we have this arrangement

So we get a connection object to create statements that we use to write to a database.  The database itself has a structured language to communicate with it, but not only 'it' but 'any' conformant database, we should therefore be able to replace DBConnection with details for any database but leave the rest of the system untouched.  By using mock statements we're able to do this.

Since the isolation point between our system and the database is:

1. At the point statements are requested.
2. The point statments are "sent" to the database

It seems logical to ensure that what our code is sending to the database makes sense, therefore it seems logical to ensure our code sends "valid" SQL since this is the common language we're using.  We ensure the writer makes valid inserts by using:

expect( mockStatements.execute( "INSERT INTO TABLE(COLUMNS) VALUES(THE VALUES EXPECTED)" ) );

We can now test the system to ensure that it's trying to add the data we expect without the need for an actual database in the test.

THE BAD WAY

Imagine now that rather than using mocks we're using an actual database.  The DBConnection, DatabaseReadHandler are no longer mock objects and they're all talking to an actual database, there are therefore no expects on the calls being made.

Scenario #1

The database creation fails for an unknown reason, this causes the dataWrittenMatchesFileDates tests to fail when no code has been touched!

Scenario #2 - and the worst of the two

The production system changes the database that it's using and the SQL that this new database accepts is more strict than the derby database used for the tests, or the type for TIMESTAMP changes e.t.c. The tests all pass because they're using a derby database however the data isn't really added in production and the first we know of it is when customers data is broken.  Now we see why this system is bad!

Or even the very case when this shortsighted change was made, because there's no longer any checks on the data sent to the database  if the dates come back correctly then this test will pass whereas any of the other data being written might be garbage and we wouldn't know until a customer discovered it.

Conclusion

As I said at the beginning of this article I am not sure this is the right approach, in fact I am pretty sure the units under test are too large but this is the place we're at and until we at least get this in place we can't progress further forwards.

Just as there is no spoon, there is no development and testing!

I've been a developer for a long-ish time now, well over a decade at least, and something has always surprised me a little and I've just accepted it as the way it should be, but no more!

The thing that's surprised me so much is that there are two departments development and testing.  In one department programs are written and automated tests are written to test them every time a build is run, in the other department programs are used and then tests are written to check the things that have been found will automatically be picked up the next time a build is run.

Now correct me if I am wrong but isn't there a massive intersect in these primary functions?

With cross-over like this comes great oppurtunity to collaboarate, share and generally inject a bit more enthusiasm and dare I say it competition because we all want to be the best that we can be :)

Don't take my word for it 

I just read a blog and in there it states

"...your responsibility as the God of Programming is also to be the God of Testing..."

The blog does state this is how good developers deal with bad code, so does it mean that those developers that don't also try to be good testers aren't good?  Interesting thought in a world where so many people assume anyone in testing is looking for a way to become a developer!