A day in the life of: SAP Test Manager

Phil the test manager, Going Postal

I am wary of writing this, as it may be boring, and unusually for me, contains no filth, but thought it may become something like the “Postcards From” articles we all write from time to time. Here goes:

For a living, I am an SAP  Test Manager, I only do contract work, going from project to project all over the UK and once in Belgium on a 3 year project, but it was close enough to drive home at the weekend. I specialise in the management of testing SAP.  This is complex software designed to run big and medium sized companies.  In the past I have done a few councils, a big pharmaceutical company Smith & Nephew, DVLA, OCS, Premier Foods, NATS, Xerox, Jack Wills, Wedgewood, Jaguar Land Rover, Delhaize (the biggest European supermarket chain) and currently at Heineken, 1 year in to what looks like another 3 year project.  The projects I do tend to cost the company involved many millions of pounds, around 20 million being a short one, a large scale implementation will be around the 100 million mark.

As I discussed in in the article “How to be a contractor”, it is not a nice life being a contractor, though it is rather well paid.

SAP is an integrated piece of software, it can manage a company’s finances, payroll, human resources, procurement, sales, inventory, customer relationships management, real estate, legal items, contracts, retail, documentation, plant maintenance, recruitment etc. You name it can probably do it.  Also it interfaces to pretty much any other software, for example here at Heineken, SAP interfaces to a Process Control System which manages brewing from the raw materials being mixed and fermented, maturation, filtering etc.  So for example once the SAP Quality Management system is informed a batch of cider or beer is ready for packaging, it will open the pipes, fill up enormous tanks and start pouring liquid and CO2 into cans and bottles, previously having ensure bottles and cans were available through its advanced scheduling module.  It is integrated as you typically only enter data once, and this data is used in many places.  For those of you who have been in workplaces, you will have come across many different systems, all requiring your basic details such as name and work ID number, with SAP this only exists once, in one area, and it is used everywhere else, such as when you do any transaction such as creating a Purchase Order or sales order.

With something like SAP or other ERP (Enterprise Resource Package), it is not “Off the shelf”. You cannot just install and it works.  It has to be configured to do what the business needs it to do (although all too often it is configured to what the business say they want, which means changes further down the line).

So what does an SAP Test Manager do I hear you ask?

Well at the start of a project, I may be early into the team or I maybe a late arrival, as the previous test manager was not very good (frequent occurrence this). If early, once the important things are sorted out like site safety procedures, getting an ID card, finding the loos and the tea room, I begin with meeting everyone.  Very important to know who people are.  I can always tell the finance people, they are usually the most miserable lot.  Build up a rapport with everyone, be nice, be firm, and be helpful.

If there is no Test Strategy, then I will write one. If there is one, and it is not any good, or no longer practical, I will re-write.  I put objectives of testing in with lines such as:

  • That all systems, process and interfaces affected by the solution are fit for purpose and that there is no adverse impact to systems that are unchanged, achieved by ensuring the signed-off business requirements have been delivered.

I will agree the methodology which will be “Waterfall”:

Phil the test manager, Going Postal

Some of you may have heard of “Agile” testing, this is not suitable for SAP.

I will see what Test Tool they have, sometimes I am lucky and can get approval for the best one, which is “HP Quality Centre”, sometimes I have to invent one and use MS Excel.  Either way, I will write a training course for it and then teach it to all those involved in testing, together with the basic concepts of what makes a good test scripts, how to manage defects and how to execute tests.

Test Strategies are not all the same; though I can copy/paste quite a lot from previous roles. The strategy will cover how we go about testing, which typically is:

Unit testing in the development environment: This is where new programs are written and changes to existing ones made. These are called WRICEFs (Workflow, Reports, Interfaces, Conversions (of data), Enhancements and Forms (such as a Purchase Order, Sales Order etc.).  The functional consultants (FC) and developers (code writers) will do this, usually badly, as it is done in isolation of other programs.

We start off with lots of workshops where we explain what the SAP solution does, then we work out the Fit/Gaps. If a fit, in that their new solution works as they want it to, then great, it makes life easier, if there is a gap, then basically they need a WRICEF to make it work the way they want it to.  The more changes they want, the longer the project and the more expensive it costs, as it takes time to develop and configure, then test and teach it.  This is where experienced people like me try to convince the business that they need to have it the way it is working, but every business thinks they are “special”, and I suppose they are.  This gives us the signed-off by the business scope for what the project needs to do.

When a developer reckons their WRICEF masterpiece is ready, we transport (copy and move) them to the next environment which is typically the Test Environment.

Once in the Test Environment we, as in the project team can test it. The project team will consist of people from the business, or a mixture of those and professional testers, sometimes these testers are offshore, usually India.  They test pretty much in isolation, in that their first round of testing is just checking their functionality in their particular process area, with very few touchpoints to other areas.  Here we call it   Functional Acceptance Testing, (sometimes called System testing) and we typically have a Cycle 1, which is essentially making sure the programs and changes work in isolation, with few touchpoints to other areas.  If we are doing say procurement, then you have to have touchpoints, you cannot receive goods or services if you have not created a purchase order which is approved, and even before then the PO may need an approved requisition.  So we test, but it is not really integrated.  This drives out a lot of defects.  A defect could be an actual program bug, it could be a process issue, master data (The bit where you select something on a list) not available or incorrect, or the testers realising what they asked for was not what they wanted, either way, best to get these issues out of the way early.  So defect management is a key part of my job.  They often have to be chased, and I know that roughly 25% of all “fixes” do not actually solve the problem so have to go back again.

Once a defect is raised, we deal with it, or rather I make sure it is dealt with. If it is a bug in the program, it goes back the developer, who fixes it, sends it to the FC who tests it in the development area, if they agree it is OK, they arrange to transport it to the Test Environment where the tester re-tests it, and if OK closes the defect down.  How we manage our defects is the single most important part of being able to deliver the solution on time.  I continually chase developers, FC and testers to ensure they are resolved, re-tested and closed as quickly as possible.  This is why I have to be firm, but cheerful, have enormous amounts of patience, and above all be able to maintain a sense of humour.  I imagine some of them call me a well-known GP word, but getting it delivered is my be all and end all.

At some point, we are now ready for FAT Cycle 2, which is pretty much the same tests, though this time we do them in an integrated way. The idea is to ensure that data flows from A to Z.  An example, in a typical business, before purchasing anything such as raw goods, they will have a signed-off process to do it, something like this:

Create a new material which is need to be procured, , create new costs centres, create a requisition for this new material, (run a report) approve it, un-approve it to change it, (run a report) approve it, convert the requisition to a Purchase Order (PO), (run a report), receive half of it, (run a report) pay half of it, (run a report) receive the other half, pay the other half (run a report). In this way, we involve the master data team to create materials, the business person to create the requisition, an approver to approve, and unapprove, a buyer to convert the requisition to a PO, a finance person running financial commitment reports and creating new data, the accounts payable team to make payments, a storeman to receive goods.  There are many variants of this, for example can we pay before receipt, what are the tolerances if say we order 150kg but only get 142kg or 155kg, add in quality control, some items go into quarantine before being allowed to be issued.  I even think of things like the PO, is it to be printed and posted, faxed (yes this still happens) or the more normal PDFified (I made up that word) and e-mailed.  If we only have a couple of line items on the PO, it may be on one page, and what if we do a PO with 157 line items, let’s test that the PO looks OK.  Speaking of which, where are the T&C’s going to go?  The business people will know what to do, it is the FC’s job to teach them how this new stuff works, and my job to ensure it is tested robustly.  I hold regular meetings to ensure that we are indeed integrated.  Apart from the finance people, who have a very good idea of every business process, the other functional areas will only know their part, so they tend not to know what happens before it gets to them, or what happens after they have done their bit.  I help them to see the big picture.

We finish this FAT cycle, again having cleared out a lot of defects that have come up now that we have tested in an integrated way. We now transport al the WRICEFs to the Pre-Production environment.

On to the final round of testing, User Acceptance Testing (UAT). Typically this will involve people from the business who come into the project to give the final tests.  The project team who have been testing will teach them first how the new solution works, then I will teach them how to use the test tool to pass/fail tests and create defects (though there will not be many).  UAT can quite often go on for 4-6 weeks on a big project.

UAT requires a lot of planning. You have to do things in the right order, so we would typically start off with a few sessions adding and changing master data, such as materials, customers, vendors, general ledger codes, cost centres etc.  Then move into the Finance area to set up their chart of accounts and other things before we move into the functional areas which could be anything really, procurement, warehouse management, inventory, HR payroll.  We would finish off with BW (Business Warehouse) reporting, this is where data on a nightly, weekly, monthly and yearly basis is dropped into one big database (Warehouse) and various reports are run.  Ideally we also run scenarios that can last for a few days based on “A Day in the life of the Business” scenarios.

For each test cycle, there are various reports I have to create and then distribute each evening, typically contains how many tests we should have done compared to what we have done, defect status reports, highlight problem areas. I ensure as much goods news as possible is announced, but I do not hang back on the bad news.  On a major, and very expensive project, if there are problems you have to call it out as quickly as possible.

On a normal day I start off with a short, sharp, simple 10 minute meeting and what we need to do today.  I will be helping people write test scripts, set them up in the Test Lab for them, plan dates, assign testers to each test, hold their hand. I will also recommend that they need variants of the scenario, for example they may be testing in the order A-B-C but I will know that it could be done A-C-B.  I would also ensure we do negative testing so the solution should not allow you to do certain things, this often includes SOD rules (Segregation of Duties), and so if you have rights to create/change a Purchase Order, you must not have rights to approve any or be a receiver of goods or services.  At 4pm each day I run a daily defect meeting, ensuring any we have not already are reviewed and have the right priority and severity codes and agree a target fix date if possible.

As a test manager, I do not actually test. It is not me that says it works, it is the tester.  I make this very clear, with the instructions that a test passes, or it fails, there is no in-between, if it fails then create a defect.  Or as I put it, “If in doubt, get one out” (defect that is).

I mentioned defects. No matter what kind it is, we must resolve and close as early as possible in the lifecycle.  There are usually 3 key environments, Development, Testing, Pre-Production, then we go into the live system (Production).  A defect found in the development system may cost say £10 to resolve, only a developer and an FC chatting to each other is needed, once we transport (move) the new and changed elements into the Testing Environment, it will cost £100, as we now have test people, me as the test manager involved as well.  It is fixed in development, tested, then this change transported to the Test Environment, where we test it again.  If we find a defect in the pre-Production environment, then it costs a £1,000, as we go all the way back to the development area, fixed, transported, re-tested in the test Environment by the project team, the transport to Pre-Production and re-tested by the business team.  Once we go to the live (Production) system, then this could cost several thousands (in some cases millions) to fix, and there are cases where a business actually stops, costing millions, such as in the recent case in Crossrail, where poor and late testing has been a major factor of delays: Crossrail testing started late and went badly.  You may also remember Heathrow Terminal 5 failing miserably, again a large part due to poor un-robust testing.  TSB meltdown and many others.  A list of the recent top 10 disasters is here:   computerworlduk.com-Recent Failures

Before going live with a new solution there are many check-points, or Quality Gates to pass. One of the criteria to meet is that testing has been robust and we have tested everything in scope, and I prove this by linking all the requirements of the project to one or more tests, so I know we have tested everything, also the testing needs to be complete and not too many defects, or critical defects left open.  The powers that be make a decision, I usually get a say, and it is not unusual for me to say “No-Go”, and I present my risks and issues as to why we should delay.  This is usually ignored and they go-live anyway, as it is nearly always a political decision.  My head though has never rolled, others not so much so.

I consider myself very fortunate in that I really do enjoy what I do, and I love working on projects. I would never go back to a permanent role where the days grind can be far too routine for my liking.  I have also worked for many years like a Scottish ship-builder (For Cunard) to get to where I am.  Yes I could move up into being the Project Manager, but I get paid well and I enjoy it.  This will do me fine until retirement beckons in a few more years.

Maybe you could write an article too!

© Phil the test manager 2018

Audio file