1 Tuesday, 2 May 2023 2 (10.00 am) 3 HOUSEKEEPING 4 SIR WYN WILLIAMS: Mr Beer. 5 MR BEER: Sir, before we hear the evidence of 6 Mrs Anne Chambers, I should inform you of 7 a significant development in relation to the 8 evidence of Gareth Jenkins, from whom we were 9 due to hear evidence on Thursday and Friday of 10 this week. As you know, the Inquiry served 11 a written request pursuant to Rule 9 of the 12 Inquiry Rules 2006 on Mr Jenkins seeking 13 a witness statement from him in relation to the 14 issues that arise in Phase 3 of this Inquiry on 15 31 August 2022. 16 That request contained 56 numbered 17 questions, although many of the questions 18 contained sub-questions, meaning that some 200 19 questions were in fact asked, set out over 20 17-odd pages. After resolution of the issue of 21 whether you should write to the Attorney General 22 to ask her to provide an undertaking that any 23 evidence given by or produced to the Inquiry by 24 a person would not be used against that person 25 in criminal proceedings against them was 1 1 resolved, you determined that you would not 2 presently seek such an undertaking but would 3 keep the position under review. 4 Mr Jenkins provided a witness statement to 5 the Inquiry on 6 February 2023. That witness 6 statement was short, 15 pages in length, and 7 addressed a small number of the questions that 8 had been asked of Mr Jenkins. It did not 9 address the issue of Mr Jenkins's knowledge of, 10 and involvement in, the investigation of 11 a series of bugs, errors and defects in the 12 Horizon System. That issue was the principal 13 issue raised by the Inquiry's written request. 14 In particular, it didn't address at all 15 questions 16 to 49 in the Inquiry's written 16 request, all of which went to the issue of 17 knowledge of bugs, errors and defects. 18 In relation to the questions which were not 19 addressed in the witness statement, Mr Jenkins 20 said: 21 "I wish to maintain my reliance on the 22 privilege against self-incrimination in relation 23 to certain of these questions for the time being 24 but I want to make clear my reasons for doing 25 so. As matters stand, my lawyers are working 2 1 their way through the evidence disclosed to Core 2 Participants by the Inquiry and will continue to 3 do so. I understand that, at the present time, 4 they are unable to advise me fully on the 5 privilege against self-incrimination, 6 particularly in relation to Phase 4, because 7 their review is ongoing and because the existing 8 evidence disclosed in relation to Phase 4 is 9 limited. I wish, therefore, to ensure that it 10 is understood that I will continue to keep the 11 question of whether I waive my rights under 12 consideration. 13 "My lawyers have undertaken to the Inquiry 14 to notify it in good time when they are in 15 a position to advise me fully. I will continue 16 to prepare for the Inquiry so as to ensure that 17 when my lawyers are in a position to advise me, 18 that is not a cause for delay." 19 On Friday of last week, the Inquiry received 20 a letter from Mr Jenkins' solicitors stating 21 that they and Mr Jenkins had continued to review 22 the material disclosed by the Inquiry, that 23 although disclosure was a rolling process, and 24 disclosure in relation to Phase 4 in particular 25 remained very much ongoing, they had a greater 3 1 understanding of the evidence that relates to 2 Mr Jenkins in Phase 3 and that, having had the 3 opportunity to consider the disclosure, 4 Mr Jenkins wished to indicate his willingness to 5 provide the Inquiry with a response to the 6 request that was served on him on 31 August 2022 7 in a statement which addresses all of the 8 questions asked in that request. 9 This is plainly an important development. 10 In the light of it, you decided that Mr Jenkins 11 should not give evidence on Thursday and Friday 12 of this week for three main reasons: first, that 13 it was in the interests of the Inquiry, of the 14 Core Participants and of the public that the 15 Inquiry should receive the widest range of 16 evidence possible, in particular from a central 17 figure such as Mr Jenkins. It was therefore 18 important to obtain this written evidence from 19 Mr Jenkins. 20 Secondly, it would have been unfair on Core 21 Participants, and in particular the 22 subpostmaster Core Participants, for Mr Jenkins 23 to have entered the witness box on Thursday and 24 Friday of this week without him having 25 previously provided a written account of what he 4 1 proposed to say about errors, bugs and defects 2 in the Horizon System. 3 It's important that all Core Participants 4 have the opportunity properly to prepare for 5 witnesses and giving evidence without previously 6 having made a statement on the issues of 7 substance would have undermined that 8 preparation. 9 Thirdly, if Mr Jenkins had been permitted to 10 give evidence without having made a statement it 11 would have involved treating Mr Jenkins 12 differently from other witnesses because it may 13 have allowed Mr Jenkins free rein in his oral 14 evidence to say what he wished without having 15 previously reduced his account to writing. 16 In the light of the time that it's needed to 17 produce that statement, disclose it to the 18 Inquiry, the Inquiry to assess whether it 19 properly engages with the questions that we've 20 asked and to disclose that statement in good 21 time before Mr Jenkins gives his oral evidence, 22 it will not be possible for him to give his 23 evidence in the Phase 3 oral evidence sessions 24 that remain in this part of the Inquiry, ie in 25 the next three weeks. 5 1 We will instead interpose Mr Jenkins' 2 Phase 3 evidence at a convenient moment in 3 Phase 4 of the Inquiry but before the summer 4 break. 5 Fair notice of this evidence will be given 6 on the Inquiry's website in the usual way. 7 Mr Jenkins will, in any event, be required to 8 make a written statement about the issues which 9 arise in Phase 4 of the Inquiry, in particular 10 his role in criminal and civil proceedings taken 11 against subpostmasters, and return for a second 12 time to give oral evidence about such matters in 13 Phase 4 of the Inquiry, most likely after the 14 summer break. 15 It follows that the Inquiry will not be 16 sitting this Thursday and Friday, but the 17 remainder of Phase 3 will continue in accordance 18 with the timetable next week. 19 SIR WYN WILLIAMS: Mr Beer, thank you very much for 20 explaining to everyone the decision which I took 21 last Friday and the reasons for it. 22 MR BEER: Anne Chambers, please. 23 ANNE OLIVIA CHAMBERS (affirmed) 24 Questioned by MR BEER 25 MR BEER: Good morning, Ms Chambers, my name is 6 1 Jason Beer and I ask questions on behalf of the 2 Inquiry. Can you give the Inquiry your full 3 name, please. 4 A. Anne Olivia Chambers. 5 MR BEER: Before I ask you any further questions, 6 the Chairman has a statement that he wishes to 7 make? 8 SIR WYN WILLIAMS: Mrs Chambers, you will have 9 already heard Mr Beer explain that Mr Gareth 10 Jenkins has, in the past at least, relied upon 11 what we call the principle of privilege against 12 self-incrimination. 13 Before we go any further, I should tell you 14 that a witness at a public inquiry has a right 15 to decline to answer questions if there is 16 a risk that the answer to the question would 17 incriminate the witness. 18 In short, that is the principle of privilege 19 against self-incrimination. 20 I remind you of that principle before you 21 give your evidence. I must tell you that it is 22 for you to make clear to me if you wish to rely 23 upon the privilege. If, therefore, questions 24 are put to you by Mr Beer or any other counsel 25 or by me, which you do not wish to answer on the 7 1 grounds that to answer the question might 2 incriminate you, you must tell me immediately. 3 At that point, I will consider your 4 objection and determine whether or not to uphold 5 it. I understand that you are represented by 6 a barrister and solicitors today. No doubt if 7 the issue relating to self-incrimination arises, 8 they will assist you. 9 If, at any stage of the questioning, you 10 wish to speak to your lawyers about that 11 principle, you must tell me immediately, and 12 I will facilitate that. 13 Do you understand all that, Mrs Chambers? 14 A. Yes, I do. 15 SIR WYN WILLIAMS: Thank you very much. 16 MR BEER: Thank you very much, Mrs Chambers, for 17 coming to give evidence to the Inquiry today and 18 thank you for providing a long and detailed 19 witness statement to the Inquiry to assist us in 20 our work. We're very grateful to you. 21 You should have in front of you a hard copy 22 of that witness statement. It's in tab A1 of 23 that bundle. It's in your name and it's dated 24 15 November 2022. Do you have that witness 25 statement? 8 1 A. Yes, I have. 2 Q. If you turn to page 63, please, is that your 3 signature? 4 A. Yes, it is. 5 Q. Are the contents of that witness statement true 6 to the best of your knowledge and belief? 7 A. Yes. 8 Q. For the purposes of the transcript, the URN is 9 WITN01700100. There's no need to display that. 10 Mrs Chambers, I'm going to ask you some 11 questions today and tomorrow about the issues 12 that arise in Phase 3 of the Inquiry. We're 13 going to ask you to return at a later stage in 14 the Inquiry to answer questions that arise in 15 Phase 4 of the Inquiry, in particular about the 16 role that you undertook investigations of 17 subpostmasters and giving evidence in 18 proceedings bought against them and, in still 19 further particular, the evidence that you gave 20 in the Lee Castleton case. Do you understand? 21 A. Yes. 22 Q. All Core Participants should respect that divide 23 or division if and when they confirm to ask you 24 questions tomorrow and I and the Chair will be 25 keeping a watchful eye to ensure that that 9 1 process is respected. 2 A. Yeah. 3 Q. I should make it clear that I know that you have 4 spent many hours preparing to give evidence 5 today and have been diligently looking at some 6 of the material that the Inquiry has sent you. 7 You were, I think, provided with a considerable 8 volume of material at the time you were asked to 9 prepare a witness statement on 6 October 2022. 10 You've been provided with much more material 11 since then, including in the last two weeks, and 12 I think we're dealing with such a large volume 13 of material that you couldn't have hoped to have 14 read all of it and digested it; is that right? 15 A. That's right, yes. 16 Q. If at any stage I show you a document with which 17 you're not familiar that hasn't been part of 18 your preparation, then just say so. 19 A. Yes. 20 Q. Can I start, please, with your career, 21 qualifications and experience. Do you have any 22 professional qualifications that are relevant to 23 the issues that we're going to discuss in your 24 evidence? 25 A. I have a degree in statistics and mathematics, 10 1 which I think shows that I've got a reasonable 2 sense of numeracy, and so on, which I think is 3 relevant. 4 Q. Probably more than that, a degree in statistics 5 with pure maths, I think -- 6 A. Yes. 7 Q. -- obtained from the University College of Wales 8 in 1978; is that right? 9 A. That's correct. 10 Q. Your first employment, I think after graduation, 11 was with a company called Dataskil; is that 12 right? 13 A. Yes, it was part of ICL. It was ICL's software 14 house. 15 Q. Before you joined Dataskil, did you have any 16 formal qualifications in computing? 17 A. No, I'd done a couple of computing modules as 18 part of my degree but I hadn't done a great deal 19 of computing. Like most people at that time, 20 I learnt on the job. 21 Q. You say in your statement that at Dataskil you 22 coded and supported various software packages; 23 is that right? 24 A. That's correct, yes. 25 Q. Was that software concerning databases and 11 1 statistical processes? 2 A. Yes, it was. 3 Q. To what extent is that the same or different 4 from what you went on to do at ICL and then 5 Fujitsu? 6 A. Um, it was quite similar in a lot of ways. 7 I mean, I didn't actually leave Dataskil, as 8 such, it just merged into -- it was just 9 subsumed into ICL. So I then carried on working 10 on the same type of things. As the years went 11 by, I did less coding, and so on, and I found 12 I enjoyed the support work. 13 Q. You tell us in your statement that from 1986 you 14 worked from home and were working part time as 15 a software diagnostician; is that right? 16 A. Yes, that's right. I had first one and then two 17 children and there was a group of people within 18 ICL who were all home based, our management were 19 mostly home based, as well, and, with what now 20 seems like very prehistoric comms and equipment, 21 we could actually do our job remarkably well. 22 Q. What does a software diagnostician do? 23 A. Some -- a user of a piece of software that you 24 are supporting somewhere in the world discovers 25 there is a problem with it, at the time they 12 1 would sort of fill in a paper form saying what 2 the problem was that they'd encountered, and 3 then it was up to me to look at evidence 4 provided. Sometimes it was great heaps of dumps 5 that you had to sort of work out how to read 6 your way through, to try to work out what had 7 gone wrong, identify the problem and, at that 8 time, work out how to fix it, usually by 9 applying something that was called a patch, 10 rather than actually changing the code. 11 Q. Just stop there. There's a noise that I don't 12 know whether other people can hear. It seems to 13 be coming through the air-conditioning vents. 14 Is it me or -- 15 SIR WYN WILLIAMS: No, it's not you, Mr Beer. It's 16 certainly very noticeable where I'm sitting and, 17 for a moment, I thought we were outside in 18 a storm. 19 If this is troublesome for you giving 20 evidence, Mrs Chambers, we'll try and do 21 something about it. Is it bothering you? 22 A. I think I can ignore it, as long as it doesn't 23 get too much louder. 24 SIR WYN WILLIAMS: Well, the usher is going to try 25 to make some investigations but we'll carry on 13 1 for the moment and see where we go. 2 MR BEER: Yes. 3 In the answer that you just gave, you said 4 that someone somewhere around the world had 5 found a problem with the system. Would 6 a software diagnostician always be responsive to 7 somebody else finding a problem or would they, 8 in some cases, proactively look for code faults, 9 errors or bugs in the software? 10 A. In these particular instances that I was 11 supporting at that time, it was dependent on 12 somebody reporting the problem to us. 13 Q. So it was always reactive? 14 A. Yeah, it was very reactive. 15 Q. So you would investigate error reports -- 16 A. Yes. 17 Q. -- is that a good way of describing them -- 18 filed by users? 19 A. Yes. 20 Q. Would a fair way of describing what you did was 21 produce code fixes? 22 A. Yes, produce patches to the code, yeah. 23 Q. That changed when you went on later to work for 24 the SSC. You did the former but didn't produce 25 code fixes; is that right? 14 1 A. That's right, yes. I had been doing -- it was 2 usually called fourth line support, so people 3 would have already checked for published known 4 errors, and things like that, although sometimes 5 things got through. 6 Once I moved on to the Post Office work, 7 I was third line support, where we were doing 8 a great deal of the investigation but we would 9 not actually be fixing the problem ourselves, 10 and not necessarily finding the root cause of 11 the problem ourselves. 12 Q. Can I look at this early stage, before October 13 2000, and your early involvement in Horizon. 14 You tell us in your witness statement, 15 paragraph 3 -- no need to turn it up -- that 16 from 1997 you did some coding and support in 17 respect of part of the new Pathway system for 18 the Post Office; is that right? 19 A. Yes, that's right. 20 Q. The purpose of the questions I'm about to ask 21 you is to establish what you did learn if you 22 did learn things as a result of your early 23 involvement in the development of what became 24 Horizon; do you understand? 25 A. Yes. 15 1 Q. In which team were you working from 1997 onwards 2 until October 2000? 3 A. I was still working for my offsite team. 4 I think we were called ICL Systems at that 5 point. I was still doing -- supporting other 6 systems as well as working on the -- on this 7 particular niche area of Post Office. 8 Q. Who was your manager or supervisor at that time? 9 A. I think my manager was Sheila Powell. But 10 again, as I say, she wasn't Post Office. She 11 wasn't part of the ICL Pathway team. This was 12 still part of this separate structure. 13 Q. Were you still home based at this time -- 14 A. Yes, I was -- 15 Q. -- or had you gone back into the office? 16 A. -- I was still home based. 17 Q. Did you ever go into the office in this period, 18 1997 until October 2000? 19 A. Um, as regarding the Pathway work in that 20 period, I remember going to Feltham, I think, 21 once and I remember giving a couple of training 22 sessions in different locations. 23 Q. Concerning the Post Office Benefits Agency 24 project? 25 A. Yes. Can I just explain this area that I was 16 1 working on. It was the transfer of files out of 2 the Benefits Agency. There was some 3 transformation done, so that they could then be 4 fed into the back end of the Pathway system. 5 Q. I was going to ask you about that because your 6 witness statement suggests you were involved in 7 the Benefits Agency side of the coding? 8 A. Yes. Well, it was a funny sort of lump in the 9 middle but the Benefits Agency side of it used 10 VME which was ICL's proprietary operating 11 system, or one of them, and that was what I was 12 particularly an expert in if you like. 13 Q. Was it restricted to that? 14 A. And that was -- my only involvement was the 15 transformation that was done on the data in 16 these files and the transfer into the back end 17 of the Post Office system. But I didn't know 18 any more about what happened to those files and 19 I knew nothing about the counter end of the Post 20 Office system at that time. 21 Q. How long did this work on the Pathway system, by 22 you, last? 23 A. I still had some involvement by the time 24 I joined the SSC because, by that time, there 25 was only about one file left that was being 17 1 processed. I think it was to do with Child 2 Benefit -- I think that's right -- and I was 3 still providing some level of support, for that. 4 Q. So from '97 until October 2000, all be it doing 5 other jobs for ICL -- 6 A. Yeah. 7 Q. -- involved in the Pathway system in the way 8 that you explained? 9 A. I had some involvement yes. Certainly by '99 10 there wasn't -- it was just sort of support of 11 this one file that was being transferred. So 12 I was doing quite a few other things then, as 13 well. 14 Q. The Inquiry has heard evidence that there were 15 systemic design problems with the development of 16 Horizon from the outset, including in respect of 17 the integration of Pathway and benefits and Post 18 Office systems, and has heard evidence of 19 problems with the requirement specifications for 20 the project. In general terms, in that 21 three-year period, were you aware of such 22 problems at the time? 23 A. Only in as much as the vast majority of it was 24 canned and the relationship obviously was -- 25 would appear then not to have been particularly 18 1 good. But, no, I had no direct knowledge of 2 that. 3 Q. What was your understanding of the reasons that 4 the majority of it, the project, was canned? 5 A. Just that the different bodies involved couldn't 6 work out properly how they wanted it all to work 7 together. I don't know. I wasn't involved in 8 any of the political side of it, if you like. 9 I was -- have always been very much technical 10 and not involved in the more political and 11 perhaps commercial aspects. 12 Q. But you picked up that it was a problem with the 13 parties working together that was the problem; 14 is that right? 15 A. I think that was the impression I got at the 16 time. 17 Q. Did you pick up anything else, that it was 18 a problem with the system or the quality of the 19 Horizon System? 20 A. No, because I don't think -- I mean, I don't 21 think I knew particularly what happened to the 22 data that was in the files that we were passing 23 on. So ... 24 Q. That's one way that you may have learned, 25 ie feedback on the issues upon which you were 19 1 working but I'm talking about talking to your 2 colleagues, receiving emails, attending 3 meetings -- 4 A. I don't recall any of that. 5 Q. So was the extent of your knowledge that the 6 majority of the project was canned, that it was 7 to do with a relationship problem rather than 8 technical issues with Horizon itself? 9 A. I think that was the impression I got at the 10 time, yes. 11 Q. Can we look, please, at POL00091901. That 12 should come up on the screen for you. You 13 should see that this is an "Operational Review 14 of the CAPS", "CAPS" meaning Customer Accounting 15 and Payment System, yes? 16 A. I can't remember if that's what it stood for, 17 but, yes, it could well have done. 18 Q. Take it from me. I mean, if you want to look 19 ahead to page 9 of the document and at the foot 20 of the page. "CAPS" in this sense, Customer 21 Accounting and Payment System; can you see that? 22 A. Yes, I can see that. 23 Q. If we can just go back to page 1, please. So 24 "Operational Review of the CAPS/Pathway 25 Interface". 20 1 A. Yeah. 2 Q. It's dated 26 February 1998. 3 A. Yeah. 4 Q. If you look at the distribution list, I think 5 you're on it. 6 A. Yes. 7 Q. The "O" being Olivia? 8 A. Yeah. 9 Q. We can see that on the right-hand side, second 10 entry. 11 Can we go, please, to page 61 of the 12 document and look at paragraph 6.3.8.2, the 13 second paragraph down. The document reads: 14 "Anne Chambers (ICL Systems) has expressed 15 doubt that NEXT-ACTION-TIME can actually be 16 explained convincingly as it is and that CAPS 17 and Pathway should get together to produce 18 a proper definition of the requirement. 19 A definitive specification would provide a basis 20 for reviewing the current implementation as well 21 as a document that would be useful in supporting 22 the Live Service." 23 So you're reported in this document as 24 saying that CAPS and Pathway -- is that 25 essentially the Benefits Agency part of the 21 1 programme and ICL -- should get together? 2 A. Yes, I mean, we've got a file or files that are 3 being transformed and passed over and it's 4 obviously important that both sides agree 5 exactly the definition of the data that one is 6 sending and one is receiving. 7 Q. So would this be an example of there being some 8 doubt or ambiguity, whoever's fault it was? 9 A. Yes, it didn't seem as if this had been properly 10 defined because -- I can't remember, I think by 11 the time I got involved the code had already 12 been written, but I think this was a particular 13 field that we were having some problem making 14 sense of exactly what it was meant to contain, 15 and the assumption had been made that it should 16 be like that but it wasn't clear that that was 17 correct. 18 Q. Presumably you've got no memory of this now? 19 A. No -- 20 Q. No. 21 A. -- very, very vague memories and I certainly 22 couldn't tell you how it was and how I thought 23 it should be or anything like that. 24 Q. No, but the issue of there not being a proper 25 definition of the requirement, that's the 22 1 customer's requirement, yes? 2 A. Yeah. 3 Q. Can you recall whether that was, in this sort of 4 three-year period, something that happened often 5 or a recurring issue? 6 A. I don't remember that, no. I think -- I mean, 7 it would only be at the point that these files 8 actually started being -- using the code that 9 had been produced and once they're actually 10 being processed, um, that then you'd have some 11 debate about whether these things were actually 12 as both parties had understood. 13 Q. Can we go forward to page 75, please, and look 14 at, under the heading "Question 3: Adequate 15 Resilience", 7.1, "Statement of the Question", 16 the question is: 17 "Is the operation of the interface 18 adequately resilient in terms of its ability to 19 recover from failure states?" 20 Then if we go down to 7.3.1.1, "Description 21 of the Issue": 22 "In order to pass a file to CAS(VME), the 23 CAPS software writes the file to a CAPS Out Tray 24 and passes a File Notification ... to CAS(VME) 25 via XPERT. Certain problems in the use of XPERT 23 1 have resulted in ..." 2 Then there's a description: 3 "When resolving such problems, it has proven 4 very useful to be able to pass a File 5 Notification to CAS(VME) manually. This has 6 been done by using CAS_MEND, which was provided 7 informally by Anne Chambers (ICL Systems), 8 a member of the CAS(VME) development team. It 9 is anticipated that similar problems will be 10 encountered in the future and that the same SCL 11 procedure or something very like it would prove 12 equally useful." 13 There is no need for us to explore the 14 technical details of what is being spoken about 15 there but is what is being described the fact 16 that you had yourself developed and provided 17 a workaround utility? 18 A. Yes, I think it was possibly something that we'd 19 had done for our own testing. Obviously, when 20 you're testing things you have to pretend that 21 things are happening, to some extent, and it 22 turned out that, you know, there was some sort 23 of a requirement for this. That first paragraph 24 is all stuff that was very much in the Benefits 25 Agency camp. 24 1 Q. Yes. If we can go over the page, please -- to 2 77, sorry -- the report continues: 3 "However, it was pointed out that the 4 condition that gave rise to actions, in which 5 this utility was used, was an error condition 6 and not normal processing. Such an error 7 condition should be investigated and understood, 8 the current situation recovered, and the root 9 cause eliminated to prevent repetition. 10 Therefore, occasion for the use of the utility 11 should be very rare indeed. 12 "It was further pointed out that the use of 13 the utility affects audit data for CAS(VME). 14 The CAS ICL is updated with information from 15 a File Notification specially created on the CAS 16 side of the interface. That information is 17 passed forward to the ICMF. It was queried 18 whether, in principle, a utility of this nature 19 should be provided by Pathway as a standard 20 component of the CAS(VME) product, since it 21 compromises the integrity of the audit trail and 22 its use could provide an embarrassment to 23 Pathway in any contractual dispute. 24 "A compromise position was formulated. It 25 was recognised, by Pathway and CAPS, that the 25 1 interface is not yet fully stable and that 2 problems of the kind described may be 3 encountered in the future. Such problems 4 require that there should be a means of 5 recovery." 6 Is what we're seeing described here evidence 7 that fixes designed to address errors could 8 themselves impact -- I'll just stop and start 9 the question again. 10 Is what we see here evidence that fixes 11 designed to address errors could themselves 12 impact on audit trails for the systems being 13 developed? 14 A. In theory, yes, they could and you wouldn't be 15 using something like this unless you absolutely 16 had to. It shouldn't be a standard way of doing 17 things if it then couldn't be audited or 18 whatever. 19 Q. Why was that? Because this was sort of 20 an ad hoc fix developed by you? 21 A. It was -- it wasn't developed as a fix. It was 22 something that existed that we could use, and 23 I think it was initially for -- possibly for 24 testing. I don't think it was called CAS_MEND 25 originally but it was something so we could put 26 1 an entry in a table to say "Look, here's 2 a file", to get over this error condition but 3 that shouldn't have ever been a long-term fix to 4 the problem. 5 But sometimes if you had to choose between 6 doing something like that that would then have 7 to be documented as an unscheduled sort of 8 a change, if you could either do that or a whole 9 day's benefit -- Child Benefit payments couldn't 10 go through, then that's something that has to be 11 weighed up against each other. 12 Q. Why would it compromise the integrity of the 13 audit trail? 14 A. Well, that's what it's suggesting here, isn't 15 it? 16 Q. Yes, but why would it compromise the integrity 17 of the audit trail? 18 A. I cannot now remember enough about the details 19 to say. 20 Q. Would the integrity of the audit trail be 21 an important principle to maintain? 22 A. Yes, it always is. 23 Q. Why is that? 24 A. Because then if there are questions afterwards 25 about something, you need to be certain that you 27 1 have got a proper record of what was done. 2 Q. You will see that it mentions the report -- the 3 fix compromising: 4 "... the integrity of the audit trail and 5 its use could prove an embarrassment to Pathway 6 in any contractual dispute." 7 Can you assist as to why the use of what 8 I've described as the fix could prove 9 an embarrassment to Pathway in a contractual 10 dispute. 11 A. I cannot remember enough about all of this to be 12 certain but there was obviously record kept of 13 the files that had been received and the sizes 14 and the dates and all that sort stuff, which 15 would have been, I presume, part of the audit 16 trail, and I can't be certain now but from what 17 this is saying, it suggests that however we were 18 notifying the system that there wasn't another 19 new file had come in, but the notification 20 wasn't arriving in the normal way. That can't 21 have been recorded in the normal way, I presume. 22 Q. Can you help us more broadly -- that document 23 can come down, thank you. Did you hear any word 24 amongst your colleagues or chatter or similar, 25 about how the Pathway Project had gone for 28 1 Fujitsu by the time you joined the SSC in 2 October 2000? 3 A. Um ... 4 Q. What was the word on the street? 5 A. I knew that at that point the rollout was going 6 ahead. I think when I started there were about 7 25 per cent of Post Office branches had got the 8 new Horizon System and so, obviously, it was 9 ramping up very rapidly and I certainly -- 10 I can't remember. I don't recall anybody saying 11 it was so dreadful enough to make me feel I did 12 not want to be a part of it. 13 Q. What about something less than that? Were you 14 told, for example, when you were joining the SSC 15 or beforehand, that a range of problems and 16 issues had been encountered in the design, build 17 and rollout of Horizon? 18 A. No. I mean, you would expect there to be 19 a certain level of problems and they obviously 20 needed more people in SSC. There was quite 21 a lot of recruitment going on which, by the 22 nature -- you know, that is group of people who 23 are providing support. So there was obviously 24 a need to have that group and to build it up. 25 But I didn't feel -- I wasn't aware of anything, 29 1 you know, "Oh, this is so bad we've got to have 2 so many extra people on it". It was, you know, 3 "This is an exciting new project, it's at last, 4 after many years of preparation, it's up and 5 running, great, let's keep it going and make 6 sure it's all working well and doing its job". 7 Q. Were you told that the Benefits Agency had 8 pulled out because of concerns over the 9 integrity of the data that Horizon produced? 10 A. No. 11 Q. When you joined the SSC, did you therefore think 12 you were to be providing support for a good and 13 properly functioning system? 14 A. I anticipated that it would have problems, 15 otherwise there would have been no job for me to 16 do there. 17 Q. Yes, that doesn't really answer the question, 18 Mrs Chambers. 19 A. No, I don't think anybody in -- who's doing 20 computer support work ever sort of -- you know, 21 the whole purpose of our existence was to get on 22 top of any problems that there were, and this is 23 probably going to come out wrong but, in some 24 ways, the whole -- not exactly enjoyment of the 25 job but what you're there for is to sort out 30 1 these problems so you do anticipate that, yes, 2 there will be things to get your teeth into, if 3 you like. 4 Q. But were you approaching this that this was just 5 another project in a line and that there was 6 nothing -- you weren't walking into a project 7 that had had a particularly problematic birth? 8 A. No, that was not how I saw it. I was -- for me, 9 personally, I was ready for a change and it was 10 quite a big change because, at that point, 11 I went back on site, I hadn't actually had to 12 work with other people very much for 15 years, 13 and I was moving from being very technical, 14 doing a fourth line support job, to being less 15 technical. I was also moving away from 16 supporting things on VME, which was my main 17 technical speciality, to something that was 18 using -- well, it wasn't VME-based at all, apart 19 from this one file that was left. 20 Q. So you joined the SSC. What did you understand 21 SSC to stand for? 22 A. Err -- 23 Q. We've had three variants of it. 24 A. Yeah, System Support Centre. 25 Q. Thank you. You joined in October 2000? 31 1 A. Yes. 2 Q. You stayed there for the rest of your career 3 with Fujitsu? 4 A. Yes. 5 Q. What was your job title when you first joined 6 the SSC in October 2000? 7 A. I think it was system specialist but I cannot be 8 entirely sure. Job titles did change here and 9 there. They didn't necessarily -- they were 10 usually sort of fairly vague but I think I was 11 a systems specialist. 12 Q. Were you now working full time when you -- 13 A. Yes -- 14 Q. -- moved to the SSC? 15 A. -- I had been -- I think I'd been working 16 30 hours plus quite a lot extra from home and so 17 now I was officially 37 hours a week. 18 Q. Did you now work in the office? 19 A. Yes, I did. 20 Q. Was that in Bracknell? 21 A. Yes, it was. 22 Q. When you joined the SSC who was your manager or 23 supervisor? 24 A. Mik Peach. 25 Q. But he didn't remain your manager for the 32 1 entirety of the 16 years that you worked in the 2 SSC; that's right, isn't it? 3 A. That's right, yes. 4 Q. But when he worked there, to whom did he report? 5 A. I can't remember. It was different people at 6 different times. 7 Q. Did you report to a director? 8 A. I don't -- oh. I don't think so, no. I think 9 there was several layers but I -- again, I -- my 10 interests were technical and not particularly in 11 the structure of the organisation. 12 Q. Did you ever report to the person above Mik 13 Peach or did you always report into Mik Peach? 14 A. I always reported into Mik or his successors. 15 Q. After Mik Peach left, you say in your statement 16 in about 2010 -- just for the transcript, 17 Mr Peach says it was in September 2009 -- you 18 say that he was replaced by Tony Little for 19 a few months? 20 A. That's the name I think I remember. 21 Q. And then by Steve Parker? 22 A. Yes. 23 Q. Did Steve Parker remain your manager until you 24 left in 2016? 25 A. Yes, he did, although we had team leaders as 33 1 well, so we did have an extra layer. 2 Q. Those team leaders, were they introduced by 3 Mr Parker? 4 A. Yes, from the existing team. 5 Q. Were there four teams? 6 A. I think there were four. I can't be quite 7 certain. 8 Q. Can you remember what the division within the 9 teams was -- between the teams? 10 A. They were just sort of purely for 11 administration, it wasn't for -- it wasn't sort 12 of one team supporting one particular area or 13 anything like that. 14 Q. So there wasn't specialism -- 15 A. No. 16 Q. -- team A, specialism team B? 17 A. No. Except possibly -- at some point, the 18 Reference Data Team sort of merged into SSC, and 19 I can't remember now if they stayed as more or 20 less a separate team or if they ended up 21 reporting to different team leaders. 22 Q. So there was sort of a mixed economy of skills 23 within your team -- 24 A. Yes. 25 Q. -- even though, as we're going to discover in 34 1 a moment, you specialised? 2 A. Yes. 3 Q. So to going back to the beginning then in 4 October 2000, there was essentially a flat 5 structure with one manager, Mik Peach? 6 A. Yes. 7 Q. How many people worked in the SSC at that time 8 when you joined? 9 A. I think it was around 25 but I can't be certain 10 of that. 11 Q. Were they all what I'm going to call 12 diagnosticians? 13 A. Yes, I think that's true to say. 14 Q. There was an administrator as well on top; is 15 that right? 16 A. Yes, there was an administrator and then, at one 17 point, an administrator's assistant as well, and 18 then no administrator. 19 Q. What was the function of the administrator? 20 A. Um, order the stationery; answer the door, 21 because it was a secure unit so people had to be 22 let in; answer the phone; and monitor the stack 23 of service tickets, peak calls coming in and 24 allocating them to members of the team. 25 Q. So they had a role in allocation of the PinICLs 35 1 and then the PEAKs? 2 A. They allocated them, yes, and also she'd look at 3 any KELs that had been mentioned to see if it 4 looked at a fairly superficial level, if it 5 looked as if it was the right one. If there was 6 absolutely no information on the call giving any 7 clue as to what the problem really was, then she 8 might return the call to second line and ask 9 them to get some more information. 10 Q. Was that person the same throughout this period? 11 Was it Barbara Longley? 12 A. It was Barbara Longley until she retired and 13 I cannot quite recall when that was. I think it 14 must have been before 2008, I think, or 2009. 15 Q. How did she determine to whom to allocate 16 a PinICL or PEAK? 17 A. Um, partly what sort of area it was, um, 18 somebody who hadn't got any calls on their stack 19 already, obviously -- 20 Q. So workload? 21 A. -- it would be a -- workload, interest. 22 Sometimes somebody -- because we could all see 23 this stack of calls -- sometimes somebody would 24 say, "Oh, I'd like that one", or, you know, 25 somebody might point out to her that it was 36 1 relevant to something else that had already come 2 in. 3 Q. Were there specialisms within the 25 of you? 4 A. Yes. We all -- everybody seemed to gravitate to 5 different areas. 6 Q. Was that it, the force of gravity, ie personal 7 interest, or was it anything more formal than 8 that? 9 A. Um, it was partly what people's backgrounds were 10 when they came in. Um, if Mik felt there was 11 a bit of a gap somewhere and not enough people 12 specialising in one particular area he'd 13 obviously get somebody in and say "Right, you 14 know, you're doing this". 15 Q. It's right, though, that you were each expected 16 to handle any type of -- 17 A. Yeah. 18 Q. -- ticket, if necessary? 19 A. Yes, we were. 20 Q. I think the number of 25 decreased over time; is 21 that right? You tell us in your statement that, 22 by the time you left in 2016, the number had 23 decreased to between 12 and 15 people? 24 A. I think so. It's really hard to remember 25 definite numbers, especially because, towards 37 1 the end of that time, partly we were taking on 2 some extra bits of workload, non-Pathway stuff, 3 some other teams that had been elsewhere in 4 Pathway were now either part of SSC or at least 5 sharing the same floor space as us. So it's 6 a little difficult to remember who was where and 7 which team. 8 I'd also say that I think the numbers 9 reduced a bit before HNG-X and then I think we 10 got more people on board then when the new 11 system was rolled out everywhere in 2010. 12 Q. So decreased before Horizon Online? 13 A. I think it had dropped a little bit naturally, 14 just by people leaving and not so many new 15 people coming in. 16 Q. You tell us in your statement that you were most 17 likely to deal with tickets that concerned 18 counter balancing? 19 A. Yes. 20 Q. How did that come about? 21 A. Er, I think largely because I was sitting next 22 to somebody who was an expert in that area and, 23 although she hadn't been my sort of official 24 mentor when I started, I picked up on a lot of 25 the stuff that she was doing and also, I liked 38 1 the -- you know, playing around with numbers and 2 checking that things added up. 3 Q. You say that there were five or six of you, when 4 there were 25, that would be most likely to 5 handle tickets that concerned counter balancing; 6 is that right? 7 A. Probably, yes. I mean, more of us would have -- 8 there was certainly a lot of other people who 9 might occasionally have picked up a call of that 10 type but probably the more complicated problems 11 would come down to, you know, five or -- four, 12 five or six of us. 13 Q. Can you remember who they were? 14 A. Um, yes. I mean, Diane Rowe early on; Dave 15 Seddon and Lina Kiang, who were both there for 16 longer than I was; Sudip Sur who started at 17 about the same time as me; Cheryl Card, who 18 started later; and then people like John 19 Simpkins and Mark Wright, who knew a great deal 20 about everything, wouldn't maybe be doing those 21 sort of calls so often but they had a very good 22 knowledge of the entire system, and I apologise 23 to anybody I've left out of this. 24 Q. Did your role in counter balancing mean that you 25 became a specialist in the operation of Riposte 39 1 and the EPOSS system? 2 A. Well, we all needed to know a lot about Riposte 3 anyway because it was at the heart of the entire 4 system but yes, the EPOSS system, I would really 5 perhaps -- where I've talked about counter 6 balancing, I mean, a lot of the problems were 7 more general EPOSS, counter front end part of 8 the system. 9 Q. You've told us that this specialism developed 10 because of the person that you were sitting next 11 to. 12 A. Mm-hm. 13 Q. Can I just explore with you what, if any, 14 training you had on and about the Horizon System 15 before you became responsible for investigating 16 problems and issues with it and the integrity of 17 the data that it produced. You tell us in your 18 witness statement that, in 2000, you and some 19 other new joiners attended the same counter 20 training that was providing for subpostmasters; 21 is that right? 22 A. Yes, that's right. 23 Q. How long did that counter training last? 24 A. Um, I think it was probably a week session and 25 it was a course run especially for us just in 40 1 a room on our secure floor. 2 Q. Was the training, to your knowledge, in any way 3 changed because you were the system 4 diagnosticians or were you treated as if you 5 were subpostmasters? 6 A. I think we were treated as subpostmasters 7 because it's useful to see it, you know, from 8 the end user's point of view. Although, 9 obviously, we didn't have the business knowledge 10 that any postmaster who'd been running his 11 branch using the paper systems for years, they 12 would come in with that sort of knowledge. 13 Q. In the course of that training, were you told 14 about concerns, issues or defects in the Horizon 15 System? 16 A. I don't recall being told of any during that 17 training. 18 Q. Now, the counter software used for balancing was 19 maintained by the EPOSS system within 20 development, the fourth line support; is that 21 right? 22 A. Yes. 23 Q. Did you know at this time, on joining or shortly 24 there afterwards, any internal reputation within 25 Fujitsu of EPOSS during the development of 41 1 Horizon, that it had been rather problematic or 2 troublesome? 3 A. I don't recall that, no. 4 Q. So, again, you were thinking you were operating 5 a system that was well oiled and functioning but 6 may turn up problems because, otherwise, you 7 wouldn't have a job? 8 A. Yes. I think that's true. 9 Q. Can we look, please, at WITN04600104. This is 10 an ICL Pathway report dated 10 May 2000, you can 11 see that on the top right, so a few months 12 before you took up your post, yes? 13 A. Yes. 14 Q. It concerns the results of an audit. You'll see 15 that it's titled, both at the top and in its 16 first line, "Schedule of Corrective Actions, 17 CSR+ Development Audit". Now, if we scroll down 18 we can see that you're not on the distribution 19 list and I'm not suggesting that this was shown 20 to you in any way. 21 Can we go to page 9 of the document, please, 22 and can we look, please, at the first column in 23 the table: 24 "The audit identified that EPOSS continues 25 to be unstable. PinICL evidence illustrated the 42 1 numbers of PinICLs raised since the 1998 Task 2 Force and the rate of their being raised. 3 "The EPOSS Solutions Report made specific 4 recommendations to consider the redesign and 5 rewrite of EPOSS, in part or in whole, to 6 address the then known shortcomings. In light 7 of the continued evidence of poor product 8 quality these recommendations should be 9 reconsidered." 10 Did you know, when you joined the SSC, that 11 an audit of the EPOSS had found it to be 12 unstable? 13 A. No. 14 Q. Did you know that a report had concluded that 15 EPOSS should be redesigned and rewritten? 16 A. No. 17 Q. Did you know that in May 2000, a few months 18 before you joined, that that recommendation had 19 been repeated? 20 A. No. 21 Q. Can we go to page 10 of the document, please, 22 and look at the response. It's in the bottom 23 right-hand corner. Thank you: 24 "Following response received from MJBC: 'As 25 discussed this should be closed. Effectively as 43 1 a management team we have accepted the ongoing 2 cost of maintenance rather than the cost of 3 a rewrite. Rewrites of the product will only be 4 considered if we need to reopen the code to 5 introduce significant changes in functionality. 6 We will continue to monitor the code quality 7 (based on product defects) as we progress 8 through the final passes of testing and the 9 introduction of the modified CI4 codeset into 10 live usage in the network. PJ can we make sure 11 this is specifically covered in our reviews of 12 the B&TC test cycles. Closed." 13 Did you know, when you joined, that the 14 quality of the EPOSS code, based on, as there 15 described, product defects, was supposed to 16 remain under review during the introduction of 17 the modified codeset into live usage in the 18 network? 19 A. No. 20 Q. You were part of the SSC in the months following 21 this report. To your knowledge, were people, 22 including you in the SSC, told of the need to 23 monitor the EPOSS code through product defects? 24 A. I don't recall being told that, and it's perhaps 25 something that I would have expected our manager 44 1 to have been keeping an eye on, rather than -- 2 I mean, because he knew all the problems that 3 were coming in, rather than of us -- certainly 4 people who have only just started, who will just 5 be looking at individual incidents as they 6 happen. 7 Q. Is that in fact the case: that he would look at 8 every ticket and see the outcome of it? 9 A. I cannot speak for him but I think it's -- he 10 certainly had the ability to do that. 11 Q. The ability, yes, but, to your knowledge, in the 12 16 years that you worked there, did the manager 13 perform that kind of function? There's 14 a recommendation here that this action be 15 closed, that there be no rewrite, no redesign of 16 EPOSS because there's going to be a monitoring 17 process? 18 A. Yes, but I wouldn't expect something like that 19 to be monitored by the people, if you like, at 20 the very pot of the heap. I would have expected 21 somebody slightly higher up, for example the SSC 22 manager. But I obviously cannot say "yes" or 23 "no" he did this. I think, knowing Mik, its 24 quite likely that he did, but it might have been 25 him, it might have been somebody else on -- 45 1 Q. If you didn't know about this, you wouldn't know 2 to feed back "I'm noticing a preponderance of 3 problems with the EPOSS system or the code in 4 this part of the EPOSS system", would you? 5 A. No, I wouldn't but then, as I said, I would have 6 expected that to have been monitored at slightly 7 higher level. 8 Q. Would you expect the people at the lower level, 9 as you called it, including yourself, to have 10 contributed to that, ie a monthly review or 11 a quarterly review or even a yearly review: 12 let's look at how EPOSS is performing? 13 A. Um, I don't know. I mean, no, I still feel 14 that's the sort of thing that, you know, where 15 you've got a lot of people working not exactly 16 individually, but when the information is all 17 there on the PEAKs, and so on, I would have -- 18 I think it seems much more likely and sensible, 19 in some ways, for it to be looked at by somebody 20 who's got the technical knowledge but has -- you 21 know, their job is to take the broader view -- 22 Q. But there wasn't any formal instruction to you 23 or informal instruction to you to say, "Chalk up 24 when you're dealing with a ticket, a problem 25 with EPOSS" -- 46 1 A. No. 2 Q. -- "so that it can be fed back to somebody 3 conducting an overarching review to carry this 4 recommendation into effect"? 5 A. No, we were never told to do that. 6 Q. When you joined the SSC, what was the role of 7 Gareth Jenkins? 8 A. I was aware that he was one of the technical 9 experts. I think to start with, he was -- I'm 10 not sure if he was based in Feltham then, where 11 a lot of the development teams were, but I don't 12 think I met him for -- until I'd been there for 13 two or three or four years. 14 Q. So 2003, 2004? 15 A. Possibly. It might have been slightly sooner. 16 I think I became aware of the name because you 17 saw it on documents, and so on. But SSC were 18 very much self-contained on our floor because it 19 was a skill floor so you didn't have people 20 coming and going, so we sort of, to quite 21 a large extent, kept ourselves to ourselves. 22 Q. Did you understand him to be the principal 23 Fujitsu expert on the counter application? 24 A. I probably picked that up fairly quickly, yes. 25 I don't think anybody ever told me that. 47 1 Q. Was there any process of induction to say, for 2 example, "This is Mr Jenkins, he's the chief 3 designer/architect of, I don't know, the changes 4 to POL's back end systems, that meant he works 5 a lot with the counter application and the EPOSS 6 code"? 7 A. No, I -- 8 Q. "If you have [X] problem, he's your point of 9 contact"? 10 A. No, and he wouldn't, at that point, necessarily 11 have been our next point of contact because we 12 would probably have talked to the EPOSS 13 developers about any problems in the first 14 instance and then I'd have expected them to go 15 and talk to Gareth if necessary. 16 Q. By the EPOSS developers, do you mean people 17 in-house? 18 A. Yeah. 19 Q. Did you form any opinion of the quality of the 20 EPOSS developers? 21 A. Um, I'm trying to think who was there. Yes, 22 I didn't work closely with them. As I said, to 23 start with, they were in Feltham anyway. 24 I think there's always a slight tension between 25 support and developers, who are also doing 48 1 support, because they are often actually 2 developing enhancements to the system, and so 3 on. And so sometimes, perhaps you've felt you 4 wanted them to focus a little bit more on the 5 support of an existing problem but they were 6 heads down working on something new. 7 Q. In the months after you joined, did you form 8 a view on the quality of the product, the EPOSS, 9 that they were working with? 10 A. Um, I don't think I thought of it in those terms 11 at that point. You know, this was what we were 12 looking after. We dealt with whatever came up 13 and, where necessary, we passed things on to 14 EPOSS. I can't remember in those very early 15 days -- when things were still potentially 16 settling down after the rollout, the only thing 17 that I can remember is that there were -- I can 18 remember one call in particular to do with 19 a cash account production, where it was very 20 difficult to get to the bottom of the problem 21 and to work out what the numbers on the cash 22 account should actually have been, and so on, 23 and I think there was someone called Steve 24 Warwick, who I think was involved in trying to 25 help out with that. 49 1 So I remember that as just one particular 2 call where there was a particular problem and 3 difficulty and I cannot remember what the root 4 cause of it was. 5 Q. But you didn't have an overarching view of 6 EPOSS, that it was a problematic or troublesome 7 system? The Inquiry has heard some evidence 8 already, in its Phase 2, as to the views of some 9 of those within Fujitsu and Post Office -- 10 A. Mm-hm. 11 Q. -- as to the quality of the EPOSS system, one 12 describing it as "a bag of" and then 13 an expletive. When you took over in the SSC, it 14 didn't strike you as being deeply problematic? 15 A. No, I mean, by this time, there were, I don't 16 know, perhaps initially 10,000 Post Office 17 Counters using it every day for all their 18 business, and then 15,000, and then 25,000, and 19 finally about 37,000 counters using it, and, 20 although yes, obviously, some calls were coming 21 in and some of them were EPOSS, we certainly 22 weren't being swamped with the number of calls 23 that you would expect if the system was 24 thoroughly rotten, because it just -- you know, 25 once you've ramped up to those volumes, you are 50 1 going to -- if there are problems, you are going 2 to be seeing them. 3 Q. Assuming they made it to the third line support? 4 A. Yes, but, basically, you know, this did seem to 5 be a usable system because it was being used. 6 Q. You mean because it didn't fall over? 7 A. It didn't fall over. People weren't reporting, 8 "Oh, I've pressed this button to sell a First 9 Class stamp and it's sold", I don't know, 10 something else instead. We weren't getting 11 large numbers of calls from people saying, "Oh, 12 we did this and it's not there", and so on. 13 So I think it's -- you know, it's hard to 14 put it into words, but we weren't getting, if 15 you like, the feedback from the live estate that 16 it -- that there were a huge number of 17 significant problems. 18 Q. So these fears that had been expressed, just 19 months before you joined, that there needed to 20 be a total redesign and total rewrite of EPOSS, 21 when the system was working, they just didn't 22 come to pass? 23 A. Well, it may well be -- I don't know, you gave 24 the date on the front of this as being -- 25 Q. 10 May. 51 1 A. Yes, but that was the final edition of that 2 document rather than when it was initially 3 written? 4 Q. Correct. 5 A. So it's quite possible that bug fixes and other 6 changes would have been made to the system in 7 that period. So, you know, the system wasn't 8 static, things were being fixed and enhanced, 9 all the way through its life. 10 Q. The Inquiry understands that a gentleman called 11 Matt Aris, A-R-I-S, was the EPOSS development 12 team leader; do you remember that? 13 A. I remember the name. 14 Q. Do you remember him being the development team 15 leader? 16 A. I couldn't have sworn to that if you hadn't just 17 told me. 18 Q. Can you help us: what would be his, if he was 19 the development team leader, his relationship to 20 Gareth Jenkins? 21 A. I assume that if there was -- when changes to 22 the system were -- when changes to the code were 23 happening or to the design, he would use Gareth 24 to discuss anything that needed discussing, and 25 so on. 52 1 Q. So he was more senior to Mr Jenkins? 2 A. No, Mr Jenkins would have been more senior, 3 I would have thought. 4 Q. Were they in the same team, the same reporting 5 structure? 6 A. I've no idea. 7 Q. Did you have dealings with Mr Aris? 8 A. I almost certainly talked to him. I think I did 9 talk to him. To start with, as I said, we were 10 quite a self-contained team and, if we wanted to 11 pass a ticket on to fourth line because we 12 thought there was a code problem and they needed 13 to investigate further, then the way of doing 14 that was just to assign it on PEAK, so it got 15 passed through. 16 As time went by, I have always liked to try 17 to develop some sort of relationship between 18 teams and so, certainly, once the development 19 teams had moved into Bracknell, then I would 20 quite likely walk down a flight of stairs and go 21 and talk to them about something, rather than 22 just saying, "Oh, well, it's off my desk", and 23 passing it on to them in that way. 24 Q. Can I turn, before we have the morning break, to 25 the ways in which the SSC operated in practice. 53 1 I've got ten or so issues I want to ask you 2 about, please: 3 Firstly, the data available to you. 4 Secondly, the process by which tickets were 5 passed to SSC and, in particular, the system for 6 linking them in to a KEL. 7 Thirdly, concerns about the SSC fobbing off 8 subpostmasters. 9 Fourthly, how the SSC would go about 10 establishing the extent of a problem when it 11 received a ticket. 12 Fifthly, what information was passed back to 13 subpostmasters by the SSC or others. 14 Six, some other problems with the PEAK 15 system. 16 Seven, the process of pacing 17 an investigation around a single PEAK. 18 Eight, looking at the Horizon Helpdesk role. 19 Ninth, the use of ARQ data. 20 Tenth, attributing a problem to user error. 21 Okay, so they're the ten topics we're going 22 to look at. 23 Firstly, then, the data available to you 24 when a ticket was allocated to you. 25 You tell us in paragraph 30 of your witness 54 1 statement, maybe if we can turn that up, please. 2 Witness statement, paragraph 30, which is on 3 page 8. You tell us in the last sentence of the 4 main part of paragraph 30: 5 "In relation to counter issues for Legacy 6 Horizon, the primary sources of evidence would 7 be ..." 8 Then you set out three bullet points. So 9 the first one, is that the branch data in the 10 message store? 11 A. Yes, this is all the branch transaction data and 12 various other messages that would be written to 13 the message store as well and all the reference 14 data for the branch. 15 Q. Just now, for later on when I ask you questions, 16 it's right, is it, that that that, could later 17 be retrieved from an archive via Fujitsu 18 Security and is referred to as the ARQ data? 19 A. Yes. 20 Q. Yes? Is that a shorthand summary? 21 A. Um, yes, I mean, the ARQ data could either 22 contain the whole of the message store or -- 23 well, it was a slightly -- I don't know how 24 I can explain this without explaining a bit more 25 about message stores and Riposte but you may not 55 1 want to go into that now. 2 Q. I probably don't, thank you. 3 A. Okay. 4 Q. But, for present purposes, it's sufficient to 5 note that this first bullet point contained data 6 that was archived? 7 A. That data was all archived, yes. 8 Q. Fujitsu Security could access it and a way of 9 describing it is ARQ data? 10 A. Yes. 11 Q. Okay. Then, secondly, the event log from the 12 Horizon counter application? 13 A. Yeah. 14 Q. Then, thirdly, the -- 15 A. Sorry, could I go back to the second one. 16 That's actually the Windows NT application event 17 log, so it's not just the Horizon application 18 that's writing to it. 19 Q. Okay, can you just describe, for the benefit of 20 those listening, what the Windows NT log was, 21 then? 22 A. Any events that have been generated by 23 an application running on a computer or by the 24 Windows system itself would be written to this 25 event log. 56 1 Q. So, essentially, events in the Windows product 2 that the counter application was built on top 3 of? 4 A. Yes, but also counter application events as well 5 would be in there. But it's not purely counter 6 application events. There would be events from 7 other processes running on the counter, as well. 8 Q. Then, thirdly, the psstandard.log from the 9 counter. Can you explain what that is, please? 10 A. That that was -- I think "ps" stood for 11 "peripheral server" but it got written to by 12 various things, so in that we could see stuff 13 like what had been output on the tally roll 14 printer at the branch, and so on. There was 15 also a certain level of diagnostics came out 16 somewhere, and I can't remember if they were 17 also in the psstandard.log or if I've missed 18 something and they went somewhere else. 19 Q. So the two event logs you mention in the second 20 and third bullet points there, on which servers 21 were they stored? 22 A. They weren't stored on servers; they were only 23 stored on the counter. 24 Q. They weren't stored on servers at all? 25 A. The logs -- the events were sent to the data 57 1 centre through something called Tivoli, I think, 2 and then they were stored. 3 Q. Where were those servers? 4 A. At the data centre, one in Bootle and one in 5 Wigan, but I couldn't tell you the names of the 6 particular servers that these were stored on. 7 Q. Were there back-up arrangements for those 8 servers? 9 A. Almost certainly but I don't know any of the 10 detail. 11 Q. You can't help us with what those back-up 12 arrangements might have been? 13 A. No, and I don't think that the stream of events, 14 although it was there for monitoring, and in 15 fact they were saved for posterity, they weren't 16 sort of securely locked and audited in the way 17 that the message store data that could then be 18 retrieved via an ARQ request was locked and 19 kept. 20 Q. That was my next question. What processes were 21 employed to ensure that the data on those two 22 event logs was archived and maintained securely? 23 A. I don't think it particularly was. 24 Q. You said in an answer before last that they were 25 just kept for posterity. 58 1 A. Mm. 2 Q. By that, did you mean by accident, as it were, 3 rather than by design because the archived data 4 might be needed? 5 A. Yes, I think it was more that a lot of files 6 were kept for quite a period. But data that was 7 intended for future use in prosecutions, and so 8 on, if you like, was -- that was very carefully 9 secured and then there were sort of proper ways 10 of accessing it, and so on. 11 Q. But that process wasn't extended to the data 12 archived in relation to these two event logs, 13 have I understood you correctly? 14 A. The application event log, no, and the 15 psstandard logs, they didn't go anywhere except 16 they were just on the counter, so we could 17 retrieve them, and they were only there for 18 quite a short period of time. 19 Q. So when you and the SSC retrieved data from 20 event logs and including from the archive, how 21 was that process recorded? 22 A. I don't think it was. We wouldn't -- the 23 long-term event archive was very rarely used. 24 We didn't -- I didn't know it was there until 25 2006. The stream that went through Tivoli we 59 1 could look at and I cannot remember if that had 2 anything behind it that did secure that for any 3 length of time. 4 If we pulled an application event log direct 5 from the counter or the psstandard.log direct 6 from the counter, I'm not sure that was recorded 7 anywhere that we had done that. 8 Q. Was nothing done to ensure that the retrieval of 9 data from these two sources was recorded and was 10 undertaken in a secure, auditable way? 11 A. I don't think it was, no. It was the only -- 12 the security about it was that we were in 13 a locked floor with fairly restricted access to 14 the counters. 15 Q. On the counter application, what sort of events 16 would be recorded? 17 A. Um, the one that springs to my mind is if 18 Riposte outputs -- Riposte being not part of the 19 counter application but underlying it -- if that 20 produced an error, or even just -- you'd also 21 have startup messages in there, so as the 22 counter application started up, it would write 23 various events saying where it had got to in the 24 process. 25 Q. Who programmed the counter application to record 60 1 which events? 2 A. I presume it was in the development code but 3 I've no idea. 4 Q. Do you know the decision making that had been 5 applied into which events were recorded and 6 which were not? 7 A. No. These are not -- there's a big opportunity 8 for misunderstanding here. The counter 9 application itself wrote events into the message 10 store to say when somebody logged on and logged 11 off or when they did a declaration or when they 12 produced a report. Those sort of events. But 13 those are very Riposte events stored in -- 14 sorry, not Riposte events. Well, they're events 15 that are stored in the message store rather than 16 in the application event log. 17 Q. On the NT event log -- 18 A. The NT event log. 19 Q. -- that was presumably a result of Microsoft 20 programming? 21 A. No, the counter application, if ... 22 (Pause) 23 Yes, I don't think I remember well enough to 24 explain this. If I had an example in front of 25 me, I could probably work through it and explain 61 1 things to you but, trying to remember it cold, 2 I don't think I'm going to be able to add a lot 3 more here. 4 Q. If you investigated the event logs whilst 5 dealing with a ticket, would you preserve the 6 event logs with the ticket, ie with the PEAK, or 7 alternatively in the KEL, or not preserve them 8 at all? 9 A. If the ticket needed further investigation and 10 was going on to fourth line, then, yes, the 11 event log would be attached to the PEAK, along 12 with the message store, and anything else we'd 13 found that looked useful because the SSC were 14 the only team who could get this information out 15 of the live system, so we were expected to get 16 what we could because then that was all that 17 fourth line support would be able to look at to 18 try to find the root cause, and so on. 19 If our investigation didn't find anything 20 further that was needed, for example it was 21 another instance of a known error or something 22 else, then these probably wouldn't be saved. 23 Q. If they weren't preserved in the way you've just 24 described, how long was each species of event 25 log retained for? 62 1 A. That would be up to the individual. I would 2 probably keep everything I'd looked at for at 3 least a year, if not longer, just in case there 4 was any follow-up. 5 Q. But that was a matter of individual discretion 6 amongst the 25 of you? 7 A. Yes. 8 Q. Where would you keep it? 9 A. On our secure server. 10 Q. So what would you do? Would you save it as 11 a file? 12 A. Yes, it would be saved and when we extracted it, 13 it would go into somewhere in our own area -- 14 Q. So almost saving to desktop? 15 A. Not on our desktop, no, on a remote server that 16 we had access to. 17 Q. Why did you settle on to a year to keep? 18 A. Sometimes it would be longer. If I felt I was 19 starting to run out of space, I would -- I would 20 very occasionally do a tidy-up but I wasn't the 21 tidiest person in the world. 22 Q. But it was down to your individual discretion? 23 A. I believe so, yes, I don't think anybody ever 24 said, "Oh, you must keep this". I'm sure nobody 25 ever said. 63 1 Q. In addition to the data that we just looked at, 2 when a ticket was assigned to you, if 3 appropriate, you would have had a KEL, yes? 4 A. If somebody who had already looked at it at 5 first or second line, or potentially the 6 pre-scanner, had decided that a KEL -- 7 an existing KEL looked applicable -- 8 Q. Looked vaguely relevant? 9 A. -- then, yes, they would have put a mention to 10 that on the PEAK or the PowerHelp call and then 11 it was just a hotlink to click on it and to read 12 the detail. 13 Q. If they hadn't made that association, would you 14 nonetheless check the KEL system to see whether 15 there was one? 16 A. Um, probably, yes. That was probably the 17 process. In practice, once I had been there for 18 some length of time, if it was a call, 19 an incident coming in about something that I was 20 already familiar with, I -- you know, I might 21 well know without the searching which KEL it 22 was. But, yes, certainly if something came in, 23 somebody reporting a particular error message, 24 then you'd do a KEL search for that error 25 message or whatever and, if you found something, 64 1 then that's your starting point. 2 Q. How would you do a KEL search? 3 A. Um -- 4 Q. Was it a free text keyword search? 5 A. It was a very, very free text search, so you 6 just entered a few words that you thought might 7 be relevant. Obviously, if you've got an error 8 number or something like that, that's a good 9 starting position, or an event from a particular 10 source, there would be clues in that as well. 11 So you could type any or all of these things in 12 and see what you've got. 13 Q. How accurate and reliable was that process in 14 turning up relevant KELs? 15 A. Pretty good but, like any system, it depends how 16 well they've been written in the first place. 17 But certainly for something like a specific 18 error number, yes, if there was a KEL, you were 19 very likely to find it. 20 Q. Then, lastly before the break, you also had the 21 databases of past PinICLs and PEAKs, is that 22 right, that you could access? 23 A. Yes. Again, that was a free text sort of 24 a search, I think. 25 Q. I was about to ask, how would you search the 65 1 database of PinICLs and PEAKs? 2 A. Yeah, I'm trying to think back. Certainly, by 3 the time left, I'm just about certain it was 4 very easy to search. Again, a free text search. 5 Q. Would you habitually do that? If a ticket came 6 in, would you go to the PinICL and PEAK database 7 and look at that database to investigate the 8 current ticket? 9 A. I'd be more likely to do it from the KEL system. 10 Q. So only if there was a link to past PEAKs or 11 PinICLs in the KEL, would you click the 12 hyperlink through; is that right? 13 A. Yes, probably that would be the normal way of 14 doing it. 15 MR BEER: Yes, thank you very much. I wonder 16 whether that's an appropriate moment. 17 SIR WYN WILLIAMS: Yes. 18 MR BEER: Just in relation to the noises, the first 19 noise was a waste disposal unit's pistons 20 needing oiling. That has been done. The second 21 noise was a mobile phone and that won't happen 22 again, I'm sure. The third noise was a fire 23 alarm not in this building because we wouldn't 24 be here. It was of an adjacent building behind 25 us, which had to be evacuated, but not us. 66 1 SIR WYN WILLIAMS: Such was my concentration level, 2 Mr Beer, that I didn't hear the third noise. So 3 whatever was going on between you and the 4 witness kept it out. 5 Anyway, we'll have a 15-minute break. 6 Mrs Chambers, I don't expect you to keep 7 yourself in purdah when we have these breaks but 8 just don't talk about your evidence with anyone, 9 all right? 10 THE WITNESS: Thank you. 11 ( 11.30 am) 12 (A short break) 13 ( 11.50 am) 14 SIR WYN WILLIAMS: Yes, Mr Beer. 15 MR BEER: Thank you, sir. 16 Mrs Chambers you said before the break that 17 when a ticket would come in, you would 18 principally rely on PinICLs or PEAKs that were 19 referenced in a KEL to conduct your 20 investigation. 21 A. No, you asked me if I would search through the 22 PEAKs -- 23 Q. Yes. 24 A. -- and I said probably not, you'd link from 25 a KEL. 67 1 Q. Yes. 2 A. That wouldn't be how you'd start 3 an investigation. 4 Q. No, I wasn't saying that was the entire range of 5 the data that you would look at. 6 A. Yeah. 7 Q. We looked at the data that you would use before 8 the break but, insofar as you were to look for 9 PinICLs and PEAKs, you would rely on those that 10 were referenced in the KEL? 11 A. That would be your starting point, if you wanted 12 to -- if you needed to look at another PEAK, 13 to -- 14 Q. So say there were two that were referenced and 15 they were hyperlinked there, would you think, 16 "Right, that's it", or would you, on each and 17 every occasion, look at the PinICL and PEAK 18 database to see whether there are any more? 19 A. Um, no, you -- it would depend so much on the 20 individual problem. 21 Q. What factors would determine whether you would 22 or would not rely on PinICLs and PEAKs 23 identified in a KEL? 24 A. Sorry, I'm not thinking this through very well. 25 Um, when you're investigating a problem that's 68 1 come in, you -- you're not necessarily starting 2 by seeing how many times it's already happened, 3 or whatever. That might then be something that 4 you would do later on in the investigation, but 5 you -- so you're saying, if it's a known error, 6 a definite known error that has come in, would 7 I then go and look to see how many other 8 occurrences of it there had been? 9 Q. Yes, I'm not saying that. I'm asking what your 10 practice was? 11 A. Yeah, I mean, if it's a known error and there is 12 a KEL for it already, then it is possible that 13 that should not have come over to third line in 14 any case. 15 Q. But we're necessarily talking here about cases 16 where there is a KEL associated with the ticket 17 that you're -- 18 A. There is a KEL associated with the ticket but 19 the call has been passed over to us anyway so 20 then we need to look at the circumstances of 21 this individual call and see whether the KEL 22 does relate to it. You know, you do a lot of 23 investigation before you go following all the 24 other links. 25 Q. Yes, and I wasn't looking at the issue of where 69 1 do you start; I was looking at the entirety of 2 your investigation and, in the entirety of that 3 investigation, the question is: to what extent 4 do you rely on only those PEAKs and PinICLs 5 identified in the KEL as being associated with 6 this issue, or do you look at the PinICL and 7 PEAK database to look for other PinICLs and 8 PEAKs that may be associated with this issue? 9 A. Yes, in some cases you would. 10 Q. What would determine the some cases that you 11 would and those that you wouldn't? 12 A. If it looked like it was a repeating problem, 13 that wasn't -- where you needed to get some idea 14 of how often it was happening, then, yes, you 15 would go and look at all the PEAKs and PinICLs. 16 Q. How would you know if it was a repeating problem 17 without looking at the PinICLs and PEAKs? 18 A. Because of our knowledge of the system and the 19 things that we had individually looked at before 20 and whether the KEL said this has happened here, 21 here and here, and what the implications of the 22 problem were. I mean, in some cases, you 23 would -- yeah, sorry, I'm finding this rather 24 hard to answer sensibly because it's not -- you 25 know, if you gave me -- if ... 70 1 So if we're -- you're saying a new problem 2 has been -- well, an existing problem is there, 3 we have another call about an existing problem, 4 would I always go and see how many instances 5 there had been? It would depend what -- whether 6 it was something that each instance could be 7 dealt with sensibly, individually or whether we 8 felt it was part of a, you know -- there was 9 a bigger picture that needed to be identified. 10 Q. Okay, I'll move on but I'll come back to that 11 later. 12 You say in your witness statement, it's 13 paragraph 16, if we just look at it on page 4: 14 "I am asked whether I consider that the KEL 15 system was adequate for its purpose. Overall, 16 I think the KEL system worked well although 17 there were some problems. For example, many 18 KELs documented similar symptoms, and service 19 tickets could be passed to SSC with the wrong 20 KEL quoted." 21 Yes? 22 A. Yeah. 23 Q. When you say "the wrong KEL quoted", it meant 24 that somebody in the chain before you had 25 identified a KEL that was unrelated to or 71 1 irrelevant to this problem; is that right? 2 A. Yes, or it might have looked similar on the 3 surface but they were unable to -- they hadn't 4 realised it didn't apply, and there might have 5 been a better KEL which they hadn't found. 6 Q. Was that raised as an issue of concern within 7 Fujitsu by the SSC? 8 A. No, I don't think so it was up to SSC to improve 9 the KELs so that the right one was found in 10 future. We were the ones who were writing the 11 KELs. 12 Q. But you weren't the one that was doing the 13 associations on a new ticket that was sent to 14 you, were you? 15 A. No, but if they were -- if second line, first 16 line had found the wrong KEL then, you know, we 17 would look at the KELs to see how it could be 18 made clearer in future, so they would -- were 19 more likely to pick up the correct one. That 20 was part of our job. 21 Q. Was anything therefore done to rectify this 22 problem with the KEL system? 23 A. Well, it wasn't a problem with the KEL system it 24 was a problem with the individual -- the ways 25 some of the individual KELs were written, if 72 1 there wasn't enough information in them for 2 somebody to ascertain between problem A and 3 problem B. 4 Q. That's one way of looking at it: it's the way 5 that the KEL has been written by the SSC. 6 A. Mm. 7 Q. Another way is that the people doing the 8 assigning in phase 1 and phase 2, first and 9 second line, are just misassociating KELs with 10 the new ticket? 11 A. Yes, so it is a problem that they have done that 12 and -- yeah. 13 Q. Was that raised with first and second line 14 support? 15 A. I'm sure occasionally it was passed back to them 16 that they hadn't found the right one, but 17 I don't think it was such a huge -- yes, I don't 18 think it was a huge problem. 19 Q. How was it established that the wrong KEL had 20 been quoted on the ticket? 21 A. Because when I or one of my colleagues looked at 22 the information and the problem, we could see 23 that it wasn't the right one and that there was 24 a better one. 25 I mean, we -- we wouldn't have started our 73 1 investigation only by looking at the KEL that 2 had been pointed out to us. We would have 3 looked at all the evidence available. 4 Q. If you had picked up a ticket that had the wrong 5 KEL associated with it, would you go back 6 yourself to the person in first or second line 7 support who had made that association and say, 8 "Look, you've associated the wrong KEL here"? 9 A. Probably not. 10 Q. What was the system, therefore, to ensure that 11 first and second line support did not make these 12 mistakes? 13 A. For me to rewrite the KELs as necessary, so to 14 clarify between the two problems. 15 Q. You are again focusing on saying that it's your 16 fault or the SSC's fault, rather than people in 17 the first and second line -- 18 A. Yes, yes. 19 Q. -- making mistakes? 20 A. Largely, yes. I mean, people do make mistakes 21 you have to base your systems around the fact 22 that people don't always get it right first 23 time. 24 Q. Was there any system of reporting to your 25 manager where you would log "Wrong KEL 74 1 associated with this ticket", and he would 2 collect that data up on a monthly, quarterly, 3 yearly basis and then go back to first and 4 second line support? 5 A. I don't know. I mean, we might well put 6 a comment on the PEAK saying, "It's not this 7 KEL; it's that one". Whether anybody monitored 8 for that and fed it back, I don't know. 9 Q. If there wasn't anything in the KEL or the PEAKs 10 or PinICLs to help you, did you have any tools 11 for analysing for the branch concerned a week or 12 a month's worth of data, or did you need the 13 subpostmaster to narrow the period of the 14 relevant problem down to a reasonably short 15 period of time so you could look at that data 16 line by line? 17 A. It obviously helped if the postmaster was 18 aware -- you know, had some idea of which day or 19 what sort of -- are we talking now about 20 balancing problems -- 21 Q. Yes. 22 A. -- where there's a discrepancy? 23 Q. Yes. 24 A. Because this was any a very small proportion of 25 the calls we were dealing with. So maybe I've 75 1 been misunderstanding you because I've been 2 answering in general terms, whereas maybe you've 3 been intending to ask me about specific 4 balancing problems. 5 Q. Previously I was asking in general terms about 6 the system of linking KELs to PEAKs and PinICLs. 7 A. Yeah. 8 Q. Now I'm asking about -- 9 A. A specific balancing problem. 10 The more information that the postmaster 11 could provide, the more -- the easier it was, 12 obviously, for us to focus and look at 13 a particular area of concern. And sometimes -- 14 I don't know, we'll see examples of this, 15 problems with rem in and rem outs. They 16 realised very quickly that something had gone 17 wrong while they were doing that and so then 18 obviously we'd always pull back the complete 19 message store, which contained roughly a month's 20 transactions. That varied at different times 21 but we're talking about a month's transactions. 22 Q. Just to stop you there, was that the typical 23 period that you personally would seek data for? 24 A. That is what was in a counter message store when 25 you retrieved it from the correspondence server, 76 1 because the data was retained for, I think 2 initially, 42 days and then it dropped down to 3 about 35 days, and so the message store that we 4 got back for a branch would always contain all 5 that data. We would then focus in on any 6 specific areas of problems but, if necessary, we 7 could look over that entire period. 8 Q. If a subpostmaster said that they had 9 misbalanced but they couldn't point out where in 10 the week that had occurred or where in the month 11 later on that had occurred, would you ever refer 12 them back to the NBSC? 13 A. I would always have a look to see if I could 14 narrow it down to where a problem might have 15 occurred and I can go into some detail as to how 16 I would do that, if you want me to. 17 Q. At the moment, would you ever refer them back to 18 the NBSC to provide more detail? 19 A. If -- the NBSC were meant to have taken them 20 through, to question them fairly strongly to see 21 if there were any user errors that might have 22 caused this. If we got a -- this type of call, 23 and there was no sign that it had already been 24 through NBSC, then it might well be passed back 25 but we would normally expect the first or second 77 1 line to have said, "Hang on, you need to go and 2 talk to NBSC first". 3 So, by the time it came back through to us, 4 I would almost always -- I would have a look 5 anyway just to see what I could see. 6 Q. Did you have a methodical process that you 7 applied to each ticket, in terms of steps of 8 investigation that replicated itself time and 9 time again or was it dependent on the nature of 10 the issue identified in the ticket? 11 A. It would depend very much on the nature of the 12 issue but, you know, getting the message store 13 was always one of the first things for anything 14 counter related. 15 Q. What did you do when you obtained the message 16 store? I think this was what you were going to 17 tell me a moment ago. 18 A. Yeah, you opened it up and it's this absolutely 19 enormous text file so we used a fairly good text 20 editor that would let us highlight, search 21 things, highlight lines, pull out all the lines 22 that we'd marked. So for a discrepancy call, 23 where we weren't given any other clues, I would 24 highlight all the product 1 lines -- product 1 25 being cash -- pull them out, put them into 78 1 a spreadsheet which I'd developed a bit, that 2 then -- so instead of just a very long, very 3 hard to read text line, it would pull out fields 4 of interest, which obviously would be value, the 5 mode in which this transaction had been done. 6 I would then do a column with a running 7 total to give you the system cash position at 8 any point in time. So if you say at the start 9 of the week the postmaster has balanced, so he's 10 declared how much cash he's got so you have to, 11 you know, at some point assume that that was 12 correct, so you've got a starting position, you 13 can then work out your system cash position as 14 you go through by adding on all the cash 15 transactions that have taken place. 16 Then, at the points at which the postmaster 17 declares cash or declares his overnight cash 18 holding, you can see two other figures -- well, 19 at least one. You can see what he or she has 20 declared that they are holding at that point, 21 and if it's declare cash or an overnight cash 22 holding where they calculated the difference, 23 you can also see what the system calculates the 24 cash to be at that time. 25 So going through a week or a month, you've 79 1 got all these points where you've got two or 2 three figures that you can compare to see how in 3 line they are. Now, if you've got a difference 4 between the first -- your own running system 5 total and the cash total that the system has 6 calculated at that point, if those are 7 different, then you have a system problem 8 because -- of some kind, which you can then 9 investigate and see, well, I think the system 10 cash should have been this but the system is 11 that. Why are they different? What's not been 12 included? And so there are some of the bugs 13 that are covered which would fall into that 14 category. 15 And also, if you're -- yeah. I'll go back 16 to that. But then you've got the comparison 17 between what the postmaster has declared he's 18 got and what the calculated figure is, and that 19 is your discrepancy, which you're then looking 20 for a cause for. 21 Now, if you've done this over a week, 22 sometimes you can see it's in step, as it should 23 be, these figures are all in step, except for 24 one day suddenly it jumps and suddenly you've 25 got a discrepancy of £2,000. So then, on that 80 1 day, you look at all the transactions to see if 2 you can see anything, either system error or 3 user error, that could possibly have caused 4 a discrepancy of £2,000. 5 Q. Just stopping there, how would you determine 6 whether the discrepancy was user error or system 7 error? 8 A. You can't. 9 Q. You just said you would determine whether it was 10 system error or user error. 11 A. Well, you can look. If you can -- if you can 12 see something like a rem of the same pouch has 13 gone in three or four times, then that's fairly 14 likely, either the postmaster has been -- got 15 really carried away and has scanned the thing 16 several times, which shouldn't be allowed to 17 happen anyway, or it's a good working hypothesis 18 that you have some sort of system error with 19 that. So then you need to look and see exactly 20 what has happened. 21 But if you look at all these -- I mean, 22 you'd start out just by looking at the cash 23 transactions and the different modes. If you 24 can't see anything anywhere that gives you any 25 sort of a clue, it doesn't seem to be 81 1 particularly on one particular day or anything, 2 you may not be able to -- in those cases -- and 3 it did happen -- if there's no sign of any 4 system error, the calculated system figure is 5 correct, all that is wrong is the difference 6 between the system figure and what the 7 postmaster says -- has declared that they've 8 got, then, unless you've got the knowledge of 9 what has taken place at the branch and have some 10 way of checking that what is recorded on the 11 system actually matches what happened at the 12 branch, then you are not going to get any 13 further. 14 Q. We're going to come back to this a little later 15 today but, in that case, where you couldn't 16 possibly identify a system error, was the ticket 17 written up as user error? 18 A. Not normally, no. It would normally be "There's 19 no evidence of a system error". 20 Q. What was the consequence of writing a ticket up 21 "No evidence of a system error"? 22 A. It would go back through the lines of support 23 and then it would be up to the postmaster and 24 NBSC to see if they could pursue it any further. 25 Q. What do you mean by pursue it any further? 82 1 A. Whether -- and hindsight is a wonderful thing, 2 but when I first started doing these sort of 3 things, I sort of assumed that perhaps somebody 4 within the Post Office organisation would go and 5 help the postmaster to discover where something 6 might be going wrong. 7 Q. Why did you assume that? 8 A. Because that seemed a reasonable thing to 9 happen. 10 Q. Did you have any positive evidence that that did 11 happen? 12 A. No, and from talking to postmasters when I sort 13 of said "Well, you know, maybe your manager 14 could help", I didn't often get any very 15 positive feedback to that suggestion. 16 Q. Were you told that in fact what happened was 17 that if you wrote off a ticket or wrote up 18 a ticket which said, "No evidence of system 19 error", that the consequence of that would be 20 that the postmaster would pay -- 21 A. No. 22 Q. -- would have to pay? 23 A. No, I didn't -- certainly early on, I did not 24 realise that. 25 Q. After early on, when did you realise it? 83 1 A. Um, I suppose when cases started going to court. 2 Q. Can you date that? 3 A. 2005. 4 Q. Did that affect the way that you conducted 5 yourself after then? 6 A. I don't think so because I still -- you know, my 7 job was to try to identify system errors and, 8 you know, you can't, I think, turn round and 9 say, "Oh, well, it might be a system error but 10 I can't find it", not in a case where -- not 11 when there's -- you know, there is so much 12 variability, shall we say, on the customer side. 13 Q. In any event, we'll come back to that a little 14 later on. Can we look, please, at FUJ00086462, 15 please. Can we start, please, at page 2. 16 This is a series of emails that you became 17 involved in, in 2006, concerning the data tree 18 build failure, that we're going to look at 19 later, but just to orientate yourself, this is 20 some six years into the operation of -- 21 A. I don't think this is quite to do with that. 22 Q. Oh, isn't it? Well, let's go down and look at 23 Kimberly Yip's message at the foot of the page, 24 please. You'll see you're not involved in this 25 but it's the background to it. 84 1 A. But I had -- yes, yeah. 2 Q. You're not a copy-ee yet. 3 A. No. 4 Q. You'll see that this is about performance speed, 5 I think. 6 A. Yes, it was the performance of them producing 7 their balance reports. 8 Q. And -- 9 A. But it's not the same as the data tree problem. 10 Q. As the data tree. Okay. Ms Yip sent an email 11 to Graham Welsh. Who was Graham Welsh? 12 A. Um, customer services manager? 13 Q. We've got some documents that suggest his job 14 title was Fujitsu's Strategic Services Manager 15 for the Post Office Account. 16 A. I think he had various job titles over the 17 years. 18 Q. But you would put him down as customer services, 19 essentially? 20 A. Yes, I think at that point, that was -- he was 21 part of the service management team, if not the 22 leader of the service management team. 23 Q. Anyway, Ms Yip says to him: 24 "Please forgive me if you are not the 25 appropriate person to forward this email to. 85 1 "I have been contacted again by the POL 2 Service Line to obtain an update on progress on 3 the current Horizon System performance issues. 4 "One particular branch has been escalated to 5 me [and then identity of the branch is given] 6 and last rollover timings have been sent to me 7 by Anne Chambers, see below: 8 "From 17.00 the branch started printing the 9 daily report and this continued until [about] 10 18.30. They then declared stamps and cash, and 11 pressed the Balance report button at 18.37. The 12 Trial Balance was not printed until 21.12 13 (ie over 2.5 hours later). Much of this time 14 the system was processing the month's 15 transactions. There's a gap between ... 19.30 16 and 20.05 where it may have been waiting for 17 input from the [postmaster], but I can't be 18 certain. 19 "After the Trial Balance the report was 20 abandoned, presumably because the [postmaster] 21 needed to check and resolve the discrepancies. 22 At 21.27 cash and stamps were redeclared (with 23 some variation from the original), and at 21.28 24 the Balance report button was pressed again. 25 The second Trial Balance was printed at 22.58 86 1 (1.5 hours) and the Final Balance at 23.04. 2 "I've looked at what was going on during the 3 balance report production. 4 "There was nothing out of the ordinary, 5 apart from the very large number of transactions 6 being processed (about 40,000). The number of 7 transactions processed per second was rather 8 less than we sometimes see, but not 9 significantly so, apart from the period 19.00 to 10 19.10 when the counter end-of-day processes were 11 running. 12 "Anne also provided me with some 13 recommendations which I have passed on to the 14 branch and I will ask FS to do a similar 15 exercise to the one above (ie provide timings) 16 when the next TP rollover is completed, 14 June 17 [2006], to see if there are any significant 18 improvements. I have been told about another 19 branch so I am hoping to do a similar exercise. 20 In both cases the rollover times do seem 21 excessive and my worry is that these are not 22 isolated incidents. So in terms of the time it 23 is taking branches to complete the balance 24 process, can FS provide me with details on what 25 constitutes an acceptable length of time, for 87 1 example, if it takes 4 hours then this is 2 reasonable or if it's more than 5 hours then it 3 needs investigating, etc. This will then give 4 me a better understanding on what I should be 5 passing on to FS or if I should be passing on 6 the recommendations to implement. 7 "One of the recommendations was to roll 8 Balance Period every week, can you confirm that 9 this does reduce the overall time taken to roll 10 into a new TP at the end of the period? 11 "If you need any clarification, [don't] 12 hesitate to contact me." 13 Then if we scroll back up the page, please, 14 we can see that Mr Welsh forwards this to you 15 and to John Burton. 16 "Anne, 17 "Can you please comment on the attached ..." 18 Then: 19 "John, 20 "This issue is silly in the amount of time 21 and resource being applied for a system that is 22 performing to design ... Yes I know but frankly 23 the level of grief and support required is 24 crazy!" 25 Then if we go to the message at the top of 88 1 the page, we can see that Mr Burton replies: 2 "Graham, 3 "... I see exactly what you mean. By 4 coincidence, I'm reviewing Gareth's report on 5 this issue tomorrow morning, before it's 6 submitted to [Post Office]. I gather it quotes 7 some hefty prices for making improvements, but 8 I'll be better informed after the review." 9 Then further up the page, please. Your 10 reply on the same day: 11 "John, 12 "I've looked at many branches now, and they 13 range from very slow to horrifically slow when 14 rolling over stock units. It does vary 15 depending on the particular process followed at 16 each branch, and if you break it down into 17 various components each may appear to be (just) 18 within 4 [hours] as long as the weekly rollover 19 used to be, but the impact on the postmasters is 20 horrible. 21 "There have been some piecemeal changes to 22 try to improve certain areas, but most if these 23 have made little improvement, and overall, may 24 have been a waste of effort. 25 "As I see it, there are two main problems: 89 1 "1. The balancing process repeatedly scans 2 and rebuilds the data tree. This was identified 3 as a problem at least 6 months ago, and 4 improvements to this are, I think, what Gareth 5 is proposing. 6 "2. Counters are inadequate for the 7 applications now being run on them and do run 8 generally slowly at times. This hasn't really 9 been fully investigated, and is really difficult 10 to quantify or prove that it is happening -- the 11 only evidence is what the [postmaster] reports. 12 It is however adding to the customer 13 dissatisfaction and could only get worse even if 14 we improve balancing. 15 "I am not at all happy about fobbing 16 postmasters off and telling them that the system 17 is working as designed when it is plainly 18 inadequate for the job. I am also very unhappy 19 that it has taken six months even to get to the 20 point of starting to consider whether [Post 21 Office] will pay for improvements. 22 "I too would like guidance on when 2nd and 23 3rd line support should investigate further. 24 Our current response has to be 'yes, we know 25 balancing is very slow, it is being 90 1 investigated' -- what else can we say?" 2 You said in the course of your reply there 3 "they", that's the times, range from very slow 4 to horrifically slow. How wide a sample did you 5 take when giving that answer, if you can recall? 6 A. Um, hundreds, I think. I can't exactly remember 7 but I did -- we got calls coming in about it. 8 I haven't recently seen any call that I sent off 9 to development but I'm sure we did, and the 10 initial response that we got back from 11 development was "Oh, well, it was agreed with 12 Post Office that, um, it wouldn't take more than 13 four times as long as" -- because they used to 14 have to roll over every week. So now they're 15 only having to roll over once every four weeks, 16 although they can still roll into a new balance 17 period each week, and apparently it had been 18 agreed that, as long as the overall process was 19 no longer than four times what it had been 20 previously, that would be all right. 21 But, in practice, it was having a big impact 22 on branches, which I was well aware of. 23 Q. What kind of delays are we therefore talking 24 about? 25 A. Well, you had some of the timings below, so, you 91 1 know, it was -- 2 Q. Were they typical, the timings we saw below or 3 atypical? 4 A. That was not untypical. 5 Q. You said the impact on postmasters is 6 "horrible". What was the horrible impact on 7 postmasters? 8 A. They were having to -- I'm sure the postmasters 9 could answer this one for you but they were 10 having to sit there after end of trading on 11 Wednesday and, instead of getting it all done 12 and being out of the door in an hour/an hour and 13 a half, or whatever, you know, they might still 14 be there five, six hours later. 15 Q. With a final balance, as we saw in that example, 16 of just after 11.00 pm? 17 A. Mm, and obviously, if during the process, they 18 did they'd got discrepancies, which are not 19 unusual things to happen at branches, but then 20 they would have to go back and check and perhaps 21 recount their cash and look for anything. And 22 so, you know, it wasn't a sit down, press 23 a button and off it all goes. They were having 24 to do a great deal behind the scenes. 25 Q. You say the problem had been identified six 92 1 months ago but nothing effective had been done 2 about it? 3 A. I think we'd been aware of the problem since the 4 switch from every week cash account periods to 5 where they changed to balance periods and 6 trading periods. 7 Q. Wasn't that in 2004? 8 A. Um, I thought it was later than that. Maybe -- 9 I'm not sure. I can't remember. 10 Q. But, in any event, this email suggests that you 11 knew that the problem had existed for at least 12 six months. What had been done in that six 13 months, to your knowledge? 14 A. I don't know. 15 Q. You say, "I'm not at all happy about fobbing off 16 subpostmasters and telling them the system is 17 working to design"? 18 A. Yes, I wasn't because I had spoken to quite 19 a few of these postmasters and I could tell how 20 unhappy they were. I think this email -- 21 I obviously was trying to get my point across 22 forcefully and I was slightly sticking my neck 23 out, but I felt I was a little bit closer to the 24 people who were having the problems, perhaps, 25 than someone like John Burton, who I think was 93 1 the counter development manager. 2 Q. Is what you had been doing for at least that 3 six-month period then been to fob off 4 subpostmasters? 5 A. No, I would have been answering the calls and 6 trying to explain that it was expected that it 7 would be a slower process now that they were 8 having to do, um -- now that the process had 9 changed. But I did feel it -- I was concerned 10 that it was having a big impact and that, as far 11 as I had seen, nothing very much had changed, 12 although I think there were a couple of code 13 fixes that this suggests that something had been 14 changed, that was meant to make it better but 15 perhaps didn't really help very much. 16 Q. Mr Welsh had said that the issue was a silly one 17 and that resources within Fujitsu were being 18 applied to a system that was performing to 19 design. You were unhappy about telling 20 postmasters that the system was performing to 21 design, correct? 22 A. Yes, given the impact that it was having on 23 them. 24 Q. The suggestion that you should say to 25 subpostmasters the system is performing to 94 1 design, was that indicative of a more general 2 approach that you were required to take within 3 the SSC, ie "Don't reveal the true position that 4 we know about publicly or to the subpostmasters, 5 just say that the system is working well and to 6 design"? 7 A. I said it in this case because that was what 8 I had been assured of but, no, I would not have 9 said it in other cases where they'd had 10 a problem that was caused by a system error. In 11 that case, I would say to them, um, "Sorry, the 12 system has -- there's an error here, this 13 shouldn't have happened. It's a fault in the 14 system which we'll be investigating". 15 Q. Was there pressure on you in your communications 16 with subpostmasters not to reveal errors in the 17 system? 18 A. No. 19 Q. On this instance, on this email that we've got, 20 it tends to suggest that there had been 21 a message that you were required to deliver. 22 You had been fobbing them off and said the 23 system is working to design. 24 A. Um, no, I think I would have said, "I know it's 25 awful but I am told that the system is working 95 1 to design, but we are still looking at it", 2 Something like that. I wouldn't have tried to 3 pretend that it wasn't a problem. 4 Q. Scrolling up the page, please. We can see 5 Mr Burton's response: 6 "Anne, Graham, 7 "I reviewed Gareth's feasibility report and 8 costings this morning, so understand things 9 better than I did. His report is based on 10 a great deal of prototyping work that has been 11 done over the last few months -- of the order of 12 100 man days. That work looked at a number of 13 options, and has homed in on the one that gave 14 the best improvements -- along the lines you 15 mention in your first point. 16 "The report should go into [Post Office] 17 next week. It'll then be up to them whether or 18 not they want to pay us to do the work. If they 19 decide to go ahead, we're looking at a likely 20 delivery date of first calendar quarter in 2007. 21 That would give around 2 years of useful life 22 before being overtaken by HNG-X. 23 "I understand your frustration at having to 24 deal with irate postmasters and having to tell 25 them that the system is working to its spec. We 96 1 can only hope that POL do agree to funding this 2 work, so that you then have something positive 3 to say. 4 "I can't see much point 2nd and 3rd line 5 support doing further investigation, when we now 6 know what needs to be done to make a substantial 7 improvement. Please say, Gareth, if you 8 disagree." 9 I should say that Mr Jenkins was copied into 10 that email. Were you content with that 11 response? 12 A. Yes, I think so, I think it looked, you know, at 13 least something was happening. 14 Q. To your knowledge, did the Post Office ever pay 15 for the improvements that were proposed or did 16 they instead wait until Horizon Online was 17 rolled out? 18 A. No, something did change and it did improve. 19 Q. When was that? 20 A. I can't remember. 21 Q. Can you remember the nature of the improvement? 22 A. No. I almost certainly would have looked to 23 see -- you know, to make sure that it really had 24 made a significant difference. 25 Q. Sorry, can you say that last answer again? 97 1 I was distracted. 2 A. I am sure that when the change did go in, and 3 I can't remember when that was, I would have had 4 a look to see if it had improved the time it was 5 taking for some of these worst-affected 6 postmasters. 7 Q. Can we look at a different issue but on the same 8 topic of improvements in the system, and look at 9 POL00001265. 10 You'll see that this is a PEAK dated -- 11 you'll see the opening line of the PEAK under 12 "Progress Narrative" of 27 March 2006. 13 You'll see under the summary two lines 14 above, what the summary of it is, namely, 15 a "Harvester Exception". Can you explain in 16 brief terms what a harvester exception is, 17 please? 18 A. Right. So the transactions are written to the 19 message store on the counter from where they 20 replicate to the message stores at the 21 correspondence servers. Overnight, processes 22 run to harvest transactions so they can be sent 23 on to various sources. So, for example, all the 24 bill payments would have to go off to the 25 various companies whose bills are being paid; 98 1 all the EPOSS transactions were harvested along 2 with others to go to Post Office. 3 In this case, for some reason it's trying to 4 harvest a message for EPOSS, and I can only see 5 this top bit so far, but the message written on 6 the counter presumably does not have the mode 7 field which should have been included in it. 8 Q. Because it was blank? 9 A. That appears to be the case from all I can see 10 so far. 11 Q. Yes, and then if we scroll down, please, you 12 make an entry at the foot of the page on the 13 same day. The ticket having been assigned to 14 you. Your entry of 27 March at 16.12.36: 15 "I have repaired the problem transaction and 16 will check tomorrow that it has been sent okay. 17 "As far as I can tell, no call has gone to 18 development about this. To summarise, 19 "Some messages get written with a null Mode 20 attribute. The root cause of this has never 21 been resolved. 22 "Changes have been made to the harvester 23 agents so that the messages with [then you put 24 a character string] can be [installed] when Mode 25 is missing ... 99 1 "MailsBalance messages have [and then you 2 put some more character strings]. This was 3 spotted soon after their introduction in 4 January, and I did intend to raise a PEAK, but 5 don't seem to have done so. At the time it was 6 thought to be benign. 7 "MailsBalance messages with missing Mode are 8 now causing number of missing harvester 9 exceptions (5 on the reports for 24/3)." 10 What does that mean, for 24 March? 11 A. 24 March, yes. 12 Q. "Each has to be repaired individually. 13 "So we need to sort out the [character 14 string] issue. This could be fixed at either 15 the agent, or in the Mails scripts. If it can 16 be fixed fairly soon in the scripts, I think 17 that will be the better option rather than 18 making the agent cope with what is basically 19 a typo. 20 "There are example messages in the attached 21 reports, or I can provide a messagestore if 22 required. Routing to Mails [development team, 23 essentially]." 24 Yes? 25 A. Mm-hm. 100 1 Q. Then in the next messages, if we scroll down, 2 I'm not going to go through them in the 3 interests of time, no need to read them, we can 4 see that your suggestion about the investigation 5 of the root cause was not taken up and 6 a decision was taken not to do that because it 7 wouldn't be cost effective, given the limited 8 shelf life of the Horizon counter application? 9 A. Mm-hm. 10 Q. Can we see your response to that on page 3 of 11 the document, please. At 13.54.33, that's it, 12 third up, "Response noted". You say: 13 "I never really expected the root cause to 14 be investigated or fixed. The typo which caused 15 the agent circumvention to fail was fixed a long 16 time ago. Closing call." 17 A. Mm-hm. 18 Q. You had earlier suggested that the root cause be 19 investigated and fixed. 20 A. No, I think I had said just change where it was 21 application mails, just make sure -- 22 Q. Do you want to just go back up to that at page 1 23 the last few lines? 24 A. Yeah, sorry. 25 Q. Page 1, scroll down. Thank you. 101 1 A. No, I said we need to sort out the applications 2 mails issue. That wasn't the root cause. The 3 root cause was the null mode attribute. 4 Q. So the message you wrote at the end "I never 5 really expected the root cause to be 6 investigated or fixed" -- 7 A. Yeah, that's the messages with the null mode 8 attribute. 9 Q. I see. So, in essence, you're saying here that 10 what you expected to be done was done; is that 11 right? 12 A. Yes, yeah. 13 Q. Thank you. That can come down. 14 In your witness statement, paragraph 17, 15 there's no need to turn it up, you say that the 16 SSC and fourth line support development did not 17 always know how many branches had reported 18 a particular problem because the tickets 19 reporting that problem hadn't been sent through 20 to the SSC. Yes? 21 A. Yeah. 22 Q. Was that a problem? 23 A. Um, it was if we didn't therefore have an idea 24 of the scale of a problem or how many branches 25 were being affected. 102 1 Q. Was that known within Fujitsu, that there was 2 a problem in the design of the system as 3 a whole, in that relevant information was not 4 being passed up to the SSC? 5 A. I don't know. Um ... 6 Q. Well, you've identified this as a problem. 7 A. Mm. 8 Q. Was that problem discussed amongst your team, 9 raised with your manager and then escalated 10 within Fujitsu and then within the Post Office? 11 A. Not to my knowledge. 12 Q. Why not? 13 A. But it -- 14 Q. If this was a problem in the system that meant 15 that the scale of any identified defect was not 16 known, why wasn't that addressed? 17 A. I don't know. 18 Q. It would be relevant -- 19 A. There would be other ways of finding out that 20 information. I mean, it would depend very much 21 on the type of problem but we're not talking 22 here about things that would have a -- well, we 23 shouldn't be talking about things that would 24 have a significant effect on individual 25 branches. 103 1 Q. How would you -- 2 A. But I think it was just understood that, you 3 know, this was the process, was that first and 4 second line are meant to filter out known 5 errors. That is why they are there. If you're 6 saying "Well, first and second line need to send 7 everything through anyway", then you can almost 8 say what's the point of having them? 9 Q. Would you agree, though, that with the benefit 10 of hindsight, this is a problem or a defect 11 within the system? 12 A. I think there are -- there were certainly some 13 areas where it would have been a lot better if 14 SSC or somebody had had more of an insight into 15 how particular problems were affecting the 16 entire estate. 17 Q. I mean, that's a key issue, is it not, for both 18 the Post Office and Fujitsu: a person has 19 identified a problem, to what extent has that 20 problem in the past afflicted the estate or to 21 what extent is it currently afflicting the Post 22 Office estate? 23 A. Yes, but there are other ways of finding out 24 that information besides having calls passed 25 through for every time it's reported. I mean 104 1 you could argue that then you're dependent on 2 the postmaster actually noticing and reporting 3 the problem and we'd also have other means of 4 seeing how often a specific problem had occurred 5 by, for example, checking the events for the 6 whole estate over a certain period of time. It 7 would just depend what the signature of the 8 problem was. In many cases, we would be able to 9 see where else had been affected. 10 Q. Was that habitually done, that when a problem -- 11 when a ticket came in, you would adopt that 12 approach of checking the message store for the 13 entirety of the estate? 14 A. Not when a ticket came in but when an underlying 15 problem had been identified, then certainly, 16 later on in the life of Horizon and, in 17 particular, HNG-X, yes, that was done very 18 rigorously. But I cannot say that that was done 19 the whole time -- 20 Q. When you say later on -- I'm sorry I spoke over 21 you. When you say "later on", what do you mean? 22 A. Certainly from the introduction of HNG-X, 23 I think we got a lot better at tracking down 24 every instance of problems without the 25 postmaster having to report it to us. 105 1 Q. Why wasn't this done before the introduction of 2 Horizon Online in 2010? 3 A. I think in some cases it was done but perhaps 4 not so rigorously and I don't know why not. 5 Q. Well, can you help, please? What determined 6 whether before 2010 you would say, "Well, I've 7 got a problem here, it's on this ticket. I need 8 to make a decision on whether to check the 9 extent to which this problem is afflicting other 10 parts of the estate"? 11 A. It would, again, partly depend on what type of 12 a problem it was, how easy it was to do the sort 13 of checks. Um -- 14 Q. Were there any rules on this? Anything written 15 down? 16 A. No, I don't believe so. 17 Q. Was it down to individual discretion of the 25 18 of you? 19 A. Yes, or probably more down to that. But, 20 I mean, a problem of any scale, it's unlikely 21 that it's just one person ends up looking at it 22 anyway, so you wouldn't -- it wouldn't be sort 23 of one in 25. But -- and it would depend on 24 obviously on what sort of a problem was and what 25 sort of impact it was having. 106 1 Q. What do you mean by what sort of impact -- 2 A. Well, if it was affecting branch accounts in 3 some way. 4 Q. If it was affecting branch accounts in some 5 way -- 6 A. Then -- 7 Q. -- then would you habitually do -- 8 A. Then you would -- 9 Q. Hold on. Would you do an estate-wide check to 10 see whether other branches' accounts had been 11 afflicted by the problem identified? 12 A. You might well attempt to. 13 Q. What would determine whether you might well? 14 A. Whether there was some clear-cut way of 15 identifying these branches that -- 16 Q. Was there a clear-cut way of identifying -- 17 A. It would depend on the problem. You can't 18 generalise. 19 Q. Would you escalate issues where you think this 20 is a significant issue that might afflict other 21 branches or some other branches? 22 A. Again, I think that was something we -- that 23 happened more subsequently -- with HNG-X 24 onwards. 25 Q. You addressed this, if we can look at it, 107 1 please, in paragraph 53 of your witness 2 statement, which is on page 17. You say in the 3 first sentence: 4 "I am asked whether Fujitsu took proactive 5 steps to identify bugs and/or discrepancies in 6 branch accounts caused by the same." 7 Then reading on four or five lines you say: 8 "If a bug was found to be affecting branch 9 accounts which had not caused a reconciliation 10 report entry, we would do our best to identify 11 all branches affected, as we did for Bug 3. 12 However, I cannot say that this was done 13 consistently for all bugs ever found, especially 14 in the early days of the project." 15 So my question is why was it not done 16 consistently for all bugs? 17 A. Yes, I think perhaps we -- I don't know. 18 I can't answer that question. 19 Q. To what extent was there liaison with the Post 20 Office when identifying whether there should be 21 an estate-wide search for the extent of 22 a problem? 23 A. Um, again, I know that a lot of the problems 24 that happened, HNG-X onwards, there was liaison 25 with them and discussion as to how to do these 108 1 searches and who should be undertaking them. 2 Q. What about the decade before then? 3 A. Um, I think one reason I'm finding it so hard to 4 remember is that I've seen very little -- very 5 few PEAKs and things from that period, sort of 6 2007 to 2010 in particular. So I -- really, I'm 7 finding it very, very hard to remember, you 8 know. I can't turn around and say, "Oh, yes, we 9 did this, this and this", because my memory 10 hasn't really been jogged by specific instances. 11 Q. How did you interact with the MSU in this 12 situation? 13 A. Er -- 14 Q. Did you highlight this kind of situation to them 15 so that they could let all branches know to be 16 on the lookout for a known problem? 17 A. Um, that wasn't really MSU's role. That was 18 very much a branch-by-branch basis. 19 Q. Was there any liaison with the Post Office 20 about -- I'm talking about pre-2010, here -- 21 notification of the discovery of a bug and to 22 look out for symptoms or signs of its existence? 23 A. I'm sure there were some but I've not seen any 24 specific examples in the last six months. 25 Q. You say in paragraph 212 of your witness 109 1 statement, if we go to that, please, it's right 2 at the end on page 63: 3 "A point of frustration with the system, was 4 that the users, namely the subpostmasters, were 5 not our clients and there was a practically 6 limit as to the extent to which we could work 7 together with them to investigate problems." 8 Are you, by this, suggesting each problem 9 that was called in by a subpostmaster was 10 treated separately and that as a result, there 11 wasn't any oversight of any wider system issues? 12 A. No, I think what I was trying to say there was 13 that, you know, there were cases where you just 14 wished you had some way of knowing what had 15 happened at the branch, and could -- and there 16 was some way of getting some more information. 17 Q. Did you, in the first ten years of the project, 18 take the view that tickets were sent in, were 19 addressed piecemeal, one by one, problems were 20 patched up, and there was no analysis of whether 21 the issues the fundamental issues raised by the 22 tickets needed to be addressed coherently 23 through any redevelopment or redesign? 24 A. Um, no, I think there was -- there certainly was 25 analysis done and, where we saw patterns 110 1 emerging, we did try to make sure those problems 2 were progressed. 3 Q. Can we look at an example, please, of where you 4 suggested, I think, that others take a wider 5 look at the system. FUJ00086490, thank you. If 6 we expand it. We're going to come back to this 7 PEAK in due course when we look at Bug 10, the 8 data tree field -- sorry, the data tree build 9 failure. I just want to look at it now to 10 explore couple of issues with you, just on the 11 back of this PEAK. 12 You'll see that the summary of the problem 13 is discrepancies when declaring euros or cash. 14 Can you see that? It's the sixth line in. 15 Thank you. 16 We can see, if we scroll down a little bit 17 that it was raised on 18 May 2007, under the 18 first entry on the progress narrative. Then if 19 we can look at page 3 of this PEAK, please, and 20 the entry at 10.45.06 on 24 May made by you, and 21 you say: 22 "This branch reports they have been having 23 problems since March 2006 ..." 24 Remembering we're now in May 2007: 25 "... where they do declarations, do further 111 1 transactions (usually transfers or rems), 2 redeclare and then get a discrepancy equal to 3 the value of the recent transactions. 4 "Two recent examples (all times UTC). 5 "Wednesday 16th May: 2000 euros transferred 6 out at 14.23 then reported as a discrepancy 7 during trial balance at 15.54. 8 "Monday 21st May: 8 cash transactions 9 totalling £2,465 between 9.35 and 10.16, ignored 10 when cash declared subsequently at 10.20. 11 "The problems are always in stock unit MS, 12 which is an individual stock unit (hence a 13 variance check is run automatically after a cash 14 declaration). Almost always using the gateway 15 for this stock unit. 16 "These types of errors are not that unusual 17 these days (I would say much less rare than 18 a couple of years ago?) We usually quote [then 19 you give a KEL] -- I think the outcome of those 20 investigations was that the Riposte message port 21 was possibly sometimes failing. 22 "However, given that this problem is not 23 uncommon at this branch, and also that [another 24 PEAK] found a problem within Horizon that gave 25 very similar consequences, I think this might be 112 1 worth a recheck. I appreciate that with 2 [Horizon Online] coming up, it may not be worth 3 pursuing this, but this problem is causing 4 a number of postmasters to have serious doubts 5 about the reliability of their systems since 6 they are very displaying incorrect figures." 7 If we also look at your entry on page 4 at 8 11.16.12, just at the foot of the page there, 9 thank you. So we're three months on now. 10 Sorry, two months on now. 11 "This problem is continuing to affect 12 a number of branches, especially larger ones 13 where they have individual stock units and 14 commonly where they have transferred out cash 15 (which then appears as if it is still in the 16 stock unit -- hence they think there is 17 something wrong with the transfer mechanism). 18 The [postmasters] are not all happy that the 19 system is giving them incorrect figures, causing 20 much confusion and potentially creating 21 opportunities for fraud." 22 So in this PEAK, we see how a multi-branch 23 issue involving discrepancies was suggested to 24 be brought together for a proper fix by you; is 25 that right? 113 1 A. Yes. 2 Q. But would this be right: that was as a result of 3 your essentially anecdotal experience of 4 an increasing number of calls of a similar 5 nature? 6 A. Yes, I was probably keeping an eye out for these 7 type of calls. 8 Q. You kept an eye out -- 9 A. Mm. 10 Q. -- for these calls, and some branches had 11 experienced a continuing problem for a number of 12 years and various workarounds were employed? 13 A. Yes. 14 Q. But, ultimately, the suggestion of a fix was 15 essentially ad hoc by you, wasn't it? 16 A. I'm not sure what you mean. 17 Q. I mean that it was just you keeping a weather 18 eye out for whether this was a problem that was 19 affecting consistently a number of branches 20 rather than the system that Fujitsu employed 21 itself detecting whether this was a consistent 22 and ongoing problem? 23 A. Yes. 24 Q. Was that essentially the approach adopted in the 25 SSC? It was down to you, as an individual, to 114 1 keep a weather eye out to see whether there was 2 a big problem or not? 3 A. Yes, I think that's probably true. 4 Q. Would you accept that that rather patchwork 5 approach, dependent on 25 individuals not each 6 knowing what the other is doing, not each 7 knowing what the other is seeing, may leave 8 frailties in the system? 9 A. It could do, although I would say -- I mean, 10 I wasn't just looking at the calls that I had 11 dealt with. I would have been -- have noticed 12 that some of my colleagues had also had similar 13 calls, and in fact we would have discussed it 14 together. But the process that we were meant to 15 follow was to, you know, you passed the call 16 over to Development, and then it is their 17 responsibility to produce a fix at some point in 18 the future. 19 I mean, there were teams, or there were 20 people who would see which calls were with 21 Development sort of potentially in the pipeline 22 and so on. But yes, them actually having -- 23 anybody having the knowledge of something like 24 this, where it was a continuing, potentially 25 serious, problem, that information was not 115 1 necessarily being fed in. 2 MR BEER: Thank you very much, Mrs Chambers. 3 If that's an appropriate moment for you, 4 sir, I would suggest that we break until 2.00. 5 SIR WYN WILLIAMS: Well, to give everybody their 6 full hour, 2.05, Mr Beer, provided the time on 7 my machine in front of me is accurate at 13.03. 8 MR BEER: I've got 12.59. 9 SIR WYN WILLIAMS: Really? Oh. It just shows how 10 reliable this system is. Fine. Let's 11 compromise: 2.02. 12 MR BEER: Thank you very much, sir. 13 (1.00 pm) 14 (The Short Adjournment) 15 (2.00 pm) 16 MR BEER: Thank you, sir. Mrs Chambers, can we 17 continue to examine in general terms how tickets 18 were processed in the SSC focusing in particular 19 on the extent to which it was a part of the 20 system to identify the extent to which an error, 21 bug or a defect afflicted other parts of the 22 estate. 23 Can we start by looking, please, at 24 FUJ00082274. This is a KEL titled "Multiple 25 cash declarations may cause incorrect figures in 116 1 Discrepancy, Variance and Balance Reports". It 2 was raised, you'll see, by Mark Scardifield on 3 15 July 2005, and was last updated by you on 4 27 November 2007. 5 If we look at the symptoms, please, thank 6 you: 7 "A cash declaration was made in 'Stock 8 Balancing' for the amount displayed on the 9 Snapshot. When the Cash Variance was checked 10 afterwards a Gain of £45.05 was displayed [then 11 a string of characters]. May get [postmasters] 12 calling in to state that they've been declaring 13 cash but they have been getting varying 14 discrepancies reported even though they've been 15 declaring the same amount of cash each time. Or 16 that they have done a transfer but are then 17 getting a discrepancy equal to the amount of the 18 transfer, or that the system hasn't transferred 19 the cash out of the stock unit." 20 In terms of the "Problem": 21 "The underlying problem is that we cache the 22 current trading position for a Stock Unit and 23 rely on a mechanism (in Riposte) to notify us of 24 new transactions across the outlet to keep this 25 cache up to date. When this fails it affects 117 1 Discrepancy, Variance and Balance Reports and 2 has the effect of presenting the clerk with the 3 incorrect information. This will be potentially 4 confusing and may lead to the clerk making 5 unnecessary corrections. These will in turn 6 show up as future inconsistencies (eg nothing 7 gets lost in the end)." 8 Is this right, that this is identifying 9 another problem with the Riposte system? 10 A. Yes, I think the underlying error was thought to 11 be in Riposte itself. 12 Q. Is it identifying a known problem with Riposte? 13 A. It's identifying a problem in Riposte. I'm 14 not -- it's known, in that it's now been 15 documented. 16 Q. I see. So it's recording that some figures 17 might be wrong, which might lead a subpostmaster 18 or a clerk to make updating changes, which would 19 then create a problem when the figures were 20 corrected? 21 A. If they did -- if they were to make those 22 changes, then yes, they would then have to undo 23 those extra changes. 24 Q. So in order to ensure no losses, if the 25 postmaster made the unnecessary corrections, 118 1 would they need to recognise the error and 2 reconcile it, whether by making a transaction 3 correction or getting a transaction correction, 4 or otherwise? 5 A. Yes, it would depend at what point they noticed 6 that something was wrong. In the cases that I 7 can remember seeing, including the call that we 8 looked at this morning, the postmaster was very 9 aware that these problems -- that the figures 10 were wrong and phoned in to complain and was 11 advised to reboot, and so on, to avoid it. So, 12 in that case, he or she didn't take any extra 13 action, except doing a reboot and then when they 14 started again the figures were all correct. 15 Q. The solution is given as: 16 "The Declare Cash problem clears itself 17 overnight. If the [postmaster] logs a call on 18 the day as having problems, ask him to try the 19 following workaround. The clerk should log out 20 of the affected counter. Another clerk attached 21 to a different (individual, not shared) stock 22 unit should log into the same counter, declare 23 cash for his own stock unit, then log out. The 24 first clerk can now log in to the same counter 25 and declare cash again. The variance should be 119 1 correctly recalculated. Alternatively log on to 2 a different counter and do the cash declaration 3 there. If the workaround is not successful or 4 the problem does not clear itself overnight, 5 send a call to the SSC, otherwise no call is 6 needed. November 2007: a fix is being piloted 7 and is likely to be sent to the whole estate in 8 January." 9 Presumably that will be January 2008? 10 A. Yeah. 11 Q. So that fix appears to be planned to be sent out 12 in January 2008 for a problem that was 13 identified in July 2005? 14 A. I think the problem in 2005, the one that was 15 found during testing, it's not -- I can't tell 16 if it was exactly the same problem that then 17 persisted. I don't recall -- well, from what 18 I've seen on the various bits of information 19 that -- documents that I've reviewed about this 20 problem, it looks as if it started happening in 21 Live, or at least was noticed more in Live, 22 sometime in 2006. 23 Q. But still a year and a half, two years? 24 A. It's still a very long time, yeah. 25 Q. Can you explain was that normal, it would take 120 1 a year and a half, two years, for something to 2 work its way through the system for a fix? 3 A. That was unusual, it depends so much on the sort 4 of what problem it is. I mean, if it's a very 5 contained problem, easy to reproduce, "We know 6 if we do ABC, this is going to happen", then 7 those sort of problems are usually relatively 8 easy to fix and to test and to get the fix out 9 into Live. This was not such a straightforward 10 problem. I'm not sure if anybody ever actually 11 managed to replicate it. We didn't have 12 a sequence of events that would make it happen, 13 and so, in the end, the fix, I believe, was 14 quite a large one, which involved sort of 15 redesigning and rewriting an area of code to 16 sort of completely change the way it worked, and 17 so, obviously, that's both a bigger job to do, 18 and then it has to go through a full test cycle 19 and because it's -- something like that could 20 have -- is more likely to have knock-on effects 21 than something which is very self-contained. 22 Q. At the foot of the page under "Evidence" the KEL 23 says: 24 "If there is any pattern of failure, eg it 25 is occurring on the same counter repeatedly, can 121 1 the audit log ... be recovered of this occurs 2 please?" 3 A. That's obviously the counter log that I had 4 forgotten about. I had a feeling that there 5 might have been another one and that must be it. 6 Q. Would that log not routinely be obtained where 7 there was a concern over a system failure raised 8 like this? 9 A. Yes, it probably was, because I think that's 10 probably the one that had some diagnostics and 11 things written to it. 12 Q. That's what I was going to ask. What is the 13 audit log being referred to there? 14 A. As I said, I think that must be the file, and 15 there was -- probably looking at that, I'm not 16 sure if that was a day of the week, so there 17 would only have been seven of them or if that 18 would have been a date in there, but it wouldn't 19 be something that persisted for very long. 20 Q. Why would the audit log only be of interest in 21 if there was a repeated failure? 22 A. Um, I think it's because then -- no, it probably 23 would have been of interest anyway. 24 Q. Why does it say, if there's a pattern of 25 failure -- 122 1 A. I don't know. I don't know who put that on 2 there. 3 Q. -- obtain the audit book? 4 Can you think of a reason why one would only 5 obtain the audit log if there was a pattern of 6 failure? 7 A. No. I think you'd probably obtain it anyway 8 but, as I said, I'm afraid until this moment 9 I had totally forgotten its existence. 10 Q. Can we look at a PEAK, please, POL00028746. 11 This is a PEAK 0103864 raised, you'll see under 12 the "Progress Narrative", on 3 June 2004, after 13 a subpostmaster reported problems with 14 transfers. You can see that four lines in: 15 "Summary: [postmaster] reports that he had 16 a problem with some transfers." 17 Yes? Can you see that? 18 A. Yes, sorry. 19 Q. Thank you. Was the problem on this, recorded by 20 this PEAK, a discrepancy arising after transfers 21 which were being replicated twice? 22 A. Yes, they -- you should only -- this -- 23 a transfer is when you're moving cash or stock 24 from one stock unit to another, and you should 25 only have been allowed, for one transfer out, 123 1 just to do the transfer in the one time. If, 2 for some reason -- and there were various 3 underlying reasons why it might be possible -- 4 you could do the same, accept it in twice, then 5 you would have a discrepancy and also a receipts 6 and payments mismatch. 7 Q. If we look at the foot of the page, please, at 8 the entry for the 3 June at 11.25, about 10, 12 9 lines up from the bottom, thank you: 10 "Information: [postmaster] reports that the 11 transaction log shows only one transfer out for 12 each item but the transfers in show so that each 13 transaction has been accepted into the BB ..." 14 A. That would be the name of the stock unit. 15 Q. "... twice and this has caused him a discrepancy 16 and he would like this investigated. This call 17 was passed to HSH from Tier two at the NBSC and 18 they have also requested that the problem be 19 investigated." 20 Then if we see what happened, if we go over 21 the page, please, and look at the foot of the 22 page at -- if we go down a bit please. 10.19.14 23 on 8 June, passed, I think, to your colleague 24 Catherine Obeng, yes? 25 A. Mm-hm. 124 1 Q. "All nodes were connected at the time that the 2 transfer in [transactions] were attempted. 3 "No session transfers took place that day. 4 "Event log from node 4 suggests that Riposte 5 replication had not been successful and so while 6 node 3 had successfully TI the [transactions], 7 this information was not apparent to node 4 thus 8 it was perceived by node 4 that those 9 [transactions] were outstanding waiting to be 10 TI. Therefore when the user ... was logged on 11 to node 4, he was presented with Outstanding 12 Transfer message which had to be accepted or 13 declined. The user chose to accept them even 14 though he tells me that at this stage he was 15 a little concerned because he was certain that 16 [the other user] had already transactioned on 17 node 3. This has created a discrepancy on their 18 Cash Account of £22,290.00. Also the host has 19 reported a reconciliation error ... for 20 £44,580.00." 21 A. Yeah. 22 Q. So, essentially, reading this, was this 23 a software error which had then been compounded 24 by an action taken by the subpostmaster in the 25 face of an apparent error? 125 1 A. Um, that looks like the likely reason. 2 Q. A significant sum of money was involved -- 3 A. Yes. 4 Q. -- by way of discrepancy? 5 A. Yes, and it did show up on one of our automatic 6 reconciliation reports so, even if the 7 postmaster hadn't realised what had happened, we 8 would have been investigating it anyway. 9 Q. How complete was the coverage of that automated 10 system? 11 A. Certainly anything that showed up -- I mean, 12 yes, all entries on the major reconciliation 13 reports would have calls raising for them and 14 SSC would have looked at each one. 15 Q. We can see if we go to page 4, please, that 16 two-thirds of the way down, at an entry for 17 15.11.29, Martin McConnell says: 18 "Attached will be a spreadsheet and a set of 19 transactions that I think are responsible for 20 the error. The spreadsheet is ALL transactions 21 that account to Stock Unit BB and I have 22 presented at data view for the discrepancies 23 committed. I do not even know if I am expected 24 to be doing a summary inspection nevertheless 25 the attached give a 'view' as stated. 126 1 "As far as the system is concerned, there 2 seems to be a flaw in Riposte informing 3 Counter 4 that the transfer object [a number of 4 stock unit BB is given] has been raised. This 5 is an Escher Riposte problem as far as I am 6 concerned, the transfer mechanism already has 7 a belt and basis approach. If we can't trust 8 the underlying software to replicate this 9 information through to all other counters, what 10 are we supposed to be able to do? If someone 11 wants to press for a fix, I suggest this is 12 pointed at the Escher team for them to sort 13 out." 14 The question there -- and I realise this 15 isn't you writing it -- "if someone wants to 16 press for a fix", why might that even be 17 an issue, whether you would press for a fix or 18 not? A substantial sum of money, by way of 19 discrepancy here, drawn to the attention of 20 Fujitsu by the subpostmasters themselves, what 21 would be the reasons for not pressing for a fix? 22 A. Whether it could be worked out in precisely what 23 circumstances this error had occurred, 24 I suppose. I mean, these transfers were 25 happening successfully at many branches at many 127 1 times, so I agree there's investigation needed 2 to try to find out why this particular one 3 didn't work. Could we see if Catherine said 4 anything else before she passed it over to 5 development? Let's go up a little bit. 6 Q. Let's go up to page 3. Then scroll down. 7 A. Right, so maybe there isn't. I'm sorry, I can't 8 see the whole picture. 9 Q. If we zoom out, please, then it becomes too 10 small. If you zoom in and just go to the entry 11 starting 8 June at 14.45.45, thank you. 12 A. I just wondered if there had been any other 13 information, any events or anything that gave 14 any more of a clue as to why the problem had 15 occurred, but I can't see anything else that's 16 been recorded on there. So yes, I mean, 17 obviously, you'll think you want this problem 18 fixed but, if you can't reproduce the problem or 19 get any closer to the circumstances which has 20 caused it, it's not necessarily going to be 21 fixable. 22 Q. So what, it's an error that just stays on the 23 system? 24 A. If it's something that has happened this one 25 time and we cannot see any reason for it, then 128 1 possibly, yes. 2 Q. What about if it was for a smaller sum of money, 3 for £100, and the subpostmaster was told before 4 it got to the SSC, "You just need to make that 5 up, that £100"? 6 A. No, anything like that, because -- as well as it 7 being reported by the branch, it would also have 8 been on the reconciliation reports, and we 9 looked at those regardless of the amount of 10 money -- 11 Q. There was -- 12 A. -- because it is -- 13 Q. I'm sorry. 14 A. No, because it is indicating, yes, there is 15 a fault in the system. 16 Q. Can we go on, please, to page 5, and look at the 17 second, third entry on the page, also from your 18 colleague Catherine Obeng at 16.30.17: 19 "Martin McConnell's recommendation is to put 20 the £22,290.00 which is adrift into the Suspense 21 Account." 22 Was money adrift at this point? Is that the 23 right way to describe it? 24 A. Well, it's a loss that the branch should not 25 have been liable for. 129 1 Q. "I am routing call [says Ms Obeng] to MSU to 2 raise an Error Notice to inform POL of this 3 incident. Also please notify NBSC to contact 4 the [postmaster] and advise him of this course 5 of action." 6 So we're now, I think, on 11 June, the 7 incident having arisen on 3 June. 8 A. Yeah. 9 Q. So somebody was to contact the NBSC -- the MSU 10 were? 11 A. Um, the first or second line support would 12 probably be expected to contact NBSC. 13 Q. Then NBSC have got to contact the subpostmaster 14 themselves? 15 A. That's what it says here, yes. 16 Q. How would NBSC communicate accurately what had 17 been established on the previous five pages of 18 this document? 19 A. I don't know. It's not made terribly clear 20 there that, you know, this was system error, but 21 that we had informed Post Office and that the 22 branch should expect an error notice in -- at 23 some point. There is an error there in that it 24 says calls routed to MSU to raise an error 25 notice. MSU couldn't do that, they would raise 130 1 a BIMS report, which would be sent to the Post 2 Office financial people and then they could 3 raise the error notice. 4 Q. "BIMS" being a Business Incident Management 5 report? 6 A. Could well be. 7 Q. That was the acronym used, I think, at the time? 8 A. Right, I couldn't remember what it stood for. 9 Q. Okay. So here we are at 11 June with the SSC 10 telling other parts of Fujitsu to tell the Post 11 Office to contact the subpostmaster to inform 12 him of this incident. Did the NBSC have access 13 to this PEAK? 14 A. The postmaster already knew about the incident 15 because he had raised the call in the first 16 place. 17 Q. Yes. Sorry, to inform POL of this incident: 18 "... please notify the NBSC to contact the 19 [postmaster] and advise him of this course of 20 action." 21 Did the NBSC have access to this PEAK so 22 that they could say to the subpostmaster "Don't 23 worry, it's not you, there's a bug"? 24 A. I don't know, and I don't know what the second 25 or first line support, who would have notified 131 1 the NBSC, would have passed through to the 2 postmaster. 3 Q. Then if we scroll down, please, to 22 June, 4 15.36.46: 5 "The Call record has been transferred to the 6 team: MSU ..." 7 Yes? 8 A. Yes. 9 Q. Then if we look at the entry immediately below 10 from Michael King: 11 "Reconciliation data provided to POL. 12 Routing back to EDSC, does this need to be 13 routed to Escher for a fix?" 14 From this record, can you see whether any 15 action had been taken to progress any fix by 16 Fujitsu with Escher? 17 A. Not at that point because that's Michael King in 18 MSU who had informed Post Office via a BIMS and 19 now he's routing the call back to EDSC, which is 20 SSC. 21 Q. Then if we scroll down to 11:59:27 on 23 June, 22 where Ms Obeng comments: 23 "Problem could have been avoided if the 24 [postmaster] had not accepted the second TI. 25 The Riposte Error events were apparent on SMC's 132 1 Tivoli Website, however, they took no action 2 until the [postmaster] raised the call about the 3 dodgy Transfers. In future, SMC would need to 4 monitor these events and contact the office 5 requesting that they avoid using the eventing 6 Node and reboot it. [A KEL has now been] 7 updated with info on what action SMC should take 8 if event occurs during [Post Office] business 9 hours." 10 A. Right. Belatedly, I now have the information 11 that there were Riposte error events which were 12 occurring. That wasn't mentioned earlier in the 13 PEAK, which it very usefully could have been 14 done, so I think this is Bug 2 in the list of 15 bugs -- 16 Q. Which we're going to come to in more detail 17 later today or tomorrow? 18 A. -- which we'll come to in more detail. But that 19 explains why the replication didn't happen and 20 what the cause of this particular problem was. 21 Q. But this bug appears to have been picked up by 22 reason of a call from the subpostmaster -- 23 A. Yes. 24 Q. -- not by Fujitsu's reconciliation process? 25 A. Oh, there would have been separate calls for 133 1 those as well but they haven't been linked on 2 here. But it does say further up in the call 3 that the branches appeared on I think at least 4 three of the reports. 5 Q. It appears to be accepted that there's a problem 6 with Riposte failing appropriately to pick up 7 the transactions. We've seen so far the 8 possibility of a fix by Escher has been raised. 9 Here, Ms Obeng is suggesting that this could 10 have all been avoided if the postmaster hadn't 11 accepted the second transaction? 12 A. That is strictly true but unhelpful, in that the 13 postmaster was prompted to accept it, and did 14 so. So I certainly would not say it was in any 15 way his fault. Because it was not. 16 Q. Can we go on to page 6, please, and look at the 17 entry for 11.17.09, "customer call"? 18 "Repeat call: auditor for [Post Office] site 19 had called in to see what is happening with 20 call. 21 "Advised I will call back, ringing through 22 to SSC Barbara to find out what's happening ... 23 "Spoke with Barbara [and] SSC advised 24 Catherine is on leave and she will try and get 25 someone to look at call and call me back." 134 1 So it seems like either the subpostmaster or 2 an auditor in the branch is chasing now on the 3 6 July. 4 A. Yes, what was the previous date? 5 Q. Well, the call was opened on 3 June. 6 A. Right, yeah. 7 Q. The previous entry by Ms Obeng was 11 June and 8 then 23 June. 9 A. Yeah. So it sounds as if nothing has been 10 resolved as far as the postmaster is concerned, 11 and now there is an auditor on site. 12 Q. Then the entry below, at 11.20.40: 13 "... spoke with auditor John, advised third 14 line are going to look at the call and call 15 back. Advised that I will call them back as 16 soon as I can." 17 Yes? 18 A. Yes. 19 Q. Then I think later that day, at 11.44 is the 20 first time that you become involved? 21 A. The first time that my name has appeared on 22 here, certainly. 23 Q. Yes. You say: 24 "I've checked with Mike King, the BIMS 25 report for this problem was sent to [the Post 135 1 Office] on [22 June] and should have resulted in 2 an error notice being sent to the branch. Mike 3 says he'll send a note to [the Post Office] 4 saying that the [postmaster] has been chasing 5 this issue; I've asked HSH to inform the 6 [postmaster] that they should have received an 7 error notice and to check with the department 8 that issues them. 9 "The corrected cash account that was sent 10 still had [a receipts and payments] mismatch. 11 The double Transfer In causes the mismatch both 12 because of the transfer and because of the 13 discrepancy which has been erroneously 14 generated. The host-calculated CA" -- 15 A. Cash Account. 16 Q. -- "ignores the transfer but is still affected 17 by the accepted discrepancy which should not 18 have been generated. It has not really been 19 possible to provide fully balance CA (email on 20 this subject sent by Mik Peach to Richard 21 Brunskill then on to John Moran, I have not seen 22 the outcome of this)." 23 So you say that a BIMS notice was sent in 24 fact way back on 22 June -- 25 A. Yes. 136 1 Q. -- and should have been sent to the branch? 2 A. Should have resulted in an error notice being 3 sent to the branch. 4 Q. Yes. Did you speak to the auditor at all? 5 A. It doesn't look like it, no. 6 Q. Can you see from this whether anyone spoke to 7 the auditor? 8 A. Um, the helpdesk were asked to inform the 9 postmaster but I don't think we ever had a phone 10 number for the auditor or I certainly didn't 11 have a phone number for the auditor, and HSH had 12 been the ones speaking to him. 13 Q. If we can go on to 12 July, please. Thank you. 14 If we look at the last entry for 12 July, the 15 one timed at 12.56.23: 16 "An add has been sent to PinICL [and then 17 number is given]. Tina NBSC confirms there is 18 an outstanding error notice but she could not 19 get exact details -- she will call Paul 20 Whittaker, the investigation officer on [and 21 then a number is given]." 22 So it appears that by 12 July 23 an investigations manager is now involved. 24 A. Um, it's -- it looks like it from what's 25 recorded there. 137 1 Q. That appears to be as a result of the call 2 earlier in the day, if you just look at the top 3 of the page we're looking at, at 11.58.56: 4 "Investigation manager Paul Whittaker wants 5 to confirm that an error notice is being sent 6 out for the discrepancy at the [Post Office]", 7 and then the call was transferred to the NBSC. 8 A. Yes, because error notices were not visible in 9 any way to Fujitsu. 10 Q. Then I think we've no need to look at it, on 11 5 August it's confirmed that an error notice has 12 indeed been issued? 13 A. Yes. 14 Q. Taking a step back, would this be right, and 15 realising that you had transient involvement in 16 the middle of this process, that a period of 17 time between 3 June and 5 August, some three 18 months, appears to show the subpostmaster having 19 an auditor and an investigator in his branch at 20 the same time that there was an identified 21 system problem which caused the discrepancy, not 22 him. 23 A. Yes, it does look like that, except it's only 24 two months, but I agree with what you're saying. 25 Q. Yes, you're quite right. Beginning of June to 138 1 beginning of August, you're quite right. 2 A. Yes, and I think, as far as we were concerned, 3 we had informed Post Office and the ball was 4 then in their court to produce an error notice 5 to sort out the impact on the branch, but that 6 obviously wasn't -- presumably wasn't known 7 clearly what had happened by all the people who 8 then were involved in a branch investigation. 9 Q. Was that a feature of your work: that there were 10 balls being passed between sides of the court, 11 between -- with one side of the court being 12 Fujitsu and the other side of the court being 13 the Post Office, and you would bat it over to 14 the other side of the court? 15 A. I wouldn't put it in those terms but I think we 16 felt, when we had done our side of the 17 investigation and notified Post Office of this, 18 and I -- then it was up to them, really, to 19 produce the error notice and resolve the 20 implications. 21 Q. Would you agree that a fair reading of these 22 exchanges is that the only reason this was 23 bottomed out was due, firstly, to the 24 subpostmaster continuing to complain and, 25 secondly, in fact, due to your intervention? 139 1 A. The information had been sent to Post Office 2 saying that "This branch has made a loss as 3 a result of this problem, which now -- which 4 they shouldn't be liable for". That was done 5 well before my involvement. But whether Post 6 Office -- I don't recall that we got any -- 7 well, it doesn't look as if we got anything back 8 from Post Office themselves saying "We don't 9 understand this, what should we do about it", or 10 whatever. So I can't explain the delay in it 11 being resolved after 22 June. 12 Q. Was there a system in place to ensure that if 13 a discrepancy was identified as being caused by 14 a systems error, that everyone on both sides of 15 the court was aware of it as soon as possible, 16 and that no action was taken against the 17 subpostmaster, so that he wouldn't, as in this 18 example, end up with having an auditor and 19 an investigator in his branch? 20 A. No, I think once we had felt we had done our 21 side of things, then we wouldn't find out what 22 had happened between Post Office and the branch. 23 Q. Thank you. That can come down. Can we turn to 24 paragraph 20 of your witness statement, please, 25 where you address the issue of communication 140 1 with subpostmasters. 2 Paragraph 20 is on page 5. You say you were 3 asked to consider whether the system was 4 adequate to manage active service tickets. You 5 say: 6 "I did not consider that the system impeded 7 our work but I am asked specifically as to 8 whether I think there were potential changes 9 which would have improved the systems. The 10 problem I would identify was that the system 11 assumed only one person or team needed to be 12 actively working on the PEAK ticket at any one 13 time whereas this was not always the case. If 14 the postmaster who raised the service ticket 15 telephoned for an update, HSD might put that 16 request on the PowerHelp ... system which they 17 operated and that might not get copied to the 18 PEAK, depending on which option was chosen, 19 which might not be noticed by the developer or 20 tester who owned the call at that moment, who 21 would not expect to give an update to 22 a postmaster." 23 Before we explore exactly what you mean 24 there, can you remind us of what the PowerHelp 25 system was? 141 1 A. PowerHelp was the first and second line desk's 2 call logging system. 3 Q. So it was operated and viewable by only lines 1 4 and 2; is that right? 5 A. SSC could see the PowerHelp calls, as well. 6 Q. Did you habitually look at what was on 7 PowerHelp? 8 A. Quite often, yes. 9 Q. You say: 10 "If the postmaster who raised the service 11 ticket telephoned for an update, HSD might put 12 that request on the PowerHelp ... system and [it 13 might or might not] get copied to the PEAK ..." 14 Was there not an instruction to put that 15 request on PowerHelp and to copy it onto the 16 PEAK? 17 A. Yes, the way it worked, I can't now remember the 18 precise wording but, if they added an update on 19 to PowerHelp, one sort of update automatically 20 got copied to PEAK and the other sort didn't, 21 and the helpdesk didn't always use the right 22 category. 23 Q. Was anything done about that? 24 A. I think we used to remind them quite frequently 25 to use OTI update or something, I can't remember 142 1 what it's called, but I think I've seen a PEAK 2 where we specifically say, "Make sure you" -- 3 sorry, I've seen a KEL or something where we 4 specifically say, "If you have to -- if you get 5 any more information, use this particular way of 6 doing it". 7 Q. So this problem that you describe there may lead 8 to left hand not knowing what the right hand is 9 doing? 10 A. Yes, it can do or information just not being 11 passed through to the people who are probably 12 being expected to give an answer. 13 Q. In paragraph 21, you refer to another problem 14 with the PEAK system. You say that: 15 "... if a problem had a financial impact, 16 then it was necessary to inform the Post Office 17 (often via the MSU team), whilst separately, it 18 was necessary to look for the root cause(s), 19 which might involve passing the problem to the 20 Development team for consideration of a code 21 fix, and also to check whether any other 22 branches had been or might be affected by the 23 same problem. That might result in two or more 24 people trying to work on the same PEAK." 25 A. Yes. 143 1 Q. What was the difficulty caused by two or more 2 people working or trying to work on the same 3 PEAK? 4 A. I suppose, basically, you were meant to be 5 working on the PEAKs that were allocated to you 6 and, if it was allocated to somebody else, then 7 it wasn't allocated to you. I think we saw, in 8 that previous call that we've just looked at, 9 Catherine passed the call over to Development 10 for their comments, so Martin McConnell did some 11 work on it, and it was only after he had then, 12 I think, passed it back that she passed it then 13 to MSU for them to notify Post Office of the 14 problem. 15 Now, what I've -- I might well have passed 16 it to MSU first or, as I go on to say in the 17 next paragraph, to clone it so you could send 18 one call to MSU to pursue the rectification of 19 the deficit, the discrepancy that shouldn't have 20 occurred, and then the second call can go on to 21 Development at the same time, so they can start 22 investigating the root cause. 23 Q. Further on this issue of passing things back to 24 the Post Office and then down to the 25 subpostmaster, can we look please at 144 1 paragraph 42(iv), which is on page 12 of your 2 witness statement. 3 If you scroll down, thank you. 4 This is the part of your statement where 5 you're describing the teams that were working in 6 this area and you say: 7 "... MSU monitored Reconciliation Reports 8 generated by the system each day", et cetera. 9 Then the next sentence: 10 "MSU raised PEAKs so SSC could investigate 11 cause and impact of every entry on the 12 Reconciliation Reports." 13 Then you say: 14 "My understanding is that MSU informed 15 a team in Post Office of any such errors which 16 potentially had a financial impact on a branch 17 via a BIM report and that it was up to the Post 18 Office to notify the branch and make any 19 necessary correction." 20 That's essentially what you said a moment 21 ago. 22 A. Yes. 23 Q. How was it possible to determine whether 24 a problem had a financial impact or not? 25 A. Um, by us investigating what had happened, and, 145 1 you know, whether it did actually affect the 2 branch figures in any way. 3 Q. So you could tell for certain, one way or 4 another, whether a problem had a financial 5 impact on a subpostmaster or not? 6 A. Yes, I think we could. Because, you know, if 7 the system had ended up producing figures that 8 it would not have produced if the problem hadn't 9 occurred, then it's -- there could well be 10 a financial impact. 11 Q. Why was it necessary to inform the Post Office 12 of those problems, where a financial impact had 13 occurred? 14 A. So that the branch would not be held liable for 15 that. 16 Q. Was it the case that if a problem didn't have or 17 was judged to not have a financial impact, that 18 the Post Office were not informed of it? 19 A. They wouldn't be informed via that route, no. 20 Q. What different route would be used? What 21 different route was used? 22 A. Something that -- a different sort of a problem 23 that had perhaps affected a lot of the estate 24 would probably be notified to Post Office 25 through the problem managers. 146 1 Q. Did the Post Office inform you about the action 2 that they had taken when they were notified of 3 a problem that had a financial impact on 4 a subpostmaster? 5 A. No. 6 Q. Did you know whether they therefore told them 7 it's a system error, don't worry", or in fact 8 they held them responsible for it? 9 A. No. 10 Q. Why was there no record back to Fujitsu saying 11 what had happened? 12 A. Because that was not in the process, as far as 13 I'm aware. 14 Q. Do you know why it wasn't in the process? 15 A. No. 16 Q. At this time -- certainly between October 2000 17 and mid-2010 -- were you aware that the Post 18 Office was relying on the Horizon System to 19 produce data to prosecute subpostmasters? 20 A. I was aware that there were a small number of 21 prosecutions. 22 Q. More broadly, if we look, please, at 23 paragraph 53 of your witness statement, you 24 address a range of situations in which the Post 25 Office would have been informed in relation to 147 1 the operation of Horizon. So that's page 17, 2 please. You say: 3 "I am asked whether Fujitsu took proactive 4 steps to identify bugs and/or discrepancies in 5 branch accounts caused by the same. The 6 automatic cross-checks made and reported on the 7 TPS, APS and banking reconciliation reports 8 highlighted inconsistencies which might indicate 9 a bug. These were always investigated, and MSU 10 informed Post Office via a BIM report if the bug 11 had affected the branch accounts, or accounts 12 with other third parties." 13 In paragraph 66, which is on page 21, this 14 is in the middle of dealing with Bug 1, the 15 receipts and payments mismatch bug. At the 16 second line in paragraph 66, you say: 17 "Post Office would have been informed of 18 each instance, I am not sure whether this was 19 via a BIM report or some other route. Fujitsu 20 would not have contacted branches directly 21 unless the branch had raised a call in the first 22 place." 23 In paragraph 149, please. When dealing with 24 the counter replacement bug, that's page 44, at 25 paragraph 149, five lines from the bottom, you 148 1 say: 2 "Post Office would have been sent a BIMS 3 report if there was a lasting financial impact 4 for a particular branch but I do not know 5 whether it was ever flagged as an ongoing 6 problem." 7 Were you personally involved in creating the 8 BIMS reports? 9 A. No, but, as I think I say somewhere, I did at 10 some point find out that MSU were tending to 11 copy and paste whatever we put in the PEAK and 12 put that onto the BIMS report. So, indirectly, 13 yes, the words I was using were probably being 14 passed on. 15 Q. So you knew that what you were saying might end 16 up in a BIMS report? 17 A. At some point I became aware of that and I can't 18 remember when. 19 Q. So you didn't create the report? 20 A. No. 21 Q. But you provided information which you knew -- 22 A. Yes. 23 Q. -- might be used -- 24 A. Yes. 25 Q. -- in the creation of a report? 149 1 A. Yes. 2 Q. What do you know about the process by which BIMS 3 reports were created and then communicated to 4 the Post Office? How did that happen? 5 A. That was the job of the MSU staff and I don't 6 know any more than that. 7 Q. Was there a process in place to ensure that 8 certain types of information were always 9 reported to the Post Office? 10 A. I don't know. 11 Q. Where was the MSU sat within Fujitsu? Where was 12 it based? 13 A. I think they were on the fifth floor at one 14 point. I think by the time I left, they'd been 15 sort of subsumed into SSC but -- or at least sat 16 on the same floor as us. 17 Q. But take the period before the advent of Horizon 18 Online, so up to 2010. Were they essentially 19 a separate thing? 20 A. They were a separate unit and, at some points, 21 it was only sort of one or two members of staff, 22 as well. 23 Q. What was their training, do you know? 24 A. Their training? 25 Q. Training and expertise? 150 1 A. I don't know. 2 Q. Were they IT professionals? 3 A. I -- not particularly, I don't think. 4 Q. Were they more in the nature of administrators? 5 A. I would have said so, yes, but I had no formal 6 involvement with them. I did spend some time 7 with them, trying to help them understand, in 8 particular, the way that the system was working 9 to produce the reports, and also the flow of 10 banking messages between the Horizon System and 11 the banks and various other parties involved, 12 which were all part of the massive amount of 13 reconciliation that was done for every single 14 banking transaction, and that was -- I did do 15 a fair bit of trying to explain that to them 16 because then it just helped them understand what 17 some of the different numbers on the reports 18 meant for those entries. 19 Q. But the SSC was staffed with IT professionals? 20 A. Yes. 21 Q. You were conducting detailed investigations -- 22 A. Yes. 23 Q. -- on occasions into bugs, errors and defects? 24 A. Yes, anything over the entire very complicated 25 Horizon infrastructure. 151 1 Q. The principal method of transfer of that 2 information, as a result of the investigation, 3 was in a BIMS? 4 A. Yes, going back to the specific accounting type 5 problems, yes, that -- the information about any 6 impact on a branch was through a BIMS report. 7 Q. What system was there for ensuring that the 8 right information was passed back to the Post 9 Office? 10 A. I don't know. This was all part of the 11 responsibility of MSU and presumably the 12 management of MSU and the teams that were 13 working together and I was never a part of that. 14 Q. Going back to paragraph 22 of your witness 15 statement on page 6, please. In the third line, 16 you say: 17 "Looking back I can see that basing the 18 whole process around a single PEAK (arising from 19 an incident) per problem was not good and it is 20 likely that there is a methodology in existence 21 somewhere for handling this type of situation 22 better." 23 Firstly, what did you mean that the system 24 was that the whole process was handled around 25 a single PEAK? 152 1 A. Well, if you've -- a PEAK comes in, yes, it's 2 a new system problem, we identify where it is, 3 we send it off to Development do something with 4 it. So, strictly speaking, at that point it's 5 off my desk. I don't need to do any more about 6 it. But that is not always sensible, because 7 it's not picking up on further occurrences, it's 8 not making sure we've got the fullest picture of 9 it, and I just felt it would have been better if 10 it was at that point logged as a problem, and 11 then, if you get more occurrences, you would, 12 you know, link the PEAKs to it so you do have 13 that view of what it's affecting. 14 And then, you know, you might have to have 15 several actions coming off that problem for who 16 is meant to be doing stuff to fix it, pick up 17 the pieces, inform Post Office, and all the 18 rest. But that was my personal view. 19 Q. Was that your view held at the time? 20 A. It's certainly a view I came to while I was 21 still working, yes. 22 Q. Did you tell anyone about it? 23 A. Yes. 24 Q. Who did you tell? 25 A. I'm pretty certain I had a conversation with Mik 153 1 Peach about it. But I can't be certain. 2 Q. What was the response? 3 A. Nothing, no positive outcome. 4 Q. What does that mean, no positive outcome? 5 A. Um, I sort of got the impression that, "Oh, 6 well, you know, this is the way we work, this is 7 how we're doing it". 8 Q. One might get the impression that there was 9 a series of hermetically sealed cells within 10 Fujitsu, the principal aim of which was to get 11 a service ticket off their desk, and then those 12 hermetically sealed cells were themselves rather 13 sealed from the Post Office client. Would that 14 be fair? 15 A. I would say to some extent that's fair. I would 16 also say that I personally tried to break down 17 some of those seals a bit, and I think it wasn't 18 just me doing that, and I don't think it's, you 19 know, it wasn't just a matter of passing 20 something on because we didn't want to be 21 holding it any more, it was passing it on 22 because that's where it's got to go next 23 because, you know, you don't want to be sitting 24 on a PEAK while, you know, if the next -- if 25 it's important, obviously, that it's 154 1 investigated by Development or for it to go to 2 MSU for -- to sort out the financial 3 consequences. So you -- to that extent, you did 4 need to pass them on in the right direction at 5 the appropriate moment. 6 Q. How might that, in your view, have improved the 7 position of subpostmasters in relation to the 8 problems with the system that were afflicting 9 them? 10 A. You mean, if we'd been able to -- if we had -- 11 so can we go back a step? 12 Q. Yes. 13 A. Sorry, what -- 14 Q. The methodology that you described of an issue 15 being raised, an examination being undertaken to 16 see the extent to which the problem afflicted 17 other parts of the system and results of that 18 investigation drawn together in a single place? 19 A. I think it would have made sure -- been easier 20 to make sure, perhaps, that Post Office, at 21 least, were always more aware of some of these 22 problems that were having an effect at a number 23 of branches. How much it would actually have 24 helped the individual postmasters, I don't know. 25 Q. In paragraph 42(ii) of your statement, which is 155 1 on page 11, you described, if we scroll down, 2 please, the Horizon Support Desk. 3 Many of the subpostmasters who gave evidence 4 to us in Phase 1 of the Inquiry said that they 5 believed that the staff in the Horizon Helpdesk 6 were using call scripts, ie that they appeared 7 to be reading something out, some text out, some 8 printed text. Did you ever see such printed 9 scripts? 10 A. I can't remember. I think for some issues 11 where, you know -- because they also handled 12 calls about printer problems, the screen not 13 coming on and all that sort of thing, they may 14 have had printed scripts for some of that. 15 I did spend a few days once in the Helpdesk 16 office in Stevenage but I cannot remember if 17 I saw anything actually printed out like that. 18 Q. Do you know who -- whether they were printed 19 scripts or scripts on a screen -- 20 A. I don't know. 21 Q. -- was responsible for the creation of them? 22 A. I don't know. I have no memory of ever being 23 involved in the creation of anything like that 24 myself. 25 Q. Have you got any memory of anyone else in the 156 1 SSC being involved in the creation -- 2 A. No. 3 Q. -- of call scripts? 4 A. No. 5 Q. In the paragraph above, when you're dealing with 6 the NBSC, you say in the last line: 7 "NBSC had no direct interface with SSC, and 8 the two teams had no access to each other's call 9 logging systems." 10 A. Yes. 11 Q. Why did you have no access to the NBSC call 12 logs? 13 A. I presume because they were part of a completely 14 separate organisation, because they were run by 15 Post Office. 16 Q. Would it have been beneficial for you to have 17 had access to the actual logs to see what 18 subpostmasters were actually saying? 19 A. Um, occasionally, at times, it might have been 20 but, in fact, they usually had to give the same 21 information to the Horizon Support Desk, so -- 22 but there might have been occasions where it was 23 useful to know what else a branch had been 24 talking about recently. But that might -- there 25 might also have been a lot of business issues 157 1 that were nothing to do with -- nothing to do 2 even directly with Horizon, I'd have thought. 3 Q. Before the break, can we address ARP data, 4 please. If we turn, please, to page 16 of your 5 witness statement and paragraph 51, you say: 6 "... when it came to branch problems, we 7 could only work on the basis of what was 8 recorded on the system and any extra information 9 which the postmaster or clerk could provide. We 10 could not see what had physically occurred in 11 the branch; whether for example, there was 12 an inputting error that was not evident from the 13 person of Riposte messages. Sometimes that 14 meant we could not provide an answer to explain 15 the discrepancy because none of the forms of 16 evidence available to us gave any indication 17 that there were no inconsistencies in the 18 figures recorded and calculated by Horizon. In 19 those circumstances we would have to record that 20 we could not find any evidence of a systems 21 error." 22 A. Yes. 23 Q. We have heard described in the Inquiry a species 24 of audit data called ARQ data? 25 A. Yes. 158 1 Q. What do you understand ARQ data to refer to? 2 A. ARQ data was all the messages written on the 3 branch counters, as they replicated to the data 4 centre, were also streamed off into a large 5 series of very, very large files, which were 6 then, end of each day, can't quite remember, but 7 in some way sort of securely locked and then 8 could only be accessed again through the special 9 audit systems. 10 If an audit extract was requested by Post 11 Office, they would have to say the branch that 12 they wanted and the date range, and the standard 13 information, I think, that POL got, Post Office 14 got for one of these enquiries was -- well, it 15 was two spreadsheets, it ended up as, and one 16 was the transaction data messages and the other 17 was the Riposte event messages. 18 So they would be, I think, all in date order 19 or it may still have been in the date as 20 received at the data centre. So it was a subset 21 of the whole lot but it was all the transaction 22 data. It was also possible to get an unfiltered 23 set, which would be all these messages in their 24 entirety, and I -- 25 Q. Did you say all messages? 159 1 A. All messages, yeah. I'm talking here about 2 Legacy Horizon, it was very slightly different 3 for HNG-X. 4 Q. So transaction data messages, Riposte messages 5 and also possible to get an unfiltered set of 6 all messages? 7 A. Yes, transaction data, Riposte events, that's 8 log on/log off, those sort of things. But it 9 was possible to have the unfiltered set. 10 Q. It's been suggested by some witnesses that the 11 ARQ data equated to data that recorded every 12 keystroke made by a subpostmaster in branch? 13 A. That was never captured. 14 Q. Simplifying matters, was it therefore the data 15 when a subpostmaster committed what they had 16 done to the stack, essentially? 17 A. Yes, because the messages got written to the 18 message store, the transaction messages were 19 written to the message store when the basket was 20 settled. 21 Q. Where was that data stored? 22 A. At what point? Do you mean the ARQ data? 23 Q. Yes, the ARQ data? 24 A. On secure servers somewhere. 25 Q. Do you know where? 160 1 A. No. 2 Q. For how long was it stored? 3 A. I don't know, because that was never my 4 responsibility. But it was lots of years. 5 Q. You say in this paragraph here, you couldn't see 6 what had happened in branch, whether there was 7 an error inputting that wasn't evident from 8 a pattern of Riposte messages -- 9 A. Yeah. 10 Q. -- and therefore you were left with no evidence 11 of a systems error. 12 A. Yeah. 13 Q. In those circumstances, couldn't you use the ARQ 14 data? 15 A. No, because the ARQ data -- well, I'm not saying 16 here that we couldn't see what had happened in 17 the branch. We could see from the message 18 stores that we looked at the transactions 19 messages being committed, and so on, and we 20 could also see the extra messages that gave us 21 some extra clues as to what had been going on 22 sometimes. But the ARQ data was just that, that 23 same data that's stored in a different place. 24 Q. It wouldn't have shown you anything more at all? 25 A. No. 161 1 Q. So is that why you didn't obtain such data, ARQ 2 data, when investigating what I'll call the 3 simple or common or garden PEAK? 4 A. Yeah, there was no -- we felt there was no need 5 to go to the ARQ data because we had the same 6 information that we could get out of the message 7 store on the correspondence servers. 8 Q. So your view was that there was nothing 9 additional in the ARQ data -- 10 A. No. 11 Q. -- than the data that you had access to would 12 have shown you? 13 A. It was exactly the same data. 14 Q. Was there a cost to obtaining the ARQ data? 15 A. Um, I believe it was something that Post Office 16 were charged for or they had a certain number 17 that they could ask for in a year. But, above 18 that, they had to pay. 19 Q. Did that have any effect on the extent to which 20 ARQ data was obtained? 21 A. Not as far as I was concerned. If I was 22 investigating something that had happened a very 23 long time ago and I needed to look at that old 24 data, then I would ask for it and would get it. 25 And I don't recall that anybody ever tried to 162 1 charge me for it or suggested that I should be 2 charged for it. 3 Q. In what circumstances would you therefore obtain 4 ARQ data? 5 A. If, for some reason, I was specifically asked to 6 go and look at something that had happened some 7 time -- a long time ago. I can't now, except 8 for one instance, remember doing it as -- it 9 certainly wasn't something that was done as 10 a matter of course, because normally, if 11 something had happened a very long time ago, we 12 wouldn't -- probably wouldn't be expecting to 13 investigate it. 14 Q. The one incident you can remember, is that 15 because there may have been others that you've 16 forgotten or it's likely that, in your 16 years, 17 you only had cause to access ARQ data once? 18 A. I have a feeling it was more than once, but 19 I can't -- it was possibly in relation to 20 prosecutions, and so on, or some investigation 21 that somebody wanted doing, but I cannot now 22 remember much by way of detail. 23 Q. Why would you access that data, rather than the 24 data that you would habitually use in the case 25 of a prosecution? 163 1 A. Because it had -- well, that would be -- for 2 anything that had happened more than a month, 3 five weeks ago, you would have to go -- you 4 wouldn't have any other sources of data, except 5 the ARQ data. 6 Q. It was the only place you could find it? 7 A. Yes. 8 MR BEER: Thank you. On that note, sir, might that 9 be a convenient moment? 10 SIR WYN WILLIAMS: Yes. 11 MR BEER: It's 3.10 now, I think. 12 SIR WYN WILLIAMS: So 10 minutes or 15? Which do 13 you think? 14 MR BEER: Knowing where I am in my notes, ten 15 minutes, please. 16 SIR WYN WILLIAMS: Fine. 17 MR BEER: Thank you. 18 SIR WYN WILLIAMS: Ten minutes, that's 3.20 then, is 19 it, Mr Beer? 20 MR BEER: Yes, thank you. 21 (3.10 pm) 22 (A short break) 23 (3.21 pm) 24 MR BEER: Thank you, sir. 25 Can I turn to the tenth topic under this 164 1 heading then, which is attributing a problem to 2 user error and doing so as a default option. 3 The Inquiry, Mrs Chambers, has heard some 4 evidence on the default use of user error within 5 Fujitsu in the absence of any other explanation. 6 I want to look at some materials, please, 7 that address the question of attributing a fault 8 to user error or a discrepancy to user error. 9 Can we start, please, with FUJ00082302. Thank 10 you. We can see this is "KEL acha", which 11 I think is one of yours, Anne Chambers -- 12 A. Yes. 13 Q. -- 1717T. It's raised by you, we can see, in 14 the fourth line, on 30 July 2010, and closed or 15 last updated by you on 10 February 2015. We can 16 see from the "Symptoms" section of the KEL that: 17 "The branch complained that they had a loss 18 of £240 on one day. NBSC had been unable to 19 find any reason for the discrepancy so the call 20 was sent to Horizon to check for system errors." 21 Then scrolling down, "Problem". You say: 22 "A discrepancy is the difference between the 23 cash the system thinks the branch should have, 24 based on previous balanced figure and the 25 transactions recorded since, and the cash 165 1 declared by the branch. 2 "Possible causes: 3 "[Postmaster] has made an incorrect 4 declaration. 5 "Transactions as record on the system do not 6 match what actually happened at the branch. 7 "Outstanding recover. 8 "Withdrawn products. 9 "Known system problem (these should be 10 monitored for, or be easy to spot from events, 11 etc) 12 "Unknown system problem. 13 "Solution ..." 14 Scroll down, thank you: 15 "Make sure NBSC have already investigated 16 and include details of what they investigated 17 and found on the call. 18 "Which stock unit is affected? 19 "Trading period/balance period? 20 "Which day did the discrepancy occur? 21 "How much is it? 22 "If the discrepancy was apparent only when 23 they balance, when, before this, were figures 24 correct? 25 "Do they do a nightly Variance Check after 166 1 declaring cash?" 2 Then you put a "NOTE" in, in capitals: 3 "SSC do not accept credible calls until all 4 information on the investigation performed by 5 NBSC is detailed and they have advised why they 6 believe there is a software fault. 7 "Solution", at the foot of the page: 8 "When responding, if they have given 9 specific examples that you can explain, do so; 10 otherwise make clear it is not a system problem 11 (assuming you have checked that everything adds 12 up). See [PEAK ending 446] for an example 13 response which may help with the wording." 14 So is a summary of what you're saying in 15 this KEL to the earlier lines of support, "Do 16 your homework, follow the proper processes 17 before referring a discrepancy up to us in the 18 SSC"? 19 A. Yes. 20 Q. It was therefore an attempt, was it, at 21 discouraging them from, in some cases, sending 22 cases up to the SSC? 23 A. I don't think we were discouraging them but we 24 wanted them to get as full a picture as they 25 could of whatever information was available 167 1 about the circumstances, which we could then use 2 to inform our investigation. 3 Q. If we go back to the bottom of page 1, under the 4 note, you're effectively telling them "You've 5 got to believe that there is a software fault 6 before you refer it on to us"? 7 A. We certainly felt that NBSC should help the 8 branch to rule out any likely user errors. 9 Q. Well, is the premise of my question right, that 10 you were telling the earlier lines of support 11 that, before referring a problem on to SSC, they 12 should believe that there is a software fault? 13 A. Well, that was what we were there to 14 investigate, was software faults. 15 Q. So it's a yes? 16 A. So yes. But obviously they could not tell 17 definitively if there was one or not. 18 Q. That's what I was going to ask you. How could 19 the HSH or the SMC identify or analyse whether 20 there was a software fault? 21 A. They weren't asked to do that. They were only 22 asked if they believed there was a software 23 fault and a belief is not the same as proof. 24 Q. So what quantum of proof did they need, what 25 quantum of belief did they need? 168 1 A. Well, they didn't. They only needed to believe 2 it. We wanted them to have done -- I think it 3 was felt that we shouldn't be investigating 4 these types of calls, given that a discrepancy 5 is most likely to be caused by an issue at 6 a branch. I'm certainly not saying that's the 7 only cause but, overall, I think most branches 8 would accept that, from time to time, they would 9 have a discrepancy of some amount, and so we did 10 want to make sure that the normal checks had 11 been made before it came over to us, really as 12 a last resort. 13 Q. Amongst the possible causes that are listed is 14 "unknown system problem"? 15 A. Yeah. 16 Q. I think it follows that the first two lines of 17 support would not be able to identify if there 18 was an unknown system problem, would they? 19 A. No. 20 Q. So were you expecting them to investigate 21 whether the discrepancy that they had been told 22 about was the consequence of a known system 23 fault? 24 A. We more wanted them to try to rule out 25 discrepancies caused by business errors at the 169 1 branch. 2 Q. By business errors, you mean? 3 A. Accounting errors at a branch. Well, not 4 accounting errors but, you know, could they have 5 mistyped an entry somewhere? 6 Q. Then over the page, in the instructions to, 7 I think, the SSC -- 8 A. Yes. 9 Q. -- at the foot, you say "When responding". So 10 that's when you, SSC people are responding to 11 the lower lines of support: 12 "... if they [that's the lower lines of 13 support] have given specific examples that you 14 can explain, do so ..." 15 A. Yes. 16 Q. "... otherwise make clear it is not a system 17 problem (assuming you have checked that 18 everything adds up)." 19 A. Yes. 20 Q. What does that mean -- 21 A. I -- 22 Q. -- "make [it] clear it is not a system problem"? 23 A. Once you've checked everything and have not been 24 able to find any sign of any system problem. So 25 I think I explained before that, you know, 170 1 you'll get your running totals of the system 2 cash position and you'll compare it with what 3 the system had calculated its position was at 4 a particular point in time. You'll compare the 5 two, if those two weren't equal then, yes, you 6 do have a system problem and then you've got to 7 find out if it's a known one or an unknown one. 8 But if the two figures are in step, the system 9 has calculated the discrepancy correctly, then 10 you've got to try to see which transaction or 11 set of transactions might not be right, which is 12 leading to the discrepancy and, if you can 13 identify something that looks suspect, then 14 could that have been caused by a system error? 15 And if you can't find anything looking 16 suspect, you can't find any events, anything 17 else indicating any sign of a system problem, 18 then you are left with the fact that the system 19 thinks there should be so much in the drawer, 20 that the postmaster is saying there's 21 a different amount in the drawer, the only 22 reason -- the only way you're going to find why 23 those two amounts are different is if you can 24 identify something recorded on the system that 25 doesn't match what actually took place at the 171 1 branch. 2 Q. Didn't that create a default position that, in 3 the absence of evidence of a system error being 4 provided by the subpostmaster themselves and the 5 absence of evidence of a problem fitting with 6 a known existing system error, the problem would 7 be written up as user error? 8 A. Or an unknown system error. I mean, if you've 9 got an unknown system error, it's going to have 10 to have left some trace somewhere in the numbers 11 that will give you a clue as to where it is, 12 what sort of transaction. But unless -- but 13 there's no way that somebody investigating that 14 problem with no knowledge of what has been 15 carried out at the branch can say "That there 16 doesn't match what they actually did at the 17 branch on the day". So to that extent, yes, we 18 were dependent on the postmaster being able to 19 say, "Hang on, why has that there gone in for 20 £2,500 when it was only £250?" 21 Me investigating, I'm not going to be able 22 to know that it should have been £250 but 23 actually it's £2,500. 24 Q. What weight was accorded to the statement by 25 a subpostmaster who had proactively raised with 172 1 the Post Office and with Fujitsu a discrepancy 2 that he hadn't, in that example, put in the 3 £250? 4 A. So you're saying, so if he is saying "Look, it 5 says £2,500 but I only put in £250"? 6 Q. Yes. 7 A. If he can say it's that line there then we would 8 look at everything to do with that line there to 9 see if we could see any signs of it being 10 a system problem that had caused it. 11 Q. But what if it came to him saying "No, no, no, 12 it says £2,500 there, only £250 -- it was only 13 £250?" Where he's the one that's proactively 14 raised this by calling in, what weight was 15 accorded to what he was saying? 16 A. Quite a lot of weight. We wouldn't -- we 17 wouldn't automatically say, "Oh, you must have 18 got it wrong". But if he's then got to that 19 point where he can identify it, then we can at 20 least do something about the financial side of 21 it, in that he now knows which line is wrong, 22 and we can attempt to pass this information 23 through to Post Office so that that line, 24 whatever it was, could be rectified in his 25 accounts, which, to some extent, is the 173 1 important thing to be done in the short-term. 2 Longer term, if you've had a lot of branches 3 all reporting that precise type of problem, you 4 would have to try to work out how come it could 5 be wrong but, in practice, I don't recall 6 anything specifically like that. 7 Q. Can we take a look at a practical example. 8 You'll see that this KEL mentions at the end 9 there the PEAK ending in 446. 10 A. Yes. 11 Q. Can we look at that PEAK, please? It's 12 FUJ00083493. You should see that's a PEAK 13 ending in 446. It's dated or raised on 14 12 November 2013 and the problem is described in 15 the PEAK "Summary" as: 16 "[Postmaster] doing cash declarations every 17 now and again has major loss." 18 Yes? 19 A. Yes. 20 Q. We can see from the "Progress Narrative" at the 21 first entry, exactly the same is entered. Then 22 if we scroll down to the foot of the page, we 23 get a bit more detail. Do you see the entry at 24 13.40.00? It's about halfway down this page. 25 Yes, that one. Thank you: 174 1 "[Postmaster] has had cash declaration 2 problem throughout year and is losing a lot 3 every now and again. 4 "He phone up Helpline told him can't of 5 [sic] declared properly." 6 I think that means he phoned the helpline 7 who told him that he can't have declared 8 properly. 9 A. Yes. 10 Q. "He states that he loses £2,000 then jumps 11 suddenly to £4,000, one point they lost £8,000 12 and is always losing money. 13 "[Postmaster] stated he has 3 post offices, 14 only happens at this site. 15 "Advised [I would] get this call sent to 16 PEAK to look for any software error." 17 Then a little further on: 18 "... only done cash declaration on back 19 office counter, hasn't tried on any other 20 counter. 21 "done a declaration this morning and had 22 a £6,000 loss. 23 "It shows no error message when doing it. 24 No report prints out, only printout of cash 25 declaration." 175 1 Then if we go over to page 2, please, and 2 look at the entry about five or six lines in at 3 14.01.00, thank you: 4 "Voiced [I think that made a voice call to] 5 NBSC [quoting a reference] to see what checks 6 they have done themselves before transferring 7 call to Horizon. 8 "They stated they had trainers come into the 9 office and ruled out user error." 10 I just ask you to remember that line, 11 please. 12 Then on the same page, at 14.57.05, so 13 that's -- yes, in the middle of the page now: 14 "NBSC called in for a message for Rob. 15 "NBSC states that user error checks were 16 carried out by auditors at the site and not over 17 the phone. 18 "Advised I would update the call." 19 I'd ask you to remember that message too. 20 Then we come to your entry, same day, at 21 15.50.04, and you say: 22 "When the variance is checked following 23 a cash declaration, the system compares the 24 declared amount against the expected amount 25 (ie the opening cash figure for the period 176 1 adjusted by all cash movements since then). 2 "I have checked the system figures ... in 3 the declared figures for stock unit AA since the 4 rollover on 7 November and can confirm that all 5 the variants reported since then have been 6 calculated correctly. There are no known issues 7 that would result in the variance being 8 incorrect. 9 "For example, on 7 November, the opening 10 cash figure rollover was £60,125. 11 "£22,300 was remmed in. 12 "£6,500 was transferred out. 13 "Single SC transaction for £11,183.60 in. 14 "So at this point there should be £87,038.60 15 in the drawer. 16 "Cash declared at 15.29: £81,580 17 ie £5,528.60 short (though the variance wasn't 18 checked at this point). 19 "I can't tell why the declared cash doesn't 20 match the expected figure, the branch need to 21 make sure that what they have recorded on the 22 system is correct, and investigate the 23 anomalies. 24 "Please pass this back to NBSC." 25 Yes? 177 1 A. Yes. I did also cover this in my witness 2 statement. 3 Q. Yes. Now, the final part of your conclusion 4 there is that there were no known issues that 5 will result in the variance being incorrect and: 6 "I can't tell why the declared cash doesn't 7 match the expected cash ... the branch need to 8 make sure that what they have recorded on the 9 system is correct", and that they should 10 investigate the anomalies. 11 A. Yes, yes. 12 Q. The NBSC had got the trainers to go physically 13 to the branch to establish what had occurred, 14 and, in particular, whether that was a case of 15 user error, hadn't they? 16 A. Yes. 17 Q. That was significant information, wasn't it? 18 A. Yes, it was, and I took that on board. 19 Q. Auditors had gone to the branch for the same 20 purpose, rather than seeking to find out what 21 had gone wrong on the phone, hadn't they? 22 A. It appears so, yes. 23 Q. Both had come back and said, "This is not a case 24 of user error", hadn't they? 25 A. Yes. 178 1 Q. So how -- 2 A. I -- 3 Q. -- please -- was the subpostmaster going to 4 investigate the anomaly? 5 A. I don't know. I felt by giving some specific 6 examples of a specific day, where actually very 7 little had happened, but they were already 8 £5,500 short apparently. If they could compare 9 those three, four figures that I gave, perhaps 10 they would be able to see which of those was 11 wrong in that process, in what I could see. 12 And, although I only gave the figures from 13 that very simple point from the beginning of 14 November, from the beginning of the period, 15 I had actually done this check that I've already 16 described and, in every case where the system 17 cash figure was calculated, it was in line with 18 what it should have been. So it wasn't that 19 messages were not being picked up and included 20 in the accounts; everything seemed to be there. 21 The branch was declaring how much cash they'd 22 got when every time they did that, they'd got 23 a report printed out that showed the cash 24 declaration. 25 Something was not right. And, yes, the 179 1 numbers were jumping about all over the place. 2 The discrepancies were varying because they kept 3 declaring wildly different amounts of cash. 4 Q. Was it them declaring wildly different amounts 5 of cash or the Horizon System? 6 A. It definitely did appear to be them and I did, 7 of course, wonder if it could be something that 8 was wrong with the declaration mechanism but, 9 again, they got the report printed out that 10 showed them what the declaration was and I would 11 have expected that if they had declared -- so 12 they had to declare by denomination, I would 13 have expected that they would quite easily be 14 able to see if what was on their reports didn't 15 match the piles of cash that they had sitting in 16 front of them. 17 I'm not happy -- I'm not very happy with 18 this PEAK. I don't think I handled it terribly 19 well. I was frustrated by it and I think that 20 shows, and I was still frustrated by it when 21 I wrote the bit in the witness statement, 22 because, you know, it really looked like there 23 was a genuine problem. There was no sign of it, 24 the figures, you know, when you added everything 25 up, yes, the discrepancy based on what they'd 180 1 done, what they declared, the discrepancy was 2 apparently correctly calculated. Well, it was 3 correctly calculated. 4 So what was it in the data that was going 5 into these calculations that wasn't right? And 6 I couldn't see it. And, you know, when you've 7 got three transactions and, at the end of it, 8 they're out by £5,500 but you're adding up the 9 numbers. It's not a matter of my belief or not; 10 it's objective. That was what was there. So 11 something -- something definitely wasn't right 12 but couldn't see it, without being in the 13 branch. 14 Q. An auditor and a trainer had been in. 15 A. Yes, I wish I could have worked with them to try 16 to get to the bottom of this but that was not -- 17 as far as I was aware, that was not an option. 18 Q. If we go back to page 1 at the top, please. You 19 closed this call off, top right, four lines in, 20 "Closed -- No fault in product", didn't you? 21 A. I couldn't find a fault in the product. 22 Q. Because you couldn't find one, are you therefore 23 saying this was user error by the subpostmaster? 24 A. Not necessarily user -- well, I didn't know what 25 it was. Perhaps I should have just put -- 181 1 I can't remember if there was an unknown 2 category. 3 Q. Well, maybe look at page 3, please, top box. 4 15.50.04, entered by you: 5 "Defect cause updated to ... General -- 6 User." 7 You're saying it's a user error, aren't you? 8 A. I'm saying I was very frustrated by it because, 9 if you like, I wanted to find a system error. 10 Q. That doesn't come across at all in this note, 11 does it? You've closed this off saying there's 12 no fault in the product -- 13 A. Yes. 14 Q. -- the defect is the subpostmaster, haven't you? 15 A. That's -- that's how it comes over, yes. 16 Q. Well, is there a different way of reading it? 17 A. Um, there must be something that they could have 18 spotted at the branch to give us a clue as to 19 what was happening. 20 Q. In the light of what the trainers and the 21 auditors had found in the branch, how could you 22 possibly say there was no fault in the product 23 and this was a user error? 24 A. Because the numbers added up. 25 Q. Can we look at your witness statement, please, 182 1 at paragraph 183, which is on page 52, where you 2 were asked about this entry, and eight lines 3 in -- eight or ten lines -- you say: 4 "This could be user error or potentially 5 system error but could only be identified by the 6 branch." 7 Can you see that? 8 A. Yes. 9 Q. Here you seem to accept that this could be 10 a user error or a system error? 11 A. Potentially. 12 Q. Why did you write it up as no fault in the 13 product and that it was a user error? 14 A. Because I was rather frustrated by not -- by 15 feeling that I couldn't fully get to the bottom 16 of it. But there was no evidence for it being 17 a system error. 18 SIR WYN WILLIAMS: Does that mean there was no 19 evidence that you saw? What I'm trying to get 20 at is the use of the phrase "Unknown system 21 error", all right? There are two ways I could 22 take that: one is that there is evidence of 23 a system error but you can't identify the root 24 cause; the other is that there is a system 25 error, on balance of probability, but you simply 183 1 don't know what it is? 2 A. I felt that there was just no -- something was 3 obviously wrong, in that the branch obviously 4 were getting these discrepancies that they 5 weren't expecting, but all I could see on my 6 side was that they were apparently declaring 7 these differing amounts, and I certainly knew -- 8 didn't know of any system errors that would 9 cause that to happen, or to cause -- or that 10 would take what they were declaring and not 11 record it correctly. I wasn't -- and I couldn't 12 see any signs of that happening and nobody was 13 saying, "Look, we declared this but on the 14 report it says that". 15 So I felt, on balance, there was just no 16 evidence of a system error. 17 SIR WYN WILLIAMS: But as I've understood the line 18 of questioning that Mr Beer has been putting to 19 you, it's not just that the subpostmaster was 20 making the error whatever that was, but the 21 auditor and the trainer were making the same 22 error; is that it or have I misunderstood it? 23 A. Well, yeah, I -- yes, I don't know why the -- 24 I'd have thought, if the report was showing 25 something but then on the system it was recorded 184 1 as a different amount, I would have expected the 2 auditor or the trainer to have spotted that. 3 Yeah, I mean, now, I feel maybe I could have 4 made steps to try to talk to the auditor or the 5 trainer and to try to talk through it a bit 6 more. But, no, I'm not happy with this one. 7 But I still stand by there being no 8 indication of a system error and the numbers 9 that they were recording just didn't make a lot 10 of sense. 11 MR BEER: You've been told by the PEAK itself that 12 the problem was a recurring one. 13 A. Yeah. 14 Q. It had happened throughout the year? 15 A. I'm not sure remembered that, but -- 16 Q. That was in the PEAK that we looked at? 17 A. Yeah. 18 Q. Did you think how likely was it that this was 19 a persistent human counting error by the 20 subpostmaster or did you think that the very 21 report itself of a repetitious error was itself 22 suggestive of some system fault? 23 A. I think I thought that it just looked possibly 24 like a certain amount of carelessness. I don't 25 know. I couldn't explain it. But I also, you 185 1 know, perhaps in my knowledge of the whole 2 system, couldn't understand, you know, this -- 3 a problem like that just affecting one branch. 4 That's very unlikely. But that's -- that was 5 probably at the back of my mind. 6 Q. What checks did you make to see whether it had 7 affected any other branch? 8 A. I'm not aware that around this same period of 9 time any other branch had knowingly reported 10 a similar problem. 11 Q. I asked a different question: what checks did 12 you make to see whether it had affected any 13 other branch? 14 A. I don't know that I made any direct checks but 15 at that point I would have seen any calls like 16 that that were coming in to SSC. 17 Q. You knew by this time, 2013, that it was not 18 always possible immediately to find a fault in 19 the system, wasn't it? You knew that from your 20 past experience of system faults with Horizon? 21 Much investigatory work needs to be undertaken? 22 A. You can usually see evidence of a fault. You 23 might not be able to get to the root of it very 24 easily but you can usually -- if a system fault 25 has occurred, there's normally some sign of it, 186 1 specifically that the numbers don't add up. 2 Q. We saw that this call was allocated to you at 3 2.34 pm and that you wrote it up as "Closed -- 4 No fault with product, user error", at 3.50 pm. 5 A. Mm-hm. 6 Q. So assuming that you picked up the call at 7 exactly the moment it was assigned to you, you 8 spent 1 hour 14 minutes on it? 9 A. Mm-hm, yes. 10 Q. What did you do in that time? 11 A. I would have done more or less what it said in 12 the PEAK -- sorry, the KEL acha17170: extracted 13 the transaction data from the branch database, 14 which is what had replaced the message stores; 15 added up the cash; checked the variance 16 calculations matched the system figure at each 17 point. 18 Q. Over what period? 19 A. Since the 7 November, I think it was. The best 20 part of a week. That was the week they'd given 21 the specific examples for. 22 Q. What about what the subpostmaster had said in 23 the PEAK, that this has been a problem 24 throughout the year? 25 A. I probably didn't go back before -- 187 1 Q. I'm sorry, you probably -- 2 A. I probably didn't check back beyond the one week 3 for which they had given sort of specific 4 examples about it varying by £2,000 and £4,000. 5 Q. Why not? 6 A. Because I wasn't sure it would give me any more 7 than showing that, you know, these -- their 8 declarations seemed to be varying so wildly. 9 Q. Does what you've done, as displayed by this 10 PEAK, indicate your more general approach, to 11 the effect that if you couldn't positively see 12 a system error, the default conclusion that any 13 discrepancy is the fault of the user? 14 A. If there's no indication at all of a system 15 error then, yes, that's the inference that you 16 draw from that. 17 Q. Was that the operating approach of others within 18 the SSC: unless there's positive evidence of 19 a system error, it is the subpostmaster's fault? 20 A. We weren't necessarily assigning blame. We were 21 saying there was no indication of a system 22 error. 23 Q. Writing off a PEAK, "user error". 24 A. In this case, yes, which I've already said 25 I probably shouldn't have done. 188 1 Q. In the Horizon Issues trial, the court was given 2 evidence that between 1 January 2011 and 3 31 December 2014 the SSC received 27,005 calls, 4 meaning that, on average, there were 563 calls 5 per month. 6 A. Mm-hm. 7 Q. Does that sound about right: between 500 and 600 8 calls or tickets a month? 9 A. It sounds like quite a lot but it's probably 10 about right. 11 Q. The Inquiry has heard evidence that the SSC was 12 under pressure to close PEAKs because of the 13 pressure of work. Is that how you felt when you 14 worked in the SSC? 15 A. No. I mean, how many of those calls were 16 counter related? 17 Q. The evidence doesn't disclose that. Was there 18 any pressure at all on you to close PEAKs 19 prematurely? 20 A. No. 21 Q. In paragraphs 27 to 29 of your witness 22 statement, there's no need to turn them up at 23 the moment, you address the prioritisation of 24 tickets within the SSC. 25 A. Mm-hm. 189 1 Q. Were there any financial implications in respect 2 of the prioritisation of tickets? 3 A. I was trying to remember and I think in there 4 I say, yes, there may have been something but 5 couldn't remember it. I think I've seen -- read 6 more documents since which have reminded me that 7 some types of call -- some of the reconciliation 8 calls involving sorting out where people's bank 9 accounts might be adrift or where bill payments 10 might not be going through, those had to be 11 resolved pretty quickly. 12 That meant making sure that the financial 13 implications were resolved, but if there was 14 a root cause that needed further investigation, 15 that would still be done after the financial 16 side of things was dealt with. So, yes, we were 17 aware some calls you couldn't leave lying around 18 for long but that didn't mean that they had to 19 be closed. 20 Q. We can take the witness statement down from the 21 screen. Thank you. 22 How quickly did the different priority 23 tickets need to be resolved before any financial 24 penalty was imposed on Fujitsu for late 25 resolution? 190 1 A. Um, I can't remember the details. As I said, 2 I don't think it was all calls; I thought it was 3 just a certain type of calls. And a priority 4 was probably within 24 hours to get the 5 financial side of it sorted. 6 Q. If a ticket wasn't given the priority that you 7 thought it needed, would you formally change the 8 priority or would you just get on with the 9 ticket? 10 A. It would depend where it was going. If it was 11 something that I was dealing with, I might not 12 change it, but I would still treat it as 13 I thought it needed doing. 14 Q. Were performance statistics kept of your team, 15 in terms of resolution? 16 A. You'd have to ask my manager. 17 Q. Were you aware, in the 16 years you worked 18 there, that performance statistics were kept? 19 A. No. 20 Q. Was there any element of performance-related pay 21 in the SSC? 22 A. No. 23 Q. Can we turn, please -- the last topic for today, 24 I think -- to remote access. I want, please, to 25 look at the nature and extent of remote access 191 1 within the SSC, the extent to which individual 2 branches were notified of issues that affected 3 them and who was responsible for such 4 communications. 5 Can we start, please, with FUJ00088036. 6 You'll see this is a document entitled "Secure 7 Support System Outline Design". 8 A. Yes. 9 Q. It's dated 2 August 2002, so about two years 10 into your tenure at the SSC? 11 A. Yes. 12 Q. At the foot of the page, you can see that one of 13 the approval authorities is Mik Peach? 14 A. Yes. 15 Q. But you're not on either the distribution, the 16 approval or the reviewer list, okay. Was this 17 the kind of document that would be filtered down 18 to you? 19 A. Um -- 20 Q. Just go back to the top. 21 A. Yeah, I have no recollection that I ever saw it. 22 Q. That could be because 21 years have passed. 23 A. It could well be, yes. 24 Q. But is it the type of document that would be 25 provided to you? 192 1 A. It would probably be input on our website for us 2 to read -- 3 Q. So an intranet? 4 A. -- which isn't the same thing -- so yes, 5 an intranet. But, as I say, I've no idea 6 whether I did read it at the time. 7 Q. Can we go to page 15 of the document, please. 8 Let's see whether what it says is correct. It's 9 paragraph 4.3.2, "Third line and operational 10 support": 11 "All support access to the Horizon systems 12 is from physically secure areas." 13 You told us, I think so far as the SSC is 14 concerned, that's correct? 15 A. Yes. 16 Q. "Individuals involved in the support process 17 undergo more frequent security vetting checks." 18 Were you aware that you underwent more 19 frequent security vetting checks than some other 20 unnamed people? 21 A. No. I know I was vetted when I first joined, 22 but I wasn't aware of -- 23 Q. Being re-vetted -- 24 A. -- being re-vetted. 25 Q. -- in the 16 years afterwards? 193 1 A. Yes. 2 Q. "Other than the above, controls are vested in 3 manual procedures requiring managerial sign off 4 controlling access to post office counters where 5 update of data is required. Otherwise third 6 line support has: 7 "Unrestricted and unaudited privileged 8 access (system admin) to all systems including 9 post office counter PCs; 10 "The ability to distribute diagnostic 11 information outside of the secure environment; 12 this information can include personal data (as 13 defined by the Data Protection Act), business 14 sensitive data and cryptographic key 15 information." 16 Is it correct that at this time, 2002, you 17 in the SSC third line support had unrestricted 18 access to all systems including post office 19 counter PCs? 20 A. If that's what it says, yes. 21 Q. Was it your experience that that is correct? 22 A. We had access. I didn't ever test it to find 23 how unrestricted it was. 24 Q. It says that such access was unaudited. Was 25 that correct as well? 194 1 A. To the best of my knowledge, yes. 2 Q. The paragraph continues: 3 "The current support practices were 4 developed on a needs must basis; third line 5 support diagnosticians had no alternative other 6 than to adopt the approach taken given the need 7 to support the deployed Horizon solution." 8 Was that your understanding of why you had 9 been given unrestricted and unaudited privileged 10 access to all systems? That it wasn't planned, 11 it wasn't part of the design, it wasn't the 12 architecture; it was reactive to events? 13 A. I don't -- I'm not sure that I thought about it. 14 I joined the SSC and I was shown, you know, as 15 I said, you know, "If you need to access 16 a counter, this is how you connect up to it", 17 and so on. These were systems that I wasn't 18 particularly familiar with, so I just did what 19 I was told on the occasions when I did need to 20 access a counter. 21 Q. So you didn't address your mind particularly to 22 how it had come about that you had -- 23 A. No. 24 Q. -- such unrestricted and unaudited access to -- 25 A. No, I -- 195 1 Q. You just took it as given that you got it? 2 A. Yes. I probably didn't question whether it was 3 audited or unaudited at that point. It was 4 something that was needed at times to do our 5 job. 6 Q. Can we move forwards a year then, please, to 7 FUJ00088082. This is a document described as a 8 "Support Guide" for the OpenSSH. Can you recall 9 what the OpenSSH was? 10 A. I do remember at some point the way we accessed 11 counters, and other systems as well, changed, 12 and SSH rings a bit of a bell. So yes, this is 13 what we switched to. 14 Q. So this document again is not signed off by you, 15 reviewed by you or, on its face, distributed to 16 you, but the process would have been something 17 that you would have been familiar with by June 18 2003; is that right? 19 A. I imagine so, because we would have changed to 20 using the method that's documented in here. 21 Q. We can see the date on the top right of 30 June 22 2003. If we go to page 6, please. It appears 23 that this was an attempt to -- I'm going to call 24 "sort out" the unprivileged -- sorry, the 25 unaudited and unrestricted privileged access 196 1 that the SSC had. Can you see in the second 2 paragraph of the introduction it says: 3 "It is necessary, for security and auditing 4 purposes, to provide a method that allows 5 datacentre and counter systems to be managed 6 interactively but for all these management 7 actions to be captured. When these actions have 8 been captured (or logged) it must be possible to 9 audit the actions. This, in turn, means the 10 logs must be in an easily understandable 11 format." 12 A. Yes. 13 Q. The report then goes on -- I'm not going to go 14 through it -- to explain the different ways the 15 system produces an audit trail for all actions 16 performed during what I'm going to call remote 17 access. 18 A. Yes. 19 Q. What was your understanding of the process that 20 had to be undertaken after the introduction of 21 this process in June 2003? 22 A. Um, I think I was aware that everything would be 23 captured and logged somewhere, but I don't 24 know -- didn't know and don't know any more 25 details than that. 197 1 Q. Can we go, please, to FUJ00138355. Can you see 2 that this is a document created by your manager, 3 Steve Parker, on 8 September 2011 -- 4 A. Yes. 5 Q. -- and was last updated by somebody called Adam 6 Woodley in February 2021? 7 A. Yes. 8 Q. You'll see what Mr Parker said: 9 "GDPR regulations require that access to 10 personal data remains within the European Union 11 and PCI data security standards mandate physical 12 security restrictions must be applied where 13 [data] access is allowed to user data. 14 Currently the only units which fulfil all these 15 requirements for data access are the SSC and ISD 16 Unix." 17 Do you know what ISD Unix was? 18 A. I think they were the operations team based in 19 Belfast. 20 Q. "The responsibility for data correction is 21 vested in the SSC although ISD sometimes act 22 under SSC authorisation (via the application of 23 a tested script). 24 "Corrections to live system data must be 25 authorised via Account change control and 198 1 auditable. Any correction requiring APPSUP role 2 is to be witnessed by a second member of the 3 SSC. Both names must be recorded on the change 4 control for audit purposes." 5 A. Yes. 6 Q. This appears to describe in 2011 the so-called 7 "four eyes" approach, or "four eyes" method, 8 namely one person from the SSC logs on to the 9 counter and makes the -- I'm going to call them 10 corrections to the branch's transactions whilst 11 another person watches, doesn't it? 12 A. Yes. 13 Q. They were supposed to write a narrative account 14 of what they'd done and record the names of both 15 people, weren't they? 16 A. Um, yes. I can't remember if the second person 17 was meant to add their own name themselves, but 18 I can't remember. 19 Q. Just look under "Data corrections": 20 "Support investigation may indicate the need 21 for data correction. In this context data 22 correction is any support action that results in 23 the modification or removal of Post Office data. 24 If any correction is required then details must 25 exist in the form of a narrative on, or 199 1 attachments to, a PEAK incident to provide 2 a clear audit trail. An approved Account change 3 control entry ... must also exist and be 4 cross-referenced from the PEAK." 5 Then "Financial Data": 6 "Changes to financial data are rarely 7 required. Where a requirement exists, such 8 changes must be made via contra journal entries 9 to maintain audit trail and the change must be 10 made using the two man rule. The 'two man rule' 11 (sometimes called the 'four eyes rule' in 12 security circles) specifies that there must be 13 two individuals that must act in concert in 14 order to perform the same action. Further, each 15 individual should have comparable knowledge and 16 skill in order to detect attempts of subversion 17 initiated by the other. 18 "Within the SSC, one member of the SSC will 19 perform the data correction whilst a second 20 member of the SSC will witness the change being 21 made. Both names must be recorded on the change 22 control for audit purposes." 23 A. Yes. 24 Q. So this process depends on people in SSC 25 depending -- or carrying out faithfully and 200 1 recording what they're supposed to have done? 2 A. Yes, or what they did do. 3 Q. Yes. 4 A. Yes. 5 Q. You remember the 2003 document that we looked 6 at. I didn't take you through its details, but 7 do you recall that that, if it had been carried 8 into effect, would have required that a securely 9 managed ID user had been recorded and logged in 10 real time against each and every action 11 performed, rather than an after-the-event 12 narrative description of what we had done? 13 A. I'm sorry, I don't remember that document. That 14 was the SSH, OpenSSH document? 15 Q. No, it was the document that I took you at some 16 speed through, in the interests of time. 17 A. Sorry. 18 Q. We can go back to it. It's FUJ00088082. 19 A. Yes, the OpenSSH -- 20 Q. You remember that? 21 A. Yes, it was the OpenSSH Support Guide. 22 Q. If you turn to page 8, I'm afraid you're just 23 going to have to read that quietly to yourself, 24 the "Basic Procedure". 25 A. That procedure is just talking about creating 201 1 the users, not the users using the systems. 2 Q. Yes. Then if you go on to page 10 on the 3 "Logging Server" and scroll down, and then look 4 at 5.2. 5 A. Yes. Sorry, I can't ... 6 Q. Sorry, you said you can't? 7 A. I'm not sure what I'm meant to be looking for in 8 here. 9 Q. Okay, then go on to 5.3. 10 A. Right, so the logging gets written to a file on 11 the SAS Server. 12 Q. Then over the page to 6.1 and 2. 13 A. Yes. So this is connecting to counters or 14 central servers. Yes, I remember doing those 15 things. 16 Q. Then down the page to 6.2.1. 17 A. Yes. 18 Q. Then over the page, please. Then scroll down to 19 the bottom of the page. As I'd read this, 20 Mrs Chambers, the system, if it had been carried 21 into effect, required by this document, would 22 have first required users to log in using their 23 own names and passwords, then, when connecting 24 to the counters, they logged in using a special 25 user ID. That special user ID would be logged 202 1 in real time against each and every action that 2 the user then performed, yes? 3 A. Yes, I think that was the case, although I can't 4 recall -- I don't think I ever saw any of this 5 logging. But yes, we did. But if we go back to 6 what you were then asking me about for data 7 changes on HNG-X, those weren't made on the 8 counter. That wasn't -- because the data was no 9 longer held on the counters. 10 Q. So that explains the difference, does it, 11 between what we read in Mr Parker's minute of 12 2011 and what is described here? 13 A. Possibly, yes, because I think this was -- there 14 was a lot -- all this background logging going 15 on, which we were really almost meant to be 16 unaware of. And I'm not sure if it did ever get 17 checked to see, but there was this background 18 audit of everything done via an SSH connection. 19 Now, I can't remember whether we -- this implies 20 that also we used SSH for connecting to the 21 central systems. I'm not sure now if that 22 included the Unix -- the Oracle databases, which 23 is what the 2010 document was talking about -- 24 changes to that. 25 So it may well be that if it was still via 203 1 SSH, then that would have been an additional 2 level of logging was going on automatically 3 anyway but I really don't know. This was not my 4 area at all. 5 Q. I understand. Can we look at what you did 6 understand as to the process, and look at your 7 witness statement at page 57, please. In 8 paragraph 199 at the foot of the page you say: 9 "I am asked whether a person using a branch 10 terminal would be expressly notified by Horizon 11 if changes were made using these access rights. 12 It is hard to remember now what was done so long 13 ago. If the branch had raised the service 14 incident in the first place, via HSD, then 15 I would probably have told them if I was going 16 to make a change to their system to correct the 17 problem. If they had not raised it, but instead 18 we knew there was a problem because of 19 a Reconciliation Report entry and the branch had 20 not yet done their balance, I probably would not 21 have contacted the branch, but would have 22 informed my manager or the Fujitsu Problem 23 Manager. If the problem was notified to the 24 Post Office and they had authorised the change, 25 I would expect Post Office to decide whether or 204 1 not to notify the branch." 2 So you say you would probably have notified 3 a branch if you were going to make a change 4 using remote access rights, if the postmaster 5 had themselves initiated the incident. 6 A. Yes. As I said, it's really hard to remember 7 what would have been done, what was done in each 8 case. I've seen various PEAKs in the 9 preparation for this, some where I definitely 10 did talk to the branch. 11 Q. And others where sometimes you didn't? 12 A. And others where I didn't, yes. Also, remember 13 that, you know, making the changes would not 14 necessarily be changes to financial data. One 15 thing we did have to do sometimes was, where 16 a branch was unable to use one of their counters 17 because it had been left in a slightly 18 inconsistent state, thinking it was still in the 19 middle of balancing, and they could sort it out 20 for themselves if the same user logged back on 21 to that counter. But if that user had gone off 22 on holiday for two weeks, that wasn't very 23 helpful. So there was a -- by rewriting a -- 24 sorry, by updating an object in the message 25 store, basically writing a new version of the 205 1 object, we could clear that flag, which would 2 then enable them to log on to the counter and 3 proceed as normal. 4 That had absolutely no financial effect. So 5 the number of changes where we were actually 6 making a financial correction were very few and 7 far between. 8 Q. You refer in this paragraph to your manager, 9 ie in the cases where the problem had been 10 picked up by a Reconciliation Report entry, you 11 say you probably wouldn't contact the branch but 12 you'd have told your manager that you had made 13 a change on the subpostmaster's counter. 14 A. Yes. 15 Q. Are you referring there to Mr Peach and then 16 Mr Parker? 17 A. Yes, it would have been. And if it was a change 18 that had any financial implication, then it 19 would have needed the -- whatever the change 20 control system at the time was, following. 21 Q. You say or you might have informed the Fujitsu 22 Problem Manager. Who was the Fujitsu Problem 23 Manager? 24 A. They varied over the years, but back for Legacy 25 it was Mike Stewart and Mike Woolgar, I believe. 206 1 Q. Sorry, the second name? 2 A. Woolgar. 3 Q. Was it your understanding that any of those 4 people, Mr Peach, Mr Parker, or either of the 5 problem managers, would have contacted the 6 branch to tell them that corrections had been 7 made to their data? 8 A. No, I think it's very unlikely that they would 9 have contacted the branch. 10 Q. You say that in relation to -- in the last part, 11 that if the problem had been notified to the 12 Post Office and they'd authorised the change, 13 you would expect the Post Office to decide 14 whether or not to notify the branch. What would 15 affect whether or not the Post Office told the 16 branch that their financial data had been 17 altered? 18 A. I don't know. I think that's a question for 19 Post Office. 20 Q. Did you know of some cases where the Post Office 21 did not tell a subpostmaster that their 22 financial data had been altered remotely by 23 somebody within Fujitsu? 24 A. Yes, I think that definitely did happen. 25 Q. Did you follow the Group Litigation at all? 207 1 A. No, not until I realised that I was being 2 mentioned quite a lot, which was very, very near 3 the end of it. 4 Q. Did you read in that, did you see in that, that 5 the Post Office was seeking to deny that the 6 remote access that we are speaking about could 7 occur? 8 A. I didn't see that at the time, certainly, and 9 I haven't read up on it thoroughly. 10 MR BEER: Sir, that's an appropriate moment -- 11 SIR WYN WILLIAMS: Yes. 12 MR BEER: -- if it's convenient to you. 13 SIR WYN WILLIAMS: Certainly, yes. Can we take 14 stock, so to speak? Are you still confident 15 that all your questions and those of other legal 16 representatives can be completed tomorrow? 17 MR BEER: In respect of the first clause in that 18 sentence, yes. The second clause, I'm in the 19 hands of others. I've canvassed some views from 20 some of my friends and I'm covering a lot of the 21 ground that they've asked me to cover. 22 SIR WYN WILLIAMS: Right. Well, would like you to 23 discuss that briefly with your colleagues for 24 this reason: that having said on Friday that we 25 wouldn't sit on Thursday and Friday of this 208 1 week, I made an appointment for myself on 2 Thursday morning which I do not now wish to give 3 up, because they're difficult to come by, and so 4 I want that to be factored in to any 5 arrangement, and I don't want to inconvenience 6 Mrs Chambers more than is strictly necessary. 7 So I'd be grateful if we could work out 8 whether we will finish tomorrow and, if not, 9 what we do. 10 MR BEER: Sir, we will liaise, and I'll report back 11 to you overnight. 12 SIR WYN WILLIAMS: Thank you. 13 (4.27 pm) 14 (The hearing concluded until 10.00 am 15 the following day) 16 17 18 19 20 21 22 23 24 25 209 1 I N D E X 2 3 HOUSEKEEPING .........................................1 4 5 ANNE OLIVIA CHAMBERS (affirmed) ......................6 6 7 Questioned by MR BEER .........................6 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 210