1 Thursday, 17 November 2022 2 (10.00 am) 3 MR BEER: Good morning, sir. Can you see and hear me? 4 SIR WYN WILLIAMS: Yes, thank you very much. 5 MR BEER: Can I recall Charles Cipione, please. 6 SIR WYN WILLIAMS: Yes. 7 CHARLES CIPIONE (sworn) 8 Questioned by Mr Beer 9 MR BEER: Please take a seat, Mr Cipione. As we established 10 on the last occasion, your report is 174 pages in length 11 and is divided into two parts, parts 1 and 2. Part 1 is 12 between pages 12 and 64, and part 2 is between pages 65 13 and 160. You're giving evidence today, as you know, 14 about part 2 of your report. 15 Can you take open part 2 of your report, please. 16 It is at page 65 in the hard copy and on screen it's 17 EXPG0000001. Thank you. 18 That runs as I say between pages 65 and 160. Can 19 you please confirm the following. Firstly, that you 20 have made clear in part 2 of your report which facts and 21 matters referred to are within your own knowledge and 22 which are not? 23 A. Yes. 24 Q. Secondly, that those facts in part 2 of your report 25 which are within your own knowledge you confirm to be 1 1 true? 2 A. Yes. 3 Q. And thirdly that the opinions you have expressed in 4 part 2 of your report represent your true and complete 5 professional opinions on the matters to which they 6 refer? 7 A. Yes. 8 Q. Thank you. Can we turn to the next page, please which 9 is your methodology. In this section of your report and 10 over the next six pages you address the methodology that 11 you and your team adopted to analyse the primary 12 material that you were provided with; is that right? 13 A. That is correct. 14 Q. Now, rather than going through each of the paragraphs on 15 those six pages, I would be grateful if you would 16 describe in summary terms the process that you and your 17 team adopted to the very large cohort of material that 18 you were given. 19 A. Absolutely, absolutely. So we'll take this section by 20 section. The first section 7.2 is the analytics work 21 stream. What I am describing here is the fact that we 22 wanted to try to structure the information in the PEAKs 23 and PinICLs in a way that we could then do further 24 analysis on that. 25 Effectively, what we did was we took a number of 2 1 HTML and PDF files and converted them into a more 2 digitised form so that we could do searching and 3 whatnot, and then feed it into further processes. So 4 effectively that is what 7.2 is describing. 5 Q. So it's the very first stage, getting the primary 6 material into a more readable and searchable format? 7 A. Exactly. 8 Q. Yes. And after that analytics work stream, what was 9 next? 10 A. So as further downstream process following the analytics 11 work stream, we also wanted to incorporate a couple of 12 technologies for the review purposes of the project. 13 The two technologies are Relativity and Brainspace. 14 Relativity is an E discovery platform for viewing 15 documents. So we wanted to populate the information 16 into Relativity, all of the PEAKs and PinICLs, as well 17 as the KELs and monthly reports if we needed them in 18 there for monthly purposes. 19 Q. Just to remind us from last time, I think you received 20 56,489 PinICLs and 16,530 PEAKs. Does that sound right? 21 A. That sounds right, yes. 22 Q. Thank you. Sorry, I interrupted. 23 A. So Relativity is just a tool to help for the review 24 process, as simple as that. Brainspace is a tool to 25 help organise the searching methodologies that we were 3 1 using for some of the work that we were doing. 2 Effectively it is a machine learning tool that allows us 3 to group the information especially as it came from the 4 analytics work section to help us narrow in and identify 5 specific documents or groups of documents as needed 6 throughout the review. So effectively we loaded the 7 information into there, into Relativity and into 8 Brainspace, as part of the process. 9 So we actually loaded in the HTML documents, but 10 we associated them with the work that was performed in 11 the analytics work stream. So they were paired up. 12 Q. Tied with each other? 13 A. Exactly. Perhaps it might be interesting. You know, 14 some of the information that we were interested in 15 seeing, especially in the PEAKs and the PinICLs, were 16 bits of information out of the PEAKs and PinICLs 17 referred to as response categories and defect causes. 18 They were just a standard bit of information that were 19 in the PEAKs and PinICLs. 20 So we -- I do reference where we had our reference 21 information from there and, if you look on figure 7.1, 22 it shows how that would have been displayed in the 23 Relativity system as the team was looking through their 24 initial review to identify documents for me to review. 25 Q. Thank you. I think you also received 656 KELs; is that 4 1 right? 2 A. That is correct. So we did a similar process with the 3 KELs. We ran it through this analytics work stream and 4 we made that information available in Relativity and 5 Brainspace for review purposes. 6 Q. I think you were only able to examine a proportion of 7 the KELs that you identified in the period that you were 8 looking at, and the reason for that was that you were, 9 I think, told that some KELs had been deleted. Do you 10 know why the KELs had been deleted? 11 A. Let me make sure. 12 Q. It's a footer 69, 7.4.4, 7.4.5. 13 A. Yes, my understanding was some of the KELs were deleted 14 and weren't available for me to receive for review. 15 Q. Did you, from the material that you did examine, 16 understand why the KELs may have been deleted or not? 17 A. Other than they were deleted they could have timed out. 18 Perhaps they weren't relevant for retention from 19 Fujitsu, but past that it would be speculation on my 20 part. 21 Q. Thank you very much. You received a series of 22 management reports, I think two, and you gave us the 23 numbers of those on the last occasion and the different 24 species of the management reports, monthly Pathway ICL 25 reports and the like. They are set out in the previous 5 1 section of your report in 6.4. I'm not going to go 2 there. 3 You explained overall on the last occasion that 4 what you described as a brute-force approach was not 5 possible. Can you just remind us of what you meant by 6 brute-force approach not being possible. 7 A. Right. So the brute-force approach was not a practical 8 approach to review the PEAKs and PinICLs. 9 Q. Yes. 10 A. However, for the management reports I did review those. 11 I reviewed all of those. So the review process for the 12 management reports was -- I read them. I read all of 13 them, sometimes several times. For the PEAKs and the 14 PinICLs it was more of the Relativity review process, 15 and we can get into that a little bit more if you want 16 as far as, you know, we would target particular words, 17 we would have Brainspace help us identify themes that we 18 were interested in looking in, and have Brainspace 19 assist us in identifying the PEAKs or PinICLs that were 20 relevant to the particular theme that we were 21 investigating. 22 So PEAKs and PinICLs, brute force, too many 23 documents, not enough time, and we did not do that. For 24 the monthly reports and the KELs we did review all of 25 those. 6 1 Q. Thank you very much. 2 In terms of the limitations of the exercise that 3 you undertook, you took us through on the last occasion 4 the limitations of it. Just to remind us of some of 5 them, firstly, the material that you were examining came 6 principally from Fujitsu rather than the Post Office. 7 A. That is correct. 8 Q. Therefore you don't have, as it were, the Post Office 9 side of the house, side of the story, insofar as 10 concerns the issues that you speak to in your report. 11 A. That is correct. There are a few of the monthly reports 12 that were jointly issued by the Post Office and by 13 ICL Pathway, but the majority of the information that 14 was used for the themes and opinions I arrive at come 15 solely from ICL Pathway. 16 Q. On occasions, the ICL Pathway-issued monthly reports 17 attribute a view to the Post Office, but that's coming 18 from ICL Pathway rather than the Post Office itself? 19 A. That is correct. To the extent that I cite anything 20 that refers to the Post Office, unless it's from the 21 joint report, that is ICL Pathway's view concerning the 22 Post Office. It's not the Post Office's view. 23 Q. We'll come to look at an example of the joint report 24 where it's co-authored by the Post Office and Fujitsu in 25 a moment. 7 1 Secondly, in terms of the limitation of the time 2 period of the exercise you were undertaking, it 3 restricted to which years? 4 A. I believe it was 1996 through 2000 was the material that 5 was covering that time period. 6 Q. Now, I believe you have been given access to the witness 7 statement of the witnesses who have to date given 8 evidence in Phase 2 of the Inquiry. 9 A. That is correct. 10 Q. You have been given access to the transcripts of 11 witnesses who have to date given evidence in Phase 2 of 12 the Inquiry? 13 A. Yes. 14 Q. Has anything in the witness statements that you have 15 been given or the transcripts that you have been able to 16 read given you cause to revise or change any of the 17 opinions or the contents of your report in part 2? 18 A. I have not read anything in the witness statements or 19 the transcripts that gives me pause concerning any of my 20 opinions. 21 Q. Before we get into the detail -- and we're going to come 22 back to this at the end, when looking at some of your 23 overall conclusions that are expressed in the executive 24 summary -- at a high level, could you summarise it for 25 us, the impact, if any, that the written and oral 8 1 evidence that you have been able to read had on the 2 views expressed in your report. That is a very broad 3 question. 4 A. Yes. Could you ask me the question one more time, 5 please. 6 Q. Yes. Looking at the witness evidence that you have 7 read, both the witness statements and the transcripts, 8 can you summarise for us what impact, if any, that 9 evidence had on the views expressed in your report. 10 A. As far as the views expressed in my report, I believe it 11 confirms most of the views that I've expressed in my 12 report. The witness statements and the transcript 13 information that I've read so far seems to run in 14 parallel with the opinions that I've expressed here. 15 Q. So would this be a fair way of putting it: it solidified 16 the conclusions that you reached? 17 A. Yes. 18 Q. In the light of the witness statements and transcripts 19 of the oral evidence that you've read, have you formed 20 any view as to what you now know as to the use of some 21 of the data that the Horizon System produced for 22 criminal justice purposes, i.e. to seek to prove beyond 23 reasonable doubt that a subpostmaster committed an 24 offence of theft or false accounting? 25 A. Are you asking me if I have an opinion on whether there 9 1 was a high level of integrity within the data that was 2 used? 3 Q. Yes. 4 A. Yes. Based off of the information in the witness 5 statements and the transcripts, I think that the 6 integrity of the information used would be suspect. 7 Q. You told us on the last occasion that part of your 8 background, a significant part of it, involved the 9 design, creation and maintenance of systems that are 10 used in legal proceedings. 11 A. That is correct. 12 Q. Can you just remind us in summary form what that was -- 13 A. Yes. 14 Q. -- or what that is. 15 A. What that is. So the firm I work for, AlixPartners, 16 does a lot of work in the turnaround restructuring space 17 in the United States. That is a legal proceeding that 18 requires a lot of reporting to the court, and it 19 requires basically presenting defensible data as far as 20 the court is concerned. When I say defensible, what 21 I mean is the data has to be complete and accurate and, 22 to support that completeness and accuracy, it's 23 important for the people producing the data and any 24 analyses, reports that are associated with that data to 25 maintain a high level of transparency and auditability 10 1 within those datasets. 2 Q. What do you mean by a high level of transparency and 3 auditability? 4 A. What I mean is it's apparent what happened with the 5 data, and you can look at that two ways. When I say 6 transparency, I mean you're talking about all the 7 processes, all the manipulations, it's apparent through 8 the coding or the queries or the reporting exactly what 9 manipulations are happening there. 10 Q. And transparent to who? 11 A. Transparent to anyone that wants to look at it: 12 transparent for the court, transparent for the debtors, 13 transparent for the creditors, which are the two sides 14 in the Bankruptcy Court in the United States. 15 Q. How does the design of a system that produces data that 16 may be used for the purposes of legal proceedings, 17 including criminal legal proceedings, affect the design 18 of the system? 19 A. I'm going to go back to the concepts of: the results 20 have to be complete and accurate, and that's kind of 21 lifted from the accounting world, completeness and 22 accuracy. Accuracy you can think of as at the very 23 atomic level of everything is correct, everything is 24 correct. Completeness means it's correct for 25 everything. 11 1 So you want to -- so, for instance, if you have 2 a set of transactions and you are trying to display 3 perhaps a sub-set of them and do a manipulation, do some 4 sort of calculation and then summary information on 5 that, what you want to be able to prove is, number 1, 6 I've started with the complete set of data. The way you 7 do that is you show, perhaps not all of the data is used 8 in the final analysis, but you show that there's 9 basically a waterfall of: here's the data -- here's 10 everything, here's this set of data that is being used 11 in the analysis, and here's the set of data that is not 12 being used in the analysis. 13 A completeness check on that would be the set 14 that's not used in the analysis adds up to £100, the set 15 that is being used in the analysis is £110, and I can go 16 back and confirm that, showing that the complete set is 17 110 pounds' worth of data. That would be just a very 18 simple view of completeness. 19 Accuracy would be: perhaps I'm calculating an 20 interest rate on some of the transactions and I want to 21 show that, you know, there's basically there's 22 a manipulation of that. So perhaps I'm doing a time 23 series, and over time interest accrues to a particular 24 amount. I want to be able to show that that calculation 25 is accurate. So I want to show that it's accurate at 12 1 the detail level, and I want to show that it's complete 2 at the very highest level, so that there's no doubt that 3 the results of the reports that I'm doing are complete 4 and accurate. 5 I can say that they are complete and accurate, and 6 you can either believe me or not believe me. But, if 7 I have a system or a process that is both transparent 8 and auditable, it allows anyone that wants to look at it 9 to confirm for themselves that, okay, I'm looking at the 10 process that happened and, for purposes of transparency, 11 that might be looking at the code that I used to do all 12 the manipulations. 13 Alternatively, if it's like a multi-step process 14 and you want to see the waterfall going from step 1 to 15 step 2 to step 3 and what the results are for the 16 complete set of data, having all of that recorded from 17 step 1 to step 2 to step 3 to step 4 and available for 18 review, that would be an auditability check. 19 So if I have one of them, it probably would 20 provide comfort to everyone. If I have both of them, it 21 just increases the comfort, not only for them but for me 22 because I want to be able to make sure, as I'm doing the 23 process, that I'm doing it correctly. I want to make 24 sure that there is a great amount of integrity in the 25 results that I produce, especially if it's in 13 1 a commercial court or a business litigation court. 2 I would expect that that level of custodianship is 3 probably even more important in a criminal court. 4 Q. So does it come to this, that the answer to my question 5 is that, yes, it does affect and it affects in the way 6 that you have described the way that you design, build 7 and run a system, if you know that the data from it may 8 be used for the purposes of legal proceedings? 9 A. I would say it would affect it regardless of what system 10 I'm using but, as the stakes go up, yes, it's even more 11 important to adhere to those principles. 12 Q. Thank you. Can we turn to the topics that you have 13 addressed in your report, and they start on page 72. 14 There is a series of topics -- and we're going to go 15 through them one by one -- that you have identified as 16 being relevant and important. 17 A. Yes. 18 Q. The first of them is disclosed by the heading 19 "Subpostmaster training experienced difficulties during 20 National Roll-out." 21 In paragraph 8.1.1, you address the importance of 22 training. Can you help us at a general level, without 23 addressing the Horizon IT System for the moment, how 24 important is training when a new IT system is 25 introduced? 14 1 A. Of course it's very important for the users of the 2 system to know how to use the system. The system is 3 there for the users. The system is there to support the 4 business processes that have been part of the 5 requirements process for building the system. 6 The system could be great but, if the users don't 7 know how to use it, it loses all its value. So it's 8 critically important that the users be very comfortable 9 with operating the system. 10 Q. In the case of the Horizon System, was there anything in 11 particular that heightened the need for such training? 12 A. I would say generally my statement is a blanket 13 statement. People need to -- you know, users need to 14 know how to use the system. I believe that there might 15 be certain circumstances in the user community for the 16 Horizon System that probably heightened the attention 17 that might need to be spent on making sure that this 18 user community is attended to properly. 19 The first one is my understanding is that, prior 20 to the Horizon system, this was a manual process. So 21 not only are you doing a change of process, you are also 22 introducing the concept of technology to assist in the 23 process. So any time you change that could be 24 a challenge for some people. 25 But going from a very paper-based process to 15 1 a computer-based process just adds another dimension of 2 unknown to the formula, and it must be accounted for. 3 Q. This was happening at the close of the last century. 4 A. Yes, in the last millennium. 5 Q. Yes. You highlight in the last sentence of 6 paragraph 8.1.1: 7 "It is apparent from reading the ICL monthly 8 reports that there were significant problems in training 9 the SPMs as they adopted the Horizon IT System." 10 On the basis of what you read, looking at it 11 generally, is it fair to say that was consistent theme 12 throughout period that we're looking at? 13 A. Yes. 14 Q. That was recorded in the ICL reports at a significant 15 level of concern? 16 A. Yes. 17 Q. In paragraph 8.1.2, you make the point that this was not 18 just a challenge of training users on a new system, it 19 was actually a challenge of training users on computers 20 in general. Is that a product of the age that we're 21 talking about? 22 A. Absolutely. 23 Q. You extract a quotation from, I think, material 24 published by Fujitsu as part of its promotional and 25 sales literature and, in the last five lines of the 16 1 quotation from their promotional material, you say, or 2 they say: 3 "Training was provided to 63,000 staff members 4 from the ages of 16 to 87 with various skills ... 5 Approximately 5,000 calls were received each week by the 6 Helpdesk, due to the counter staffs' lack of computer 7 experience." 8 Is it on that basis that you make the point that 9 the group of people that were being trained naturally 10 had a variable level of skills? 11 A. Yes. 12 Q. 5,000 calls a week with a cohort of 63,000, did you draw 13 any views on that? 14 A. That's a high rate, for sure. I think it's a function 15 of, like I said, not only are we changing the system 16 from paper-based to computer, but we do have a wide 17 spectrum of what I would expect to be some people are 18 very computer savvy at that point, but there might be 19 some people that have moderate abilities in computers, 20 might be some people that don't have any ability to use 21 computers, and there might be some people that are very 22 resistant or just don't want to use computers. 23 That's a wide range of users to try to 24 accommodate, and I would expect that that probably was 25 part of the calculus that went into those numbers. 17 1 Q. You tell us in paragraph 8.1.3 that Pathway was aware of 2 the importance of training the subpostmasters and that 3 that was recognised by them early on, and you note that 4 Pathway itself noted in April 1999 that the 5 subpostmasters were facing difficulties in transitioning 6 from a paper-based balancing process to an automated 7 balancing process, and that ICL Pathway responded by 8 saying that a greater emphasis should be placed, in 9 training, on the practical experience of balancing. 10 That was the answer that was given at the time; is that 11 right? 12 A. Yes. 13 Q. Did it appear that that was successful, i.e. the answer 14 from ICL saying a greater emphasis must be placed on 15 practical experience of balancing, given the reports of 16 difficulties in balancing persisted throughout 1999? 17 A. I apologise, Mr Beer, what was the actual question? 18 Q. Did it appear to you that the response to the problems 19 that subpostmasters reported of facing difficulties in 20 balancing, namely, to say, "Well, a greater emphasis 21 must be placed on training, on practical experience of 22 balancing", was successful, given the reports of 23 difficulties in balancing persisted throughout 1999? 24 A. I would say that it didn't appear to alleviate the 25 problems. 18 1 Q. Can we look at some examples over the page, please, in 2 the extracts from the management reports, including ICL 3 Pathway's monthly report. In fact, just before we do 4 so, could I look at what you said in paragraph 8.1.4 at 5 the foot of the previous page. 6 You tell us in the last few sentences that what 7 we're about to look at are verbatim extracts that you 8 have highlighted some matters in bold, that the views 9 expressed are those of the authors, being principally 10 ICL Pathway documents, but in some cases joint ICL and 11 Post Office documents. 12 A. That is correct. 13 Q. If we then go over the page, please, do you in the 14 second column of this table make clear whether this is 15 an ICL document that you are quoting verbatim from, or 16 a jointly issued document from ICL Pathway and the 17 Post Office? 18 A. Yes, it's clear. So, for instance, the first line, ICL 19 Pathway monthly report April 1999, that's an ICL 20 Pathway-authored -- a solely ICL Pathway-authored 21 document. 22 Q. The rest of that page, I think, is all the same? 23 A. That's correct. 24 Q. Then, if we just skip over the page -- 25 A. Yes. 19 1 Q. -- and look at the first entry. 2 A. Yes, the first entry there, ICL and Post Office monthly 3 joint implementation report. So that would have been 4 co-authored by ICL and the Post Office. 5 Q. Then I think the rest of the documents on the page are 6 ICL Pathway alone-issued documents? 7 A. That's correct. 8 Q. I think we can see that in evidence, not only by your 9 description of the report as being an ICL Pathway 10 monthly report, but also from the extracted text. If 11 you look at the second report for May, Pathway wrote: 12 "POCL are shaping up to hit us on service level 13 agreements and training." 14 A. Yes, that would have been ICL Pathway's view. 15 Q. So the "us" there is the ICL Pathway view? 16 A. Yes. 17 Q. Was there anything in the content of the reports that 18 suggested, where they were authored by ICL Pathway 19 themselves, these monthly reports, that POCL were 20 provided with copies of them? 21 A. I don't believe I ever saw any indication of that. 22 Q. From the other evidence that you read from PinICLs, 23 PEAKs, KELs or other documentation that you read, was 24 there anything in that which suggested that those ICL 25 Pathway monthly reports were provided to POCL? 20 1 A. No. 2 Q. Now that formulation that we read at the end of 3 paragraph 8.1.4, if we just go back to that, at the foot 4 of the page, is the same formulation, give or take a few 5 words, that you use in succeeding sections of your 6 report, when you're about to introduce the table of 7 verbatim extracts from PinICLs, PEAKs, KELs, monthly 8 reports? 9 A. Yes. 10 Q. And does what you just said apply equally to all those 11 succeeding sections? 12 A. It does. 13 Q. Thank you. Go back to the substance then, if we can 14 just briefly look over the page, please, at September 15 '99. Scroll down, please. The first entry ICL records: 16 "Although national roll-out rates have risen to 17 2,000 (sic) Post Offices per week, the level of issues 18 occurring on installation day and the level of training 19 scheduling failures puts achievement of the 300 offices 20 per week roll-out rate required in 2000 at risk. 21 Knowledgepool are introducing new scheduling software 22 and a plan of activity to remove/reduce the causes of 23 the other issues is being put in place for the November 24 to January break in National Roll-out." 25 Can you help us who were Knowledgepool? 21 1 A. I believe Knowledgepool was the partner with ICL that 2 assisted with the training of the SPMs. 3 Q. Essentially a subcontractor of ICL Pathway that provided 4 the training? 5 A. Yes. 6 Q. We can see that at the second entry for September 1999: 7 " ... currently a serious issue relating to the 8 scheduling of training events within the implementation 9 programme. The training scheduling system of Pathway's 10 training subcontractor, Knowledgepool, has been 11 struggling to scope during the early part of national 12 roll-out, although a planned system replacement was 13 imminent." 14 So this isn't Horizon itself. This is a separate 15 system used for scheduling training; that's how you 16 understood it? 17 A. Yes. 18 Q. During September the training scheduling system crashed 19 resulting in a loss of data and some data corruption, 20 and then we can read the rest. 21 Then, if we go over the page, please, the joint 22 report covering a period of September '99 to nearly the 23 end of October '99, training continues to cause major 24 difficulties, and then in May, the two lots of May 2000 25 entries, if you just cast your eye over those, please. 22 1 The first: 2 "POCL perception of SLAs and training, and also of 3 our commercial attitude to risk-taking on new business: 4 all negative as epitomised by the recent Dave Miller 5 letter. Risk remains that POCL will extract commercial 6 concessions out of us ..." 7 Did you track that through, the suggestion that 8 POCL would seek money from ICL Pathway in relation to 9 the provision of training? 10 A. When you say, did I track that through -- 11 Q. Yes, did you see that in later documents? 12 A. I believe that that was a running commentary, yes. 13 Q. We can see in the entry then of December 2000: 14 "A settlement for the projected shortfall in 15 training courses against the contracted, number arising 16 from low course-occupancy levels, has been agreed with 17 the Post Office. As part of a package to achieve 18 relaxations against existing service level agreements, 19 pathway will pay the first £1 million of the training 20 shortfall." 21 That's what I was -- 22 A. Yes. 23 Q. -- referring to. 24 Did what you read in these reports, these 25 management reports, reflect what you found in the 23 1 PinICLs when you came to read them relating to training 2 issues? 3 A. Yes. 4 Q. If we look at the foot of the page at 8.1.5, you start 5 a summary of some of the PinICLs and include verbatim 6 extracts from them. I'm not going to go through them, 7 but you summarise them in 8.1.5: 8 "A review of the PinICLs and PEAKs [I think these 9 are just PinICLs in fact] reinforces the theme that the 10 subpostmasters were reporting that the lack of training 11 was problematic in the execution of business activities. 12 Additionally SSC staff [remembering back, that's the 13 third tier of support] were also raising concerns about 14 the ineffectual nature of training." 15 You embolden some sections. You then set out in 16 paragraph 8.2 a series of PinICLs. As I say, I'm not 17 going to go through them. Can we go forward to page 76, 18 please, and look at 8.1.6, which is towards the bottom 19 of the page. You say that you: 20 "... surveyed the PinICL and PEAK population for 21 any final defect cause being assigned 'General - User' 22 or 'General - User Knowledge' ..." 23 Can you just explain what you mean by surveying 24 the population for any final defect cause? 25 A. Certainly. So at the beginning of our conversation 24 1 today, we talked about the analytics work stream and 2 evaluating the PEAKs and PinICLs information, and a part 3 of that analytics work stream was to help identify 4 certain data elements that were contained within the 5 PEAKs and the PinICLs. 6 One of those data elements was a defect cause and 7 a defect cause is essentially the -- as an error was 8 raised and entered into the PEAK or PinICL system, it 9 was evaluated, and part of that evaluation was ICL would 10 make a determination on what the cause of the particular 11 defect that raised that error was. 12 This is a representation of that analysis. So, 13 looking through a population of PEAKs and PinICLs that 14 we had in our possession, we looked for the final defect 15 cause. So perhaps it might be good for me to talk about 16 that. 17 So the defect cause has the opportunity to change 18 values as the error is being investigated. It might be 19 thought that there might be a particular, an initial 20 evaluation that a defect cause is one thing, but upon 21 further investigation it has the opportunity to change. 22 The analysis here is looking at what the final 23 value of the defect cause was across the PEAK and PinICL 24 population, and I identified two values that I thought 25 were pertinent for this section. 25 1 Q. They are General-user and General-user knowledge? 2 A. Exactly. 3 Q. That resulted, in the period you were looking at, 435 4 PinICLs being identified where those two values had been 5 applied? 6 A. That is correct. 7 Q. You say: 8 "Please keep in mind that the SMC was supposed to 9 resolve user issues. These PinICLs and PEAKs were 10 promoted to the SSC." 11 What did you mean -- what sits behind those two 12 sentences? 13 A. So we'll talk about in greater detail, I'm sure, as the 14 conversation continues. But the SSC who is in charge of 15 the PEAKs and PinICLs, that's more of a technical error 16 evaluation and remediation system. User questions 17 should have been identified by the Helpdesk or by the 18 SMC, which are the first and second levels of basically 19 help to the users. Only real technical errors are 20 supposed to arrive for the SSC to evaluate but, if it 21 does get to the SSC, they raise a PinICL and it starts 22 to be documented. 23 So what I'm trying to convey here is that this is 24 what actually didn't get resolved at the first and 25 second level. This population could be much greater if 26 1 we went and looked back at the actual Helpdesk tickets, 2 because the Helpdesk is what precedes anything that 3 might get into the PEAK or PinICL system. 4 Q. The 5,000 calls a week? 5 A. Yes, exactly. 6 Q. What's the significance of this finding in 8.1.6? 7 A. There are a lot of problems with user knowledge on how 8 to use the system. 9 Q. In 8.1.7, you say: 10 "This figure indicates a wave of user issues in 11 September '99, March 2000, June 2000 and November 2000 12 during the national roll-out period." 13 If we go over the page, please, if we can blow 14 that up slightly, you depict the overall volumes 15 between, I think, May '99 and December 2000. 16 A. Yes. 17 Q. The spikes that you speak about, September '99, which we 18 can see before, October '99, March 2000, June 2000 and 19 November 2000 -- I suppose, July as well -- the colour 20 coding on the right-hand side, does that refer to the 21 final defect cause? 22 A. Yes. 23 Q. So the thing that you've just spoken about? 24 A. Exactly. 25 Q. Given the process of progress in national roll-out, did 27 1 you think about whether this is a function, these PEAKs, 2 of the increasing number of users being admitted to the 3 system? 4 A. It absolutely is part of the function, yes. 5 Q. Did you draw any overall conclusions from what you saw 6 in the monthly reports and the PinICLs, as to the 7 training difficulties that were experienced during 8 national roll-out? 9 A. Just the general concept that the training didn't seem 10 to go swimmingly well. The user base had a lot of 11 problems with understanding how to use the system or, 12 throughout the national roll-out period, that's the 13 conclusion that I would draw. 14 Q. Thank you. Can we turn to section 9, please, of your 15 report on page 78. The heading "Hardware issues were 16 problematic during national roll-out", and if we read 17 9.1.2: 18 "In the national roll-out of the Horizon IT 19 System, there was a discrete period (August 1999) where 20 hardware issues rose to the level of being ' a serious 21 Acceptance Incident'. According to ICL Pathway's 22 monthly reports some issues persisted through 23 October 1999, but appear to have subsided to an 24 acceptable level by January 2000." 25 Now, the material that you relied on to say that, 28 1 I think, all came from the ICL Pathway monthly reports 2 which you set out in table 9.1; is that right? 3 A. Yes. 4 Q. If we go forwards, please, to page 80 and read that very 5 short paragraph at the top of the page: 6 "It should be noted that in May 2000, there were 7 still hardware issues being raised in PinICLs." 8 Then, if you then look at the table at 9.2, you're 9 essentially checking what you read in the monthly 10 reports again against the PinICLs; is that right? 11 A. That is correct. 12 Q. Now, despite what was said in the ICL Pathway monthly 13 report about the issue subsiding to an acceptable level 14 at the start of 2000, you record that in May 2000 there 15 were still hardware issues being raised in PinICLs; is 16 that right? 17 A. That is correct. 18 Q. Did you conduct a survey of PinICLs and PEAKs between 19 July 1996 and the end of 2000, where the product at 20 fault in the PinICL or PEAK was shown as desktop or 21 hardware? 22 A. Yes. 23 Q. If we turn over the page to page 81, please, and just 24 scroll down to 9.1.5 and the table underneath, did you, 25 using that product-at-fault value, find that there were 29 1 1,281 PinICLs or PEAKs that had that value ascribed to 2 them as the final fault? 3 A. Yes. 4 Q. Did you set those out in your table between the period 5 I mentioned, July '96 and end 2000, at 9.1? 6 A. Yes. 7 Q. Does that show a spike during the national roll-out 8 period? 9 A. Yes, it does. 10 Q. Again, that might be a function of the fact that there 11 were more users being brought online? 12 A. Absolutely. 13 Q. If we look at 9.1.6, you say you again surveyed the 14 PinICLs AND PEAKs for product groups listed in the next 15 figure's legend, where the product-at-fault value was 16 hardware, ISDN, ISDN adaptor or driver. 17 Just explain again what you are doing there. 18 A. So there's a value within the information that 19 I received called "product at fault" and, to the extent 20 that it existed and we identified a value for that, what 21 we did was we basically pulled out any product-at-fault 22 values that were either hardware, ISDN or ISDN 23 adaptor/driver, all of which I would associate with 24 a hardware issue. 25 Q. Do we see, if we go over the page, in 9.2, a similar 30 1 pattern of a PEAK during national roll-out -- 2 A. Yes. 3 Q. -- in table 9.2? 4 A. Sorry, yes. 5 Q. In 9.1.7, you tell us that you looked at 332 KELs of 6 which about 10 per cent were coded as responsive, i.e. 7 the nature of the KEL was concerned with 8 a hardware-related issue; is that right? 9 A. Yes, that is correct. 10 Q. Looking at all of those sources, the monthly reports, 11 the PinICLs and PEAKs, and the KELs, did you form any 12 view overall, as to the preponderance of hardware issues 13 during national roll-out? 14 A. The hardware issue -- first of all, there absolutely was 15 an issue with hardware during national roll-out, and it 16 impeded the roll-out certainly. 17 Q. Could such issues affect the reliability of data held on 18 the system, and could such reliability issues lead to 19 what would seem to be inexplicable financial imbalances? 20 A. That would -- yes, there's a possibility that that could 21 happen, absolutely. 22 Q. Is it normal that there are hardware issues when you 23 roll out a big system? 24 A. Yes, and we need to remember the time range that we're 25 talking about. I would say, over the last 25 years, the 31 1 reliability of hardware used in technical systems has 2 improved drastically. It wasn't as good. It certainly 3 wasn't -- the hardware wasn't as good in the late '90s 4 as it is now, as far as reliability and durability is 5 concerned. So I am extremely aware of that. 6 Notwithstanding that statement, that probably was 7 little comfort to anyone that was experiencing the 8 hardware issues during that time, and it definitely 9 posed a problem for the national roll-out. 10 Q. Could anything have been done to warn or inform end 11 users, subpostmasters, of the issues that might be 12 caused by hardware problems; so hardware-generated 13 keystrokes, for example, or phantom transactions? 14 A. Yes. So to the extent that ICL Pathway was aware of 15 such a specific thing as that, definitely communications 16 could have been made. I suspect that that was a small 17 fraction of what I'm showing here. I think what I'm 18 showing here is more of a catastrophic, holistic failure 19 of either the communication, the ISDN line, or the 20 counter set-up. It could be that both your hard drives 21 failed. You know, unfortunately that's just things that 22 happened during that time. I mean, drives could fail 23 right now certainly also, but the prevalence of that was 24 much greater 25 years ago. 25 Q. Overall I think you told us that you formed the view 32 1 that hardware issues caused problems, and problems for 2 subpostmasters, and it was problematic for them during 3 the national roll-out. 4 A. Absolutely. 5 Q. Can we turn to section 10, please, page 83, where you 6 address the disconnection of many Post Office branches 7 from the central system during national roll-out. You 8 tell us that in 10.1.1 that: 9 "The ambition of Horizon was to allow branches to 10 communicate their information to a central system" -- 11 Reminding us of an earlier passage in your report, 12 the Horizon campuses. 13 "It also allowed for software and reference data 14 updates to be distributed from the campuses to the 15 branches. 16 "To accomplish this ... a telecommunications 17 system was incorporated into Horizon." 18 This depended on ISDN lines or satellite links 19 being installed at each branch with BT or Energis 20 providing the backbone infrastructure to utilise this 21 hardware. It relied also on each branches equipment to 22 be available for polling. 23 Can you tell us what polling means, please. 24 A. Certainly it's basically having a remote computer being 25 available to communicate with a central computer, and 33 1 the polling is usually described as when the central 2 computer reaches out to the remote computer to see if 3 it's available to communicate. So that would be 4 polling. 5 Q. How does that work? Can you explain in a little more 6 detail how that works. 7 A. Certainly. So in this system the network was not 8 constant -- always on. It was not like systems are now 9 when you're -- well, even when you're on your browser, 10 you're not really actually always on necessarily. 11 But effectively what they were -- the mechanism 12 that they were using was they had a communications link, 13 either through a satellite or through an ISDN line, that 14 utilised BT or Energis' backbone or the other service 15 providers, which allowed the movement of data between 16 the counter information at the branch and the data 17 servers at Bootle or Wigan. 18 What would happen is every night the counters 19 needed to basically communicate with the central branch 20 to accumulate all the transactions that happened, right? 21 So that would be the direction of: I've conducted my 22 business. Now I want to make sure that all of the 23 records of those transactions are managed centrally at 24 the data centres. 25 In order to accomplish that, the data centre and 34 1 the counter needs to communicate. Since it's not 2 a persistent connected communications link, what would 3 happen is a polling would have to occur. The central 4 system would need to reach out to each one of the 5 counters and say, "Number 1, are you even there and, if 6 you are there, I need to grab some information from you. 7 I need basically to collect all the transactions that 8 you performed that day so that I can do the downstream 9 processing of all that information." So effectively 10 that's what the polling information is. 11 There are occasions also where the central system 12 needs to push information down to the counters. That 13 might be: I'm updating reference data which is -- if you 14 remember, it's part of the system that allows the system 15 to work correctly and, in order to maintain 16 synchronisation of how each system works, all of that 17 information needs to be passed down to the counters, as 18 well as just straight software updates, that would need 19 to be passed down to the counters. 20 Q. You tell us in 10.1.3 that: 21 "The Monthly Reports indicate that throughout '98 22 and '99 Pathway was concerned with their ability to 23 effectuate this design feature: they were concerned with 24 BT's coverage of the UK as well as other technical 25 issues related to their standards." 35 1 Is that concern by ICL Pathway before national 2 roll-out? 3 A. Yes. 4 Q. This is even before national roll-out. Then if we look 5 at 10.1.4: 6 "During the national roll-out these problems were 7 realised." 8 So, true enough, ICL Pathway's worries turned out 9 to be correct? 10 A. Yes. 11 Q. "Hardware, network availability, and user issues 12 combined to create a situation where ICL Pathway was 13 occupied with a higher than expected amount of 14 non-polling branches." 15 Given what you have said, that there was concern 16 before national roll-out about ICL Pathway's ability to 17 effectuate this design feature, i.e. to ensure polling 18 with hardware installed at the branch, what do you mean 19 by: it was higher than expected amount of non-polling 20 branches? 21 A. I would imagine, in any contingency planning that anyone 22 was doing with a system that was designed like this, 23 that they would expect that some counters wouldn't be 24 available for some periods of time. But based off of 25 the reading that I performed, it was much higher than 36 1 they expected as far as non-polled counters. 2 Q. You say in the third sentence: 3 "This was problematic because Horizon relied on 4 this ... design aspect to not only collate and 5 centralise information ... but ... also for efficient 6 updates of software to the branches." 7 What level of problem is this? 8 A. It's a giant level of problem. Let's take it to an 9 extreme. Let's say that I never am able to poll 10 a counter. Effectively that counter is out of the 11 network. So, if I am not able to communicate between 12 the counter and the data centres, that's breaking the 13 system. That's taking all of functionality that was 14 purported to be in the Horizon System and throwing it 15 out on a counter-by-counter basis. 16 Q. For bug fixes, did that rely on the ability to poll? 17 A. Yes, yes. Any communication between the central servers 18 and the counters relied on that communication mechanism, 19 which I'm referring to as the polling. The polling is 20 actually: are you there? Once you get an affirmative 21 answer back, then you can move information between 22 either the counter and the data centre or the data 23 centre and the counter. 24 Q. You say in 10.1.5: 25 "Additionally ICL Pathway was compelled to raise 37 1 and resolve an issue for any branch whose non-polled 2 status was 24 hours in duration." 3 A. Yes. 4 Q. Just explain what that means. 5 A. So I believe this is related to an SLA requirement for 6 ICL Pathway, just as part of the requirements process. 7 They needed to make sure that they were polling 8 different counters, and that's regardless of why -- you 9 know, there could be things that were completely out of 10 ICL Pathway's control that would stop that polling from 11 happening, such as the counter wasn't turned on. If 12 that counter's computer is never on, it's never going to 13 be polled and, unless ICL Pathway sends someone out to 14 turn it on, it's never going to be turned on. So there 15 were definitely some aspects of that that were outside 16 of their immediate control. 17 Q. You set out extracts from the monthly reports that 18 essentially establish what you said in your last few 19 answers. Can we go on to page 86, please, and look at 20 the table 10.2. Could explain what this table shows, 21 please. 22 A. Certainly. So this table is describing all of the days 23 in the month of November of 2000, and it is creating 24 a set of categories of how many counters or how many 25 offices weren't able to be polled across a different set 38 1 of days. So, for instance, we see the first figure of 2 86. On November 1, 2000, there were 86 counters that 3 had not been polled in a day. 4 Q. You use language interchangeably of counters and 5 branches, and counters and offices. Is it counters or 6 is it offices? 7 A. This says offices. So maybe we should also refresh our 8 memory on exactly how the Horizon System worked at that 9 time. So there might have been many particular counters 10 at each office, but there was only one counter that was 11 hooked up to the communications system that allowed for 12 polling. So this would be office, not necessarily 13 counter. This would be per branch or per office. 14 Q. You were telling us that, for 1 November, there were 86 15 offices that did not poll for one day? 16 A. Exactly. 17 Q. There were 83 that hadn't polled for 2 or 3 days? 18 A. Yes. 19 Q. There were 28 that hadn't polled for between 4 and 9 20 days? 21 A. Yes. 22 Q. So it goes on? 23 A. Exactly. 24 Q. Did you draw a conclusion from this level or extent of 25 non-polling offices and the duration? 39 1 A. This was an issue. Especially as we look further into 2 November, you see the numbers becoming very high there. 3 So, even if there were, you know, 20,000 offices -- 4 sorry, 20,000 branches -- let's pretend that there's 5 20,000 branches live in the network -- that's 6 a significant -- you know, towards the end of November 7 that's a significant portion of those branches that are 8 not communicating with the central data servers. 9 Q. Does this show a consistently serious problem across the 10 entire month essentially? 11 A. Yes. 12 Q. You then again cross-refer to what the monthly reports 13 had shown to the PinICLs and PEAKs; is that right? 14 A. Yes. 15 Q. If we look at the foot of 87, page 87, at 10.1.8, you 16 say: 17 "I surveyed the PinICL and PEAK population for 18 entries where the Product at Fault contained 'ISDN' or 19 'VPN' within their values." 20 That's because they are both related to 21 connectivity. 22 "This query resulted in the 4,733 entries shown in 23 the figure below, with their specific Product-at-Fault 24 value shown in the legend. This problem manifested 25 during the national roll-out period." 40 1 Then over the page, please, to the graph. The 2 colours on the right-hand side or the key on the 3 right-hand side, I think, shows that the preponderance 4 of the problems related to the shade of blue that 5 relates to ISDN; is that right? 6 A. Yes. 7 Q. The fourth colour down on the right-hand side? 8 A. Yes. 9 Q. Again, 4,733 product-at-fault values with ISDN or VPN, 10 did you draw any conclusion as to whether that signified 11 anything about the size or severity of the problem? 12 A. It indicates that there's absolutely an issue that was 13 related to the physical connection or the implementation 14 of the Horizon package at this time. 15 Q. Thank you. 16 I'm going to skip over for the moment sections 11 17 and 12 of your report and go straight to section 13, 18 please, on page 98 where your heading of the issue is: 19 "The persistence of reference data management 20 degraded the integrity of Horizon." 21 Can you explain, before we get into the detail of 22 this, what your overall finding was. 23 A. My overall finding was -- so just to remind everyone, 24 the reference data is a component of the system that 25 I referred to at my first hearing as data-driven logic. 41 1 It could represent something as simple as just a price 2 list, you know, a first-class stamp costs this much. 3 That information was not hard coded into the Horizon 4 System. It was passed to the Horizon System via what 5 I'm referring to as reference data. 6 Now, my understanding is there are a lot more 7 complicated partitions of the reference data that 8 I won't go into now, but it was essential for the 9 Horizon System to operate correctly for that to have 10 integrity and to be consistently completely and 11 accurately delivered to each one of the branches in 12 order for the Horizon System to operate. 13 What I'm saying here is there were a lot of issues 14 that I read about that were referring to the integrity 15 of the reference data as it pertained to the actual 16 operation of the Horizon System. 17 Q. I think on the last occasion you identified, or you told 18 us that a design of a system that utilised data-driven 19 logic was in principle a good thing. 20 A. Absolutely. 21 Q. You say in 13.1.3: 22 "The advantages of data-driven logic rely upon its 23 custodianship. If the 'data' in the data-driven logic 24 is not timely, accurate or complete, the system it 25 supports will not operate as intended." 42 1 I think that speaks for itself. 2 A. Yes. 3 Q. You then in 1.4 and 1.5 set out issues raised by ICL 4 Pathway as early as early 1997, concerning POCL's 5 reference data incorporation into the system. You say: 6 "By late '97 ICL Pathway characterised its 7 contractual obligations regarding reference data as 8 poorly defined but acknowledged the significance of the 9 issue as crucial." 10 What do you mean by that? 11 A. So, if this is a vital component of your system, you 12 have to make sure that it's right. What I've read in 13 the materials that were provided to me was that this was 14 a split responsibility between ICL Pathway and the Post 15 Office Counters Limited. 16 Ultimately, a lot of the information needed to be 17 sourced from POCL. I don't think all of it, but a lot 18 of it was sourced from POCL. What I'm trying to say 19 here is that I don't know that the communication of 20 requirements or the -- I believe that what ICL Pathway 21 is saying here is they don't think that the Post Office 22 is understanding the importance of this issue as ICL 23 Pathway is trying to develop and deliver a system. 24 Q. Thank you. You say in 1.6 that you are going to extract 25 from the monthly reports the data that supports what you 43 1 have just said. Then, if we go over the page, please, 2 to the foot of the page -- keep going, please -- sorry, 3 two more pages to 102, and then foot of the page, 4 please. 5 In 13.1.7 you say: 6 "A review of the PinICLs and PEAKs supports the 7 contention that Reference Data was a cause of problems 8 in the Horizon IT System." 9 You set those out in your table at 13.2. Go over 10 the page, please, and then over the page again to 11 page 105. You say that you: 12 "... surveyed the PinICL and PEAK population for 13 any Product-at-Fault value where Reference Data was 14 indicated." 15 You found 1,863 such PinICLs or PEAKs, and that 16 led you to the conclusion that reference-data problems 17 began manifesting themselves in 1998 and were prevalent 18 during the national roll-out period. 19 Can you just identify for us in the table -- thank 20 you -- it's the dark blue, second from bottom; is that 21 right? 22 A. Yes. So the dark blue second from bottom says 23 "reference data" explicitly, yes. 24 Q. But the other product-at-fault values identified on that 25 table are product-at-fault values that also in your view 44 1 relate to reference data? 2 A. Yes. 3 Q. I think they show a persistence of the issue right until 4 the end of 2000; is that right? 5 A. That is correct. 6 Q. How serious or problematic is this? 7 A. This is a big problem. This is a main component of the 8 system. If the reference -- 9 Q. I missed that. 10 A. This is a big problem. This is a serious problem. If 11 this is not done correctly, the system is not working 12 correctly. 13 Q. Going back to 13.1.8, you say: 14 "Interestingly, [I suppose that's a relative 15 value] a Product at Fault value of 'POCL Reference Data' 16 seems to appear in February 2000 and from that point 17 forward occupies a significant portion of the chart." 18 Without teasing you too much, why did you find it 19 interesting? 20 A. I found it interesting that, I believe, ICL Pathway was 21 trying to clearly delineate whether they thought this 22 reference-data issue was a Post Office issue or not 23 a Post Office issue. 24 Q. I see. So they are being more precise as to the 25 attribution of where they believed the reference-data 45 1 issue arises from and attributing that to POCL? 2 A. That is the conclusion I've drawn, yes. 3 Q. Is that the sky-blue colour in the chart at 13.1 -- 4 A. Yes. 5 Q. That's why we don't see it before because it's a new 6 product-at-fault value? 7 A. That's right. Hypothetically, before that it was still 8 could have been caused by POCL, but they weren't 9 tracking it at that level of precision, as you say. 10 Q. That brings us to the end of section 13 of your report. 11 Sir, might that be an appropriate time to take the 12 morning break? 13 SIR WYN WILLIAMS: Yes. 14 MR BEER: We lost you at the end of your sentence there. 15 SIR WYN WILLIAMS: I muted myself too quickly. I said 16 that's fine, Mr Beer. 17 MR BEER: Thank you very much, sir. Can we say half past, 18 please. 19 SIR WYN WILLIAMS: Yes, of course. 20 MR BEER: Thank you. 21 (11.17 am) 22 (A short break). 23 (11.31 am) 24 MR BEER: Good morning, sir. Can you see and hear us? 25 SIR WYN WILLIAMS: Yes, thank you. 46 1 MR BEER: Mr Cipione, can I go back before we move on to 2 section 14 to something you said in section 13 of your 3 report. If we just go back to 13.1.1 on page 98, you 4 say in 13.1.1: 5 "This design feature [that's data-driven logic] 6 it's benefit was efficiently to update the Horizon IT 7 System's functionality without the need to develop, 8 design, test, and deploy new versions of the software." 9 I think you are saying that as a positive thing 10 there. 11 A. Absolutely, yes. 12 Q. Would you accept that the use of reference data in this 13 way, whilst it lowered the need for development, design, 14 testing and deployment of new versions of the software, 15 in practice it may have led to a need for increased 16 testing? 17 A. If the integrity of the reference data or the mechanism 18 for distributing the reference data wasn't good, it 19 could absolutely create a situation where there are 20 a lot of errors in the system, and it might not be 21 readily apparent where those errors are arising from. 22 So that, in effect, could cause more testing, more 23 cycling, more uncertainty around the causes and 24 remediation efforts required to make the system work 25 properly. 47 1 Does that answer your question? 2 Q. Yes, I think it does. 3 Can we turn to section 14 of your report, please, 4 page 106 and your subject heading is: 5 "The Horizon IT System Helpdesk was often the root 6 of Service Level Agreement issues with POCL." 7 Again can you just explain in overview what this 8 section of your report is concerned with. 9 A. This section is concerned with the front-line support 10 that was provided to the subpostmasters and 11 subpostmistresses at the branches that were using the 12 Horizon System. There was -- one of the service level 13 agreements, probably many of the metrics in the service 14 level agreements, required ICL Pathway to have 15 a Helpdesk that was responsive when calls came in, and 16 that's what I'm referring to here. 17 Q. Here you're referring to the HSH Horizon System 18 Helpdesk. That's the first line of support? 19 A. That is the first line of support. 20 Q. You say that in the third line of 14.1.1: 21 "The monthly reports discussed the failings of 22 which HSH, in regard to SLA [failings] for a significant 23 amount of the review period. Concerns were first raised 24 in September '98 and carried through the national 25 roll-out." 48 1 On your review of the material, did those concerns 2 persist at the same level throughout that period? 3 A. They were a consistent topic of discussion in the 4 material I read throughout the period. 5 Q. You say: 6 "The same issues that triggered SLA concerns also 7 'dented' confidence from POCL." 8 Where did you get that information from? 9 A. The monthly reports. 10 Q. So is that ICL saying that those issues dented POCL's 11 confidence, so their perception of POCL's confidence in 12 them? 13 A. Yes, those are ICL's words. 14 Q. "In May 2000, ICL declared an 'own goal' ..." I think 15 that's the word in the report itself, or words in the 16 report itself; is that right? 17 A. That is correct. 18 Q. "... based on Horizon System Helpdesk performance." 19 They replaced their management team, and improvements 20 were noted in June 2000. 21 A. Yes. 22 Q. You then set out in your table at 14.1, on that page and 23 the next, extracts from the monthly reports that 24 substantiate the conclusions that you had reached. Then 25 at the foot of the next page, you tell us that you 49 1 didn't examine the PinICLs and PEAKs to substantiate or 2 undermine what was said in the monthly reports. Just 3 explain why. 4 A. I wouldn't expect any discussion of front-line SLA 5 response rates to exist within the PinICL or PEAK 6 documents. 7 Q. Can we turn to section 15 of your report, please, 8 page 108, where your conclusion is essentially in the 9 section heading: 10 "The Horizon SMC was frequently cited for not 11 properly filtering calls to the SSC. This lack of 12 filtering delayed the SSC from resolving technical 13 problems." 14 So you are here addressing the second to the third 15 level support lines; is that right? 16 A. Yes, that is correct. 17 Q. You remind us in 15.1.1 of the error-escalation process 18 where users were directed to level 1, the HSH first. 19 "The SMC [at level 2] was responsible for 20 determining if the problem required the SSC to become 21 involved. If the issue was deemed necessary for 22 escalation ... it would then be recorded in the PinICL 23 system. 24 "The SSC was responsible for the maintenance of 25 PinICLs". 50 1 What's the importance of a tiered escalation 2 process such as this? 3 A. It would create a very efficient process to handle any 4 issue that was raised by anyone using the Horizon 5 System. The efficiency I'm referring to here relates 6 to: a user is having a problem. That problem could be 7 that the user doesn't know how to use the system or 8 something very simple, in which case it would be good to 9 have at the first period of triage someone there that 10 could help with very simple answers to the user 11 community. That would have been the HSH, which was the 12 first line. 13 But it's not efficient as far as the Helpdesk 14 management process, or the HSH, to have a global 15 understanding of every issue that might or might not 16 come up. So they need to have someone to refer to if 17 their -- because usually the first line of Helpdesk is 18 usually provided with a number of scripts that allow 19 them to diagnose information that was received by the 20 callers, and either help the caller immediately, or at 21 some point that end of the script it's, "Okay, I don't 22 know enough about the system right now. I need to pass 23 you on to the next level." 24 The SMC is the next level who, theoretically, are 25 a little bit more knowledgeable about the technical 51 1 issues of the system, as well as the just general 2 usability of the system. It's their responsibility to 3 make sure that calls that get beyond the initial 4 Helpdesk are created or routed correctly. 5 So, if you recall, in my first testimony, the SSC 6 is not really in charge of any hardware issues. So, if 7 a call gets to the SMC, and it's clear to the SMC that 8 this is hardware issue, I believe that there was 9 a different group that helped the hardware issue. So 10 perhaps the front line would not know exactly where to 11 send that, but the SMC would. 12 The SMC was supposed to only send calls on to the 13 SSC, if they were a technical -- a non-hardware issue 14 technically related to the Horizon System. So that was 15 their purpose and that is generally a good construction 16 of a helpdesk process. 17 Q. What did you see in the monthly reports when the topic 18 of the SMC filtering calls was discussed? 19 A. When I did read about the SMC in the monthly reports, it 20 was usually in regard to the SMC was sending calls to 21 the SSC that they should have been able to handle, or 22 the Helpdesk should have been able to handle, or 23 certainly was not the responsibility of the third line 24 of support, the SSC, the technical portion of the 25 Helpdesk. 52 1 Q. What are the consequences of that? 2 A. The consequences of that are: now you have a group of 3 people at the SSC -- that's the third line -- who are 4 responsible for resolving actual technical issues of the 5 Horizon System. That's a difficult job for multiple 6 reasons. 7 The situation that is created when the SMC is not 8 filtering calls correctly is now they are having to deal 9 with not only technical issue calls but non-technical 10 issue calls, but they don't know that necessarily when 11 they're getting it. If the SMC promotes an issue to the 12 SSC, they think it's a technical issue. So they start 13 investigating it and probably spend a good amount of 14 time on it. If they find out that this was something 15 that should have been handled by the first or second 16 line of support, their attention has been diverted from 17 actual technical issues to non-technical issues that 18 should have been resolved prior to it getting to the 19 SSC. 20 Q. I think you found, you tell us in 15.1.4, that that 21 problem persisted throughout the national roll-out? 22 A. Yes. 23 Q. You also, if we go to page 110, please, examined at the 24 bottom half of the page extracts from PinICLs. The 25 heading to that table says "Verbatim extracts from 53 1 monthly reports". I think that should be from PinICLs 2 and PEAKs? 3 A. Yes. 4 Q. To see whether that which you had read in the monthly 5 reports was supported, and I think you found that it 6 was? 7 A. It is. 8 Q. We can see some of the things which I suppose the SSC 9 were writing emboldened. The first entry, 27 May '99, 10 "Why was the call sent to the SSC?" 17 June '99, "I'm 11 unsure why this was sent to the SSC." 13 October '99, 12 "I don't understand why this call has been sent to the 13 SSC." So it goes on right up until December 2000. 14 A. Yes. 15 Q. So despite I think the issue having been recognised it 16 persisted; is that right? 17 A. That is correct. 18 Q. Can we turn, please, to section 16 of your report. 19 "The SSC was overwhelmed with PinICLs and PEAKs 20 but was earnest in its effort to perform its duties." 21 Is this related to section 15 that we've just 22 looked at? 23 A. Yes. Yes, this is the third line. We discussed them 24 a little bit in the prior section. 25 Q. In terms of the SSC being overwhelmed with PinICLs, was 54 1 that related to the SMC failing properly to filter calls 2 up? 3 A. That was an aggravating condition. 4 Q. What were the other causes of the SSC being overwhelmed 5 with PinICLs and PEAKs? 6 A. There were a lot of errors being raised about the 7 Horizon System. 8 Q. You record in 16.1.2: 9 "The ICL Monthly Reports often call attention to 10 the workload of the SSC." 11 You reviewed the PinICLs and PEAKs and recognised 12 the complexity of some of the issues that SSC was 13 responsible for resolving. What did you mean by that, 14 that you recognised the complexity? 15 A. So just -- you know, in the course of doing my review, 16 of course I looked at quite a few of the PinICLs and 17 PEAKs, and I needed to gain an understanding, as best 18 I could, of the issues that were being raised, meaning 19 that there was a PinICL or PEAK created, and the 20 remediation process that was performed to attempt to 21 remediate these errors as they arose. 22 In the course of that review, I had to basically 23 dive as deeply as I could into understanding many of 24 these PEAKs or PinICLs. Through the course of that 25 reading, it became clear to me that some of these issues 55 1 were difficult to identify what the cause was and, 2 therefore, reading the narratives, I just wanted to 3 comment that it was -- it was a good effort on the 4 people -- on the PinICLs and PEAKs that I read, for the 5 people that for the most part -- now this is not 6 a blanket statement, but this is a statement for the 7 ones that I reviewed -- for the most part of the review 8 process that I observed through reading the narratives 9 of the PinICLs and PEAKs, there appeared to be a very 10 earnest effort to try to resolve the issues. 11 Q. What are the parameters of that view in terms of when or 12 for what period? 13 A. Are you talking what was the period of documentation -- 14 Q. No, the view that there was an earnest attempt to 15 resolve problems by the SSC. 16 A. So my parameters were: was attention being given to it? 17 Did it seem to have been evolving? You know, was extra 18 information needed? Was it provided? What was done 19 with that information and, you know, was there some sort 20 of resolution at whatever level achieved in those 21 PinICLs and PEAKs? 22 Q. It's my fault. You just answered what values did you 23 use. My question was: over what time period did that 24 conclusion relate to? 25 A. Sorry, yes. So the information that I was looking at 56 1 was from 1996 through 2000. 2 Q. So the entire period? 3 A. Yes. 4 Q. You extract for us in 16.1. and the table at 16.1 parts 5 of the monthly reports that support those conclusions. 6 Then, if we go over the page, please, to 16.1.5 on 7 page 114, which is at the foot of the page, you 8 introduce your figure 16.1 by telling us that the 9 following figures line shows the open balance of PPs, 10 i.e. PinICLs and PEAKs, by day. Can you just explain in 11 broad terms what the table is then? 12 A. So an open -- so just to go through the life of one PEAK 13 or PinICL -- 14 Q. Yes. 15 A. -- it's entered into the system. There's an entry date 16 into the system. It's worked on for one to many days. 17 At some point the PEAK or PinICL has closed. So what 18 this chart is showing is, as PEAKs and PinICLs are 19 entered and relieved from the system, there's 20 a particular open balance on any particular date. This 21 chart is showing what the open balance was of the PEAKs 22 and PinICLs on any particular day throughout the period 23 in the chart. 24 Q. Can we look at the chart and you speak to it. 25 A. Certainly. So we can see that the chart starts in the 57 1 latter part of 1996, and the information as far as new 2 PinICLs and PEAKs existing for entry into this chart 3 runs through, you can see right there, at the end of 4 2000. 5 Q. Yes. 6 A. So it might be a little difficult to see, but there is a 7 little bit of a different colour at the beginning of 8 2001. 9 Q. Yes. So 9 January 2001, that line there is dark -- 10 A. It's a bit darker. So for up to the end of 2000 there's 11 the possibility of a new entry into the open balance. 12 By the time we get to the beginning of 2001, I've run 13 out of my source material. But what I wanted to show on 14 this chart was that there were still open PEAKs or 15 PinICLs that were sourced from 1996 to 2000 that still 16 hadn't been resolved. 17 So I'm running the chart out until when all of 18 these PEAKS and PinICLs are resolved. But the open 19 balance is probably, or is absolutely most pertinent 20 from the end of '96 through the end of 2000 on this 21 chart, where it's just the run-out after 2001 begins. 22 Q. The graph shows a downward trend leading to a minimum 23 number at that time in July '99 and then a very rapid 24 rise; is that right? 25 A. Yes, that is correct. 58 1 Q. And you describe in your report that cresting in 2 May 2000? 3 A. Yes. 4 Q. What conclusions overall did you draw from that analysis 5 of the open balance of PinICLs and PEAKs by day? 6 A. Well, the first thing is there seems to be a lot of 7 them. You know, everything's relative on different 8 systems, you know, what allows a PinICL or PEAK to be 9 created, at what level of granularity is it. But having 10 read it, there seems to be a lot of errors being raised 11 regarding the Horizon System. So that's just an overall 12 comment about the chart. 13 You will notice that it looks as though, you know, 14 in the middle of August of '99 the balance seems to be 15 dropping. But as we're approaching or as we're getting 16 to the end of August, all the way, you know, through 17 looks like, well into the next year, there is a 18 significant increase in the population of open errors in 19 the system that need to be resolved. 20 Now that could be because that happens to concur 21 with the national roll-out period, so just the volume of 22 counters that exist, absolutely will have an input to 23 this. But it also indicates to me that there are still 24 significant issues being raised about the efficacy of 25 the Horizon System. 59 1 Q. You conducted an analysis of the time taken fully to 2 close a PinICL or PEAK, and that table is shown over the 3 page, please. Can you explain what figure 16.2 shows. 4 A. So this is trying to represent how long from the 5 inception of a PinICL or PEAK to its resolution. It's 6 basically showing its age. You know, what are the age 7 categories of the PinICLs and PEAKs that were available 8 for me to review? 9 The first bar chart indicates that, you know, more 10 than 30,000 of them took from zero to 20 days to 11 resolve, from opening to closing. 12 The next bar is 20 to 40, and it goes on, you 13 know, from there. But there is -- you know, at the end 14 of the chart there is a box that represents everything 15 that took more than 180 days to resolve. So this is 16 just showing a distribution on how long it to look to 17 resolve a particular PinICL or PEAK. 18 Q. So that last chunk at the end there, more than six 19 months to resolve, does that equate to some 3,000 20 PinICLs and PEAKs requiring more than six months to 21 resolve? 22 A. Yes. 23 Q. Did you form a view or draw any conclusions in relation 24 to this analysis? 25 A. It appears to me that these take a long time to resolve. 60 1 The amount of -- basically the ageing of these PinICLs 2 and PEAKs seemed to be very long, in my opinion, on how 3 to resolve them. Of course every one is unique. There 4 could be a particular issue that is outside of their 5 control that they didn't feel comfortable in actually 6 officially closing a PinICL or PEAK down. That's fine 7 for one or two. But what I'm getting from this chart is 8 that it takes a lot of time to actually close these 9 down. So that doesn't -- it's not a good look. 10 Q. You also, I think, analysed those people, I think all 11 within ICL, who were involved in resolving 1,000 PEAKs 12 or PinICLs or more, and there was a list of 48 of them, 13 I think, and this is a sort of a league table; is that 14 right? 15 A. Yes, that is correct. 16 Q. Barbara Longley, the first amongst them, an employee of 17 ICL Pathway; is that right? 18 A. That's my understanding, yes. 19 Q. Can we turn to section 17 of the report, please. 20 SIR WYN WILLIAMS: Mr Beer, before we do that, can I just go 21 back to the table at 16.1, just either to confirm what's 22 in my head or for Mr Cipione to dispel it. 23 He looked at the opening of PPs up until the end 24 of 2000. So I've taken from that, Mr Cipione, that if 25 we look at the chart which is more or less opposite 61 1 9 January 2001, those are the open PEAKs at that point, 2 and it actually takes until, well, between September and 3 December 2002, which is getting close to two years for 4 some of them to be resolved. Have I got that right? 5 A. You have that right, sir. 6 SIR WYN WILLIAMS: Fine, thank you. 7 MR BEER: Can we turn to section 17 of your report, please, 8 on page 118. Again the topic disclosed by the heading: 9 "Acceptance Incidents [AIs] were a gating issue to 10 the financial success of ICL Pathway. A persisting 11 issue related to AI376." 12 Again to introduce this topic, can you explain 13 what you're addressing in this section of your report, 14 please. 15 A. Yes. So my understanding from reading the material was 16 that the acceptance was a term that POCL and ICL Pathway 17 used to confirm that the system was good enough to be 18 put into practice. It was good enough to be deployed 19 and, more specifically, it actually triggered, I 20 believe, the first payment to ICL Pathway for 21 constructing the system and for implementing the system. 22 Q. You tell us in 17.1.2 that: 23 "Acceptance was financially significant to ICL 24 Pathway. [Because] ICL Pathway was paid once acceptance 25 was achieved. [And] it received a high degree of 62 1 attention by ICL Pathway." 2 The "it" there, are you referring to acceptance or 3 the financial significance attributed to acceptance 4 or -- 5 A. Acceptance. I'm referring to acceptance. 6 Q. You tell us that the monthly reports describe 7 outstanding or existing AIs, and ICL Pathway's efforts 8 to resolve them, and 376 caught your attention. It 9 concerned accounting integrity. Why did AI376 catch 10 your attention? 11 A. The Horizon System, while it is not the actual official 12 financial accounting system for the Post Office, it is 13 the source of all information for the financial -- for 14 the accounting for the Post Office. If there are any 15 issues with accounting integrity that are sourcing from 16 the Horizon System, that seems like -- I mean, that's 17 the purpose -- you know, that's one of the main purposes 18 of the Horizon System, to make sure that it is providing 19 accounting data with integrity to the Post Office. 20 So when I saw that, that caught my attention, that 21 that was a term associated with AI376. 22 Q. You point out that 376 was one of the final AIs to be 23 closed. 24 A. That is correct. 25 Q. In terms of the chronology, you tell us that: 63 1 "24 September 1999 marked the day that Acceptance 2 was granted, triggering [an invoice for payment of] 3 £68 million ... to be paid within 30 days." 4 Then: 5 "In November '99, at least one full month and 6 possibly two full months after acceptance ..., ICL 7 reported that 'POCL have come round to the understanding 8 that dealing with residual AI376 concerns in the short 9 to medium term will rely on processes and tools but no 10 new software features as such'." 11 Why did that catch your attention? 12 A. I keep going back to, you know, the integrity of the 13 accounting data that is being sourced from the Horizon 14 System. That's an extremely fundamental concept, and 15 that is probably the most important item that -- most 16 important feature that the Horizon System should 17 deliver. 18 The fact that this is still an issue troubles me 19 when I read about it. 20 Q. Why does it trouble you? 21 A. Because, if that data doesn't have integrity, the system 22 is not performing it's proper function. 23 Q. In 17.1.7, if we scroll down, please, you record that: 24 "In February 2000, ICL Pathway declared that the 25 POCL Acceptance Manager has left the project and 64 1 transferred the residual actions to 2 'business-as-usual'". 3 It was unclear to you exactly what took place to 4 close AI376. 5 You say: 6 "The reading of these entries leaves much room for 7 interpretation." 8 What did you mean by that, please? 9 A. I was not clear from the materials that were provided to 10 me exactly how this acceptance issue was closed. I know 11 it's a big deal. I know it's a big deal. I keep 12 seeing, I keep reading about it's a big deal, and then 13 it's closed without a lot of commentary on exactly: was 14 the system fixed? Is it perfect now? I did not derive 15 anything from the material I was reading as to what 16 happened to actually allow that particular Acceptance 17 Incident to be closed. 18 Q. You just asked yourself a question there or posed 19 a question. "Is it perfect now?" Is that what you 20 would have required for accounting integrity? 21 A. I certainly would have required that I thought it was 22 perfect. I mean, there's no perfect system. There's 23 always the opportunity to introduce errors in any 24 computer system. So perfect is not an achievable goal 25 for any system. 65 1 However, you should think that it's pretty 2 perfect. You should think that there's no reason that 3 is out there that you will be introducing errors into 4 your financial system, based off of how the technology 5 is working. 6 Q. You then set out in 17.9.1: 7 "Regardless, the fact that accounting integrity 8 was a persistent issue in the national roll-out of 9 Horizon cannot have been the intention of the sponsors 10 nor the goal of ICL Pathway." 11 That might be stating the obvious. It's plainly 12 not a goal. 13 A. Of course not. I'm positive that ICL Pathway did not 14 have that as their intention. I'm positive that 15 Post Office, as the sponsor of the system, was not 16 expecting that. Perhaps I wasted a few lines by writing 17 that, but I just wanted to confirm that I don't think 18 that this was an intended feature of the system to have 19 problems with the accounting integrity. 20 Q. But your focus has been instead on what was done to 21 resolve it. 22 A. Yes. 23 Q. Is the state of your conclusion that you're unsure 24 exactly what was done to resolve the issue on AI376? 25 A. From this report, from the materials I was provided for 66 1 this report, I could not figure out what was actually 2 done to close that incident. 3 Q. Have you read or heard evidence since then which goes to 4 that issue? 5 A. Yes. 6 Q. What was done; what is your understanding of what was 7 done? 8 A. My understanding of what was done was that the benchmark 9 for closing it continued to change throughout this 10 period. There were a lot of tools and techniques and 11 mechanisms that were required to be wrapped around the 12 Horizon System to check for errors that the Horizon 13 System might have been passing along through its 14 processing. So I guess, when I read about that -- so 15 let me explain. 16 You have the source of information which is 17 Horizon, which is supposed to be creating the accounting 18 information that's fed into the financial systems. We 19 know through the PinICLs and through this AI376 that 20 that was not operating correctly, at least during this 21 period. So one option, which I think would have been 22 the best option, was to go and make sure that the actual 23 Horizon System was transacting correctly, was processing 24 the data correctly, so that it had high integrity. 25 What I understand, from reading from the materials 67 1 that I have been provided with subsequent to writing my 2 report, is that there was a lot of difficulty in 3 figuring out exactly what was introducing these errors 4 in the Horizon system. So as a plan B, there were a lot 5 of processes and procedures and tools that were trying 6 to ring-fence the Horizon systems, which was known to be 7 generating information that did not have the highest 8 level of integrity, to try to catch those errors before 9 it hit the Post Office's financial systems. 10 Q. What was your view as to the introduction of a tool or 11 tools that sought to recognise that there was a lack of 12 financial integrity or accounting integrity? 13 A. Having a safety net around any financial system is fine. 14 It's always a good practice. 15 However, if you know that the source of 16 information that's creating the errors is your own 17 system, that being ICL, I think that's where the 18 effort -- there should have been a lot of effort in 19 making sure that that was rock solid, that everything 20 was being done to make sure that that system functioned 21 exactly the way it needed to function, because that's 22 one of the basic features of this system. It's a very 23 important feature for any financial system. 24 You have to have integrity with the data, and 25 I think, from the materials that I've read, it sounds 68 1 like they were going away -- at least during this 2 period -- going away from that particular effort to 3 trying to just make sure that they knew errors were 4 being generated, let's have a plan B, a plan C, a plan 5 D, to try to make sure that, if an error existed, it was 6 caught before it actually got into the Post Office's 7 financial records. 8 Q. So, rather than putting effort into or all effort into 9 identifying root causes, you are focusing on catching 10 the consequences? 11 A. Yes. 12 Q. As a process what's your view of that? 13 A. I think that having those type of controls is always 14 a good practice, provided that you're really trying to 15 make sure that the genesis of the information is 16 correct, has efficacy to it. If you're relying on your 17 safety nets to protect you, I would suggest that that's 18 poor practice. You need to be -- sorry, if that's one 19 of your main reliances, that's poor practice. You 20 should always rely on your safety net. Safety nets are 21 there for a reason. It's always a good thing to have 22 a safety net, or guard rails, or even a couple of layers 23 of safety nets or guard rails. But if you are -- from 24 the materials I've read, it sounds like this was still 25 an issue. This was an issue with the Horizon System. 69 1 If you continue to allow those safety nets to 2 cover functionality that should be present in the core 3 system, I think that's a terrible practice, rather than 4 correcting the core system in a way that you know will 5 persist. 6 Q. Did you form a view whether the issues relating to AI376 7 were sufficiently serious to be bound up in whether the 8 system should have been accepted or not? 9 A. Accounting integrity, absolutely. I'd say that's 10 probably the number 1 thing for this system, the 11 accounting integrity. 12 Q. And, therefore, what was your view as to whether or not 13 the issues with AI376 were, on the face of it, 14 sufficiently serious as to affect acceptance? 15 A. It absolutely should have affected acceptance. 16 Q. Can we turn, please, to page 129 of your report. This 17 is skipping over extracts from monthly reports 18 concerning AI376 in particular. 19 At the foot of the page, you say you: 20 "Surveyed the PinICLs and PEAKs for the pattern of 21 'Acceptance Incident' followed by numeric or 'AI' 22 followed by a numeric to identify PinICLs and PEAKs that 23 dealt with Acceptance Incident issues. The following 24 figure shows that 358 PinICLs and PEAKs were related to 25 Acceptance Incidents and 223 PinICLs and PEAKs were 70 1 specifically associated with AI376." 2 If we go over the page, please, I think the 3 lighter colour green relates to AI376; is that right? 4 A. Yes, that is correct. 5 Q. It's slightly difficult to tell. Does that show that 6 they persisted into after June 2000? 7 A. I also am having a hard time seeing the colours, but it 8 looks like they are gone -- it looks like the last entry 9 for those are July of 2000. 10 Q. Thank you. Is there anything else you wish to say about 11 Acceptance Incidents and in particular AI376? 12 A. The accounting integrity of the Horizon System, I think, 13 is the most important part of the Horizon System. It is 14 the reason -- you know, beyond any -- just the 15 operational efficiency of the branches being worked, 16 sure, that's important, but no-one would ask for 17 a point-of-sales system that they knew were connected to 18 your financials if there was any doubt about the 19 integrity of the accounting information that was being 20 derived from it. 21 Q. Can we turn to section 18 of your report, please. Thank 22 you. With the heading: 23 "Payment and receipt imbalances were common 24 symptoms with varied causes." 25 You say that what you previously discussed 71 1 directed you to identify examples of accounting issues 2 within the PinICLs and PEAKs, and in this part of your 3 report you explore selected examples of accounting 4 issues as represented by payment and receipt imbalance 5 issues. Just describe to us why you are focusing on 6 this, please. 7 A. I thought it was important to try to, number 1, support 8 the last, you know, the prior theme that we just 9 discussed. But I also thought it was important to 10 illustrate how these issues were manifesting themselves. 11 I think as we get to talk through the examples that I've 12 put in it's going to show that it wasn't as there was 13 a particular section of the Horizon System that was 14 causing integrity issues as far as, like, the 15 development code was concerned. 16 The causes were all over the place, in my opinion, 17 as to what created these payment and receipt imbalances, 18 and I thought it was important to just show that there 19 were a variety of reasons that would illustrate how 20 things could be out of balance. 21 Q. You remind us in 18.1.2 of the Cash Accounting Period 14 22 to 15 and then into 16, example that you gave earlier in 23 your report which was, I think, an example showing how 24 it should work. 25 A. Yes. 72 1 Q. You have updated that to show what happens when you 2 introduce a bug into the system which causes the 3 brought-forward balance to be incorrectly calculated. 4 So, if we just scroll down, please, and look at fig 5 18.1, thank you. Can you talk us through it, please. 6 A. Certainly. So what I'm showing here is that, if the 7 Horizon System was to inaccurately calculate the opening 8 balance for cash and stock, in this example, just 9 multiplying those values by three, as this particular 10 receipts and payments account was reported out, it would 11 show an imbalance between the receipts and the payments. 12 So in this instance there's, you know, approximately 13 £11,000 difference or discrepancy on this particular 14 report. 15 So this is just an illustration. This is 16 something that I thought would be helpful for the chair 17 just as an example. 18 Q. So this shows an imbalance of £11,000 where payments are 19 greater than receipts? 20 A. Receipts would be greater than payments, I believe. 21 Maybe I wrote that wrong. 22 Q. I was going from over the page -- 23 A. Sorry, yes. 24 Q. Let's just go back to the table and check. 25 A. I think that I might have switched receipts and payments 73 1 here. But the important part is there's an imbalance 2 here. 3 Q. So this would show as, what, a surplus of £11,000? 4 A. Yes. 5 Q. To what extent could you ascertain whether that 6 information that's shown on this page here was visible, 7 assuming this was a real-life example to 8 a subpostmaster? 9 A. I will say that, during the course of my review this, in 10 particular, was a little bit difficult to tease out from 11 the information that I had. I did have some technical 12 manuals, I did have some documentation available to me. 13 None of the reports that were available in those manuals 14 as representative of the reports that the branches might 15 see looked as straightforward and clean as this. So 16 I thought it was, just for the concepts, important to 17 put this -- I guess this is the way I would have shown 18 it, this is to show the concept, but this certainly this 19 not, I believe, anything close to what the people at the 20 counters would have seen. 21 Q. If we go over the page, please, and look and 18.1.5, you 22 say there are various other issues that could result in 23 am imbalance: payments that were not recorded in 24 Horizon; payments that were erroneously recorded in 25 Horizon; receipts not recorded or erroneously recorded 74 1 in Horizon; carried forward balance incorrect because 2 the cash and/or stock were not correctly declared by the 3 subpostmaster, or there have been cash and stock changes 4 that couldn't be accounted for. 5 You then describe a methodology. Can you explain 6 what you did then, please, in relation to this issue. 7 A. Certainly. So in order to come up with the examples 8 that eventually exist in this particular section, we 9 wanted to basically search all of the PEAKs and PinICLs. 10 Effectively what we did was we went through a number of 11 iterations, looking at one or two trying to, manually, 12 trying to understand the way they would look, and then 13 start feeding in concepts or words into the Brainspace 14 technology that I described at the very outset of this 15 hearing. 16 Q. Are they described in figure 18.1? 17 A. Yes, yes. So, for instance, in this we're looking for, 18 you know, the concept of error, cash, issue, fail, and 19 others. So what we're looking for -- basically what's 20 happening is Brainspace is taking entirety of the PEAKs 21 and the PinICLs, and coming up with a bunch of words and 22 phrases that are available for searching on. What this 23 is showing is, you know, some what Brainspace calls 24 a concept which is exactly how it sounds. What does it 25 think -- just without being told anything, what concepts 75 1 could exist within this body of documents? One of them 2 was error, one of them was issue, one of them was cash, 3 and one of them was fail, that we have in the light blue 4 highlighted here. So that's the start of the process. 5 Q. So that gave you your first run of data, namely 38,803 6 PinICLs or PEAKs? 7 A. Yes. 8 Q. You explain that only a small number of these were rated 9 highly relevant to each of the displayed concepts. Just 10 explain what you mean by that. 11 A. So within these concepts, so as the Brainspace 12 technology runs and includes different numbers of 13 documents underneath each of these concepts, it has 14 a grading for how closely aligned to the concept is this 15 PinICL or PEAK and how weak aligned is it. What I'm 16 saying here is that, although we have a large 17 population, since this is kind of the outset of the 18 process, as expected, not a high percentage of any one 19 of the populations for each one of these concepts was 20 considered very tightly aligned to that concept. So we 21 needed to go on to another step. 22 Q. You explain that in 18.1.18: 23 "A review of those that top of the distribution 24 revealed an issue mentioned in several PinICLs and PEAKs 25 namely cash or stock not balancing. The common phrase 76 1 being use was 'receipts vs payments'." 2 So did you use that concept for a new search? 3 A. Yes. 4 Q. Then, if we go over the page, please, what was the 5 result of that new focused search? 6 A. So that provided a very focused set of documents. I 7 believe it was 67 documents that was the result of the 8 population underneath the receipts versus payments 9 concept. 10 Q. Can you explain what 18.1.9 means, please. 11 A. Okay. So this was just another phrase, and it looks to 12 be a technical term that was also associated here within 13 the search that happened -- it's basically -- it's 14 something that came up to be considered, to be looked 15 at, and it looks as though it was -- so it says: 16 "It's worth noting that 'EnteredBBF' is commonly 17 found in PPs when pasting in error messages from 18 a manual migration message store." 19 So from my understanding of the readings of the 20 PEAKs and PinICLs, oftentimes the SSC will try to pull 21 in information from the message stores, and they're also 22 entering -- they're trying to document the process that 23 they're using to fix it. When they happen to paste in 24 information from the message store, this particular 25 phrase happened to have been included in that 77 1 information. So it shows up a lot. That's all we're 2 saying. 3 Q. So the search returned 67 documents, and you actually 4 read those? 5 A. Yes. 6 Q. You say that a CMML model was created. What does that 7 mean? 8 A. I'm going to have to refer back to even refresh my 9 memory of what the actual words are. But essentially 10 what that means is we're going through an iterative 11 process because this is a machine-learning mechanism. 12 The first step that we talked about was an unsupervised 13 machine-learning process, but every step beyond that is 14 what I'm going to be describing as a supervised 15 machine-learning process. 16 What does that mean? That means that we are 17 giving a particular problem to Brainspace to solve. We 18 are allowing Brainspace to provide what it thinks the 19 best answer is, and then we're grading that answer and 20 feeding the results back. So when I say grading that 21 answer, so, for instance, I might say I want to look for 22 a particular concept like error -- you know, that was 23 that beginning of this process -- and let's pretend that 24 there was a document population of 100, and Brainspace 25 came back and said, "I think 15 of these documents are 78 1 related to the concept of error". It's now incumbent 2 upon me as a reviewer to grade Brainspace. 3 So I might say, of those -- what did I say, 4 11/15 -- of the 15 that Brainspace thought was an error 5 concept, I will say, "I agree with you on 11 of them and 6 I disagree with you on 4 of them", and feed that back 7 into Brainspace and let it do the process again. 8 So now it not only has an idea of what I think is 9 an error, it also has an idea of what I think is not an 10 error, and it can then basically incrementally improve 11 its search to align with what I think the concept of an 12 error is, as opposed to what it thinks the concept of 13 error is. 14 Q. Thank you. You say that three additional rounds of 15 training were undertaken, as set out in your table. 16 Just summarise what the additional rounds of training 17 were for. 18 A. The additional rounds of training were the process I was 19 just talking about, except for it was more than once. 20 We kept doing it over. We kept telling Brainspace what 21 we wanted, we'd get it back, we would grade and we would 22 do it again, and get it back, and do it again, and get 23 it back, until we get to the end process here which was, 24 I believe, a 386 document. 25 Q. Then scroll down, please. I don't think we need to 79 1 address 1.12 or 1.13, do we? 2 A. No, no, they're just technical descriptions. 3 Q. Therefore, if we come to your analysis, which is divided 4 between quantitative and qualitative, starting with the 5 former, can you explain what you're describing in 6 18.1.4, please. 7 A. Certainly. So the result of the prior process was that 8 we had identified 399 PPs. 137 were selected for 9 review, and 127 as relevant. So that just sets -- 10 Q. Relevant to the concept, the idea of the payment and 11 receipt imbalance? 12 A. Exactly. So for that population the first thing we 13 wanted to do was just to quantitatively describe those 14 PPs, those PinICLs and PEAKs, across two of the data 15 elements that are associated with those. One is the 16 response category and one is the defect cause, and 17 I think we've discussed those already today but I'd be 18 happy to discuss them again if you would -- 19 Q. If you would, just by reference starting with response 20 category to the table at [18.16], please. 21 A. So as a PinICL or PEAK is processed, they start tracking 22 what's happening with this. We're going to identify -- 23 you know, we're going to put different values in what's 24 called the response category, which is basically how 25 each individual that's reviewing the PEAK or PinICL is 80 1 describing it. This is the category -- this is me as 2 a review, this is my response to the categorisation of 3 what I think this particular -- the stage of this 4 particular PEAK or PinICL is. So that's one dimension. 5 The defect cause, that's a little bit more 6 straightforward. As I said before, that can change 7 throughout the process, but ultimately there is a final 8 value associated with the defect cause which is supposed 9 to represent what the SSC thought the actual defect was 10 for this particular PEAK or PinICL. 11 Q. In terms of the response category, I think you concluded 12 that 19 per cent of the closure reasons, those which are 13 described as "Other" in figure 18.3 and "Administrative 14 Response" in 18.3, didn't provide much or any insight 15 into the investigation process; is that right? 16 A. That is correct. 17 Q. Just explain what the significant of that is. 18 A. So, you know, these data elements should provide 19 a little bit of information about exactly how these 20 PEAKs and PinICLs were resolved, you know, what the 21 response was. When I see something like 22 "Administrative", it almost sounds like an "other" or 23 catch-all. It's just not very in informative to me as 24 to what the issue is. 25 So, you know, for instance, in the chart in figure 81 1 18.3, one of them is "no fault in product". That tells 2 me a lot more than "administrative" response. At least 3 I understand what the point of view was from the SSC on 4 that particular PEAK or PinICL. 5 Q. Thank you. Over the page, please, and scroll down. 6 Second element of the analysis, analysis of defect 7 cause, and I think from this you found that 8 a significant proportion of these PinICLs and PEAKs had 9 defect causes that were recognised by ICL Pathway as 10 being related to the design or development of Horizon. 11 That is 45 per cent on the pie chart. Tell us which 12 elements take you to the 45 per cent. 13 A. Certainly, and it's in the footnote. So it's, if the 14 defect cause is design, design/high-level design, 15 development/code, development/low-level design, 16 development/reference data. 17 Q. That takes you to your 45. 18 A. Yes. 19 Q. You say in the narrative: 20 "This indicates [to you] that there were 21 acknowledged bugs, errors or defects in Horizon that 22 were capable of giving rise to a payment and receipt 23 imbalance." 24 A. Yes. 25 Q. Why did that indicate that to you? 82 1 A. Because the corrective -- because the cause -- 2 identified by the SSC was it was a development or 3 a design aspect of Horizon. 4 Q. So it's self-describing? 5 A. Yes. 6 Q. What's the consequence of that? What's the significance 7 of that? 8 A. The significance is this was an error in the Horizon 9 System. That's what was causing these imbalances. 10 Q. But also an error in the Horizon System giving rise to 11 or capable of giving rise to a receipt or payment 12 imbalance? 13 A. Absolutely, yes. So these are all receipt and payment 14 issues that we're talking about now. So everything 15 we're -- everything that we're going to discuss in this 16 section has to do with that concept. 17 Q. You turning to analyse the time that it took to resolve 18 PinICLs and PEAKs in this category. We can turn over 19 the page to 18.5, and you tell us in the text underneath 20 that only 26 or 20 per cent of them were fully closed 21 within a week; 55, 43 per cent took five weeks or more 22 fully to close; and 37, i.e. 29 per cent, took nine 23 weeks or more fully to close. 24 A. Yes. 25 Q. What, if any, conclusion or observation did you make on 83 1 that material? 2 A. Well, the best case scenario on this -- let's talk about 3 the nine weeks or more. Let's talk about the errors or 4 defects that relate to those 37. The best case is that, 5 after the initial error was identified and entered into 6 the PEAK or PinICL system, there was another nine weeks 7 where that error could have been manifesting itself 8 throughout system. That's the best case scenario unless 9 something outside of this process was occurring that 10 mitigated that issue. 11 Q. Can we turn over the page, please, to your qualitative 12 observations. You say in 18.2.2 that it's useful to 13 examine individual PinICLs and PEAKs. You selected 14 seven that highlight the varied causes of payment and 15 receipt imbalances. How did you select them? 16 A. I picked them from the population that we were just 17 discussing. So I picked them for purposes to illustrate 18 the various possible causes of this particular symptom. 19 Q. So to give a fair cross-section of illustrative causes? 20 A. Yes. 21 Q. You summarise at a high level the PinICL or PEAK. You 22 took some excerpts from the PEAK or PinICL and then you 23 gave observations. We can see that in practice, if we 24 go over the page, please, to look at the first of the 25 seven examples. 84 1 Summary at the top, then the Chronology, and this 2 isn't all of what's in the, in this case, PinICL. We've 3 seen that some of them are very long indeed, running to 4 many, many pages. 5 A. Yes. 6 Q. But you have taken that important parts of the 7 chronology; is that right? 8 A. Yes. 9 Q. And then over the page, please: 10 "My observations". 11 If you scroll down, please, you have asked 12 a series of six questions which I think are the same 13 repeating through the seven examples, and we can see 14 what those questions are. 15 So that was the way that you were structuring your 16 examination of the seven examples. 17 A. Yes. 18 Q. If we go back to page 137, please, at 18.2.4 you make 19 the following general observations. Can you talk us 20 through those five observations please. 21 A. Certainly. So many of these PEAKs and PinICLs seem to 22 have been raised as a result of internal reconciliation. 23 So that would have been what I was describing before, 24 perhaps, you know, one of those safety nets that exists 25 to detect errors. So these PinICLs or PEAKs weren't 85 1 necessarily raised by an SPM calling up and saying, 2 "I have a problem." They were more raised by the 3 detective controls that had perhaps ringfenced the 4 Horizon System. 5 Q. Thank you. The second observation? 6 A. So and I think I said this in one of my prior themes. 7 Reading these, I think that the team that's trying to 8 resolve these is earnest in their effort to trying to 9 fix these problems, at least as they saw the symptom. 10 So I did not see any indication that they weren't trying 11 to solve them or trying to not do their job, and I think 12 that's important for the chair to know. 13 Q. Third observation. 14 A. Each one of the PEAKs and PinICLs had a lot of different 15 teams that were involved. So oftentimes there would 16 be -- so there's different development teams that are 17 being -- so there's the SSC, and then there's, you know, 18 different development teams within the Horizon group 19 that are working together to try to solve it. 20 So the SSC could try to solve something by 21 themselves, but they might need to reach out to someone 22 on the EPOSS development team or on the APS development 23 team or on the LFS development team, depending on, you 24 know, what time period you were in. 25 So this is just describing that there is a lot of 86 1 conversation back and forth between people in the 2 various groups. 3 Q. Then your fourth observation, you have told us about the 4 earnest effort to investigate issues, identify a root 5 cause and mitigate future recurrences. How successful 6 was the earnest effort? 7 A. I think that the effort, the earnest effort that I was 8 talking about, fixed something. It fixed something. 9 But there's -- perhaps we can kind of step away from 10 this particular example but just talk about what causes 11 errors and at what level do you remediate errors. 12 There are symptoms, and there are probably lots of 13 different levels of causes of those symptoms. The 14 further up the food chain you can get to the cause, the 15 more likely it is you are going to eliminate future 16 occurrences of those particular symptoms. 17 What I'm describing here is: I know that something 18 was fixed, for sure. I know that perhaps there was 19 a correction entered into the system. So, for instance, 20 there was a receipt and payment imbalance. Someone 21 understands what that is and understands that at the 22 Post Office that needs to be corrected. So perhaps one 23 fix would be: I'm going to correct this particular 24 instance of this symptom. That would probably be really 25 just attacking the symptom. 87 1 Sometimes it might be: well, I know that there was 2 a reference-data issue related to this, and I know that 3 there was one specific part of the reference data issue 4 reference-data system that was at fault here. So I'm 5 going to go correct that, and also provide a correcting 6 accounting entry to the Post Office. 7 But what if I know that the management of that 8 reference data is really the root cause of that 9 reference-data issue existing to begin with? There 10 might be another level that we need to go to fix that 11 and there might be another level about that. 12 So what I'm trying to describe here is: these were 13 all addressed in an earnest way to actually resolve 14 these particular PEAKs and PinICLs. I'm not clear on 15 whether the appropriate activity to get at the ultimate 16 genesis of all these problems was really ever addressed 17 across the spectrum of all these, because the SSC has 18 got a lot of work to do. We discussed that already in 19 one of my themes. Their job is to fix this issue, and 20 they did a good job on that. 21 What I'm not clear on is whether the true cause 22 that's creating all of these error instances is being 23 addressed or not, and it's just not apparent to me in 24 each one of these very specific errors that are being 25 addressed. 88 1 Q. That's why you say it's not evident that the identified 2 issue was resolved? 3 A. Yes. 4 Q. So the symptom might have been cured? 5 A. Yes. 6 Q. Or the next proximate cause to the symptom might have 7 been addressed? 8 A. Yes. 9 Q. But there wasn't evidence that any root causes were 10 addressed? 11 A. Well, I guess that really depends on what we define as 12 the root cause. So we can have one degree of separation 13 of root cause, two degrees. I mean, if we know that 14 just one degree of separation actually is truly the root 15 cause, there could have been things that were absolutely 16 addressed. But I'm just not sure of what the degree of 17 separation is here, because I'm only looking at the 18 PinICLs and PEAKs information. I don't know anything 19 more the system for sure. 20 Q. Your fifth conclusion: in the majority of these PinICL 21 and PEAKs, the root cause is related to Horizon. What 22 do you mean by that? The riposte to that that might be: 23 of course it's related to Horizon. What are the other 24 alternatives? 25 A. The first example we're going to get to is not 89 1 a Horizon-generated issue. So that one doesn't have 2 anything to do with Horizon; the rest of them do. 3 Q. Okay. What's the significance of that? 4 A. The significance of that is these are not environmental 5 factors that are causing these imbalance issues; it's 6 Horizon. 7 Q. It's the system? 8 A. Yes. 9 Q. Can we turn then to the first example, please, on 10 page 138 and 139. If we could display those together, 11 please. It may be that we won't be able to display all 12 of them together and, therefore, at least in a readable 13 way. We'll do them page by page. 14 Thank you. 15 Can you talk us through this first example, 16 please, concerning ECCO migration. 17 A. If it's all right with you, Mr Beer, I am going to read 18 it verbatim, each line and then -- 19 Q. Then give an explanation? 20 A. Yes. All right. So the first line is: 21 "[The] system call related to TIP 1052. Please 22 route this call to John Moran in EDSC." 23 I believe EDSC is another term for the SSC. 24 I think that's correct; I'm not positive that's correct. 25 But what this is indicating to me is, related to TIP 90 1 1052. So I know that TIP is the Post Office's 2 accumulator of all the transactions for its accounting. 3 That's where Horizon is delivering its transactions to, 4 at the Post Office. 5 So this would be an example of: since it's coming 6 from TIP this indicates to me that this is not coming 7 from postmaster. This is coming from somewhere down 8 dream saying that there's a problem, okay. 9 Q. Yes. 10 A. All right. So, MSU, John Moran: 11 "The following has been copied from Business Call 12 [I'll skip that incident]. Discrepancy in Cash Account 13 for week 46. A comparison between values received 14 within the cash account files and those derived from the 15 transaction stream ... identified the following 16 differences." 17 So it's showing differences. So what this is 18 saying is: it looks like the cash account is showing 19 different numbers than what the transactions that have 20 already arrived in TIP are. So I'm imagining that there 21 is a balancing that's being done at the branch. It's 22 being sent up to the Post Office. The Post Office 23 already has, in theory, all the transactions, and they 24 are comparing what they think that should look like to 25 what the branch is sending it, and there's 91 1 a discrepancy. 2 Q. There's a shortfall of £97.39? 3 A. Yes. So now we're asking the SSC to investigate. So 4 that gets us to the next line. 5 All right, so we see some colour commentary here: 6 "This is an illustration of the stupidities that 7 ECCO software allows." 8 So just to refresh everyone's memory, ECCO was a 9 point-of-sales system that I believe existed in some 10 branches prior to the Horizon system. So that's why 11 this is an example of an issue that is not really 12 a Horizon issue, okay? 13 "A clerk can transfer cash and cheques between 14 stock units without bothering to make sure that they 15 match up. The result shows up when the transaction data 16 is migrated to Horizon, which insists on clear 17 demarcation between cash and cheques. In this case the 18 critical transactions are Transfers In of two cheques 19 whose total amount exactly equals the discrepancy 20 noted." 21 So what this is indicating to me is that ECCO 22 allowed someone to call -- let's see -- ECCO is allowing 23 cash to be called a cheque. So that's where they're out 24 of balance. The Post Office is thinking: I should have 25 some cash and cheques, and Horizon is getting -- this is 92 1 probably an initial load of data into -- this is 2 probably an initial month in, at this particular branch, 3 and when they migrated the data from ECCO to Horizon, 4 ECCO incorrectly put everything into cheques instead of 5 dividing it between cash and cheques. So that's what 6 the issue -- that's the source of this particular issue. 7 Then it goes on to say: 8 "I think Steve Warwick is aware of the ECCO 9 problem here but as a matter of course I will route the 10 call to him to allow him to comment." 11 Steve Warwick says: 12 "This issue is well documented in previous 13 incidents with TIP. The effect is that Pathway system 14 reports the values of the affected products incorrectly 15 on the Cash Account for the migration CAP although the 16 cash account still balances. TIP then uses the Cash 17 Account figures from the migration CAP as the start 18 point for validating the next Cash Account received from 19 the outlet and report a discrepancy between the 20 transactions received in week 2 and Cash Account for 21 week 2. This is a user error pre-migration of ECCO+ 22 Office." 23 This is not a Horizon issue. This is not 24 a Horizon-generated issue. This is an issue that 25 they've seen before when they are migrating from the 93 1 ECCO system to the Horizon System. So then they do 2 their passing back to MSU for issue of RED. I think 3 that's a reconciliation database. I can't recall 4 exactly what "RED" stands for but my understanding is 5 that's kind of a documentation method that the SSC uses 6 to or Horizon ICL Pathway uses to show the Post Office 7 that everything's all right, and they issued it to the 8 customer, no data error, close the call, and then they 9 closed the call. 10 Q. Your observations, please. 11 A. So if I assume that the issuing of a RED notice to the 12 customer resolved the PEAK or PinICL, then yes. So -- 13 and that's my assumption but I just wanted to note that 14 the resolution was that they documented it and sent it 15 on to -- 16 Q. So the immediate issue was fixed? 17 A. Yes, exactly. 18 Q. On that assumption? 19 A. Yes. 20 Q. Was a defect or root cause identified? 21 A. Yes, this was a migration -- this was an ECCO-generated 22 issue and it was identified. 23 Q. And was that correctly recorded? Was that defect or 24 root cause correctly recorded in the PinICL? 25 A. I said yes on this. I think they indicated in the text 94 1 here that this was because the user made that transfer, 2 so I think that this is an appropriate identification. 3 Q. Next question: evidence that the defect or root cause 4 was addressed. I think you said you wouldn't expect it 5 to be addressed? 6 A. I doubt it. 7 Q. Then more general observations; can you help us with 8 those? 9 A. Right. So what I wanted to bring up here is that this 10 was -- this was generated through an internal 11 reconciliation, which is fine -- and this would have 12 been one of the reasons why having that ring-fence 13 around, you know, the system was good. This wasn't an 14 issue really that was generated from within Horizon, but 15 it was an issue to make sure that the integrity of the 16 data was right and that particular safety net worked 17 correctly and appropriately in this case. 18 Q. And no response to an SPM was required. So he or she 19 wouldn't know about this fix? 20 A. That's right. This wasn't raised by an SPM; therefore, 21 there was no need to call an SPM about this. 22 Q. So the "changes" in, in inverted commas, the SPMs' data 23 would be known without the SPM realising that there was 24 a discrepancy or the cause of the discrepancy or the 25 correction of the discrepancy? 95 1 A. That's right. They would have been blissfully ignorant 2 of this in an appropriate way. 3 Q. Yes. And then lastly observations on the defect root 4 cause, you say that general user defect cause was 5 correct? 6 A. Yes. 7 MR BEER: Sir, that might be an appropriate moment to take 8 the break before we move to example 2 of 7 because that 9 will take a little while. 10 SIR WYN WILLIAMS: Yes, of course. 11 MR BEER: So might we say 1.55? 12 SIR WYN WILLIAMS: Yes, fine. 13 MR BEER: Thank you very much, sir. 14 (12.56 pm) 15 (Luncheon Adjournment) 16 (1.55 pm) 17 MR BEER: Good afternoon, sir. Can you see and hear us? 18 SIR WYN WILLIAMS: Yes, I can. 19 MR BEER: Thank you very much. 20 Mr Cipione, can we look please at page 140 of your 21 report, EXPG0000001, 140. We're dealing here with the 22 second of the seven exemplar PinICLs. Can we do as 23 before, please; you talk us through. 24 A. Yes. As before I'm going to read and then explain. 25 Q. Translate, yes. 96 1 A. The first line: 2 "Incorrect CA value. Live trial the CA sub file 3 for org units 12609 ..." 4 Which is a particular "FAD", which I understand to 5 mean is a branch number identifier. 6 "... CA week 21 contains an entry for line 2050 7 with a value of £17,181.05. However, TIP has calculated 8 from the transactions it has received that the value of 9 the line should be £17,642.31. There is leaves 10 a difference of £461.26." 11 So there's a problem. All right, so it moved on: 12 "Barbara, I have just spoken to John Pope 13 (Requirements). This is classified under Acceptance 14 Incident 376. Would you please raise the level of AI 15 incident. Would John Simpkins please take a look, then 16 send to EPOSS." 17 Q. Just: "Would you please raise the level to an A/AI." 18 A. Yes, "to an A." 19 Q. Do you understand that to be the categorisation of A, B 20 and C? 21 A. That is what I understand, where Category A would be the 22 highest priority. 23 Q. Yes. 24 A. Right so John Simpkins responds: 25 "I have checked the agent boxes at Wigan for any 97 1 T_HV_ALL event for this office between 12 August and 2 18 August and did not find any." 3 Then Jim Anscomb says: 4 "There is a null value transaction Mode on 5 [a particular number] for a cash credit of £143.22 6 though this is now not a problem for the harvester. No 7 delays shown in the APR db." 8 I don't understand everything that's in there 9 except for what I've honed in on, this null transaction 10 mode, which sounds to me as though there is a problem in 11 the reference data for this transaction. 12 Q. Thank you. 13 A. All right. So the next line: 14 "The erroneous message was [a particular number] 15 not [the number that the person before was 16 referencing] -- in case anyone else is relying on this 17 [information]. We released a fix for this on 20 August 18 into WP5406 [which I assume means work package, which is 19 like a quick software release] which went to OTT and is 20 due to be released in Tivoli package 21 EPOSS_COUNTER_CORE~..." 22 This is very technical and I can't explain 23 everything, but it sounds like they are trying to 24 release a software fix, or they are releasing a software 25 fix, but it indicates it hasn't made it to live, yet 98 1 which means I don't think it's in production, meaning 2 it's not actually being used right now. 3 "The problem message is unfortunately an Existing 4 Reversal message. So the harvester's automatic 5 assignment to Serve Customer is likely to provide 6 problems, someone will need to amend this. Routing to 7 EDSC for them to solve the procedural problems and check 8 when the Tivoli package is due for release." 9 So okay, the problem message is an existing 10 reversal message. So it sounds like the regular process 11 at the harvester would just be a step along the way of 12 getting the information from the counter, all the way up 13 through the data centres, then eventually on to 14 Post Office. It sounds like the existing path -- it 15 sounds like they think it's going to be a problem; you 16 are not going to be able to do the regular thing. 17 Then we have Jim Anscomb saying the total 18 discrepancy is 461.22 -- 19 Q. 26? 20 A. 26, thank you. Indicates 143.22 has been accounted for 21 above. Then it looks as though this next line is 22 indicating that it's a high priority call. It's just 23 indicating that, and with an encouragement to progress 24 on this rapidly. 25 Then John Pope says: 99 1 "Just a thought, but the sign reversal mentioned 2 above (serve customer sent to TIP instead of Existing 3 Reversal) may explain 2 x 143.22 [because that's] 4 286.44. Can anybody help with 174.82?" 5 So up to this point I think they've got some 6 ideas. They're trying to figure out what the problem 7 is. I don't think we've identified exactly what problem 8 is as of yet, just in case anyone thinks that they have. 9 Q. Yes. 10 A. All right. So the next line Steve Warwick: 11 "It may be of interest that the value of the 12 discrepancy between the TIP and Pathway figures appears 13 to correspond to 2 x £230.63. During the balancing of 14 stock unit AA on [18 August] a stock adjustment was made 15 to reduce the value of Cheques by this amount, with 16 a corresponding increase in Cash. These two stock 17 adjustment records were later individually reversed, 18 generating a further four transactions for £230.63, 19 three against cash and one against cheques. Therefore, 20 in total four cash transactions (two positive, two 21 negative) and two cheque transactions (one positive and 22 one negative) were written. 23 "Given that there have previously been issues with 24 TIP's rejection of 'Existing Reversal' transactions 25 where the reversal settlement contained no 100 1 cross-reference details, is it possible that this has 2 caused the reconciliation failure? According to the 3 message store data, the Cash Account for CAP 21 reported 4 total receipts equal total payments, indicating that the 5 message store data is complete and accurate." 6 Okay, so here there's some more speculation going 7 on as well as an indication that in CAP 21 everything 8 appears to be fine or, sorry, in CAP 21 the total 9 receipts and total payments are equal. So Steve Warwick 10 comes back again the next day: 11 "From further information received from TIP, the 12 sequence of events seems to have been as follows ..." 13 And I'm not going to read that time and date: 14 "A stock adjustments was carried out to reduce 15 value of cheques by £230.63. This wrote two 16 transactions -- one to reduce the value of cheques, one 17 to increase the value of cash by the same amount, both 18 transactions carried mode 'SAN'." 19 Then: 20 "... a reversal of the cash settlement transaction 21 to the cheque adjustment took place resulting in two 22 transactions being written against cash, one to reduce 23 the value of cash, one to increase the value of cash to 24 settle the reversal. Both transactions carried the mode 25 'ER'." 101 1 And then: 2 "... a reversal of the cheque adjustment 3 transaction was carried out, generating two 4 transactions -- one to increase the value of cheques and 5 one to reduce the value of cash by the same amount." 6 So he's telling us what happened from his 7 investigation: 8 "These transactions are recorded in the message 9 store with the correct signs ..." 10 Because that's probably where he's getting this 11 information." 12 "From the information supplied by TIP, it seems as 13 though they have received/treated the 14 transactions ... (a reversal of a previous reduction in 15 the value of cheques) as though it was a reduction in 16 value rather than an increase in value, thereby 17 calculating a discrepancy of twice that amount." 18 So he's still speculating on this last sentence: 19 "Either the sign on the transaction value sent to 20 TIP was incorrect, or TIP have misinterpreted the data 21 sent." 22 So what this is telling me is they've identified 23 where the difference is now. So that's good. So they 24 found at least what the symptom is, and now they are 25 speculating on what happened in between to make those 102 1 differences. 2 The next line: 3 "Looking at the TIP file there were two reversals 4 for £230.63 in quick succession. The first is 5 translated for TIP as balancing plus and minus entries. 6 The second, however, is translated into two positive 7 entries, which would account for the error. See extract 8 of TIP file and message store attached." 9 So what TIP received is not what TIP was intending 10 to receive. So the next line: 11 "Changes to be made to clsEPOSS and clsTransaction 12 in the EPOSSCore. Fix applied ..." 13 So what they are saying here is they are changing 14 code. To me this sounds like they are changing code 15 modules. 16 Q. Sorry, they are changing code modules? 17 A. They are changing code, in particular, in the modules. 18 I think that those are module names or the class names. 19 I don't know what the code looks like, so I'm 20 speculating. 21 "You should get in the attribute grammar for 22 a cash settlement for an ER transaction the additional 23 data of ..." 24 And it's talking about the cross-reference mode. 25 "The harvesters need this." 103 1 So this indicates to me that there was probably 2 something wrong in the attribute grammar and, if you 3 recall from our first session, that's the format that 4 the reference data -- 5 Q. It's written in? 6 A. Yes. Okay, so the next line: 7 "Testing of this should include transacting in 8 each mode." 9 So, if you recall, they will have been talking 10 about two modes. One is ER mode and one is SAN mode. 11 So that's really more for them, just on the technical 12 level, but those are the two modes that they are talking 13 about. 14 "Testing should include transacting in each mode 15 and the message should be as they were. Then performing 16 a reversal of each mode an checking that the new 17 attribute grammar exists in the cash settlements of the 18 reversals." 19 So they are just saying, when you test this, make 20 sure you are using both modes and make sure you are 21 doing a reversal, which is trying to recreate the set of 22 circumstances that are associated with this error. 23 The next line: 24 "Link tested okay on CSR dev counter. 25 Performed" -- 104 1 Q. Just stopping there, do I understand it correctly that 2 the problem that had been identified so far is with 3 EPOSS? 4 A. That is the way I'm reading it, yes. 5 Q. Thank you. Sorry. 6 A. Okay. 7 "Link test OK on CSR dev counter. Performed 8 a transaction followed by existing reversal for each of 9 the following modes: 10 "Serve customer, Rems (all modes) [They list out 11 the modes]. 12 "On each existing reversal the message store was 13 checked for the new attribute grammar. 14 "CrossReference.OMode -- followed by the 15 corresponding mode of the reversal." 16 So the next line: 17 "WP_5766 has been applied to live. Routing call 18 back to call logger for closure." 19 So they think they've fixed it, they think they've 20 identified the problem, made the changes and applied the 21 fix. Then it says: 22 "We have seen that when a call is the subject of 23 an Acceptance Incident (as this call is) then there is 24 no point in us ringing up the originator to ask for 25 closure. They always say that such calls are the 105 1 subject of regular discussions between John Pope ... and 2 Martin Box ... Eventually somebody at TIP rings us with 3 a list of calls which can be closed. Accordingly 4 I shall send this call to our holding stack to await 5 such closure." 6 Then it looks like it's closed. So my inference 7 from this is, since this is a high-priority issue for 8 both the Post Office and Pathway, there's already 9 a mechanism going on behind the scenes to validate 10 whether this can be closed or not. 11 Q. So just looking at the chronology then, this was opened 12 on 20 August, and then closed on 30 December 1999? 13 A. Yes. I will say that -- yes, I think it looks like they 14 started waiting for -- so it was officially closed on 15 30 December but I believe they think it's been 16 resolved -- 17 Q. They being at the end of October? 18 A. Yes, exactly so just to be clear. 19 Q. Then your observations. 20 A. Okay. "So was the immediate issue fixed? Yes." They 21 fixed the issue and they received the proper closure 22 instruction. 23 "Was the defect or root cause identified?" Yes, 24 they identified the root cause. It was an issue being 25 the signage and the grammar attributes sending the 106 1 information to TIP. 2 "Was the defect root cause correctly recorded in 3 the PinICL or PEAK?" It was recorded as code, 4 development code, which we saw that they changed. So 5 yes. 6 "Is there evidence that this defect root cause was 7 addressed?" So I'll just read what I said: 8 "The text indicates that a software update was 9 made and tested, albeit the text only indicates that the 10 testing occurred, not the results of this. However, 11 I note that the text does not indicate that the test 12 failed, which I would expect it to have said if that was 13 the case." 14 So I guess what I'm saying is: I saw the testing 15 documentation here. It said it was ready to go live and 16 they had a package. So I'm going to say: yes, it was 17 addressed. 18 "Observations on the management and closure. The 19 closure on the PinICL once the fix had been implemented 20 and the original raiser's approval was obtained seems 21 appropriate to me." 22 Then: 23 "Observations on the root cause. There is 24 evidence in the ticket that the fix was implemented in 25 LHITS to remediate ..." 107 1 So I think that this was addressed. It seemed to 2 be addressed appropriately. So that's my observation. 3 Q. Thank you very much. Can we turn to the third example 4 on page 144, "Reference data delivery issue". 5 A. Sure. 6 Q. Again, if you can walk us through the text this one is 7 much shorter. 8 A. "Office 91008 reports a difference in its receipt and 9 payments totals for CAP 18. Please send this call to 10 John Moran." 11 So John Moran gets the call, and he indicates: 12 "I know what caused this problem. It was because 13 reference data was not sent to the outlets concerning 14 P&A products -- the cash settlement was mapped to the 15 cash account, but the corresponding transaction was not. 16 If these transactions were recorded by in the Counter 17 Transaction Exceptions report I could supply POCL TP 18 with this information myself, but they have not been 19 recorded. 20 "Can you supply the offending non-mapped 21 transactions to this PinICL in message store extract so 22 I can reconcile with Chesterfield." 23 And then: 24 "The difference in the receipt and payments totals 25 was caused by the fact that non-core reference data was 108 1 not delivered to this office in time. Reference data 2 was for OBCS products 177 to 185, and this reference 3 data included primary mappings for these products. 4 These products cannot be mapped to the cash account at 5 stock unit rollover. That is what caused the difference 6 in the receipts and payments total. 7 "This incident is related to 9 others all caused 8 by this same problem. All the offices affected were 9 migrated to Horizon on 20 July. All the offending 10 transactions took place on 21 July when there was not 11 reference data at the outlets. The correct reference 12 data was delivered for business on 22 July. 13 "I have provided with the final BIM report an 14 Excel spreadsheet (with the same file name as the BIM 15 report) listing the offending transactions which were 16 not mapped to the cash account." 17 Then it goes on to say: 18 "Caller has raised BIM BASED ON THE EVIDENCE 19 EXTRACTED AND so call has been closed." 20 So this is the reference data for -- non-core 21 reference data, which my understanding is there were 22 certain products that Post Office sold at every branch, 23 and there were certain products that were sold only at 24 some branches. Those products level into the non-core 25 reference data. 109 1 Q. Over the page, please. 2 A. All right. 3 "Was the immediate issue fixed? Whilst the text 4 states that the BIM was raised and the impacted 5 transactions were identified and included with the BIM, 6 it is not possible to determine from the PP [from the 7 PinICL or PEAK] whether the transactions were updated to 8 map them correctly. If I assume that the BIM process 9 would rectify this issue, then it appears that the 10 appropriate steps were taken ..." 11 So I'm going to correct myself here on rereading 12 this. I believe that the BIM process was more of 13 a process to actually give correct transactions, not the 14 process to update the reference data. 15 Q. I understand. 16 A. All right. 17 "Was the defect/root cause identified? The text 18 indicates that this was a known issue and a clear root 19 cause is provided (i.e., the product reference data 20 required to correctly map the transactions to the cash 21 account had not been delivered to ... so the 22 transactions were not mapped)." 23 "Was the defect root cause correctly recorded?" 24 It was recorded. 25 "The root cause recorded in this PEAK is 110 1 "General-unknown". The root cause is clearly defined in 2 the PEAK and, therefore, the root cause is more akin to 3 'Product reference data not delivered in time'. So that 4 would seem like [a better description]. 5 "Is there evidence that this defect/root cause was 6 addressed? The text references the fact that the 7 branches had now received the required reference data." 8 So yes. 9 "Observations on the management and closure of the 10 issue. This specific issue was fixed in that the 11 reference data was delivered to the branch. 12 "Other observations. There's evidence in the 13 ticket that a fix was implemented in LHITS to remediate 14 the identified issue. It's unclear to me why the 15 reference data was not timely delivered to the branch 16 and nine others." 17 So that would be kind of the next step up of, you 18 know, what the root cause was. I mean, we know that the 19 root cause of the specific transaction issue is because 20 it wasn't reference data there. We know that to fix 21 that in the future the reference data should be there. 22 So that's one level of fixing, you know, of addressing 23 the root cause. 24 But then the next level up of addressing the root 25 cause is: why was that reference data late arriving? So 111 1 they fixed it. I don't know that they -- I don't know 2 what was happening to fix the timeliness of all 3 deliveries of reference data. 4 Q. Thank you. Can we turn to the fourth of the seven 5 examples? 6 A. Certainly. This one is titled "Stock unit deletion". 7 Q. On this one the text is relatively long. 8 A. Yes. 9 Q. Almost three pages. So could you pick out from the 10 text -- I'm not sure whether that's possible, 11 Mr Cipione, or not, the key entries? 12 A. Yes. Let me rest my voice while I look for the proper 13 first citation. So the first entry just describing 14 discrepancies. 15 Q. Just describe for us what the issue was. 16 A. So if we go, I think the meat of it is on page 147, the 17 first entry for Steve Warwick. It's the bottom of the 18 middle of the page: 19 "The £155 error reported by TIP at FAD is almost 20 certainly related to the MVL transaction. A number of 21 these transactions took place in the week, and there was 22 also a loss declared the previous week for this value 23 against cash. The value of £155 was also transferred 24 between two stock units during the week and a gain of 25 £155 was recorded when balancing the end of CAP 18 112 1 (offsetting the loss of £155 declared at the end of CAP 2 17). 3 "Since there was no failure of the office to 4 balance its cash account, it would seem that either one 5 of these transactions had not been sent to TIP or TIP 6 having miscalculated the value of the transaction." 7 So that's the beginning of the diagnosis. 8 Then at the bottom of that same page it says: 9 "The cause of this mis-balance ... was the 10 deletion of Stock Unit ZZ on 29 July before the [end of 11 day] marker for the outlet had been written. This meant 12 that the transaction carried out on the stock unit 13 totalling £155 in CAP 19 was not reported to TIP." 14 Q. Then maybe the Steve Warwick entry -- 15 A. Four up from the bottom? 16 Q. Yes. 17 A. "The original response given to TIP was based on the 18 fact that the symptom of the call appeared to be similar 19 to other calls which had been identified as being 20 a signing problem. This initial view was provided along 21 with the statement that the incident was still under 22 investigation and that once the evidence had been 23 examined, the root cause would be determined. The root 24 cause has now been determined and John Pope updated the 25 spreadsheet shared with POCL regarding AI376. Closure 113 1 will be agreed ..." 2 And the root cause to TIP was the development 3 code. I'm just referring back to the header of this 4 particular sheet, so ... 5 Q. Okay. Then your observations. You split these 6 according to FADs. The FAD code is, I think, a numeric 7 code which uniquely identifies a Post Office; is that 8 right? 9 A. That is correct. That is my understanding anyway. 10 Q. So you collect together the first three -- 11 A. Right. So this is a bit unusual, because it seems that 12 there's -- I think this is one where there seems to be 13 multiple -- there seem to be multiple discrepancies 14 included on one particular log file, which in theory -- 15 if they were all the same type of error, they could all 16 be together. But, if there are different errors, they 17 should have different PinICLs. But I think in this case 18 there are two different issues going on here which we 19 lost because I didn't read every particular line. But 20 just keep that in mind. 21 So for three of the FADs there was an RED notice 22 to resolve the issue. So we think that that was fixed. 23 For the FAD that we really read the meat about, for what 24 we read, the reason we think it was fixed is because we 25 know that David Salt, which was one of the items we 114 1 didn't read -- someone wrote back and said that it was 2 able to be closed. They agreed to closing the call. 3 That would have been the third line up from the bottom. 4 Q. Yes. 5 A. "Root cause identified". Okay. So -- 6 Q. Over the page, please. Yes, the next page. 7 A. So for the first three there was an issue with mode 8 attributes being null. 9 Q. What are mode attributes? 10 A. I believe that's part of the information in the 11 reference -- information they are sending back. There 12 was a piece of data missing when they were trying to 13 send the information up, so it couldn't be processed. 14 On the second set, the root cause is because the 15 stock unit was deleted before the end-of-the-day marker 16 had been written. So it was an ordering of operations 17 issue, where they deleted a stock unit internally. This 18 stock unit probably should have been deleted after the 19 end of the day, or recognised -- there is an order of 20 operation issue that created the problem. 21 So was the defect/cause recorded correctly? The 22 defect cause was recorded as development code, and 23 I assume this applied really to the second item. 24 Q. So 523? 25 A. Yes. If this assumption isn't correct, this seems like 115 1 an appropriate defect code, but it's unclear if this 2 also applied to the other three which were a different 3 issue. I'm not sure, besides the fact that they 4 actually sent the RED in, which my understanding is 5 that's helping to correct the transactions. I'm not 6 sure if they addressed what the actual problem was at 7 least in this particular PEAK. 8 Observations on closure, here I'm noting that 9 we're talking about two different problems. So it's 10 a little difficult to follow, but it does appear as 11 though they have addressed both problems, one with the 12 RED and one with the development code. 13 Then ultimately on the root causes or, sorry, 14 observation on defect root cause, for the first three 15 I don't have any information on what went on there. So 16 I can't comment on that other than I don't have any 17 information. I know that they are aware of it, and they 18 sent corrections for the symptom, but I'm not sure what 19 they did on the null mode. 20 The last one it's identified -- it seems 21 appropriate to me based off of the reading of the text. 22 Q. Thank you. Can we turn to number 5, please, over the 23 page, which concerns a brought forward balance 24 multiplied by causing payments and receipt imbalance. 25 This one is relatively long. I appreciate that asking 116 1 you to summarise from your perspective is not the most 2 helpful thing to do, because it means that you skip over 3 text that's important to explain or qualify your 4 observations. But do take your time, if you just need 5 to look or remind yourself of the problem or the report 6 of the problem from the customer call. (Pause) 7 A. I think the first item that we should be looking at is 8 at the top of page 151. 9 Q. Yes, "Initial investigations"? 10 A. "Initial investigations have shown that the problem 11 arose at the time that the Office Trial Balance report 12 was produced. On the Office Trial Balance report the 13 brought forward value was £71,000 instead of £14,000. 14 This appears to have been caused by the creation of 15 a correctional stock unit which was additional to the 16 normal stock unit. Due to an error in the code, when 17 the stock unit balance records are read, the first stock 18 unit (22 -- first in alphabetical sequence) is correctly 19 identified as having no 'Brought Forward' value from the 20 previous week, the system then incorrectly assumes that 21 this must be a migration week and generates a brought 22 forward value for the stock unit which is incorrect. 23 The error is being investigated for urgent correction." 24 So this is a code error. 25 Q. I think you can probably see the response to it over the 117 1 page on page 152, second entry from Steve Warwick. 2 A. "The doubling (or multiplying) of the brought roaring 3 value on the cash account has been traced to an error in 4 the changes which were delivered to correct the previous 5 problem related to the previewing of the 'Final' cash 6 account." 7 So what I'm reading from this is there was 8 a different error. A code correction was made for the 9 error, but now it's creating this particular error. So 10 they have identified this error as being related to 11 a previous code correction. So that's effectively what 12 the cause of this particular error is. 13 I'm happy to read the rest of it, but I understand 14 you trying to be efficient with time. So I could skip 15 over the rest of that but that's the cause. 16 Q. Then go to page 153 then, your observation. 17 A. "Was the immediate issue fixed? I don't see evidence in 18 the PinICL or PEAK that suggests that the issue at the 19 FAD was fixed although the text does not that the FAD 20 completed their roll-over, presumably with the imbalance 21 still in place. Perhaps the alert to the Horizon 22 Helpdesk indicates that the corrective procedures were 23 communicated ..." 24 So what I'm saying here is I'm just not sure if it 25 was actually fixed. It wasn't clearly indicated in 118 1 here, but it certainly doesn't mean that it wasn't 2 fixed. 3 All right. 4 "Was the defect root cause identified? Yes, the 5 root cause was an unforeseen consequence of the current 6 LT1 fix", which is what I just described. 7 "Was the defect root cause correctly recorded? 8 [It] was recorded as development code. This seems 9 appropriate since a software fix was required to [fix 10 the problem]." 11 "Is there evidence that this defect/root cause was 12 addressed? Yes ... 13 "Observations on the closure. The root cause was 14 identified, addressed in an upcoming LT2 release, and 15 a workaround was communicated to the Helpdesk." 16 Then that's pretty much the same observation I had 17 for the overall observation on the root cause and 18 defect. 19 Q. Thank you. Penultimately example 6 on page 154, please, 20 which is "Navigating to a different mode whilst 21 transactions are on the stack." Can you just explain 22 what that means. 23 A. So what's happening here is, if you remember from my 24 last testimony, there's the ability for the counter user 25 to start adding things to the stack, which is basically 119 1 the customer list. I have got a whole bunch of things 2 that I want to perform for a customer. So what it looks 3 like here is that they're in the Serve Customer mode, 4 they have started adding things to the stack, and then 5 for some reason they're wanting to switch a mode. I 6 believe that that's what is creating the issue; it's 7 changing from a Serve Customer mode to some other type 8 of mode while there are items on the stack. 9 Q. You can see that this afflicted 28 offices from the 10 first customer call on 5 November '99. 11 A. Right. It would appear that way from the beginning part 12 of the text of this PinICL. But they have identified 13 that they know that 24 of them were already -- that 14 these differences were already accepted at migration. 15 So they were -- 24 of them are known migration issues 16 and they have already actually addressed that. So we're 17 left with four FADs that we need to deal with. 18 I will preview that I know that one of these FADs 19 also is a migration issue. So it's really three FADs 20 that have this actual mode-change issue related to their 21 problems. 22 Q. Then, if you go over the page, I think you can see how, 23 by reference to the FAD code there, what the response 24 was in relation to each of those three. In fact, 25 there's a fourth as well. 120 1 A. Yes. So the first one it says it's a housekeeping 2 transaction. It was carried out for a value and it was 3 not settled in the user -- so the first FAD, the third 4 FAD and the fourth FAD in this list, they are all 5 indicating that they had transactions on the stack, they 6 weren't settled, and the user at the branch navigated to 7 a Revaluation Up menu, and that's what caused the 8 problem. For the second one it was just a discrepancy 9 at migration. That's the one I was indicating. 10 So those are the three that really had the issue. 11 Q. Can we go over the page then to your obligations. 12 A. Sure. 13 "Was the immediate issue fixed? For 25 of the 28 14 FADs the difference had been handled at the time of 15 migration. For the remaining three, I don't see 16 explicit evidence in the PinICL that suggests that the 17 issue with the FADs was fixed, but ..." 18 And there was more text in here about POCL's 19 response or non-response. 20 "... but POCL's exclusion of these FADs from their 21 'list of call to remain open' seems to indicate that the 22 issue was resolved to their, [meaning POCL's], 23 satisfaction." 24 So there must have been a mechanism between ICL 25 Pathway and POCL for indicating: we know there's an 121 1 issue and then having an open-issues chart at the end, 2 and these weren't on it. 3 "Was the defect root cause identified? The root 4 cause for 25 of the 28 related to initial migration. 5 For the remainder, the root cause was identified, namely 6 that Horizon allowed a user to navigate to a different 7 mode while transactions were on the stack, which would 8 then be subsequently settled incorrectly. 9 "The defect ... was recorded as reference data. 10 I have here "Without a comprehensive understanding 11 of which component of the system contained the error, 12 it's not possible for me to say whether this is an 13 appropriate code". I would say it seems like it could 14 be an appropriate code, especially since I know or since 15 I believe that the modes are governed by the reference 16 data. So they put the reference data there. 17 "The PinICL states that the error was corrected in 18 live software." That's for "Was it addressed?" 19 "This one was closed without positive confirmation 20 that the imbalances were addressed", but "the root cause 21 appears to have been identified and remedied." 22 Q. Lastly number, 7 please, page 157, PinICL concerned with 23 a data tree build failure. I think you can see Customer 24 Call, first entry, what the issue is. 25 A. Yes. So the cash account line comparisons -- so let's 122 1 go: 2 "The office had big problems with its receipts and 3 payments. CAP 5, 6 and 7 did not match." 4 Then it lists the differences. And so we get down 5 to the first substantial Steve Warwick entry: 6 "The cause of the problem in all 3 CAPs at this 7 outlet was the fact that Stock Unit DD's rollover 8 records from CAP 5 and 6 represented a 'nil balance' 9 (the total stock holding was nil, no receipts or 10 payments transactions were recorded) despite the fact 11 that that stock unit had been trading normally during 12 the period. [Said] This issue was raised [earlier] and 13 is still under investigation ..." 14 Then it says: 15 "The fact that the Stock Unit DD's transactions 16 and stock holdings were omitted from CAP 5 Cash Account 17 meant that the Brought Forward value for the office in 18 CAP 6 was incorrect. This caused CAP 6 Cash Account to 19 mis-balance. I am still investigating by CAP 7 cash 20 account mis-balanced, but I note that the office 21 returned to a balanced position in CAP 8." 22 Q. Then I think if you go over the page. 23 A. Yes. 24 Q. And scroll down to John Moran's last entry -- 25 A. Which one would you like me to start at? 123 1 Q. I mean, this is more for you than me, but the 2 12 July 2000 entry. 3 A. Okay. So John Moran is asking: 4 "I need to know what correct Cash Account figures 5 should have been were it not for the Dataserver 6 failure." 7 So in between our things they've identified a data 8 server failure. 9 Q. Yes. 10 A. "Can these be derived from the transactions in the 11 message store ..." 12 So John is asking questions in order to help 13 remedy this situation. 14 "2. The diagnostic code which was delivered 15 before this incident happened was promised to aid in 16 investigating the cause of the problem. Has this code 17 helped? How?" 18 "At some point a work package was delivered would 19 alert the user that there was a problem with the [stock 20 unit] rollover and the user would be prompted with 21 a message to redo the rollover. Has this been 22 delivered? If so, why did the rollover process not 23 cease and prompt the user to try again?" 24 So just a little colour. This indicates to me 25 that they have seen this type of issue before, and 124 1 they've tried to put in some safety procedures at the 2 actual Horizon level to help the user make sure that 3 they can't generate this problem. 4 So I'm going to go down to the first entry on 5 page 159. 6 Q. Thank you. 7 A. Just at the end, the final paragraph, not the three 8 numbered: 9 "I don't think I'm being premature in revealing 10 that we think we know why these failures with the 11 Dataserver or occurring. Steve Warwick experienced such 12 a failure on a rig he was testing against and found the 13 root cause was that the Archiving was active during 14 a riposte query; this only occurs 'out-of-hours' at end 15 of each working day. Archiving will occur 'in-hours' 16 should the counter have been switched off overnight for 17 seven consecutive days and hence the sporadic nature of 18 these incidents." 19 So what this is saying is there's an unexpected 20 issue when archiving is occurring, and there's Riposte 21 activity, and Riposte is the part of the system that's 22 helping manage the message store, which is where they 23 are keeping all the transactions. It's not -- it seems 24 to be that there's some sort of conflict going on when 25 archiving starts up, and they are trying to do this. So 125 1 that's what's suspected to have actually caused this 2 problem. 3 If we go to the middle of the page, they are 4 describing here essentially that they are having 5 problems trying to actually tease out the actual details 6 to walk forward and everything. I don't think we need 7 to read it all, but they are saying that they are having 8 problems looking at the actual detail to resurrect -- 9 because remember this a CAP 5, 6, 7 issue. They are 10 having trouble figuring out how they could actually see 11 everything that was going on in each one of those CAPs. 12 Then at the bottom it says: 13 "I'm not sure it's worth spending time trying to 14 resurrect the other CAPs. The method I have derived 15 assumes that the CashAccLines for the previous CAP" -- 16 The cash lines, cash access lines. Cash account lines 17 probably. 18 "... for the previous CAP. I see from Steve 19 Warwick's analysis that CAP 6 was not correct as well. 20 Now, if I rerun the tool I have developed on CAP 6 it 21 will use the base line in the Cash Account Line figures 22 from CAP 5 which we know are wrong and I have just 23 recalculated. I think therefore enough time has been 24 spent on this problem and it's not cost effective to 25 proceed further. However, in future where the problem 126 1 is just one CAP, we should be able to resurrect the 2 figures more easily." 3 So what they are saying is, since this was 4 a multi-CAP problem, there's no way for them to go back 5 and resurrect information from CAPs that are more 6 than -- that are further than the most recent CAP. 7 Q. Your observations, please. 8 A. "Was the immediate issue fixed? It appears that CAP 5 9 account was reconstructed, but that CAP 6 and CAP 7 10 accounts weren't." 11 I don't know if that's really a problem or not, 12 but it's something that they acknowledged. 13 "Was the defect root cause identified? A root 14 cause was identified, namely that the archiving process 15 was active during the Riposte query. 16 "Was the defect cause correctly recorded?" 17 It said the development code was recorded, which 18 I think is appropriate. 19 "Is there evidence that this defect root cause was 20 addressed? The reference" -- there was a release 21 a reference to a release that suggested that it would 22 address this particular issue, and something that we 23 didn't read, and then it appears that the investigation 24 was carried out with -- there was a lot of discussion. 25 So this would have been an example of: look, they 127 1 are working towards finding at least an answer for this 2 particular situation, and they think that this next 3 release should take care of it. So that's my 4 observations. 5 Q. Thank you. That can come down from the screen now. So 6 stepping back, having looked at those seven examples 7 are, they evidence that payment and receipt imbalances 8 were commonly reported? 9 A. Yes. 10 Q. Taken together with the wider dataset, the non-examples, 11 things that we've not looked at, at examples, but which 12 you found through your search methodology? 13 A. Yes. 14 Q. Are they evidence that the payment and receipt 15 imbalances that were commonly reported had a variety of 16 causes? 17 A. Oh, yes. 18 Q. Are they evidence that there was an earnest attempt by 19 the SSC, in particular, to seek to resolve the issues 20 raised? 21 A. Yes. 22 Q. But do they also evidence that the majority of PinICLs, 23 six out of seven on the examples, all had as a root 24 cause a problem with Horizon? 25 A. Some were within Horizon, yes. 128 1 Q. Have you got any overall observations to make in 2 relation to this section of your report? What does your 3 work on this tell you, if anything? 4 A. When I see evidence of issues where there seems to be 5 a lot of different reasons for causing the same problem, 6 number 1, that that obviously makes it harder for the 7 people that are trying to track down to fix it because 8 all they are working off of is: Here's the symptom, you 9 know. "Doctor, I'm tired. What's wrong with me?" You 10 know, that type of thing. Well, it could be a lot of 11 things wrong with you. Let's investigate a little bit 12 more. But it takes that investigation process. So 13 that's one issue. 14 Second is, there seems to be a lot of different -- 15 at least at some level of separation, a lot of potential 16 different causes for these problems, which is troubling, 17 because you wouldn't -- especially for this type of 18 issue, it just looks like the system has a lot of 19 instability in it, and it's not apparent to me looking 20 at these PinICLs that it's isolated in one place. 21 Therefore it would probably be difficult to just focus 22 on one little place to fix those instability issues. 23 Q. Thank you. Can we go back to sections 11 and 12 of your 24 report which I skipped over. 11 is on page 89. Thank 25 you. This section of your report is given over to the 129 1 observation by you that financial concerns were 2 considered by ICL Pathway throughout the time period 3 reviewed. You observed that ICL Pathway is 4 a profit-seeking entity and you therefore assume that it 5 was motivated to deliver the system and to make 6 a profit. 7 In the light of that observation, which is, if I 8 may say so, a statement of the obvious, why have you 9 included this in your report? What stood out for you, 10 if anything? 11 A. So the materials I relied on for this analysis included 12 the monthly reports, and there seemed to be a lot of 13 discussion about financial issues, you know, costs, 14 timelines, and how it might affect, or acceptance and 15 how might affect payments. It seemed to be prevalent in 16 the monthly reports, and I agree, it's obvious. But 17 those were the documents I was given, and I felt as 18 though, if I didn't say something to that effect, it 19 might look like I'm not discharging my responsibility 20 correctly. 21 Q. You say at 11.1.3 that: 22 "The financial success of Horizon relied on 23 Pathway orchestrating many different constituencies: 24 Sponsors, suppliers and their many internal groups. 25 Benchmarks relating to acceptance, the timing of 130 1 roll-out coverage, adherence to service level agreements 2 requirements, were topics of discussion as they were 3 directly connected to revenue, cost, and profit 4 realisation." 5 Is that always the case in an IT project? 6 A. Yes. 7 Q. "The balancing act between operational and technical 8 activities and their financial ramifications was 9 highlighted in the April 98 report: 'Should the 10 pressures mount, the temptation to hold to NR2 dates at 11 all costs is immense. If we were to (purely 12 theoretically) compromise NR2 quality in order to hold 13 timescales, we would almost certainly be worse off in 14 the long run.'" 15 "Acceptance was received achieve on 24 September 16 '99, triggering the first invoice ... roll-out coverage 17 incentive for 1,800 outlets was achieved on 18 5 November 99. The first payment (£105 million) was 19 received in early December '99." 20 You say in 11.1.5: 21 "The items [you then include in the following 22 table] illustrate some financial wins and some financial 23 losses from ICL Pathway's perspective. Financial 24 concerns were weighed against the resource allocations 25 to deliver the Horizon IT system. It's my opinion that 131 1 the financial aspects of delivering the Horizon IT 2 System affected the decision-making process." 3 Can you explain or expand upon that last sentence, 4 please. 5 A. Yes. Since ICL is a profit-seeking entity, obviously 6 they are wanting to make sure that this is a profitable 7 endeavour. When they are discussing concepts such as 8 making sure that they get acceptance, and how that 9 relates to timelines, and then how time slippages are 10 creating concerns with their particular partners, their 11 vendors, all of that together just tells me that that's 12 obviously on their mind, which it should be of course 13 and, like I said, this is stating the obvious, but 14 I think I would be remiss not to comment on it, since 15 there was quite bit of text in the monthly reports 16 concerning it. 17 Q. To what extent, if any, did you ascribe Fujitsu's 18 financial concerns to the fact that, initially at least, 19 the programme was being run and financed under a private 20 finance initiative, and then one of the significant 21 partners from which much revenue would be obtained 22 withdrew? 23 A. Could you repeat that question, please. 24 Q. To what extent did you ascribe Fujitsu's financial 25 concerns to the fact that, at least initially, the 132 1 programme was being run and financed under a private 2 finance initiative from which one of the major partners 3 withdrew? 4 A. I will say that, you know, as far as the private finance 5 initiative is concerned, this is my first exposure to 6 what even a private finance initiative is, and it 7 probably didn't really resonate with me that much, 8 because I'm not sure that I actually really took that 9 into account. 10 I think, now that I maybe understand that a little 11 bit better, I think that, you know, that would explain 12 why they didn't get paid up until a particular 13 acceptance point, just from the witness statements that 14 I read after I submitted the paper. So that has helped 15 contextualise the acceptance period. 16 I think I was more focusing on their not getting 17 paid until they get acceptance, without really 18 understanding the linkage between that particular event 19 and the fact that it was originally a private finance 20 initiative that they went into with another partner. 21 I also don't know that I had much understanding of 22 the impact of the BA's withdrawal either. It wasn't 23 really on my radar. I didn't understand the 24 implications of it as much as I do now. So I would say 25 it didn't really impact it a lot. 133 1 Q. Thank you. That's very fair. In which case I won't ask 2 you any further questions about that. 3 Some witnesses to the Inquiry have suggested in 4 the course of their evidence that a consequences of this 5 being initially a private finance initiative 6 run-and-financed programme, and then it being a legacy 7 of a private finance initiative was that the Post Office 8 was not allowed to see the high or low-level design of 9 the Horizon System, and was required instead to trust 10 Pathway to produce acceptable outcomes. 11 In the design delivery of IT projects, have you 12 encountered what has been described as a black-box 13 approach such as that before? 14 A. Have I experienced it personally, no. I mean, 15 I understand what it is, and I certainly can appreciate 16 that sometimes that's okay, but personally I've never 17 engaged in a project where it's been a completely black 18 box. 19 Q. Thank you, in which case I won't ask you questions about 20 that either. 21 Can we turn to section 12 of your report, please. 22 You refer to "The tenuous relationship between ICL 23 Pathway, their sponsors (initially the Benefits Agency 24 and POCL) and suppliers were often topics of concern for 25 ICL Pathway's management team." Can you describe 134 1 overall what this section of your report is addressing. 2 A. Certainly. 3 Q. What was the issue for you? 4 A. The issue is this is a very complex system that ICL 5 Pathway is trying to create and, notwithstanding what 6 you just said, they're not trying to create this in 7 a vacuum. They need to have a good working relationship 8 with their sponsors as well as with all of their 9 co-suppliers of this system. 10 So, if there are problems in communications, 11 either because of expectations being misaligned or for 12 whatever reason, that's going to cause problems. It's 13 just not a healthy working environment. Everyone needs 14 to basically be rowing in the same direction in order to 15 get something like this done on time and efficiently. 16 So overall my commentary around here is that it 17 doesn't -- it seems that there were times where there's 18 a lot of tension between ICL Pathway and either its 19 sponsors or potential tension between ICL Pathway and 20 some of their partners in delivering the system, and the 21 reason I know that is because, you know, it was talked 22 about in the monthly reports. That just struck me as 23 something that I'm sure everyone wishes wouldn't have 24 happened, but certainly couldn't have provided the most 25 healthy environment in making sure that the Horizon 135 1 System was the sole focus on getting produced. 2 Q. So far as you could see, was this a consistent problem 3 over the entirety of the period that you looked at, even 4 after the Benefits Agency withdrew? 5 A. I don't think it was a consistent problem over the 6 entire period, but I think that there were points in 7 time, based off of the information in the monthly 8 reports where it was a concern. 9 Q. Thank you. Can we turn back, please, to page 4 of your 10 report. This is your Executive Summary on pages 4 and 5 11 and I would like, if you would, for you to speak to 12 these nine or so statements that you make. So the 13 first: 14 "Horizon IT envisaged modernising the UK's Post 15 Offices branches. This was an ambitious goal placing 16 hardware and software in about 18,000 branches to allow 17 subpostmasters and mistresses the convenience reliably 18 to store and transmit electronic records of their daily 19 business activities." 20 The words "this was an ambitious goal", what did 21 you intend to convey by that? 22 A. I wanted to employ language that would indicate the 23 complexity of this roll-out of this system. It is 24 ambitious. There are a lot of moving parts in this 25 system, a lot of moving parts. In any system, 136 1 especially a technology system, the more moving parts 2 you have, the more opportunities there are for problems. 3 There are a lot of managerial tasks both strategically 4 and tactically that need to be tackled. There are a lot 5 of logistical tasks. There are a lot of training tasks, 6 and there's a lot of technology tasks. The technology 7 can be spread across infrastructure, hardware, software, 8 any number of things. 9 The issue with computer systems is that, like I 10 said, there are a lot of moving parts, and any one of 11 those can cause a problem. So it's ambitious to say 12 that I'm going to make sure that all of these parts are 13 co-ordinated in a way that it's going to deliver exactly 14 what you want me to deliver. So that's why I think it's 15 ambitious. 16 Q. Thank you. Second: 17 "The technical aspects of the Horizon IT System 18 were significant but did not account for all of the 19 complexities of a successful implementation." 20 What did you mean by that, please? 21 A. So the technology certainly is the focus and is there. 22 But even aspects such as training -- do my end users 23 know how to use the system -- that's not necessarily 24 a technical issue but it's a very important part of the 25 system with a capital S. You know, are we going to make 137 1 this system work correctly? 2 Training, you know, even delivery of replacement 3 hardware, not what I would think of as a technical 4 issue, but logistically very important to the process. 5 Are we making sure that this systems up all the time? 6 Do we have good management communications amongst all of 7 the different players that are here to either provide 8 the requirements, implement those requirements through 9 our design and development process, or support those 10 requirements, like an infrastructure backbone such like 11 BT or Energis making sure that everyone understands 12 what's going on? 13 So those are kind of the issues that are orbiting 14 the fact that even just the technology itself is 15 complex, there's a lot more to this system than just 16 sitting down and coding it. I would say that was the 17 system with a small S, and making sure the whole thing 18 works is the system with the capital S. 19 Q. You say a sentence on: 20 "Both the technical and organisational dimensions 21 of the Horizon IT System also required vigilance. An IT 22 system is a 'living' entity. It needs care and 23 attention beyond its initial roll-out." 24 Can you expand on that. 25 A. Yes, this is not a point in time thing. Let's say that 138 1 I've delivered to you the perfect system today. That 2 doesn't mean that you will be satisfied with it in six 3 months, because perhaps you might want some changes to 4 it, you might want to either expand the volume of 5 activity that can be done on it, or you want some more 6 functionality or features. So there's going to be some 7 sort of feedback loop process which will most likely 8 require me to continue augmenting the system. 9 That's another dimension that continues to 10 complicate making sure that this system with a capital S 11 works right because now, not only have I delivered you 12 a system right now, we're talking about evolving the 13 system and having it mature over time, but making sure 14 that it's stable throughout that whole period. 15 That's the living system and, like I said in my 16 discussion before, it's not just the technology. It's 17 the technology and the people and the processes that 18 that technology support, all being co-ordinated and 19 making sure that they are in concert with each other as 20 opposed to being at odds with each other, and that's 21 difficult. 22 It just is. It's difficult to have that many 23 different items working together correctly. So does 24 that answer your question? 25 Q. Yes. Could you help: does the level of care, attention 139 1 and ongoing vigilance that one is required to give to an 2 IT system, in part, depend on what has occurred in 3 design, development, testing and roll-out, i.e. how easy 4 a birth the product had? 5 A. Can you roll back and ask me the question again, please. 6 Q. To what extent does the level of care, attention or 7 ongoing vigilance that one has to give over to an IT 8 system depend in any way on what occurred in design, 9 development, testing and roll-out? 10 A. I will say that, in a perfect world, the design phase of 11 any system is very much a partnership between a sponsor 12 of the system and the team that's delivering the system, 13 and they need to be very closely aligned and in very 14 good communication, in order to produce a final product 15 that satisfies the sponsor's needs. 16 Having said that, oftentimes, you know, we can all 17 of a sudden start seeing an introduction of new 18 requirements in the design phase if we're not 19 disciplined about it. You know, there's got to be 20 a point in time when you say, "This is what we're going 21 to do and this is what we're not going to do", because 22 we could be on an endless cycle of, "This also would be 23 nice, this also would be nice, this also would be nice." 24 So it's very important, I believe, for the 25 sponsors to have a really good idea of, hopefully, 140 1 generally what they want and then be committed to 2 specifically talk in detail of whether it's okay to 3 deliver it in this manner, or whether it's okay to be 4 delivered in this other manner. 5 That's a discussion to be had. Unless the 6 sponsors know exactly what they want at a very detailed 7 level at the beginning, in which case they can say, 8 "Here's exactly what we want, go do it and you don't 9 need to talk to us until it's done." That's quite rare, 10 but -- I mean, as a programmer, I would love that. You 11 tell me -- you know exactly what you want and don't make 12 a change to that, I would love that because then I just 13 have to go and actually execute against that. 14 But that's not typically what happens in the real 15 world because usually discussion -- you know, you didn't 16 know what was possible. I had a couple of comments about 17 what you asked for, and it might change your mind about 18 what you want. So there's obviously a dialectic going 19 on there, but that should be very short in time to the 20 point where you say, "Okay, I know that there's a wide 21 range of possibilities available, but for right now 22 let's focus on what I want immediately. Let's get that 23 done, make sure it works right, and then we can move on 24 to the next stage." 25 Q. Thank you. Paragraph 1.1.3. You say: 141 1 "The Horizon IT System had multiple constituencies 2 that needed to be both strategically and tactically 3 aligned." 4 What do you mean by that strategic and tactical 5 alignment? 6 A. I think I was actually just referring to what I just 7 said. So strategically that's a very high level, right? 8 We know we want -- you know, in this case we know we 9 want a network system that can do our point of sale and 10 interface with our accounting system. Okay, great. 11 That's a good strategy, and there's a business reason 12 behind that strategy, and I'm more than happy to supply 13 that to you if I was the supplier. 14 The tactical part though is where a lot of the 15 heavy work is. We need to know specifically how to do 16 things. We need to specifically understand what you're 17 expectations are. 18 For instance, you know, if we go back to my first 19 testimony when I said, "Do you have a set of business 20 processes that are mature and you want this system to 21 support?" Okay, let's talk about those, let's make sure 22 that the system that we're developing supports those, 23 and then we need to go back to our team, to my team and 24 all of my partners, and I need to make sure that we all 25 understand exactly what the needs of those business 142 1 processes are and how we're going to first architect 2 a solution and then actually build and deliver that 3 solution. 4 Q. Thank you. You say in point 4: 5 "The systems design and implementation needed to 6 account for real-world contingencies. Many designs are 7 very elegant, but they only maintain their elegance if 8 the implementation withstands practical realities and 9 concerns." 10 I think that probably speaks for itself unless you 11 need to expand on it. 12 A. No, I think that speaks for itself. 13 Q. Thank you. 14 1.5: 15 "The user support mechanism needed to assist the 16 users as they migrated from a paper-based process to 17 computer-based process or switch from using one system 18 for another." 19 That latter is a reference to -- 20 A. ECCO. 21 Q. To the new EPOSS system: 22 "Continuous training for all versions of the 23 system needed to be available. The support structure 24 needed to cater for end-to-end users who might struggle 25 to adapt to changes from the manual processes they had 143 1 undertaken for many years. The support structure needed 2 to be able to service a high volume of users. The 3 support structure needed to be designed to adapt to the 4 needs of the users as the IT system evolved through its 5 different versions." 6 I think you have spoken to that already and 7 therefore we needn't expand upon it. 8 You say, sixthly: 9 "The internal error resolution mechanism needed to 10 be able to quickly resolve reported errors through 11 identifying root causes, methodically correcting these 12 errors and distributing the remedies in a timely and 13 efficient manner." 14 Was there any particular reason why that was 15 necessary in Horizon as opposed to any IT system? 16 A. This is for certainly for any IT system. This is, you 17 know, some of the material I was asked to review, so it 18 definitely needed to have a mention. 19 Q. Seventh: 20 "The system's functionality needed to maintain 21 accounting integrity. It was the origin of sales and 22 inventory information that flowed into the financial 23 systems of POCL. Consequently, it was intended to be 24 a 'source of truth' for these fundamental accounting 25 facts. Any errors deriving from the IT system would, if 144 1 not the otherwise rectified, be reflected in all 2 downstream processes and systems." 3 I think that probably speaks for itself. 4 You say in your paragraph 8 that: 5 "Throughout [your review you] identified 6 shortcomings in each one of these key areas." 7 The first of them is "constituency alignment". 8 Can you speak to each of these six points, please, the 9 first, constituency alignment. 10 A. Yes. You know, through the reading of the documents, it 11 was obvious to me, at least through my reading of the 12 information, that there were some strained relationships 13 between ICL Pathway and POCL. It was actually fairly 14 straightforward, and that's just not an ideal situation 15 when you are trying to work together to develop 16 a system. 17 Q. You say: 18 "The Helpdesk was often the root of Service Level 19 Agreement issues with POCL; AIs were a gating issue to 20 the financial success of Pathway." 21 I think you have spoken to those already, haven't 22 you? 23 A. Yes. 24 Q. On design and implementation you say: 25 "Hardware issues were prevalent during national 145 1 roll-out. Many post offices were disconnected for 2 extended periods of time; the persistence of reference 3 data management (sic) degraded the integrity of the 4 Horizon system." 5 Collecting those three things together, how 6 serious a concern were they? 7 A. I think that they impacted the actual performance in the 8 real world of the system. We saw in the examples that 9 we just went through, you know, some of it was reference 10 data information that were creating the situation where 11 there were some receipts and payments mis-balance 12 information. 13 At a more understandable layman's term level, if 14 your hardware's not working, if you have just delivered 15 a new system to me and even if the software is perfect 16 but the hardware's not working, that doesn't matter to 17 me. I really want is something that works. I want 18 something that works correctly, reliably, consistently 19 and, from an end user's point of view, I don't know that 20 it a matters if it was a hardware issue or a software 21 issue or a service provider issue. If the system in my 22 perception is not working correctly, it's not working. 23 It doesn't really matter why it's not working, and 24 ICL Pathway, you know, at this point is in charge of 25 delivering of that system. So regardless of whether 146 1 they're personally -- you know, if it's within their 2 particular enterprise responsibility, or if it's more of 3 an umbrella partnership with other enterprises, they are 4 going to be the one that get the focus if it doesn't 5 work. 6 Q. "User support. Postmaster training experienced 7 difficulties during national roll-out; Helpdesk was 8 often the root of SLA issues with POCL." 9 You have explained that already I think. 10 "Error resolution: The SMC was frequently cited 11 for not properly filtering calls to the SSC. The SSC 12 was overwhelmed with escalated issues ... but were 13 earnest in their efforts to perform their duties." 14 You have explained all that. 15 "Accounting integrity: the persistence of 16 reference data mismanagement degraded the integrity of 17 the Horizon System." 18 Was that a view that you formed as being 19 applicable even at the end of the period that you looked 20 at, i.e. as at after roll-out? 21 A. Yes. 22 Q. "A persisting issue related to AI376 (accounting 23 integrity); payment and receipt imbalances were common 24 symptoms with varied causes." 25 We have just examined those in section 18 of your 147 1 report. 2 You came to those conclusions without, I think, 3 seeing any or all of the reports created at the time by 4 developers, programmers, coders, engineers, within 5 either ICL Pathway or POCL, and went instead off the 6 PinICLs, PEAKs and KELs, and the monthly reports and 7 other management reports. 8 I think since then you have seen, through looking 9 at the witness statements and reviewing some of there 10 was transcripts of the evidence, a range of other 11 reports or documents referred to; is that right? 12 A. That is correct. 13 Q. Amongst that material that you have reviewed, was there 14 anything you would, in particular, wish to highlight as 15 being relevant to as undermining or supporting the 16 conclusions that you have set out in those six 17 sub-paragraphs? 18 A. There is nothing that I read that does anything but 19 support my confidence in the conclusions that I've 20 written in my report. 21 Q. Amongst the material that you examined, did anything 22 stand out in particular to you? 23 A. I will say that there was a lot of discussion in many of 24 the witness statements and the transcripts that I read 25 that was talking about the design and development 148 1 process that went on at Pathway for the Horizon 2 programme. Much of it was talking about what appeared 3 to be a lack of design process and more of a "Let's just 4 go develop it", without necessarily having a good set of 5 requirements or a proper design facility to help inform 6 that development process. 7 Now, that's what I've read. I will say that that 8 certainly could substantiate the amount of PinICLs and 9 PEAKs, the volume of PinICLs and PEAKs that were there. 10 It certainly could substantiate the fact that there 11 are -- when I was looking at the payments and receipts 12 imbalance and noticing that there were a lot of 13 different causes for that, if in fact there was not 14 really a good design -- and I have no reason to not 15 believe what I read in the witness statements -- that 16 certainly would explain why there were so many errors in 17 the system as represented by the PEAKs and PinICLs, and 18 why, even when you fixed one, something else popped up, 19 and then you fixed one and something else popped up. 20 That certainly is a consequence of not having an 21 extremely disciplined design-to-development process. 22 Q. We've heard evidence, in particular, concerning what was 23 called the reverse engineering or reverse documentation 24 of the part of the EPOSS system. Have you read about 25 that evidence? 149 1 A. Yes. 2 Q. Sorry, have you read that evidence? 3 A. Yes. 4 Q. Did you read the evidence concerning the EPOSS PinICL 5 Task Force? 6 A. Yes. 7 Q. Did you read the EPOSS PinICL Task Force report? 8 A. Yes. 9 Q. I wonder whether we could look at that then, please. 10 That's FUJ00080690. Is that the same document that we 11 are both talking about? 12 A. Yes. 13 Q. Rather than take you through individual parts of it at 14 the moment, I am going to ask you to look at a couple of 15 bits in a moment, what if any was your reaction to 16 reading it? 17 A. It was disturbing to read that. If everything in here 18 is as represented, it would indicate to me that, number 19 1, there was no design at least for the EPOSS section of 20 the Horizon System and, considering that's what a lot of 21 the PinICLs and PEAKs errors were that we just went 22 through that specifically led to the imbalance issues, 23 it's -- that's fait accompli. If you're not going to 24 design something properly, it's not going to -- if you 25 don't have a good design, it's not going to work 150 1 properly. 2 So, if there's no design and that design -- first 3 of all, there has to be a good design process. Second, 4 that design process has to be aligned with a very well 5 known set of requirements from the sponsors. Only when 6 those two things are done can you start thinking about 7 the construction and development of the software. 8 Otherwise, it's very difficult to make sure that the 9 system does what your customers want it to do. 10 Q. Can we look at a couple of aspects please, page 6 of the 11 document. Scroll down, please, to EPOSS Documentation. 12 "The document suite supporting EPOSS consists of 13 three main elements: 14 "EPOSS functional specification. 15 "High-level design ... 16 "Several low-level design documents." 17 And the authors wrote that: 18 "All of these were developed by reverse 19 engineering the EPOSS product code at that time." 20 We've been told that that means, not that the code 21 was reverse engineered, but the documentation was 22 reverse documented to fit the code that had already been 23 written. Is that what you were just speaking about as 24 being ill-advised? 25 A. Yes, because what that implies is you didn't know how to 151 1 construct the system from the beginning to make sure 2 that it aligned with the customer specifications. If 3 you need to do this, that means there was not the proper 4 design process for a system of this ambition, because 5 you can't -- this is not like sending three people into 6 a room and asking them to code a system. This is many 7 teams, many teams of coders, and then many other teams, 8 you know, for infrastructure and for telecommunications 9 and for training and for all of that. 10 If you are doing that by basically putting people 11 in a room and coming in and saying, "I think this is 12 what they want. Why don't you code for three months and 13 show me what you have." It's just ... it's 14 undisciplined. 15 Q. Can we go forward to page 15, please. Paragraph 7.1.1 16 the authors record: 17 "The EPOSS product was originally developed using 18 RAD techniques." 19 You told us about RAD on the last occasion. When 20 you wrote your report, did you know whether or not the 21 EPOSS product had been developed using RAD? 22 A. No. 23 Q. Do you have any view or comment on the appropriateness 24 of using RAD techniques to develop which EPOSS product? 25 A. RAD is very good for developing something small, for 152 1 developing something quickly that is just for 2 illustrative purposes. But for a heavy-duty enterprise 3 system that needs to be viable over the long-term, it's 4 terrible, because you're not planning it. RAD almost 5 means no plan other than, I think I know what the goal 6 is, and I'm going to get to that goal as fast as 7 I possibly can, and there's good situations where that's 8 needed. But this is not the right situation for that. 9 You need to do everything in a very methodical 10 almost militaristic way to make sure that everything 11 works properly together, because I think I mentioned 12 before: there are so many moving pieces. This has to be 13 highly co-ordinated. RAD is the other end of the 14 spectrum. It's not highly co-ordinated or it doesn't 15 support co-ordination amongst big teams because that's 16 not the purpose of RAD. RAD is: get it done quick, get 17 a little bit done quick. If you have 80 teams doing 18 their own little part independently, you can't put that 19 back together and think that it will work as well as if 20 they were all very co-ordinated from the beginning. 21 Q. Thank you. 22 Then lastly on this document page 17, please, foot 23 of the page, from paragraph 7.3 onwards there are four 24 examples of some code that the authors have cut into the 25 document. I think, when you prepared your report, you 153 1 were not asked to look at any of the underlying code, if 2 it's still available for Legacy Horizon, were you? 3 A. That's correct. 4 Q. Here you see some example code that the authors have 5 included in their report. Did you read this section of 6 the report? 7 A. I did. 8 Q. You read the code? 9 A. I did. 10 Q. What did you make of it? 11 A. This is terrible code. This is terrible code. So even 12 the one you have up here -- so it does what it's 13 supposed to do, but it's like, you may think building -- 14 I can't remember the term -- the overly engineered 15 mousetrap. This has to be a joke. I mean, this has to 16 be a joke, because this is a ridiculous set of code to 17 change the sign on something. I'm sure that, even if 18 you don't code, you would understand that, if I want to 19 make something a negative, I just times it by negative 20 1. You don't need an "if" statement. You don't need to 21 do anything but that. This is, you know 1, 2, 3, 4, 5, 22 6 -- six lines in what could be one line. 23 So it's inefficient, at the very least it's 24 inefficient or it's actually a joke, because it's 25 terrible. 154 1 Q. Then very shortly then, if we go over to page 19, two 2 more examples of code there. An example of some 3 unreachable code and what's described by the authors as 4 bad practice. Did you form a view on either of these 5 two codes? 6 A. Yes. So the second one is bad practice. It's just not 7 the right structure, and it indicates to me that they 8 don't understand what those particular structures are, 9 and just take my word for it. They don't understand 10 what the structures are. 11 The first one, I think, is probably a little 12 easier for everyone to understand why this is bad, if 13 you actually can scroll back up, please. So the first 14 one, the highlighted items, when I read this and looked 15 at it, essentially what's happening here is they are 16 saying, "if" -- what they are showing here is, first, 17 they are checking to see if a particular variable was 18 either 3013 or 3016. So that has to evaluate to true. 19 So, in order to get into that code, you already 20 know that "lstockrootnode" is either 3013 or 3016. 21 Further in the codes then they are checking that same 22 stock root node to see if it's 2493. It's not going to 23 have changed -- let me make sure. 24 There's nothing in here that has changed the value 25 of that particular variable. So there's no way that it 155 1 will ever get inside this particular condition. 2 So I don't know if this is an artifact of 3 something old where perhaps the first condition was 4 changed and now it's changed it again. But, even if 5 that was the case, there's no comment on this. I don't 6 know what they are checking for in this first condition. 7 So either this is written by someone not so smart in 8 here, or there's been multiple updates to this code, and 9 perhaps the first statement used to be: if it's 3013 or 10 if it's 2493, because I could see that as being 11 a possibility. 12 But then further on someone went and changed the 13 condition. But then they didn't account for what 14 happens with 2493. So now am I ignoring what should 15 have happened correctly to 2493, or am I just completely 16 divorced from mathematical logic and put this in here? 17 Either way it's a bad example. 18 Q. And the third example at the foot of the page. 19 A. So this is a little bit more -- 20 Q. Niche. 21 A. -- art, art-ish. It just has to do with the 22 construction of, am I going to do -- why would I put an 23 "if" statement in a "do" statement? I'm not sure that 24 this would resonate so well with anyone that's not 25 a programmer, but this is a poor construction. 156 1 Q. Over the page, please. 2 A. Okay. So, if we remember my first testimony, I was 3 talking about hard coding information into code, and 4 then let's instead use a data-driven logic. In this 5 code, this is an example of some hard coding, and what 6 this is, this has a whole bunch of values in here that 7 have been hard coded. Why have they been hard coded 8 that way? I have no idea because I don't see any 9 comments around here. 10 I believe that the author of this document 11 speculated that there was a particular, very specific 12 condition that was trying to be resolved by doing this 13 hard coding, which means essentially they are looking 14 for any one of these numbers, and then they are doing 15 something, if they find any one of those numbers. 16 Why are they looking for those numbers? I have no 17 idea. I don't think anyone that would come after them, 18 after this particular author was there, would have any 19 idea why those numbers are hard coded in there. Can I 20 take them out? What will happen if I take them out? 21 No-one knows. 22 It's just very odd that you wouldn't put this into 23 a variable to at least allow some flexibility. First of 24 all, there's no flexibility here at all. It's just 25 these numbers other than, if I wanted to start typing 157 1 some more numbers into the case statement, but more 2 importantly that's really it. It's not very often that 3 you see hard coding in somewhere unless they are trying 4 to quickly fix an issue. 5 Q. Thank you. That can come down off the screen. Then my 6 last question just before the break, please, can we go 7 back your report at page 5. And your overall conclusion 8 at 1.1.9, you say: 9 "In my opinion, the stability of the Horizon IT 10 system was affected by these shortcomings. The 11 sometimes conflicting expectations between ICL Pathway 12 and POCL introduced a disruptive element at the 13 management level. The effects of these disruptions 14 manifested itself throughout the implementation of the 15 Horizon IT System. Other ICL Pathway self-inflicted 16 wounds included the suboptimal support from ICL 17 Pathway's program for training of SPMs, the Helpdesk 18 support of SPMs and the Helpdesk of ICL Pathway's error 19 resolution function. A noticeable symptom of these 20 issues was a recurrent balancing problem experienced by 21 the SPMs which directly degraded the accounting 22 integrity of the Horizon IT System." 23 Given everything that you've read since you made 24 that statement, is there anything that you wish to say 25 to change or to qualify it? 158 1 A. No, no. Everything I've read has been -- no, I'm 2 confident in this statement. 3 Q. Thank you very much. Might that be an appropriate 4 moment for the break? I wonder whether we could take 5 ten minutes, in which case I think there will be a good 6 chance of completing Mr Cipione's evidence today. 7 SIR WYN WILLIAMS: Certainly. I certainly have no wish to 8 bring Mr Cipione back for a very short period of time 9 tomorrow, so I will sit as reasonably long as is 10 necessary to finish him. 11 MR BEER: I don't think that's going to be necessary, sir, 12 but thank you anyway. 13 (3.34 pm) 14 (A short break) 15 (3.45 pm) 16 MR BEER: Sir, can you see and hear us? 17 SIR WYN WILLIAMS: Yes, I can. 18 MR BEER: Mr Stein first. 19 Questioned by MR STEIN 20 MR STEIN: Mr Cipione, I ask questions on behalf of a large 21 group of ex-subpostmasters and mistresses and managers, 22 a group of 156. Mr Cipione, you have given evidence 23 over now some days, your time when you came to the 24 Inquiry before and obviously today. Virtually all the 25 questions I was going to ask you have now been asked. 159 1 So I can really cut this down. 2 You were asked number of questions today by 3 Mr Beer about the fact that the system should have been 4 designed to be able to support court proceedings. Do 5 you remember the questions I think earlier this morning? 6 A. Yes. 7 Q. The way that my learned friend Mr Beer put that was that 8 he was talking about cases that are being prosecuted; in 9 other words, prosecuted in the criminal courts. 10 A. Yes. 11 Q. Your evidence was that you're familiar with such 12 structures because, in fact, you're engaged in the 13 design of those types of systems yourself; is that 14 right? 15 A. Yes. 16 Q. Now, just so that we can round this particular question 17 out, you're aware, obviously, that there are both 18 criminal courts and civil courts. 19 A. Yes. 20 Q. The criminal courts work to what the lawyers call the 21 higher standard of proof, which is beyond reasonable 22 doubt, and the civil courts work to a different standard 23 of proof, which is, in the UK, on the balance of 24 probabilities. I think in the States it's the 25 preponderance of evidence; is that right? 160 1 A. Yes. 2 Q. Okay. Now, so that we make sure we're all understanding 3 your evidence about this, when you're designing a system 4 that should be capable of supporting cases in the 5 courts, essentially do you agree it should be designed 6 to fulfil a high quality product, making sure that 7 material or evidence from it comes from a system which 8 is capable of holding high data integrity? 9 A. Yes. 10 Q. Now, a number of different issues have been highlighted 11 with you as to the problems that were existing within 12 the Horizon System. Now, in the last part of your 13 evidence you were discussing the coding and the 14 difficulties that you were seeing in relation to 15 particular examples. You will be pleased to hear -- 16 probably everyone will -- I am not going to try and go 17 near those. 18 Let's just think about other issues other than 19 software. So let's start with hardware. So hardware 20 issues -- is this correct -- such as connection issues 21 or de-connection issues, going offline, power cuts, 22 environmental issues that relate to the weather, all of 23 those issues can also affect the operation of the system 24 in different ways; is that correct? 25 A. Yes. 161 1 Q. So using just one example, if there's a power outage, an 2 unexpected power outage, that could cause issues, if 3 there is mis-communication within the system, with 4 balancing; is that right? 5 A. Certainly it's a possibility. 6 Q. There are also issues that you mention within your 7 report about training. Now, do you accept that this 8 goes in perhaps two different ways: First of all, if 9 you get your training wrong, you're training people 10 badly, then people, users of the system, in this case 11 subpostmasters, may themselves get it wrong when they 12 are using it; is that right? 13 A. They could operate the system wrong; that's correct. 14 Q. Then there's the other side of it, which is that the 15 training itself has not been sufficient, or incomplete, 16 or not sufficient for the people that are operating the 17 system so they've not been able to get it or understand 18 it or use the system adequately, yes? 19 A. Yes. 20 Q. So that's another level of issues that come up within 21 the system, all of that entirely innocently from the 22 perspective of subpostmasters; do you agree? 23 A. Yes. 24 Q. Now, one final thing regarding Helpdesk issues. We've 25 seen some evidence -- and I can take you to it if we 162 1 need to -- that there were scripts which were 2 inaccurate; in other words, scripts being provided to 3 the Helpdesk advisers that were inaccurate, in other 4 words, that the Helpdesk people were using scripts that 5 were inaccurate and then giving inaccurate advice; okay? 6 A. So you are saying that Helpdesk scripts were inaccurate 7 or the help was not correct? 8 Q. Right. So that can lead to yet another level of issues, 9 because the subpostmaster may accept such inaccurate 10 advice and operate the system badly based on that 11 advice; do you agree? 12 A. Certainly. 13 Q. It also will lead to confusion no doubt within the 14 subpostmasters and their ability to operate the system 15 more generally? 16 A. Yes. 17 Q. The final part of that is: it will probably lead for 18 more contacts back to the Helpdesk? 19 A. Yes. 20 Q. Now, you mentioned that we are looking at a system that 21 was designed and started in operation at the end of the 22 last millennium. That was part of your evidence today. 23 If the system was designed so that a customer 24 going into a post office branch was able to have their 25 transaction completed and verified back to the Horizon, 163 1 if you like, mainframe, back to the main servers, and 2 then confirmed to the subpostmaster so that all of that 3 was done in one basic transaction, so confirmed from 4 both ends that the information is correct, that there 5 has been receipt of that information at the branch, 6 confirmed back to the office, if you like, and confirmed 7 yet again back within that same transaction to the 8 subpostmaster, would that be the only way that you could 9 design a system at that time, so that everyone knew 10 everyone's checking going, "Yes, you got it right, it's 11 all clear"? 12 A. I think your goal of the confirmation is a good goal. 13 How you would effectuate that in the late '90s, I think, 14 would require some thought on my process, but 15 I appreciate what your goal is, and I think that would 16 be a good goal. 17 Q. Was the system designed to do that? 18 A. I really can't comment on exactly what the mechanics of 19 the system are. I could say that I don't think it was 20 designed to do that, based on my limited understanding 21 of the technical specifications underneath. 22 Q. Can I point you in one direction that may help. We know 23 that the system was designed so that, even if there was 24 connectivity lost, the branch could continue to operate, 25 working on the basis that the branch, when reconnected 164 1 to the main system, would then catch up; in other words, 2 there would then be a conversation. 3 A. Yes, I'm aware of that functionality. 4 Q. So that tells us that, in my example of a customer 5 coming into a branch that, if the system happens to be 6 operating offline at that particular time, there 7 wouldn't be a capable way of actually communicating 8 confirmation and acceptance, would there? 9 A. No, no, there would not. 10 Q. Now, you've been referred by Mr Beer at 11 paragraph 7.4.5 -- if we need to go to it, it's the 12 reference to correspondence that some KELs may have been 13 deleted. You were asked a couple of questions about 14 that. 15 Just so we understand the detail about this, what 16 you say in your report is that some KELs may have been 17 deleted. That's your understanding from the 18 correspondence. 19 Were you given any information about whether they 20 were or were not deleted? The word "may" is rather 21 loose. 22 A. I think I'd have to refer you to Mr Beer. I didn't have 23 any direct correspondence with Fujitsu. 24 Q. So we need get back to the Inquiry and ask them the 25 detail about that conversation. 165 1 A. Yes. 2 Q. Lastly, this happens to be at 6.2.20, you discuss the 3 failure by Fujitsu to provide PinICLs in a timely 4 fashion. Can you help any further with that and how 5 that happened? 6 A. What specifically are you referring to? 7 Q. In your report at 6.2.20 -- if you can have the report 8 up on the screen, please -- so reference EXPG, thank 9 you, and then pagination Relativity pagination page 60. 10 We're looking for 6.2.20. If you can highlight 6.2.20 11 just further down. Thank you. 12 To this is the point I am referring to: 13 "The PinICL archive databases were received late 14 in my review. I therefore decided, in consultation with 15 the Inquiry team, not to fully investigate the databases 16 as this would have unduly delayed the completion of this 17 Report, and could have had knock-on consequences for the 18 Inquiry's timetable. In addition, I noted that Fujitsu 19 had not produced these PinICL archive databases in 20 response to the original Rule 9 request submitted by the 21 Inquiry in December 2021. Therefore, I deduce the 22 incremental information not to be responsive to the 23 original request from the Inquiry." 24 If we just quickly take this into bits, first of 25 all did you ever receive those PinICLs? 166 1 A. Yes, I have those PinICLs and perhaps an explanation 2 might clarify that. So we received the 50,000 or 60,000 3 PinICLs and PEAKs in PDF or HTML format. Subsequent to 4 that, we were delivered this database which was, for the 5 same PinICLs and PEAKs, just in a structured format. It 6 wasn't -- it wasn't new PinICLs and PEAKs; it was just 7 in a different format. 8 Q. So in the end the confusion about this was cleared up? 9 A. Yes. 10 Q. You were able to look at the PinICLs fully? 11 A. Yes. 12 MR STEIN: Thank you, sir. 13 Questioned by MS PAGE 14 MS PAGE: Laura Page, I represent a number of postmasters as 15 well, and just a few questions to see if there are 16 a couple of areas you may be able to help me with; you 17 may not. 18 One of your 7 examples that you talked through 19 with Mr Beer earlier is dated from June 1999, and it was 20 an experience that's common to a number of the 21 postmasters. It involved the bringing forward of 22 balances and, when you brought them forward, they 23 doubled up. Then you might bring them forward again and 24 they'd double up again, and that's something that we 25 have heard a lot of from the subpostmasters. 167 1 Your example was from June 1999. One of the 2 subpostmasters that I represent, she had that problem in 3 2000 -- Nichola Arch. So not very long afterwards but 4 nonetheless perhaps a year or so afterwards. Then we 5 see it happening again and again. 6 In the example that you looked at, it appeared 7 that a fix had been applied, but we see the problem 8 coming back. 9 Now is that something that we can relate to this 10 terminology that we've heard "code decay", where you fix 11 on fix on fix, and you end up sometimes making the 12 problem worse, or it comes back in a different form? 13 Does that make any sense to you? 14 A. It's a possibility. Of course I can't tell you exactly 15 why, because even in the examples I reviewed, I didn't 16 see the code that -- if that example was code related. 17 If some of the witness statements and transcripts 18 I've read indicate that, you know, regression testing 19 wasn't done properly. It is possible that the issue 20 that I was describing in the PinICL was corrected and 21 then for some reason someone went back in and did 22 another correction to that code but actually reverted it 23 back; that's a possibility. I have no idea if it 24 happened but it's certainly a possibility. 25 Q. Thank you. 168 1 We also heard from a witness, Mr Andrew Simpkins. 2 I don't know if you read his testimony, but he talked 3 about an element of the design which he saw as a pretty 4 fundamental design flaw in which the problem was that 5 subpostmasters couldn't see where or when discrepancies 6 occurred in their accounts. It was completely opaque to 7 them. So numbers just kind of came out, but they didn't 8 know the calculation behind that or where those numbers 9 had been put in. 10 He said that that would have compounded a problem 11 because, when they called for assistance, they wouldn't 12 be able to identify exactly when the problem occurred, 13 leaving the Helpdesk unable to -- I think the word he 14 said, was that "replication" of the fault is so 15 important to try to get to the bottom of the fault that 16 this may well have compounded the problems that the 17 Helpdesk were having because subpostmasters weren't able 18 to say, "Well, it happened exactly like this". 19 Is that something that you would agree with, that 20 was a kind of a design flaw, if you like? 21 A. I would say that that was a usability design flaw 22 certainly. I don't know that that is necessarily 23 a technical design flaw. I mean, I'm kind of picking at 24 nits here but, from a user perspective, absolutely. If 25 I could have seen a register of everything that the 169 1 system was recording that I had entered, even if 2 I hadn't entered it, it certainly couldn't have hurt the 3 investigation process. It absolutely most likely 4 probably would have helped the investigation process. 5 Q. Thank you. Just a couple more. Again, I hope this 6 isn't a lengthy one but it may assist if we bring up 7 a document, please. The reference is POL00043744. 8 SIR WYN WILLIAMS: Ms Page, could I ask you to speak 9 a little more into the microphone, please. 10 MS PAGE: Certainly. Is that better? 11 SIR WYN WILLIAMS: Yes, it is. Thanks. 12 MS PAGE: So the document that I've just brought up if we go 13 to page 30 and we go down, please, towards the bottom of 14 the page and going over to the next page, we see 15 a section that is headed "audit and Fraud Management" 16 and this deals with -- this is a document dated in 2001 17 and it deals with the fact that when the Benefits Agency 18 came out of the tripartite agreement, one of the things 19 that happened was that a fraud management system that 20 they had wanted and had sort of put in for benefits 21 fraud was removed. 22 If we could just perhaps read about one of the 23 possible implications of that, it says: 24 "There are a number of possible implications on 25 the Horizon audit solution. 170 1 "Is there a need for the Audit Server to harvest 2 and store information from the NBE? If so, then the NBE 3 will need to support either NFS or NT file shares. 4 "Audit Server sizing will need to be examined. 5 There will be a significant increase in the data that 6 needs to be audited." 7 If we could go over, please: 8 "Will there be an increase in the number of 9 requests to retrieve audit information? The current 10 Audit solution is designed to service a limited number 11 of requests (a maximum of 50 per year). Any dramatic 12 increase in requests will require the current solution 13 design to be re-examined." 14 Then: 15 "A fraud management system was developed for the 16 PAS/BES system, but removed from the Horizon solution 17 when these applications were dropped. It is necessary 18 to consider whether there will be a requirement for 19 fraud investigations and control with Network Banking. 20 Although the current Audit solution will hold the data 21 that may form the basis for detailed fraud 22 investigations, it is not designed for, nor can it cope 23 with, an ongoing fraud management approach based upon 24 statistical analysis of transaction patterns over a wide 25 variety of Outlets." 171 1 Now, I don't pretend to understand all of that and 2 I don't know whether you would be able to help us with 3 all of it either, but what it appears to be getting at, 4 as far as I can tell, is that with the removal of the 5 fraud management system, there's then set up a sort of 6 an arrangement we've heard about where up to around 50 7 requests a year for audit were done in a sort of a much 8 less organised fashion, and those 50-odd requests could 9 be used by Post Office for all sorts of things. It 10 might be just an investigation, it might be what's been 11 told, a fishing expedition, or it might involve actually 12 going forward to a criminal prosecution. 13 What I am trying to understand from this set of 14 paragraphs is, is there something about the maximum of 15 50 a year? Is there something magic about that? Does 16 data degrade in some fashion if there's going to be more 17 than 50 requests a year? 18 A. Who was the author of this document? 19 Q. This appears to be an ICL Pathway document trying to 20 deal with the Network Banking, the next phase. So it's 21 a later phase and they're talking about moving forward 22 into that phase but one of the things they are dealing 23 with is this issue around how do we support prosecutions 24 and investigations, how do we provide audit trails? 25 A. Okay. So, I'm sorry, what was your question again? 172 1 Q. So I'm trying to understand whether this 50 -- in the 2 bullet point that's still on the screen: 3 "The current audit solution is designed to service 4 a limited number of requests (a maximum of 50 per year) 5 any dramatic increase in requests will require a current 6 solution design to be re-examined", can you sort of see 7 any reason why there was -- is there going to be 8 a problem with extracting data if you don't have a 9 proper solution for it and is there some magic around 10 the number of 50? 11 A. First, I don't even know what this system is, so I don't 12 know that any of my commentary is helpful or not. 13 Q. No. 14 A. I don't know why that 50 is in there but I also don't 15 know what the system is, so anything I say would just be 16 speculation. 17 Q. All right. Then just finally this also might be a bit 18 speculative, but it's just in case you are able to 19 assist. 20 On the non-polling issue, when I put that section 21 of your report to a witness who was a network witness 22 (that was his specialty), Mark Jarosz -- I don't know, 23 again, if you were able to hear much or read much of 24 his -- but he explained that when non-polling occurred 25 there was a procedure which ought to have been in place 173 1 which ought to have involved a special laptop being 2 brought to the branch in question and he said that there 3 was a solution, a Day D solution. 4 I should say that the subpostmasters we've heard 5 from, I don't believe any of them have experienced this 6 occurring but is there something that -- does that ring 7 any bells? Is that something that would have worked if 8 a special laptop has arrived that would have helped with 9 the data reconciliation? 10 A. I'm not familiar with that. I suppose they could have 11 brought another piece of hardware and connected it up, 12 but I'm not familiar with what you're discussing. 13 MS PAGE: All right. Thank you. Those are my questions. 14 MR BEER: Sir, I believe that's all of the questions for 15 Mr Cipione. Can I thank him on behalf of the Inquiry 16 for his report and the two occasions when he's come to 17 this country to give evidence. Thank you. 18 SIR WYN WILLIAMS: Well, you can do it, Mr Beer, but I think 19 I'm not just duty-bound but it's my pleasure to give 20 Mr Cipione my considerable thanks, and I break it down 21 in this way, Mr Cipione. First of all, for the 22 production of a comprehensive report in a very tight 23 timescale, so my thanks for that, and I dare say thanks 24 are due not just to you personally but to the team that 25 you had helping you in that regard. 174 1 Then, secondly, I'm very grateful for you keeping 2 your eye on the Inquiry and reading evidence as it's 3 unfolded so that you could verify or alter or amend in 4 any way you thought appropriate evidence contained 5 within your report; and then, lastly, for coming twice 6 to answer a great many questions for which you, again, 7 have my considerable thanks. Thank you very much. 8 A. Thank you, sir. 9 MR BEER: Sir, I think we return next Wednesday at 11.00 am. 10 SIR WYN WILLIAMS: That is correct. Thank you very much, 11 Mr Beer. 12 MR BEER: Thank you. 13 (4.08 pm) 14 (Adjourned until 11.00 am on Wednesday, 23 November 2022) 15 16 17 18 19 20 21 22 23 24 25 175 1 I N D E X 2 CHARLES CIPIONE (sworn) .......................1 3 Questioned by Mr Beer .........................1 4 Questioned by MR STEIN ......................159 5 Questioned by MS PAGE .......................167 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 176