Good morning. Uh, welcome to the Well, actually, is this the organizational meeting? This doesn't look correct. Um, welcome to the regular meeting of the audit committee for August 24th. I am Elliot Payne and I'm the chair of this committee. I want to offer a friendly reminder to all members, staff, and the public that these meetings are broadcast live to enable greater public participation. These broadcasts include real-time captioning as a further method to increase the accessibility of our proceedings to the community. Therefore, all speakers need to be mindful of the rate of their speech so that the captioners can fully capture and transcribe all comments for the broadcast. We ask all speakers to moderate the speed and clarity of their comments. At this time, I'll ask the clerk to call the role so that we can verify a quorum for this meeting. >> Member Al Mwafak >> I. >> Committee member Chuck Tai >> present. >> Angelhart >> present. >> Hoseby is absent. Peterson >> present. >> Vice Chair Singleton >> present. >> Chair Payne >> present. >> We have six members present. Let the record reflect that we have a quorum. Colleagues, our agenda is before us. May I have a motion to adopt the agenda? >> So moved. >> Second. >> Uh all those in favor say I. >> I. Oppose say nay. >> That carries. Next we have the acceptance of minutes from our regular June 22nd meeting. May I have that motion? >> So moved. >> Second. >> All those in favor say I. I. Those opposed say nay. That [snorts] carries and the minutes are accepted. Uh we have four items in new business today. Item number three is uh the 2025 state of Minnesota office of the state auditor management and compliance reports. Karen Stacken, audit director from the office of the state auditor will join us to give that report. Uh welcome. All right. Good morning, Chair Payne and members of the audit committee. My name is Karen Stacken. I'm with the Office of the State Auditor and I'm here to present the results of the 2025 audits. Um, first I want to thank management and staff of the city of Minneapolis, the Minneapolis Park Board, and the Minneapolis or the Municipal Building Commission for their cooperation and assistance during our audits. We audited the city of Minneapolis, the Municipal Building Commission, and the Minneapolis Park and Recreation Board for the year ended December 31st, 2025. We issued our opinion on the cities and the park board's financial statements and issued management and compliance reports for the city, NBC, and the park board. Um, the NBC does not have a separate opinion because their financial statements are included in the city's report and they're not separately issued. We performed our audits in accordance with auditing standards generally accepted in the United States of America, the governmental auditing standards, and for the city of Minneapolis and the park board uniform guidance, which are the federal requirements related to federal programs. Our opinions on the financial statements were dated June 24th, 2026 for the park board and June 25th, 2026 for the city. Our opinions are included in the respective entities's annual financial report and our opinions for the city and the park board were unmodified which means it's a clean opinion and it's the best opinion that can be issued. This means the financial statements are fairly presented and conform with applicable accounting standards. We also audited the city and the park board's major federal awards and have issued our opinion on compliance for federal programs that we audited. Our opinion on each of the city and park board's major federal programs were unmodified. Um this means that the city has and the park board have complied in all material respects with the compliance requirements that were direct and material to the major federal programs. Um throughout our audit, any findings related to internal control, compliance, Minnesota legal compliance, federal program internal control, overcompliance, or federal program compliance were discussed with the respective management and staff. At the end of our audits, we discussed the written findings that would be those that were included in the management and compliance report um in detail for each of the respective entities with their respective management and governance. Our management and compliance report for the city includes a corrective action plan and a summary schedule of prior audit findings, both which were provided by management. No findings were reported for the park board or NBC in the current year. The corrective action plan is the management's plan for resolving the current year findings and is provided to us just before the report is issued. We do not audit the information included in the corrective action plan, but we are required to follow up on those findings in the next audit. The summary schedule of prior audit findings lists the findings that were reported in the prior year's management and compliance report and provides the status of each of those. They're either fully corrected, partially corrected, or not corrected. In our 2025 management and compliance report for the city, we reported four findings and we were able to resolve three of the findings from the previous year. Um, our schedule of findings and question costs includes um the following information about each finding. The criteria, which is the basis for what we were measuring against, the condition, which is a description of the issue. the context which is additional information to give you more perspective. The effect which is the um condition or what it could lead to and then the cause is why the finding exists and this was reviewed with um finance management and then the recommendation of um how would we would um suggest to improve on that for the future. So our four findings they related to service organization controls reporting um in the federal awards and prompt payment of invoices. The city's provided its corrective action plan for the findings reported and as with all the current year findings we will review them again at the next audit and management will be required to provide a current summary schedule of prior audit findings with the status of those. So the findings that we did have um there was one related to internal control. Um this was finding number one and it was related to service organization controls. Um there was two federal program findings. Both of these related to the CDBG program and they were both related to the reporting compliance requirement. And then there was also one related to legal compliance and this was due to the prompt payment of invoices. the findings that were resolved from the previous year. Um the first one was related to an audit adjustment. The second one was related to um suspension and debarment compliance requirement for the um state and local fiscal recovery funds program and the last one was related to subreient monitoring and that was also related to the state and local fiscal recovery fund program. So our our management and compliance report for the park board and NBC did not have any current year findings. Um does anybody have any questions? >> Do any members have questions? >> Oh um member uh Alma flag. >> Thank you chair and thank you state auditor. Uh I have just few questions regarding the recurring SOC1 finding. For example, this is a second year of the state auditor has reported the lack of SOC1 type two or SOC1 report for the convention center service organization. What specifically prevented the city from resolving the finding during 2025 and at what point would the city consider replacing the vendor if it cannot provide COC one report? So the vendor was not um did not have a review done that would give us that sock report. Um the city did implement some additional controls. So they provided some additional testing that they had done to try to show that the controls were working and it did cover many of the additional controls but not everything that we would get with that. Um they call them a sock report. So the city is my understanding still trying to work with the vendor to try to get that report and our office is looking into options of other um ways to resolve the comment if that is not available. >> Okay. I have a follow-up question regarding this. Uh beyond a particular vendor for this vendor, does the city maintain a complete inventory of third-party service organizations that materially affect financial reporting? And do we verify annually that appropriate SAK one type two reports are obtained and reviewed for all the service organizations? >> You were asking if they're obtained for all of the service organizations. If we have if we have uh a record or for example inventory for those service organization and shows for example if they provide s one report you're asking if there's a resource out there that will show if they provide them. >> Not that I'm aware of. Um I believe that that is something that has to be requested from the vendor. >> Okay. >> And I'm not aware of a resource that will show you if they have one or not. >> Thank you. Uh, another question regarding the FFATA. The audit found that all six tested 2025 CDBG subawwards were reported late. Three three older subawwards had not been reported and one reported amount was incorrect. What control should have detect these issues before the state auditor identified them? >> What control should have been in place? Yeah, before it's coming just when we have the audit and then we detect this for example findings. >> So I think in most cases um because we do look at a lot of federal awards and they do have a lot of reporting in place and oftentimes they have a calendar with a schedule that will like remind them of when the reports are due. Um I think in some of these cases um there is action that has to be taken by the federal agency first and then once the federal agency takes action then you can go in and do the reporting and so I think what happened is that it got updated at the federal agency but the federal agency doesn't always notify the local agency that it's updated and ready and I think they had been checking back um but not as frequently as needed to be to be for the report to be done in that time. So I would say you'd want to have you know like a calendar reminder or something to go back and frequently check that um just to make sure that it gets done timely. >> Thank you. My last question, the state auditor reports the municipal city did not qualify as a loss risk audit. >> Yeah. what specifically prevented the city from qualifying and what would need to change for the city to re regain low-risk audit status. >> Um so there are multiple things that can cause you not to be a low-risk audit for the year. Um things like you have to have an audit in the previous two years. Um you have to have an audit without there being any material weaknesses um or having any of the programs modified. and that has to be for two consecutive years. Um, so in 2024 all of those requirements were met, but in 2023 there had been a finding that was considered a material weakness. >> Um, so if there are no material weaknesses in the next reporting, in the next audit report, um, and all of the programs receive unmodified opinions, then the city would go to being a low-risisk audit again. So you're on your way to doing that. it it was met this year and so hopefully after the next audit it will return to a low-risisk audit. >> Thank you very much. >> Thank you. Any other questions? Uh thank you, Miss Tacken. >> All right. Thank you for your time. >> Uh seeing no further questions or comments from committees, I will direct the clerk to receive and file these reports. Uh next is item number four, which is the software inventory and access audit. We will be joined by Sedartha Pujal, director of internal audit for the for this presentation. Welcome. >> Good morning, Chair Payne and the members of the audit committee. My name is Zartha Bodell and um I'm the director of uh internal audit. The office of city auditor recently completed an audit of the information technologies department software and system inventory practices as well as access management processes for one of the primary systems managed by them and I'm here to present the results of it. Uh so this is the overview of the presentation. uh I'm going to present about the background, audit objectives, scope and methodology, conclusion, findings and recommendations and management action plans. So why audit the system and processes? Because accurate system inventories and effective access management are foundational controls that help city safeguard data, manage technology risks and ensure consistent IT governance all across departments. Uh to give the audit committee some background related to these um what we audited. So the city of Minneapolis relies on a wide range of software applications and technology systems uh to support operation, deliver services and maintain critical infrastructure. Um so so vendor management and contract services this is one of the divisions within the IT department. They maintains the city software and system inventory and then customer support and relation another division uh manages access control uh process as well. So because service now is the both the system of record for inventory management and one of the primary systems managed by it this audit includes an assessment of access management activities within service now itself. So basically we selected that particular system to um audit as well because all the software and systems are inventoried within that system itself. And this figure shows the IT department's work chart. Uh I have also highlighted the specific divisions we primarily worked with in the for this audit. So on the left side at the bottom is the customer support and relations. Uh which is also called collaboration division. And then on your right bottom is the vendor management and contract services. So this audit represents an initial review of system inventory and access management practices within the IT department. In future phases, we plan to expand this work using risk based approach to evaluate additional departments and systems across the city because since this was the initial audit, we wanted to focus on it and their processes and their systems. Once we have done that uh we know there are other systems um in the city as well. So for future audits we'll select additional departments and systems and keep this work um going in the future as well. In terms of the audit objective the objective of this audit was to answer the following questions. Is there a centralized system in place to maintain complete, accurate, and consistent enterprisewide software inventory? Do the information security policies and procedures exist to comprehensively govern access management, user access, account management, acceptable use, password security requirements, software installation restrictions and production of city systems and also ease user access to inscope software application which is service now complete, accurate, authorized, actively used and removed timely when no longer needed. So for the scope and methodology. So this audit focused on assessing accuracy of systems software management inventory and evaluating the adequacy of access control including but not limited to existent completeness and accuracy of the centralized inventory policy procedures tools and governance practices used to maintain it and provisioning deprovisioning which is adding and removing and periodic access reviews processes associated with service now system. Our scope covered activities from January 1st, 2025 to May 31st, 2026. As a part of this review, we met with IT management and process owners to gain an understanding of the processes and related activities. We conducted walkthroughs of Service Now system to review how information is tracked. We requested, received and reviewed relevant documentation and performed testing of controls. The key risk areas we covered rel to this audit were um operational and IT risks. So the based on our review the conclusion we came up with is the audit the IT department should strengthen system inventory governance and access management practices. Addressing the gaps identified through the recommendations will enhance the city's ability to maintain accurate records of its technology assets, ensure appropriate user access and support a more consistent and mature technology governance framework. There are two findings uh which we'll cover in the next few slide uh slides and associated recommendations as well. So the first finding uh is related to software asset management. So the um finding is the IT should maintain a centralized up-to-date inventory to provide accurate visibility to software assets. We observed that the city's centralized software systems inventory is incomplete and does not fully capture all the soft software and systems related to information used across departments. The department does not have a proactive process in place to regularly confirm new systems or tools implemented. Although workflow exists, it also has not fully documented procedures needed to maintain, validate and update the centralized inventory. So I understand they do have a process uh in place for any new systems there during the RFP process they do perform uh their technical due diligence as a part of the uh bringing in the new systems but if the other departments go and purchase their own systems and software on their own without letting it know about those things then at that for those systems and tools it is unaware of especially if it's like web- based tool um so that's what we're recommending uh to come up with a process to follow up on those on a regular basis and also documenting that process. And the second finding is related to the access management. Uh it should establish and maintain a periodic review process for service now users to ensure access remain appropriate, current and aligned with roles and responsibilities. So ongoing reviews to confirm that user permissions remain appropriate over time are not occurring right now. Uh moreover it only performs ad hoc reviews of the number of users when they exceed the licenses purchased. Uh however those reviews are not properly documented. So anytime so there are like certain number of um licenses that it has bought for the city. So whenever we go beyond or around that number so we perform any reviews whether any accounts are no longer needed so we need to remove those accounts but those are not regularly performed and documented as well. So basically those these two was the finding uh presented to the IT management and in terms of uh management action plan uh IT management agreed to recommendations and develop uh corrective action plans accordingly uh management will request information from departments on additional software uses documented software management procedures and conduct an annual documented reviews of all service now accounts. So the plan completion date for the first one is um December 31st, 2026 and the for the second one is uh March Oh, the first one is March uh 31st, 2027 and the second one is December 31st. So I just put the like the the latest like the most longest period and um it so based on what uh IT management provided as a part of the management um plan uh we reviewed them and looks um that action plan will sufficiently remediate the findings identified in this audit and the detailed corrective action plan is included in the report. So I stand uh before here for any questions. >> Thank you Mr. Puja. And before we go to questions I'll note that we've been joined by member uh Hosbin and any questions? Uh member Mafak. >> Thank you chair and thank you the auditor for that report. Uh it is the first report for the IT systems and that is a great start for that. Uh I'm just concerned about the completeness of this software's records. We know we you and you reviewed 822. Do we have a sense of how many for example other systems are not reported to the IT or not kept in the records of the IT? >> So we don't have that because we have limited time. So we couldn't do once we identify it's not complete. We stopped at that point. So basically what we're trying to letting them know as a part of the process is um once at least have that process in place in the back end to ask for the departments um any systems or software or tools are newly introduced and one of the controls that it has is so not everyone can u download and install those systems in the computer because you need to have u elevated access which is like admin access so even I and download a new system from internet and use start using that. Uh so that so they do have some of the uh primitive controls to make sure some of those things doesn't happen but we're now asking even in the back end they need to have a process in place to make sure uh there are no other systems or tools used by the department besides without it's knowledge. I I have a followup just question here. Uh in the action plan the IT department mentioned that they will request this information from the department. So it will be self-reported to IT department. Is there any way that we can find what other system through procurement system from for example financial records just to confirm those other system that are not in the IT inventory. Uh basically uh one of the other ways that we have come up with it like because the office of city auditor also performs um like a throughout the interviews and the questions as a part of the enterprise annual risk assessment process as well. So we can introduce that as a part of the controls to know and just disseminate those information to IT to make sure if their system is complete as well and we haven't or the IT hasn't explored other like a systematic approach. Um and maybe Bob or yeah Paul maybe yeah so Paul Cameron the CIO so he'll be able to answer question better. Y thank you. Uh good morning. Uh my name is Paul Cameron. I'm the chief information officer for the city of Minneapolis. Thanks for your question. Um so a little bit about the software that we don't have cataloged. Um we have I would say very effective controls with procurement uh for uh like if a city is spending money on something we we have knowledge of that. Uh I would say there is a gap where there's maybe a small purchase using a PECard where we are aware of it and we don't track that today but that's something that will be changing going forward. Um but any of the larger software that gets purchased will um will get tracked for it. I think the probably the bigger challenges we have and this gets at the self-reporting nature of it um is uh web-based software or there are also tools that people can download and install uh that do not require uh elevated access on the computer and and so if it's free software of that nature, it's very difficult for us through uh the normal sort of monitoring of what's happening on the computer to be able to uh understand what that is. And so that's where this the self-reporting aspect comes in. >> Thank you. I have also another question. We have now this list of all softwares and many of them they are from third parties or they are third party softwares or for example cloud based system. Do we know if we have the uh suck two type two reports for this system just to make uh to make assurance of the IT the internal controls or the controls in those systems because they are they are include some of the sensitive data of munab city. So how we can make sure that we are receiving from those software companies at least the sec the s two type two reports. >> Yeah, thank you for the question on that one. Um uh there are uh uh requirements um and I I don't know exactly when those have been added to our RFP process that require that uh companies be able to provide that. we don't um as a normal course of action sort of request that on the normal um portion of that. I'm also wondering if uh uh director arro may have better information around know the specificities for the sock 2 um for uh for what that process is. There are also uh pieces of software that we have in which I would think of them more as utility software. they're not uh storing city data. And so those from a risk perspective, we won't look at as closely as say um something like uh that's holding financial data uh or uh sensitive uh city information data within that process as well. Um so hopefully that gives you a little bit of additional detail on that. >> Yeah. And if if that also can be added to the audit plan for example for next audit plans that if you can check also the SU reports if they those third party softwares they have those SU reports >> and al and also um we performed an audit um last year regarding the third party um software as well as a part of that we did recommend uh anytime we are bringing in a new systems that the sock report be reviewed. Ed um and also reviewed on a regular basis as well and also based on my conversation uh with IT department at one point instead of reviewing the the sock to report. Uh we for the governance sector they have fed ramp state ramp and gov ramp which is performs like similar type of assessment for these systems as well. So the plan was to maybe partner with them and get that membership and uh review uh what that system or that assessment is does said. So it's similar to uh sock 2 but sock 2 is performed on annual basis but the state or gov ramp is performed on a regular basis and not so because things can change before a year. So that was the planned in the past. Uh I don't know if that has changed but maybe director Arco >> good morning welcome. >> My name is Robert Arco. I am the director of vendor management and contract services within IT. To answer your question we are moving forward with um contracting with Govramp. Uh that's moving forward in uh the next council cycle or two. So um that will afford some additional protections. In addition uh as as CIO Cameron had uh specified, we have enhanced our RFP process to include requirements for uh the sock 2 type 2 be part of that RFP. So captured as part of the contract. >> Thank you. >> Thanks. Thanks. >> Any other questions? Member uh Hosein. >> Thank you, Chair Payne. This is um maybe more of just a comment more than a question around recommendation number two on page nine and the manager report or uh a management response on page 11. Um there's a line on here uh that says these discrepancies are by you know not by themselves indicators of inappropriate access. I would say agreed. We don't know what those are because it hasn't been done. Um, I just am curious if annual is the right answer knowing that you don't know what you're going to find starting with the process. So, I would almost have a recommendation to have some sort of um review of what these are, especially around terminated employees. Like the fact that that access could still be live um just seems like annual might not be the right answer as you move through this first time of audit. And also no thank you through the chair odd committee member who has been like even in terms of like um terminated employees sometimes what happens is there uh so for example for for like election judges right they work and then they leave and their access is terminated so they show up in that terminated list but they come back again. So due to the timing of it, it might show they might be terminated but they might not. But we wouldn't know that without performing that um periodic access review, right? So just because we someone shows up as terminated but it's still in the list doesn't necessarily mean it's not right. But at least we have to perform that review to make sure those are actually terminated, terminated and not. they came back to the city to perform additional work. Does annual feel like the right um cadence based on your review of >> normally uh for the period periodic access reviews because it also because we just recommend >> and it's up to the management how and so normally we see barely at least every six months because I think that's what nest u says from my perspective but it's up to the management because they need to look at their resources as well and how much time So it's a combination of both like how much risk that this the management is willing to take and have that cadence of the review. So it's up to them to do that. >> Okay, understood. And I think maybe that's the the the recommendation that if you do see it to adjust that to a six month um and not just set we're going to do it annually not knowing what those findings are. Um um and I think that's like a good scope of how you mitigate risk, right? If it's higher risk, you do it every quarter. if it's lower risk, you do it every year or whatever that is. So, okay. Thank you. >> And that's why normally during our like for our recommendation, we just say periodics and not mention any specific timeline. >> Okay. Perfect. Thank you. >> Thank you, >> Member Peterson. >> Thank you, Chair, and thank you, uh, auditor. Um, a question kind of around that the periodic reviews. Um I think if I read it correctly that um it referenced that there's like group managers are like responsible like within the departments to have some monitoring um over their user access. Um is there any way that it there can be a requirement or in their process that yes they are doing like a bianual review or an annual review and then it can still do a sampling or a review. um whatever their frequency is, but they could be testing then to see that the department themselves or those managers are doing some reviews and you know and then we kind of get multiple hands looking at it and um maybe you know catch some inconsistencies or things. I don't know if that would be a possibility. Yeah, I think like u through the chair um a committee member Peterson I think that that's accurate because if it's IT employees then probably they have a good handle on who needs to have access versus who don't but for the other departments I think that's the way to go about because it doesn't know which but the one thing I want to mention about the service now is like all the employees has access to uh this system so it's not based on the needs like even for the residents audit committee member like there is a separate account for all the because if there's any um issue with your email and whatnot, right? So you have to call in. So so it's a little bit different than other systems because for other systems those are based on needs and maybe they are especially working in those areas that's why they need access to it. But for this one it's more of like all the employees whether like parkboard or city employees they all have access to this system. So, it's a little bit different compared to the other systems. >> Seeing no further questions or comments from committee members, I'll direct the clerk to receive and file this report. Uh, and I will now direct staff to publish this report. May I have a motion? >> So, moved. >> Second. >> All those in favor say I. >> I. Oppose say nay. >> Eyes have it and that carries. Uh, thank you, Mr. Puel. Thank you. >> Uh, up next is item number five, the shotswatter performance audit report. We will be joined by Michael Kwaters, director of special reviews and advisory services. >> Actually, Mr. Timberman. >> Thank you, Chair Payne. Uh, I'm just going to give a little background on these two audits. Um, and then we can invite Mr. Kwaters up. Uh, good morning members of the audit committee. I'm Robert Timberman. I have the pleasure of serving as the city's, uh, city auditor. Uh today we are presenting two reports that examine how the city of Minneapolis uses public safety and surveillance technologies and why careful independent review of these tools is essential to maintaining public trust and ensuring real public safety outcomes. Across the country, cities are grappling with the same question that we face here in Minneapolis. When new technologies promise faster responses, improved detection, or better data, how do we verify that those promises translate into meaningful results for our community? And just as importantly, how do we ensure these technologies are deployed responsibly with clear oversight, transparency, and documented safeguards? The reports before you today speak directly to these issues. The first audit focuses on Shot Spotter, the city's acoustic gunshot detection system. Minneapolis continues to rely on Shot Spotter alerts for police and emergency response. Yet, the city lacks independent mechanisms to verify the accuracy of those alerts, evaluate whether they meaningfully improve public safety, or ensure that their use aligns with city policies and community expectations. concerns from residents, advocates, and national research, including a recent July 2026 independent study, which found no significant reduction in gun violence attributable to gunshot detection systems highlight the importance of scrutinizing both the effectiveness and broader impacts of these tools. Our message here is straightforward. When a technology carries significant implications for surveillance and policing, it also carries an obligation for rigorous oversight. Right now, the city does not have the systems in place to meet that obligation. The second audit looks at data security practices across city's camerabased surveillance technologies. While Minneapolis does not use certain private systems that have raised regional concerns, such as Flax Safety's license plate reader systems, the public's reaction to those systems and more recently to questions about federal access during Operation Metro Surge underscores a growing community expectation. Surveillance technologies must operate within clear, consistent, and transparent safeguards. Our findings show gaps in contract oversight, inconsistency policy application, outdated records management requirements, and uneven training for staff who are responsible for data governance. These are structural issues that if addressed can strengthen trust and ensure the city's technologies are used responsibly. Together, these audits tell a broader story. Technology alone cannot build safety. Thoughtful policy, meaningful oversight, reliable data, and clear accountability are what make technology trustworthy. Our recommendations today aim to support the city in achieving those goals. Now, I will hand it over to um Special Review and Advisory Services Director Michael Kwaters. He has uh taken the lead on both of these reports and will present the findings and recommendations. Thank you, Rob. >> Chair and committee members, good morning. Uh, my full name is Michael. Please just call me Mac. It's more than fine. Okay, let's get started. I know that you're anxious to hear what we found. So, traditional structure here on the presentation background. I know a lot of you have information about sound thinking and the shot spotter system. So rather than going through the full background which we record in our report, let me highlight something specific from this slide and that is currently uh as the city has implemented this system. Uh sensors cover areas in both both north and south Minneapolis. The reason why I want to highlight this is because those locations are data driven uh approximately based on volume. Uh and that means that they do not align with the neighborhood. That means it crosses through communities uh which is of course significant when you're trying to compare data because there isn't a clean comparison even within the community itself. And I I think that it is important to the public to be considered as a community as well as members of the city of Minneapolis. Um, so it isn't just the general public, but how they experience it, of course. But, uh, as an auditor, I'm very interested in the data. And when you cross through communities and you don't get that clean comparison, uh, what I'm really talking about in terms of data analysis is [snorts] you have problems with variance where the variance isn't consistent even within a given community. And so that makes regression models more complicated. When you perform that type of analysis, you would have to take on certain assumptions. So again, it there's playing the concept of accuracy even with something as statistically significant as a regression model. So I want to highlight that in background because it did affect our audit. It's affected other audits. Um our audit objective uh of course was focused on the city's implementation of the system uh and is shot spotter achieving its intended public safety outcomes. Uh I'll touch on that again in a second. But pursuant to that objective, let me talk about the scope that we set for this audit. Uh we focused on processes, city processes. We of course don't have perview over sound thinking. We audit the city and so we looked at the design of the controls related to the city's implementation of this system. Uh it we got really great collaboration from many stakeholder departments OCS, 911, MPD, IT. I mean they made themselves as available as possible. Um there is not a a structured query for KPIs related to this system. Uh there isn't an established data table specific to the system. Uh so we had to uh quilt together data and they did a great job of giving us access uh as much as they could. Uh but because of the challenges with analysis, we had to perform other types of audit tests including gap analysis uh which is really important for the processes that we looked at including policies. Uh and we relied heavily on our SMEES. We interviewed over 20 of them uh all the way up to the uh then chief of police uh the deputy commissioner of OCS, the director of 911 [snorts] uh and many of whom we uh interviewed repeatedly. Uh let me also touch on this slide about uh the period of review. It is significant. We limited our period of review so that we could have as controlled a comparison as possible. uh we looked through calendar years 23, 24 and 25. So we looked at uh some data from before the 2024 changes per the amendment and then we looked some at some data post those changes. Uh but because of challenges with uh data in time, we had to rely heavily on criteria that is established in uh directives and documentation. So, we really relied heavily on the contract and its various amendments when we were trying to uh scope out our requirements uh which I'll talk more about how we implemented those requirements in our attribute testing during this audit. And then we had to rely heavily on the documentation that comes from the stakeholder departments uh policies and procedures in including uh MPD and 911. I touched on many of these points already about the methodology. I do want to touch on something else. Uh we did a judgmental sample of one event per month during the period of review. Uh that's 36 events. That sounds like a very small sample, but most of our tests were compliance tests. So a small sample is actually okay because if one equals no, then you don't have 100% compliance. uh it would be tempting to think that maybe if we had sampled a larger group that it would be more reflective but without having the complete data set you can't actually say that it's reflective of reality. So um a larger sample may not have been as useful as uh one might think because of the way we structured our methodology. So the the tests were purposeful uh so that we could look at uh practices that were uh occurring in the city. uh as I touched we did uh on in the last slide we did attribute testing where we essentially took the requirements of various policies procedures and the requirements of the contract and we looked at those 36 events from our judgmental sample and we just went straight down the line how' they perform so I talked about the objective the scope the methodology and the basic conclusion that we record in our report is that uh we did not review evidence that demonstrated the shot spotter system is achieving all intended public safety outcomes. So we saw where it achieved some, not all. And because of the challenges with data, it's difficult to get more specific than that. Let's talk about the findings and recommendations because we believe that this is a significant report and that our findings are accurate and reflective of city practices. Uh the first I want to draw the distinction between the first and second finding as we dive into the first finding. Uh we write that the city relies on shot spotter data for performance oversight. We also talk about how Shot spotter data is implemented for performance strategy and operations. So that's the second finding. I want to draw the distinction that the city is responsible for uh contract oversight and that's what we're trying to communicate here. Now some of the practices related to city oversight we know do occur. uh they meet at least annually with the shot spotter sound thinking representative. Uh we know that there is uh coordination between multiple departments including OCS, MPD and IT where if there are issues with the vendor uh it can get escalated. Um some of that was difficult to gauge the consistency. Um and even more so uh we want to highlight here that a lot of that coordination and a lot of that effort relies on what Shot Spotter tells the city. So the reports related to system performance which would be evidence of uh fulfilling contract requirements comes from sound thinking and uh we did not review uh comprehensive programs that either validated that data or ensured its accuracy and so we we want to make clear that it's they're relying on reports which is not a judgment about whether or not those reports are actually accurate. Again, we don't audit sound thinking, right? So, I I couldn't say and we didn't bother looking because it wasn't in the scope of this review. What I am commenting on is uh we didn't see independent effort by the city for validating that information. Many of these points I've touched on [snorts] uh and look at the second bullet point in the first sub bullet there. Uh the city does have data systems. Uh it has data systems for 911. It has data systems for MPD uh which include reports and raw well not raw data but uh data that's entered that would be analyzable. Uh so there is an opportunity for the city to independently validate those shot spout reports for the purpose of contract oversight. Um the contract has certain KPIs written into it related to uh location accuracy which is essential for MPD operations. Um timeliness of response to inquiries. Again we know these things occur. We just didn't find conclusive evidence that uh it is done in a comprehensive way by the city. And our recommendation of course it's essentially a follow-up to that finding which is uh there if the various stakeholder departments think that it's best for the purpose of independent verification to use city data we recommend that they do so. Um we recommend that they establish uh documented performance oversight practices uh that include communication across departments to ensure timeliness uh in terms of follow-up with the vendor. So sometimes when something is an issue or is an apparent issue and I'm not speaking specifically to sound thinking here. I'm speaking generally about contract oversight uh where there are opportunities uh for improvement or questions or concerns. the the easiest and first step is vendor engagement. Uh we know that IT liazes with sound thinking. We know that MPD liazes with sound thinking. Um having it structured to ensure like timely follow-up would be a boon. Um and then uh the last bullet on this slide I want to draw special attention to as well. Um during the course of our audit uh we did not review uh a comprehensive list of sensor locations uh and we recognize that there's a balance between security uh but also contract oversight. So we recommend that the city maintains a complete list of shot spotter or sensor locations uh which is essential for the purpose of contract oversight related to the currently documented requirements. Okay. Hey, finding two. The city relies on shot spotter data in public safety response and operations. Gez, that sounds like similar wording to finding one. As I mentioned, there is a distinction and the distinction here is okay. So, not just the performance of the system, but the performance of the city as it implements the system. That's what we're talking about in finding two. Uh and it we highlight here that the city is not collect collecting sufficient independent data to conduct oversight of the city's shot spotter program. The city's shot spotter program. Um some of that might be from our review of the directives. Uh and we highlight in the report where we think there may be gaps in processes that should those gaps be changed or closed that maybe reporting would be captured in a way that the data would be more meaningful for the purpose of cities the city's oversight of its uh shot spotter program. Uh so looking at those policies for where the process allows something that maybe doesn't get structured as a report or is a decision point and that decision is not normally documented. Uh that's what we're referring to in this bullet. Uh continuing on with our finding, um there are other uh metrics that would be useful for the purpose of uh performance oversight related to the city's implementation of this program. Uh whereas you could look at data that is established and being collected. So rather than talking about the gaps in policy where data isn't collected, we're talking about where data is currently established uh for the purpose of following up with the vendor or following up with uh various city resources. Um here we could talk about uh for example final disposition data uh which is a coordinated report between 911 and MPD and MPD being the field resource does field verification of what actually is occurring and what actually is occurring gets recorded as a final disposition data and there's an opportunity to use that data for the purpose of uh oversight related to things possibly like uh false positives Again, where we see some oversight of the program, we did see coordination of false negatives where an event that was not started by a shot spotter notification and MPD did respond and find responsive information. They could follow up and do follow up on a case-byase basis with sound thinking to see if there's more shot spotter data available. That's a false negative. uh there are other metrics that perhaps they could use similar uh field level information for the purpose of oversight. Um we also want to talk about uh the way that the city implements data for the purpose of its shot spotter program strategy driven strategies. Uh for example, there are MSTEP meetings where essentially uh public safety leadership comes together to talk about strategies on uh various goals weekly or monthly etc etc seasonally. Um those MSTA meetings uh we know occur we know shot spotter data is presented where we saw slides that were examples of those presentations. The data that is presented is bariate. So it does not distinguish shot spotter uh and its influence. Um it's rather data that uh you track uh comparatively but uh there are other factors that they could document or analyze that would uh isolate shot spotter for the purpose of strategy planning. Uh in in the examples we didn't see that they did that. We saw by varied design. So something like a volume over time for example. Um that being said, we recognize that in those meetings that uh information is presented. Uh we didn't see documentation that that information drove strategies. So that is a careful distinction I want to make in this presentation. [snorts] So recommendation two, again it this is a complicated system. So our recommendations are really for the city to find out how best to implement them. Uh so we tried to create flexible recommendations that would be useful. Uh so they essentially mirror our findings. Uh there are a couple of different ways that you could analyze uh system produced data for the purpose of strategies and operations or uh even if you want to accept uh shot spotters reports verifying that they're accurate. One thing you could do is of course request raw data from the vendor and deidentify that data and perform your own level of analysis to ensure that it's statistically significant. Um but you could also again use established city data um and use that to validate the shot spotter reports should the city wish to consider those shot spotter reports when formulating strategy. Maybe they don't. Um we would also recommend reviewing alignment between policies and practices uh so that documentation of response to shot spotter activations can be used in performance oversight uh and ensure that uh city documents that the city documents uh when and how verified shot spotter data is used in those strategies. So this was a very large audit in my opinion. Uh that's a personal opinion. Um it's a sensitive topic and uh we uh got really great responses from it was a coordinated response and uh the management response letter which we attached to our report. Uh the individual departments uh made commitments through the management action plan. Uh we highlight several significant uh items here from the management act action plan. Um, but generally I want to highlight for you that they're talking about coordinated [snorts] a coordinated plan that follows the recommendations for the purpose of responding to our findings. Of course, our report is more detailed. Please do a careful review. Uh, but in the meantime, if you have any questions for me, I stand for questions. Thank you for that presentation. Any questions from members? Uh member Alafar, >> I just have two questions >> please. >> Uh the audit was unable to conclude whether shutter is achieving its intended public safety outcomes. >> Mhm. >> What specific measurable KBIS will the city establish so that we can make that determination in the future? And what about the KBIS in the contract? >> The the second part is what I can respond to. I being an auditor, I would have to remain independent and objective and say future planning would be the city's responsibility and you know, city leadership I know through the response of the management action plan is committed to establishing the first part of your question. Uh the second part of your question about what KPIs exist. Um some of the clearest examples I've already touched on. So timeliness from the vendor uh timeliness in responding to inquiries uh timeliness in providing notifications uh those KPIs are written into the contract and I'm not a lawyer but as we understand the contract uh those have been uh KPIs that were established several years ago. Um also uh related to uh accuracy for location data. Uh and location data we saw like considerable and by considerable I mean operationally significant considerable utilization by investigators. So in MPD the folks who do detective work uh those investigators rely on that location data for evidence recovery and then evidence recovery is a great way to build cases. So, uh, there's high utilization from investigators and even with the accuracy concerns, uh, it isn't as significant a concern to investigators because if you found evidence, then it was accurate. Uh, so there's a a little bit of practical response uh, built into that. That probably is part of why there's high utilization. um we didn't study utilization rates per se and we didn't investigate in our root cause analysis why there might be higher but I'm speaking in a general informed way um but also uh location data is reliant on sensor location and so as I touched on in the presentation uh the city to conduct appropriate con contract oversight would have to be aware of those sensor locations. Um so that they could uh when engaging the vendor about concerns related to individual or trending uh accuracy reports or the accuracy of location reports uh they would have an informed position for the purpose of engaging the vendor. So those are uh probably the clearest examples that are established in the contract. >> Thank you. My my second questions is regarding also this sensor locations. >> The audit recommends that the city maintain a complete list of shock spotter sensor locations but management says it will only evaluate requesting that list. If the city needs this information to verify contractual coverage and eventual decommissioning, why isn't obtaining and maintaining a confidential inventory a firm requirement? >> That's a great question. Um, I don't know that they're not committed to that. M um I would imagine that what they need to review is that balance of security and it's security of operations. You don't want biased response. Um if we know where the sensors are located and there were bad actors, there could be biased response. Um and so I think that a responsible department would take that into consideration and seek careful counsel related to it. Um I think of course you don't want uh the folks doing the shooting to know the sensor location. So you know again in terms of balancing security it includes information security. Um and I think that uh there are operational necessities. Uh I'm not an engineer but there are engineering concerns about uh ensuring appropriate coverage so that you can maintain your service level. Uh so where that sensor location uh list is housed, who reviews it, and uh how often it's updated, I think is something that is so uh uh sensitive that they're being careful about approaching that recommendation. So >> thank you. >> Thank you for the question, chair. >> Uh member Truckai. >> Thank you, Mr. Chair. Um Director Kwater, thank you for this this presentation. um and [snorts] the the supplemental report um a few things stand out to me as areas of and I'm putting this mildly significant concerns. Um, so I'll start with asking, do you know if management, and by management I do mean the office of community safety, the police department, or 911 here today to be able to speak to management reaction and response? >> Uh, do I know >> like are they here right now is what I mean. I don't see >> I know that several department representatives are. Would you like me to call them up to? >> Yeah, I don't know if they're in the overflow room, but I do have some questions for 911, Office of Community Safety, MPD. So, if they can make their way into the chambers, that'd be really great. That's my cue that I'm going to have some questions for them. But I'll continue with asking you questions while we wait, >> please. >> Um, something that really strikes me is is this conversation about not knowing where all the sensors are. >> That scares me. um that that here at the city we don't know where all of the shot spotter sensors are, but we're making the argument that the sky will fall without Shot Spotter. That seems really inconsistent to me. The sky cannot fall without Shot Spotter and also we don't know where Shot Spotter is. Seems weird. I I appreciate your concern and I uh don't want to communicate in a way that seems difficult, but I do want to draw a distinction that we purposefully report in a a a very specific way and and that specific way is purposeful in that we're saying we didn't review it. So uh you know during the course of our review and as related to the period of review we didn't see evidence of a comprehensive list. So that that's merely what we report there. There are limitations to what we report in that statement of course but that is what we were able to observe. >> Gotcha. I I just meant in in finding number one recommendations, maintaining a complete list of shot spotter sensor locations. And you know, there are maps I've seen the administration present that show this is where we're expanding. These are uh like individual unique dots that indicate where the expansion is going to be and where exactly we're going to place sensors on different intersections. So, >> okay, >> sometimes that data exists, but you had unprecedented levels of access to each of these data systems. You know better than than anyone else does, and your auditors know better exactly what data exists, how it is used currently. And and so I actually I would trust that as the word of truth, the independent word of truth. >> I greatly appreciate your support. I think that your last point I I would uh reiterate that it's independent. It's objective. I wouldn't say that we know more than anyone. I would say, you know, a careful review of the evidence we were able to obtain. This is what we found. >> That's helpful. >> Um something that really strikes me, uh and and this isn't directed towards you. This is this is really ultimately my question for the administration here. Um I know that there are really rigorous controls in contract management um and contract oversight mechanisms that exist in certain aspects of public safety response. For example, um in our violence interrupter and violence prevention work, >> um we work with a lot of different vendors on on that body of work. We don't pay invoices until we independently verify the the data that data set that each vendor presents to us. So I see a high level of rigor in making sure that a safety strategy is working in certain areas, but in others we just take the police department's word for it. I don't I don't know of a better way of of putting that. And so it it really does concern me that we just we just believe whatever sound thinking tells us without any independent verification. Am I understanding that correctly? >> I I think so. You asked that the administration respond to it, but I'll give you my two cents because most of my career in public safety was on the operations side. I of course audited specifically in public safety as well, but a lot of it was on the operations side and a lot of uh my work in policy was in formulating some of those more proactive responses uh that you're talking about. And this is a different system. It's a an entirely different tool. uh where you're looking at something like violence interruption uh and you look at the national models that exist for those violence interrupt interruption strategies. It's more on the preventative side and uh shot spotter and gunshot detection in general is uh detection I mean and the notification is for response. So there are different sides of public safety. Um I can comment on that and I can comment on where we report we did observe oversight uh it was oversight for some of the things that you're talking about and the example that I highlighted in my presentation but I'll reiterate uh because I think it's useful for the purpose of this conversation is uh we saw where city analysts contacted Shot Spotter because they believe that they did not receive information and that's a type of oversight and it's reliable. I mean if you know your field resources made direct observation it's pretty reliable QED and uh then the analyst would follow up um and we saw examples of that communication so we know it exists. Uh those communications we report that we did not see evidence that was centrally tracked. Um so again there the city uses it the city through its management action plan appears to commit to using it in a way where they would get more out of the system. Um so I if I communicated that they weren't conducting oversight I miscommunicated and that's my fault. We know that they do some uh we believe that where they and Shot Spotter is a service and so of course as part of the service they provide reports where there are those reports we believe that the controls as designed would function better if there was independent verification of those reports. >> That's really helpful. Um I don't see anyone from the office of community safety the or the Oh, I do see someone from the police department. >> It's probably because I'm so talkative. Please let me invite them up. I've got chief of staff Cleti and uh I know that it was a great partner throughout the course of this audit as well. Uh so where we need we may reference them as well. >> Helpful >> chief. >> Good morning chair uh pay members Lesie Cleti chief of staff of MPD. Um I wanted to first thank city auditor Timberman and Auditor Kwaters. Also auditor Cam who put in so much time and effort into this. We do appreciate working with us along the way. Um I had a very limited um time period during the January 23 through December 2025 period. Um I am going to be somewhat limited in my ability to offer a thorough subject matter um approved responses. So, I would ask that I'm happy to receive questions, but I would like to respectfully request that we can provide a staff response memo to give us the time where I can work with OCS uh management as well as the subject matter experts within MPD and not provide more anecdotal responses that come from my more limited understanding um generally of the technology. Um, and as I was on the tail end of this audit itself, >> that's helpful. I mean, I think my questions for you are help h helping to understand if the city um and by that I mean 911, police department, OCS. I'm going to just that's what I'm going to mean from this point on. Um if we maintain a comprehensive list of s sensors placed um throughout the city and coverage areas. Um what type you know just like going through these recommendations um you know just a standard management response to these recommendations would be really helpful to to understand um both current landscape and uh future planning. Um, but I would say generally speaking, I am really really concerned that we are um we're taking Shot Spotter data and assuming it is the word of truth. It is like the most accurate truth uh about gun violence in our city. Um, so it would be helpful to understand what type of independent verification you're doing um to better understand both false positives and false false negatives um and and how that impacts operational decisions. I do understand Shot Spotter data along with 911 data heavily informs um safety operations and how uh things like police response are prioritized. um what communities uh response is is prioritized in and so how are we even verifying the input to determine what kind of response is needed. Um hopefully that all makes sense. That's my happy to take a a response memo. >> Thank you member Chug Tai. >> That's all Mr. Chair. >> Thank you. >> Any other questions? Uh, member Hosby. >> Thank you, Chair Payne. Um, was there any I know the answer to this. This is a silly question. I should just say it. um uh in terms of being a community member on this committee um and reviewing the public facing like what I can see um for Shot Spotter data and then knowing that there's really doesn't appear to be within your audit strategic initiatives kind of based on this and for a nice total of $3 million a year. Um, I see when I open the dashboard that every half block in this city has a dot. My kid's school, my house, my place of office, every single dot, every half block for the most part. Um, the blocks are bigger, the dots look more spread out, the dots are smaller, it looks more um, I live in a place with a block is three streets, right? So, it looks like, oh, maybe there's a little bit less gun violence here. not a reality, right? We are not going to put this the spot. I also see, you know, if I look at a couple of my key places that I go, those records are from 2013. They're from 2008. They're from other times where I don't know what I'm supposed to do as a community member for this data. So, this is more just a comment for those who are providing the management response. What is this doing for community safety? and when my kids walk to school, what do I do with this information? Um, and I feel like it's a really easy search that pops up at the top of the city website, the top of MPD that I think maybe makes it fearful versus helpful. And we're in a time in the city where we don't need fear. We need data. We need ability to have a a map that shows progress and change and diversity and all these things. And just hearing all these like, okay, a shot spotter is running right through the middle of a neighborhood that has been just decimated in the last six, eight years. How is that helpful? We don't have any strategy or ability to look at that and make sure that we're meeting objectives there. Um, so like I said, not really a question, just a comment about the community response on this in terms of if it's not helpful for the city. I don't think it's helpful as a community member. And I think in a day two where we have such scary surveillance, a few more things on light poles. Like I'm not saying it needs to go away, but I think we need to have a reason and we need to tell the community why we should be having this and what data we should be using for this to help our community. >> Chair, may I respond? >> Thank you, chair. And committee member, I recognize that was a comment. I appreciate your comments. Um, I do want to respond though because I have to compliment the people that formulated the request for this uh audit. I and I'm going to nerd out a little bit as an auditor here. When you're doing when you're talking about technology that interacts with the public, there are several facets that are normally not considered that should be and uh they things like uh psychological effects and uh the experience of the technology uh and then the solution of the technology and these three key areas were written into the request itself. And I was uh as somebody who's been an auditor in public safety for quite a while, I was very impressed at how that request was formulated. Um experience is normally uh calculated and audited in terms of value and uh valuation models aren't useful in this case. Valuation models are useful in dynamic environments. uh valuation is not useful when you're talking about human life and public safety. No one would want to put a value on that other than the highest. So in terms of auditing and and your in a way that would be responsive to your concerns, I would say that wouldn't be possible. Uh we were very careful in the design of our audit so that it accurately reflected the information that was available during the period of review. And the purpose of uh reflecting practices and events is that the members of the public, the people of Minneapolis interpret that information for the purpose of contextualizing their experience. We would not want to inform their experience. We would merely want to provide information for them that is useful. And uh I think you touched on uh city leadership's response to our audit. Uh and of course we love when people are responsive through their action plans. uh you know uh and I think that they is um a commitment there to responding to our findings and and I hope that the information we provide them is ultimately useful as well. >> Yeah. And I appreciate that. I an audit nerd always nerding out is always appreciated and I have to sometimes put my audit hat to the side and say I'm here as a community member of how this impacts me and I think for your audit the comment wasn't for that. I think the audit was probably or my comment was more for right behind you and how this is being received by management. Yes. >> Um and I think >> perhaps making sure that we're going back to that request and I love that it included I don't love it. I think I really appreciate that it um included the psychological and kind of community ties of this. Um and so I think it was like maybe more out past the audit. Um so I really appreciate that and I um and maybe that's where that gap is being felt to between like what can we do here which is the audit and I tr I don't actually have any questions about the audit. I really appreciate your comments around it. it is about the response and then the next steps. Um, and how that moves to is it city council, is it? And I think this is a lot of questions we've had. Um, and then the other piece of just we have a lot of services whether it's corporate or city that don't have a public need like they don't have a public face that is needed. We don't have public dashboards for a lot of things that the city does. Why is this one useful? And I think that's something that should be evaluated when it is something as much as shots fired in my neighborhood, shots fired in a neighborhood that we want we want students to come to our public schools. We want Minneapolis to be a great but if they're looking and going well I can't be here because there's a million blue dots around this. >> What's that doing? So I just think there is a public facing piece to this of like what is useful and if it's not useful for internal strategy is it even need to be a dashboard on the website. >> I appreciate your comments. Thank you. >> Any other questions or comments? Uh well I will note that um on city council we extended the contract for Shot Spotter. Uh they were requesting a three-year extension. We gave a one-year extension because we knew this audit was coming. Um, it's really good to see the uh completion target dates for some of these recommendations being the end of this year. That contract is up for reconsideration on March 21st, 2027. And I think that, you know, as someone who's skeptical of the value of these technologies and I think this audit reflects that skepticism, I don't think this has put that skept skepticism to bed. I I think it's going to be critically this is a comment more for management. Um I I I'm not seeing a reason to renew that contract with what we see in front of us today. Um we see some recommendations on how we might be able to independently verify the um efficacy of this technology for the the stated safety goals. Um it's going to be a really high bar to clear for me to actually see how management responds to uh identifying that independent uh performance of this of this technology. So, I'm I'm just very grateful for this audit and thank you so much. And I think this is a you know, I missed the last audit meeting, but there was a robust conversation about how we translate actions on audit and those reports and how they translate to um effective oversight within this institution and how city council can be a partner in that and I'm seeing this as a really key opportunity to identify that. So, thank you. >> Thank you. I appreciate that, chair. Thank you for your comments. Uh, seeing no further questions or comments from committee members, I will direct the clerk to receive and file this report. And I will now direct staff to publish uh this report. May I have a motion? >> So moved. >> Second. >> Uh, all those in favor say I. >> I. Oppose say nay. >> The eyes have it and that carries. Up next is item number six, which is camera data breast practices advisory report. We will again be joined by uh Matt Kwaters, director of special reviews and advisory services. >> Yeah, bad news everybody. I'm going to stay up here and keep talking. Um this is a very different audit. Uh whereas the shot spotter audit was a specific technology, this is more of a landscape type audit. Um and I hope that it is equally useful. Uh I want to uh thank you all for the request for this advisory service. Uh this was another advisory service effort that was uh it was only capable because of great collaboration from the stakeholder departments which were many public works and regulatory services and IT of course and uh MPD and it just it was a a lot of people involved and uh we appreciate their partnership throughout the course of this effort. Uh well, it's an advisory service. Again, it follows the same basic structure as our audit reports. Um very brief background. Uh we looked at the contract landscape for camerabased surveillance technologies. And there are some that uh oh, let me go back. Um I want to comment that we recently reported on and so they were purposely scoped out. So MPD's use of BWC uh was already reported this year. So we did not recreate that report. Uh same with ALPRS MPD's use of ALPRs. We did not recreate that report. Uh but we did look at the city's other camerabased services um that were contracted for the most recent period of review that we could establish. Uh I believe if memory serves this uh advisory service was accepted by the city auditor in April. So we looked at March of this year to March of last year. So we tried to essentially obtain that the most recent data that we possibly could. Um we also used I mean a great emphasis in this report is on uh best practices and we hope that's a useful product of this report uh and most of that is based on uh national leaders in uh information security and data security uh and some of those include uh private functions that uh have a great interest in data security and they publish uh frameworks and recommendations. Uh we also looked at governmental frameworks and recommendations and uh we considered that through the course of this advisory service. Our objective was to uh review what are the city's current public data security practices and internal controls. Our review was heavy on the internal controls. Uh and then the second objective was what established best practices could strengthen the city's current surveillance data security practices and internal controls. I mentioned the scope. Uh this is a list of the departments that participated. We thank them for their participation and the access to the information they provided. Um because this is a landscape type advisory review. Uh we took a slightly different methodology. Uh we relied very heavily on SMEES in this advisory review to help us collect the information that we needed to review. Uh the SME that we identified are data and records liaison. You may or may not be familiar with that role. It's a formal role that is assigned to uh at least one member of each department. We did find each department that participated had such a person assigned. Uh and we relied on them to help us uh gather information and then we interviewed them as SME as well. Um I mentioned we did process mapping that's recorded in our report. uh we did comparative analysis, but let me specify that by saying comparative analysis of uh best practices the city currently implements versus what further best practices they could consider that come from those national organizations I talked about at the top of this presentation. So and mostly what we mean is IT controls over user access, logging, monitoring and change management because we looked at the contracts. We looked at services but we also looked at the hardware. And so there are essentially like three general categories we consider hardware which is deployed and then the hardware and technology associated with transmission of data. So pointto-point security uh transmission security and the access points along the way and then uh data warehouses where that data is stored and the security and access controls associated with those various data warehouses. And our conclusion is the city has automated technology processes and departmental policies and procedures in place that safeguard the city's surveillance technology. The city's data security architecture can further incorporate established best practices and standardize these practices across the enterprise. There are more findings here. They're shorter. There are more. Um, finding one is uh related to uh public surveillance contract oversight. Um, essentially we see oversight. Uh, we see different contracts and some of that may be related to when they were contracted, but uh, some of the TNC's that are written into those contracts vary. Uh, it depends on the department and when the contract was established. Uh, in addition to contracts, uh, we know that there are agreements. Uh, there are agreements for coordinating with other law enforcement agencies. We weren't able to obtain agreement information during uh our fieldwork. It was a very short period of fieldwork. Uh so we weren't able to essentially review the scopes that might detail in those agreements data security requirements. We mostly looked at the data security requirements that were specifically written into the various contracts, 19 of which were reviewed during this advisory service. [snorts] And our recommendation is simply that the city ensures that contract provisions are enforced for all camera services vendor services uh related to the collection, hosting and sharing of public surveillance data. >> Finding two is about policies and procedures. Uh it is very readily available for internal departments to the city. Um it's an IT directive. It's called the information security policy. Section 11 specifically details many many items about uh recognizing, notifying, and reacting to um attempts to access city data from camerabased networks. Um it's detailed there. It is responsible for keeping it up to date so that reflects technology. um and the security needs that come with modern technology. Um but I do want to highlight while there were directives across the various partner departments uh they were different they were incomplete or insufficient there was an insufficient level of detail uh recorded in the directives that we reviewed. So looking at the department level policies so that they uh better memorialize or at least reference uh the information security policy would be a a good way to help ensure better data security. [snorts] Um we observed again in our recommendation we detail reference to MPD's policies and procedure manual. uh many of their procedures have specificity related to other requirements that exist outside of their department. Uh but we also saw in the same manual uh references to coordination with other federal law enforcement agencies. Uh we encourage them to further define the user access reviews in those policies. I mentioned the data and records liaison. uh there is training available for them. Uh we found that access to that training was inconsistent and uh we didn't find strong enforcement of those training requirements. Uh this is a really great resource for each department. uh this person could or could not be an IT expert, but at the very least they're an expert in what resources the city has established that the their department needs to access when they're considering technologies, implementing technologies, reviewing their directives to ensure data security related to their technologies. These liaison are a great point of contact for event response, event notifications, escalation, etc., etc. Um so we encourage that they uh have more formalized training. Uh and by more formalized I mean uh there's consistency uh related to the duties of their role. Uh that there's consistency in the adherence to data governance policies and there's uh consistency in the access to the resources that would uh inform this role. Uh we also recommend speaking of uh established resources in the city there's the IGPC um the information governance policy committee it's a it's a team that meets at least monthly of experts that uh say a data and records leaison could meet with to gain further insight or consultation. So we encourage uh engagement of that body especially for something as important or sensitive as camerabased surveillance technologies. Uh finding four, many of you have heard of this before. It's section 1305 of the Minnesota statutes. It's the Minnesota government data practices statutes. We saw directives that reference this. uh we did not see uh the type of memorialization in the directives that would communicate the requirements of that statute in the directive itself. So we could be more specific there. [snorts] We recommend that uh the city review policies and procedures to ensure compliance with the statute consistently across the enterprise. Where there are questions, of course, the city could leverage existing resources like the IGPC to accomplish these efforts. Uh and then data security guidelines. Uh this is uh findings spec that specifically reports uh the city should consider uh data security best practices. Uh we recognize that they have many of those practices in place. uh it's more so establishing structurally in their operations a review of those best practices so that they can be updated over time so that they can consider the full landscape of technology and the way those technologies interact. They can consider what the city's most recent needs are. uh the best practice resources are out there and formalizing uh a review of them and uh consideration for adoption would be useful to the city. Uh we believe uh the management action plan is uh currently being uh developed. Uh this was again a very short review uh for the purpose of our advisory service. It involves 19 contracts, many departments as you've seen. It's extremely intricate and we know that the uh city is carefully considering its response and uh the uh corrective action items that they would follow up on. So, it's in development and uh the city auditor will the office of the city auditor, excuse me, will continue to engage those departments as the corrective action plan is communicated and then we will follow up through our normal audit uh follow-up. So, uh if that presentation felt relatively brief, it is because our advisory service is meant as an overview service. Uh we hope that it's mostly informational. We hope the best practices that we recommend to the city are useful to them. Uh however if there are questions or comments I'm definitely here to stand for those questions or comments. >> Thank you for that presentation. Any questions or comments? Uh member Moawak. >> Thank you chair and thanks for the office of the city auditor for this report and thank you for the presentation. Uh my first question is the audit states that the auditor office requested but was unable to obtain a comprehensive list of data sharing agreement or MO use with federal law enforcement agencies. Does such a comprehensive inver inventory currently exist? If not, how can the city be confident that surveillance data is not being shared outside established authorization and oversight processes? Thank you for that question and chair and committee member uh I would say and I I would say from an audit perspective we don't know uh so comprehensive is difficult for me to uh report at this time uh and by we I mean specifically the team that performed this advisory service uh we didn't observe information that would indicate a comprehensive list of agreements and requirements associated with those agreements um so I don't know. Uh but that being said, uh there are many laws in place. I mean there's the state law, uh there's federal law, uh and all of these uh have separate requirements for data security. Um and accessing beyond those laws would therefore be nefarious. And um so I know that there are laws in place. We've reviewed directives for each department. uh we know those are in place and uh our best practices are to strengthen where the city has purview. >> Thank you. >> Thank you. >> My my second question is the report identified the absence of specific procedures for federal law enforcement requests >> is a matter of kiosity here. For example, what happens today if a federal agency requests access to the city's surveillance footage or data? Who approves the request? What legal review occurs? And is the decision documented and editable? >> It's too good of a question. I don't know all of those things off the top of my head. I'm sorry. Um, there are directives related to who reviews and approves access. I know that there are directives for IT review that you talked about law enforcement MPD review. I think it's system dependent on who does the review. Um and then how those reviews are documented uh and the restrictions that come with the approvals. We did not review it as part of our advisory service. So uh I'm not able to report on that at this time. Um I do think it's a good question. I think that we reviewed uh law enforcement directives that specified it it access and coordination should be in line with uh state and federal law and we saw that uh it should be aligned with uh the city's law enforcement practices. So I that I do specifically remember off the top of my head. I know that's not as useful of an answer as you would like. I'm sorry. >> Thank you. Could I actually request that the clerks um put together a memo for a thorough followup to that question? >> Chair, you were looking for a follow-up from us. >> Um perhaps uh you could be a liaison to the management team for a follow-up. >> Thank you. >> Uh were you finished with your question? >> My my last question, >> please. The audit notes that the city records management policy was last revised in 2002. Yes. Given the rapid evaluation of technology, AI, cyber security, data privacy, and data sharing practices in recent years that the city has an enterprisewide processes requiring policies and procedures to be reviewed and updated on a regular cycles. If not, should the the city establish defined review? frequencies, responsible owners and mechanisms to ensure policies remain current and responsive to technological and regulatory change. Chair and committee member, it's a great question. Uh, and I thank you for it because I think that you hit on something important. [clears throat] Um, it is a policy that we did not assess whether or not it it is sufficient. We comment that it was less updated in 2002 and there have been changes in the technology landscape since then. I mean maybe it's a perfectly written policy and procedure where it's perfectly flexible and captures everything we need. We didn't uh review its sufficiency but uh I agree with you that in terms of uh the design of the controls related to the directives uh a review of that specific policy uh is called for and and we recommend it. Uh but I also would like to highlight that there is uh quite a few more directives that are updated more recently. This was a an outlier and it's why we highlighted it in our advisory service report. Uh and then we have observed a clear interaction from city departments with the city's IT governance architecture and um the information security policy is updated frequently per IT's review cycle which um I I would be happy to have uh any member of IT comment on how frequently that's reviewed but I believe the review cycle is captured in the directive. I'm going off the top of my head, so don't hold me to it. But I think that frequency is uh documented. >> Okay. >> Thank you. >> Thank you. >> Any other questions or comments? Member Hoseman. >> Thank you, Chair Payne. Uh maybe a quick one. >> It feels weird to leave this without sort of some sort of date where you would come back with a management plan. Um is there kind of within your standard audit like this is complexity so we give them 6 months, 9 months, 12 months? like is there a date we can put on this so we can expect a follow-up? >> Yes, chair and committee member, thank you for that question. Uh we in our current engagement with the various departments for their coordinated response are looking at uh the next uh audit committee meeting. We hope to have more information for you then. Um and I know you're using a colloquial way, but we we hope that like we coordinate with these departments and and establishing that response plan. And so it does depend on them somewhat. >> Yeah, absolutely. I think there's maybe the maybe a better way to put it was >> in terms of audit followup. >> Yes. Okay. >> Update date verse they have to do this by December. Yes. >> More like a date for the expected update even. >> So they don't come back in October and say still too complex. We don't know. It's like well then we should keep doing right. So just some sort of structure around that. >> Yes. Thank you. And I I apologize if I misunderstood your question and uh I think that the engagement was almost immediate from the city and the the plan is for the next meeting. >> Thank you. >> Thank you. >> I just have kind of like a general way I think about a lot of these technologies. I think about what policies we have in place. What are the guard rails? But then often these technologies and particularly these vendors like to sell us out of capabilities are that are much more expansive than what our use case is. So there it's kind of like a layer there's layers to it. There's like what's our policy, what's the capability and then what are the vulnerabilities that we don't know about on how these technologies could be uh inappropriately accessed inappropriately used. Um, and it feels to me that, you know, a a a records management policy from 2002 there's certainly is not going to reflect the current technology climate. Um it seems that we should have a similar um refresh on how we think about these technologies at least from that kind of maybe maybe that overly simplified three-part framework is not sufficient but something that tries to capture um a comprehensive a policy approach to these technologies because I I feel like that's a big gap here. I in chair thank you and yes I think uh what you're saying is really good and uh I think that where [snorts] we had recommendations we hope that it's enterprisewide. Yeah. And I and I think um as a as a work product, you know, this is the the advisory services is a somewhat new capability that we're developing. And I think that how this work product gets used by the enterprise and specifically city council as it relates to um we've had a number of contracts come forward with a lot of promises around how it's going to uh you know magically improve the well-being of residents of Minneapolis without a lot of substance to back those claims up. And I think that um in the absence of this types of comprehensive framework um we should be looking at any vendor pitch with a high degree of skepticism and they should be meeting a framework that we establish proactively. So I think this work output should help inform such a policy. >> Well, thank you. I I genuinely appreciate that support and we'll keep auditing, you know, and that's part of the assurance for this. [laughter] So >> excellent. Well, uh, seeing no further questions or comments from committee members, I'll direct the clerk to receive and file this report and I will direct staff to publish this report. Members, may I have a motion? >> So, move to second. >> Uh, all those in favor say I. I. Oppose say nay. Eyes have it. And that carries. And our last item of new business is item number seven. receiving a request from the city council to undertake an independent audit of the Minneapolis public housing authorities use of city funds. Um, Mr. Timberman, did you want to speak to that at all? >> Chair Payne, I really don't have much to say other than I would request that the audit committee uh move forward with this request to our office. We are willing to take on this work. I would however ask that we target 2027 and we build it into our audit plan in 2027. So if you accept council's action and request from the office of city auditor, I would just ask that we add a a statement in there that it's for um to be considered with the 2027 audit plan. >> Uh so uh Mr. Clerk, should I make a motion to uh accept this request with a recommendation of 2027? >> I, Mr. Chair, I think you can just uh make a motion to request the city auditor to do exactly what the resolution stated. Um and then uh the the auditor can then work it into the work plan as he suggested. I don't think you need to put a timeline on it. >> Okay. So, I am going to move that uh the city auditor um request uh >> I'm move to request an independent audit of the Minneapolis Public Housing Authorities use of city funds. Is there a second? >> Second. >> Uh the clerk will call the role. Member Al Muak >> I. >> Committee member Chuktai is absent. Englehart is absent. Obine >> I. >> Peterson >> I. >> Vice Chair Singleton >> I. >> Chair Payne >> I. >> There are five eyes. >> That carries. And last is the report of the city auditor. Uh, and I will recognize Mr. Timberman to give that update. >> All right. Thank you, Chair Payne. Thank you, out of committee. Um, I know we've been here a little while, so I'll try to be as brief as possible, but I do want to provide some updates on open corrective action plans, talk a little bit about what uh our plan is for October, bringing some of the audites in front of the audit committee to answer some questions that we've been discussing. Um, so let's dive in. Uh, as always, we'll go through some of our work in progress. We'll go through uh those closed corrective action plans or closed findings that we've been able to identify. Uh, we're really excited to have been able to roll out the finding uh corrective action plan tracking dashboard. So, I know that some of you have reached out, asked some questions about that. Uh, I think that'll be a great tool for um for the the committee and for our office going forward as well as the administration. >> Uh, Mr. Timberman, uh, I wanted to check in with member uh, Inglehart to see if you wanted your vote recognized for the um, addition of the MPHA audit directive. >> This would be to vote to add it. >> Yeah. Uh yes I I do >> if the clerks could get that noted without objection. Thank you. >> All right. Thank you. Um so the work that we've completed we always identify in these slides just for uh documentation purposes but I won't go into them since we just had some very thorough presentations today. In progress work we've got of course um work continues. we we come to these audit committee meetings and it feels like a finish line and then tomorrow is a is the um starting line. So we we try to manage our work throughout the year and uh this is what's coming up for you in October hopefully. Uh all of our dates are always subject to a little bit of adjustment based on uh response and and uh the work that goes into these uh reports. But we've got the management of afteraction reviews. This was a a request that came in uh in relation to the um Lucer Muri review. Um the the idea here is to determine if there is a place within the city structure to perform that type of review on a regular basis. Uh right now we are doing field work having a number of conversations with the city attorney's office um expanding those out to potentially some conversations with county and state uh individuals and and stakeholders as well. The permitting process audit this is to look at the permitting life cycle from application to certificate of occupancy. uh primarily looking at CPAD with this uh and also scoping it down to single uh residential uh homes because um if we looked at commercial that would be a whole different audit and not to say we can't do that down the road as well. So hoping to have a report for you in October on that. And then finally we've talked uh for the last several audit committee meetings about a BCR audit that we've had in the works. We are currently in the fieldwork stage. Uh we've done some ride alongs with our BCR vendor. Uh and we uh hope to have a report for you soon. Since our last audit committee meeting, we've closed one investigation. Uh we do have two in progress. Um for the newer members on the audit committee, typically we we just provide numbers on these. These are are fraud, waste, and abuse investigations that are classified by the Government Data Practices Act as as non-public. So, we try to provide updates on where we're at in a process or completion or numbers. Um, but typically we're unable to share significant details on these. Um, all right. So, that brings us into uh an update on audit findings, corrective action plans that are still open, maybe past due. Um, as a preview, we've got 68 to total open corrective action plan items. Uh, those are a one:one correlation with audit findings. Uh, currently 44 of them are from 2026. It's not uncommon for the most recent to be higher in volume because they haven't even hit their due date yet. So, and and that's where we are in this case. Um just because they have not hit their due date does not mean the office of city auditor is not actively working to get updates communicating with the audites to um to encourage their meeting their targeted deadlines. Um and so you'll see throughout uh the next two months there will likely be some corrective action plan items that get closed within the tracking database. So if you are keeping an eye on that you may see it. Since our last audit committee meeting, we closed uh in the tracking database the five findings that we validated as a part of the unresolved management corrective action plans report that we presented to you last audit committee meeting. Uh and we've also closed and validated five additional findings which we will cover on the next few slides here. Uh so I will transition to those now. First we'll get into closed audit findings. uh these are audit findings uh as I mentioned that we indicated in that report uh in June that we had sufficient evidence we validated um but the stage of where we were in the development of the database we had just not gone into the database and actually closed them yet. So these five on this slide are the ones that have been closed. These five are new audit uh closures, new corrective action plan closures. Um so the first two of these were items on page nine of that report that was on the the chart saying here's the ones that management has committed to closing this summer. And the second column was here are the ones we plan to invite in October because we've not seen progress on the audit uh corrective action plans. Uh so two of these they've they've met their targeted um goal of closing them this summer. We validated those uh three of them with Helix. I believe these either did not have original due dates or they were not past due, but these uh we have also validated and confirmed to be closed. Uh you'll see in a moment Helix that report still has two open corrective action plan items and so we will be inviting them um to the October audit committee meeting. Okay. So, we uh had gotten a commitment from MPD to provide an update on uh a few of their open corrective action plans. They had said that they would be closed by uh the time that we have this audit committee meeting. um we were unable to validate or or get additional detail from MPD on those open corrective action plan items. Uh that said, we've had active conversation with uh Chief of Staff Clet um and others within the department. And so those items will be added to our October list. We have several open corrective action plan items uh and we will incorporate those into the invitation for them to speak to you about in October. Uh that is also true with the fleet management audit. Okay. So getting into the audit response dashboard, we're going to start slowly taking less or or removing some of this information from our auditor updates because you now have access to the dashboard. What I'd rather do is just have you reach out if you have any questions, if you want specific information, or bring them uh to an audit committee meeting. We'd be happy to dive in uh dig into the original finding or recommendation. Um but because this is becoming so robust and uh there there's a lot of data within it already and we're only looking back a couple years. Um I I want to be mindful of time in these updates. Um but as far as the dashboard goes, a couple quick updates. Um you do now now all have access. We are also connecting this dashboard to another dashboard that the legislative research and oversight division is creating within the clerk's office. That dashboard um identifies open legislative directives for this um for the administration to respond to. Um, Director Hawkins has been gracious enough to allow us to enter our information with open findings uh into that dashboard so that the administration has very clear um communication on what is still open. Those are also public dashboards and so the public will also know what legislative directives, what audit findings have not been resolved. All right, with the dashboard, we're still working on some technical updates. Um, we want it to have some uh automation in its requests that go out to the administration. um our partner in IT who helped develop this dashboard um has has just left the city of Minneapolis and so that work has gotten transitioned to a couple other individuals and so we're not quite sure the timeline yet on when we'll have that and then also some built reporting functionality um but we are continuing those conversations to see um to see when we can have those upgrades All right. So, really quickly here, I'm just going to go year by year of what we have for past due open corrective action plan items because again, these uh these divisions or or departments will all be invited to the October audit committee uh meeting to provide updates. But 2019, we have police off duty. Uh there are two findings that are still open there. Finding one and two. If you remember from June when I said we were going to close finding one, um we based on your uh comments, we have kept that open and we'll include that in the request for updates from MPD in October. Um the next is 2023. There are several that are still open uh from property and evidence. Uh that's also an MPD and and finance type um department that will come to us in October. Revenue and collections audit phase one. There's one finding still open. Uh they did update their remediation date uh which is soon. So that might be resolved before uh needing to come in front of the audit committee meeting or the audit committee. the sexual assault examination kit, uh, special project audit that still has some unresolved corrective action plans. That one I I think we are going to have to take in close session. The property and evidence we may need to take in close session as well. So, here's my heads up to the uh, city attorney and to the clerk's office that we'll have to do some collaboration to make that happen. 2024, we've got fleet management findings that are still open. Um the remediation dates are are not the same for all of the findings. That's why there's a couple bullet points here, so we'll need to get an update from them in October. Uh 2025, we've got a number of findings. I won't go through them um word for word, but they are here in your information if you would like. They're also in the database. And then 2026, as I mentioned, there's none that are passed due yet. On the screenshot here, it indicates that there are three afteraction items that are passed due. The reality is is none of these have actually hit their target date. That's just uh within this system. We've either not entered a target date or we entered the target date of our last audit committee meeting. I I don't recall why this is, but we need to make an update in the database related to those. Uh and that said, um we don't wait until the audit committee to to get updates. So if we get updates that a corrective action plan has been completed, we will validate it. We will update it within the database and then we will at the next audit committee meeting also included on a slide for your awareness. Okay. So I've talked a lot about the upcoming October audit committee meeting. Uh we'll have a few audits to release but we also will invite um uh it looks like six audits that have 17 findings and it will be MPD um probably finance and property services as well as health. I believe those will be the three uh departments that we invite. If there's more, um we'll we'll make sure to invite them as well. But Chair Payne and I plan to get together shortly after Labor Day to draft letters to the departments, formally inviting them to the audit committee. Um the questions are those that were at the end of the report that we um presented last audit committee meeting. uh and this is what we'll be asking them to provide is is really reasonable efforts that they have taken to mitigate the risk. Um if it's a budget issue, what have you done to address that? Uh if it's a cross- departmental issue, what are the coordination efforts that you have taken? If it's related to a policy and procedure, what's the progress you've made on those uh updates? If it's related to tech to technology, what are the roadblocks? um if it is needing just additional resources either from the council or the audit uh the office of city auditor or anyone, what is the holdup? What is the delay? And so we will uh within the letter we will include these and we will expect the department leaders to respond to each of these items. I do know and I will share that there are uh council members uh and chairs of other uh city council committees that are very interested in uh getting updates on some of these open corrective action plans. Um I I spoke with one council member specifically and and just asked that they wait to bring some of these departments in front of their committee until after October because we have built over the last 10 to 12 months this this process to get to where we are. um at least in that conversation um there was general agreement that we would wait to see this process play out. But I do think based on what we've talked about and and committee member Hosine, you brought this up uh today, we updated our audit charter in December of last year to strengthen this committee's ability to move things to the city council. And that's kind of what we're building towards. And so we will see from those conversations in October from departments if there is satisfaction to your concerns about delay and if there is not we will then move those forward through the chair and working with the clerk's office to ensure that they get um seen in front of a city council committee. All right, just a few other administrative updates for the office of city auditor. Um, we've been doing a lot of work internally to strengthen our own processes, procedures, controls. Um, I think that, uh, an audit shop never lives in a glass house and so we need to make sure that our house is in order. Um, right now we have invited the Association of Local Government Auditors, which is an association of of offices like ours. We've invited them to perform a peer review of our adherence to professional auditing standards. They should be uh coming here um in I believe January, if not February, to perform that work. My hope is to have a report from them for you by February, but it might be at the the following audit committee. just depends on their timing. Um but that answers that that age-old question of who audits the auditor and we will get some good feedback and just like our audites uh we hope for some um some critical um review of our work. We hope for any recommendations and and suggestions that they have for us and frankly we hope to have no findings. Um the other work that we've been doing uh is is kind of leading us up to that point. We've completely uh rewritten our own book of policies and procedures and developed an audit manual for our office. Um we've redeveloped our standard internal audit program which is basically the a toz on how to perform a performance audit within the office of city auditor and we've also internally developed a new audit software. So, we've used Power Apps um to create our own internal audit software, which um quite frankly, leading into budget discussions, I will brag, will save the city $30,000. That was not our original intent. Our original intent was to save $30,000 for one year to leverage that into purchasing a new audit software in 2020 um in 2028. My hope is we'll still be able to do that if if the budget allows for it. But um of course we're we're always thrilled to be able to share that we're playing our part in budget reduction. Okay. So the I believe last item is we received a a suggestion um the audit committee received a suggestion from one of our audit committee members. Thank you so much for that. Um, I I want to be mindful that I'm never going to call out an individual audit committee member for any communications, but whoever submitted this, I I would be happy for you to speak up and own this. I wanted to make sure to get it on an agenda so you all could talk about it. Uh, the idea is simply that the audit committee on an annual basis uh have a discussion, maybe even a documented discussion about auditor independence and the resources for the office of city auditor. um to discuss matters related to the independence of the office if we have the resources necessary to effectively carry out our responsibilities. thinking about uh unrestricted access to records and personnel. I think the idea here is to ask the question, are we getting push back on that requirement? Um are we getting cooperation from the departments that we're auditing? Uh do we have staffing and bud uh budget concerns for our own work? And then are we facing any potential challenges that could affect our independence? So, I'm going to pause momentarily. Maybe the author of this request um will speak up. If not, I would suggest that um generally we we do this. So I think it's a great idea and I would suggest that February when we release our um annual audit and strategic plan would be a great time for this discussion. >> I will recognize member Alma Flack. >> Thanks Mr. Chair and thanks uh the city auditor. I think that is very important for us as a committee just to ensure the independency of the auditor of the city auditor and it is great just to get this confirmation in annual basis just to have a conversation with the auditor about the resources about the challenge that they are facing that can impact the independency of the auditor and I would like to to ask do we have for example annual written independence statement that are provided in every year for example. >> Uh chair and audit committee members. We we do a number of things to document our independence. The type of statement that um I think is being contemplated here. We may we may want to develop to be more of a an officewide statement that we include in our annual uh audit and strategic plan. I know that we have statements uh regarding independence in there already, but I think we could strengthen that. Each of the auditors including myself um sign an annual independent statement. Each of the auditors including myself on each project sign an independent statement. And so we have a number of documented um controls in place to uh to ensure that we as individuals and the leaders within the office are ensuring that independence. Um that said I do think that building a an actual statement in our strategic plan on an annual basis would have value. >> Thank you. So can we have it for example al in February meeting? >> We we have it documented. I think that um we'll make sure to include it on every February going forward. >> Okay. Thank you. >> Yeah. >> Anything else? >> Thanks. >> All right. Thank you, >> member uh Hoseby. >> Thank you, Chair Payne. And I really appreciate this coming forward. I think it's going to be a really nice um like parallel to the updates to the charter >> cuz we'll see that, you know, without throughout this fall and then some of these questions I think will really get it that was the charter successful. So, I really appreciate that in addition to the other pieces that this brings. But yeah, thanks for this coming forward. >> Member Peterson. Um just a quick question about the peer review. Is um has there been peer reviews in the past or is this going to be the first peer review? >> Uh so I will share with you in full transparency the history of peer review. The office of city auditor has not had a peer review requested since 2009. Um that was when it was the um internal audit division or department within the city administration. In 2021, the office of city auditor was created by uh the change in city government structure with that city charter change. Um and it was recognized early and immediately when I came on that one had not been done. I will be further transparent in saying we did not pass that peer review in 2009, which is exactly why we are going through all of the rewrites of our audit manual, creation of our audit manual, um rewrites of our standard internal audit program and transition from IIA redbook standards to GAO yellow book standards. Um it is paramount that the overseers are um acting with the utmost integrity. It pains me to share this news with you. Um but I do think that um the work that we have put in to prepare for the peer review coming up in uh quarter 1 is going to lead to success. And as I mentioned, I hope it leads to some recommendations for us to further that uh development of the work within our office and uh I hope it leads to a a strong uh no finding review as well. And I I feel strongly that it it very well could. >> Oh, thank you so much. >> Yes. Um, I will just note that the mayor's recommended budget did have some cuts to the department and I see that as a direct challenge to your independence and I want to work really closely with you and your team on making sure that you are resourced adequately cuz I know that this this um committee has been asking a lot of this team has been increasing um the volume of work um and you've been able to meet us at those requests and I I think that uh we've been building some momentum here that uh is really critical to the functioning of this institution and so uh we'll have further conversation about that but I want to let you know that I'm very committed to the strength of your team and your capacity cuz you've been able to deliver some really remarkable work so far. So >> chair Pin, thank you for that. I really appreciate it. I look forward to talking. >> No other questions or comments? Um thank you for the presentation. Sorry. Go ahead. I had forgotten that I I always do this thing where I thank my office because I think it's really important. I mean the uh the work of the auditors within the office of city auditor and and Cher Payne, you just recognize this and I appreciate that. Um they work tirelessly to get these reports out. They have uh a lot of um time constraints thrust upon them, especially as we're getting closer to audit committee meetings. and I will tell you that they respond every two months under pressure and deliver and so I very much appreciate them and their work. Uh I also want to recognize a new auditor who's joined our office, Lucy Chapman, who is in the front row here. Uh Lucy joins us with uh significant financial audit experience from the private sector which uh as you know we don't perform the financial audit of the city of Minneapolis because we heard from um the organization that does that. Uh and the city also contracts out um to get their their statements uh audited on an annual basis. But that's a skill that we need because every single performance audit has some financial component. So we're thrilled to have Lucy join us. Um that said, now I am done to your pain. >> Thank you for that presentation and thank you team for being here. Uh I will ask the clerk to file the report and seeing no further business before us and without objection, I will declare this meeting adjourned. Thank you. Thank you.