The Digital Transformation Agency (DTA) has released its report of its whole-of-government generative artificial intelligence (AI) trial with Microsoft 365 Copilot, citing some “efficiency gains” but also “broader concerns… that require active monitoring”.
The trial, which ran from 1 January to 30 June this year, was made available to over 7,600 staff members across more than 60 government departments and agencies. Of these staff members, 77 per cent were satisfied about integrating an AI tool into their work environment, 86 per cent said they hoped to continue using it, 69 per cent said they had seen a noticeable improvement in the speed of completing tasks and 61 per cent said the AI tool “enhanced the quality of work output”.
“The DTA has always been clear in its goal to not shy away from innovation,” Lucy Poole, General Manager of Strategy, Planning and Performance, said.
‘To swiftly provide the APS with tools to experiment with generative AI in a safe and responsible way, we identified a tool that would quickly integrate into most workplaces with minimal disruption. We were particularly pleased to see that participants saved an average of an hour of work a day handling note taking and administrative duties.
“Even more pleasing was that 40% of that time was dedicated to building skills through activities like mentoring; or it was dedicated to staff engagement and team building.”
However, the majority of efficiency enhancements resulting from the trial were contained within “business-as-usual and office management tasks” including capturing notes, action items, minute taking, and kicking-off document writing or planning.
“The evaluation measured three key activities performed by APS staff: summarising written content and items such as meeting transcripts, drafting and rewriting documents, and searching for information,” Vivek Puthucode, General Manager for Public Sector, Microsoft Australia and New Zealand, said.
“Over these three activities, participants reported saving up to an hour a day using Copilot, with 40 per cent saying they used this time for higher value activities like mentoring colleagues, strategic planning, engaging with stakeholders and enhancing products. One of my favourite anecdotes was around spending less time playing ‘corporate archaeologist’ searching for information, and more time thinking strategically and performing deep analysis.
“While these use cases offer a great baseline comparison, there is significant potential within the public sector for broader impact, such as deeper trend analysis to inform policy recommendations, fast-tracking Cabinet submissions and accelerating skill development for early-career public servants.”
“As we are testing these tools at such an early stage, there are clear opportunities for tailored solutions to be developed that can handle highly technical material,” Poole said.
“The evaluation points to the importance of agencies carefully considering detailed and adaptable implementation of these solutions. They should consider which generative AI solutions are most appropriate for their overall operating environment and their specific use cases. We’re pleased that a lot of the recent work released by the DTA helps government agencies identify and address these very considerations.
“We are highly aware of the realities of bias creeping into these services. Bias can emerge in data when it’s incomplete, unrepresentative, or mirrors societal prejudices. AI models might replicate these biases from their training data, resulting in misleading or unfair outputs, insights, or recommendations.
“That is why we continue to reiterate the importance of keeping a human in the loop. This is why we raise the importance of transparency, explainability, and thorough review across so much of our guidance. Keeping with new innovations ensures the APS remains a competitive workplace. Making tools available that improve the efficiency and quality of some types of work goes a long way in retaining new talent.
“This cannot be outweighed by losing key skills around processing and synthesising concepts or knowledge of the work undertaken by an agency. These tools should instead enhance strategic thinking and making connections between disparate pieces of work undertaken by government. And always with the aim of improving outcomes for all people and businesses.”
The report also indicated several other concerns about the implications of generative AI on APS jobs, the potential bias of LLM outputs towards western norms and the impact of generative AI on the APS’ environmental footprint.
“The potential for generative AI to improve inclusivity and accessibility in the workplace was of great interest to some agencies. Our recent work tries to arm the APS with the tools they need to fulfil this potential in generative AI. Agencies should look to ensure their AI solutions incorporate:
- Fairness. Their systems should seek to minimise disproportionate impacts on individuals or groupsm,
- Reliability and safety. Special consideration should be given to the underlying data sources, risk identification and mitigation practices, as well as ongoing monitoring and intervention controls, and
- Privacy protection and security measures. These ensure entities have the proper authority to use this data and that regular assessments are undertaken to determine any potential impacts.”
“There were some participants in the trial who expressed caution around using Copilot and the perceived risks to data security. This reinforces the importance of transparency around generative AI services that sit within an organisation’s secure environment, and the important step required to educate users before they access the service. Working with agencies during the trial and beyond, we have learnt that rather than create greater risk, Copilot can proactively work to preserve the privacy and security of sensitive information,” Puthucode said.
“It’s also worth noting that some APS staff were concerned AI could displace and change roles, undermine skills such as writing and impact the environment. Every successive tech advancement presents both opportunities and important considerations. At Microsoft, we’re thinking deeply about the short and long-term impacts. We believe that infusing AI into our work tools will both create net new job opportunities in the longer term and increase job satisfaction in current roles. Early research is showing that AI is augmenting productivity, freeing up workers to focus on higher-impact, creative work and increasing job satisfaction.
“Finally, the report notes some unintended but positive outcomes. One is that ‘generative AI could improve inclusivity and accessibility in the workplace, particularly for people who are neurodiverse, with disability or from a culturally and linguistically diverse background’. This is an important area for Microsoft, and Copilot presented a rare opportunity for us to incorporate accessibility right from the inception of the product.”
