Microsoft365 Copilot is yet to be ingrained in the daily habits of Australian Public Service (APS) staff after a 6-month trial by 60 Government agencies, and many have stopped using Copilot because of a poor first experience with the tool or found it took more time to verify and edit outputs than it would take to create them otherwise.

Adoption challenges were blamed for the moderate use of Copilot during the Trial which ran from January 1 to June 30, 2024, with only a third of 7,600 trial participants using Copilot daily.

A report into the trial found that “Overall, the usage of Copilot is in its infancy within the APS.”

It also found there was inappropriate access and sharing of sensitive information due to “Poor information, data management practices and permissions.”

SharePoint was assigned the blame for shortcomings in data classification that resulted in access to unauthorised content.

One of the trail participants interviewed for the evaluation report said, “Their information management in SharePoint is not great which has resulted in end users finding information that they shouldn’t have had access to, though this is a governance and data management issue - not a Copilot issue.”

Trial participants raised instances where Copilot surfaced sensitive data that staff had not classified or stored appropriately. This was largely because their organisation had not properly assured the security and storage of some instances of data and information before adopting Copilot.

Overall, trial participants across all job classifications and job families were satisfied with Copilot and the majority wish to continue using it. The largest percentage of survey responses came from those working in ICT and Digital Solutions and Policy roles.

This group also estimated the highest efficiency savings of around an hour a day when performing summarisation and document drafting activities.

Overall, 77% were optimistic about Copilot at the end of the trial.

Post-use survey respondents remarked they felt they spent less time playing ‘corporate archaeologist’ in searching for information and documents and more time in strategic thinking and deep analysis.

Some of the adoption barriers highlighted by the trial included:

- Uncertainty regarding the need to disclose Copilot use, accountability for outputs and lack of clarity regarding the applicability of Freedom of Information requirements were barriers to Copilot use – particularly for meeting transcriptions.

- trial participants remarked that they often forgot Copilot was embedded into Microsoft 365 applications as it was not obviously apparent in the user interface.

- Poor Excel functionality and access issues. Focus group participants noted that it was unlikely that trial participants had the newest versions of Outlook and were therefore unable to access Copilot features in Outlook

- Copilot could not emulate the standard style of Australian Government documents. Some noted that heavy re-work was needed to meet the tone expected by senior stakeholders within their agency and of government more broadly. For this reason, focus group participants noted they would not use Copilot for important documents or communications.

- Due to fears of hallucinations, many reported combing through Copilot’s outputs to verify its accuracy. In some cases, this involved reading the entire document Copilot produced to check for any errors which significantly reduced any efficiency gains.

- Focus group participants also noted that while Copilot attaches sources to its outputs, this is currently limited to 3 documents and does not provide visibility on why the documents were selected. Observations from Home Affairs also identified that Copilot appeared to be unreliable in its approach to referencing information provided against data sources.

- lack of Copilot integration with third-party software, in particular with Janusseal, a software that enables enterprise-grade data classification for Windows users. Interviews conducted by the DTA noted a lack of integration with Janusseal could lead to APS staff gaining access to information they did not have permissions for. Microsoft has advised that this is a third-party labelling issue, not a security issue, and that Copilot has an in-built fail safe to protect against this issue. It should be noted that such integrations were out of scope for the trial and Microsoft has further advised that a more permanent fix to the labelling issue is in the pipeline.

‘As we are testing these tools at such an early stage, there are clear opportunities for tailored solutions to be developed that can handle highly technical material,’ said Lucy Poole, General Manager of Strategy, Planning and Performance at the Digital Transformation Authority (DTA).

 ‘The evaluation points to the importance of agencies carefully considering detailed and adaptable implementation of these solutions.’

‘They should consider which generative AI solutions are most appropriate for their overall operating environment and their specific use cases. We’re pleased that a lot of the recent work released by the DTA helps government agencies identify and address these very considerations.’

The full report is available HERE.