AI in action - Video series

First question for all of you is if your AI
journey to date was a movie, what would the
title be? And what was the real world problem that
pushed you to create an agent? Well, I guess I've
got the you go first. I actually I did pre-prepare
some of these and I got some assistance. So,
for the you know PCO we produce legislation
of course. So, The Last Consolidation,
A Question of Authority, The Semantic Divide, All the Versions we Leave Behind. My favorite,
The Fragment Strikes Back and,
that speaks a little bit to,
you know, how we approach this, like I mean,
this was a year ago, when we started this,
and we didn't know what we didn't know. And
so really the effort at that time was to
get moving, to get started with something, to figure out how are we going to work in
this in this space. And one of the things
that really came out of that is that in terms
of the production of legislation. We're
very much anchored currently into a document
centric world that came from you know legislation
and statute books. And for this new era,
we really needed to look at how we structure
and organise our legislation differently,
into fragments. So I'll leave it
there, more to say. Thanks, Joe. I did ask my
question this team. I'm not sure if I liked the
answers. I think they might have used ChatGPT.
But they did say that if it was a title of a
movie, it would be the Fast and Furious Regulator.
But I did ask also some of them came back
and said that if it was an existing movie today
it would be Indiana Jones
Raiders of the Lost Ark because we were constantly
learning, lots of puzzles to solve, sometimes
felt trapped in the maze while we were doing the
work but we got there in the end. So the big
real world problem that we were trying to solve
was our regulatory system was under significant
pressure. We had a growing backlog of applications
to undertake activity on DoC land. A number of
those applications had the potential to unlock
significant economic value from New Zealand,
for New Zealand, and we were under significant
pressure from both our Ministers and the public
to do better because of the length of time it
was taking to process the applications. So the big
focus around the tool that you saw some of what we
did was how do we speed up and support regulatory
decision making, not replace regulatory judgment
but actually support it so that we could make
faster, better decisions. Oh, kia ora koutou. I think
we talked about this and we, I, did kind of
start with True Lies, it was a bit
of a drama because I think, you know, some of the
things that came out we really learned all about
some of the hallucinations in the work that we
were doing. So that was really interesting. But
in the end we went for a bit of a documentary
theme around Pilot to Proof and I think that's
really as we started to find out a little bit more
about it and how things worked we got that proof
of concept which was really what we were after.
I guess the main problem that we were looking
to solve the Education Review Office does bucket
loads of reports, hundreds of schools, thousands of
early learning services, and then national reviews
every year. And we were like well that's a lot
of stuff we could do something with but we're a
pretty small agency. We tried a whole lot of things
and to manually just analyse those reports to
get system insights and then individual provider
insights was just impossible. We tried things
like WordStat and other things. It just didn't
work for us. So we partnered with the lovely
Fintan and we just tried really a small prototype.
What we wanted to know was could we use this
unstructured data in a really useful way. So
we started small which was just with some of our
school reviews using public information.
Things that were publicly found and then we just
looked at what our recommended actions were in
our improvement actions and we wanted to know did
schools use them, were they useful? Did they put
them in their plans? And so that was really
the key element that we had here and it just
meant that it was a really scalable useful way
of being able to get across over time as
we build this out hundreds of schools. And you know,
reports and use those insights both operationally
and strategically. Brilliant. Thank you. Okay. And
for me well it's actually the Lego movie and
I use my trusty friend Copilot. Sorry Fintan
to help with that. And really what that's about,
I mean, we built iteratively. We encouraged
experimentation. And it's about sort of just
empowering everyday innovators and just being
innovative. Just try it and have a go and,
build it out from there. So it works.
So one of the functions at WorkSafe that
I manage is our guidance products area. We
have 440 guidance products. A number of which
need updating or most of which need updating.
We've also got an amendment act to the Health and
Safety at Work Act which is going through Select
Committee process now. There's a number of
changes to sort of key concepts in that which is
you know on the day that that's enacted our
440 guidance products in effect become out of date
and so we've been exploring what are
the ways that we can update that guidance more
quickly and how can we tackle that doing
the sort of the manual way that we would have in
the past will just never get there in the time
that we need. So we have a, we've sort of built
two agents. One is a currency agent that looks at
how we update our guidance. And then the other
one is very similar sort of feedback type process,
submissions process, very similar to the one that
that Fintan showed you in the in the video.
Sorry. All righty.  I'm going to have to be
selective in my questions with our available time,
but Andy, you, at PCO, you did something
quite interesting in that you actually, I believe,
worked with maybe five different providers
on five different poc's at the same time,
but or in a short period of time. But then you
also have been quite transparent about how you
shared what happened through that process. So, I
just wonder if you could talk to us a bit about
how that idea came about and why you chose
to operate the way you have. Yeah, I mean,
it's interesting because we keep hearing this
is something interesting and different, but it
didn't really occur to me at the time that it was
anything particularly different if I'm honest.
But I mean, we're we're a very small organisation.
We're 100 people. We don't have developers. We
don't have the technical ability to do any of this
work ourselves. We didn't have clear ideas of
exactly where the opportunities were. We knew we, I
mean, we had use cases, right? We had our staff
were involved in coming up with use cases where
they thought that there could be value. And so
instead of just you know rolling the dice and
picking one you know we had to go through a
process of procurement we had to go through a
process of setting a project up and I was like
look if we're going to do this right how much extra effort is it to you know run the same
process but just pick five of them. Well the
obvious problem was, well that's going to cost more
money. But what we decided to do was just
pure R&D. And so what we did is we said,
"Look, we've got $25,000 per use case. That's a
fixed price. Surely there are some really
interesting companies around New Zealand who would
love to be given not a specification but a
you know a user story. So we we didn't spend time
producing a big document of requirements. We
had a single use user story about what we thought
the outcome was. And then we basically looked
for companies who looked creative and innovative
and were willing to spend, you know, no more than
$25,000 worth of their time to help us figure
out, you know, which of these might make sense.
And we ended up with five proofs of
concept, the poc's, for that amount of money.
And I would say, and there are some vendors
in the room, the people we worked with,
the company, I mean, we've got an amazing talent
here in New Zealand, and they put in more effort
than we paid for, if I'm quite honest,
but I like to think that they got something from
it because part of the, I guess the approach
we took, we set up right up front, is that we want
all of this to be in the public domain. So,
all of the specs, all of the research reports,
all of the code, it was built into our agreement
that we could talk about it openly. We would,
we would share it. And it actually had one
of the the other benefits of, it was, an
incredible way to communicate across government.
You'd like to think that we've got better ways
of talking to each other, but by simply putting
this out in the open, that actually got a
lot of attention and helped us, you know, build
on some of that with other collaborations.
So yeah I think it was just out of need really
being a small organization with limited budget
we just had to think you know how can we do this
differently. Brilliant. Thank you. Melissa, I
wanted to ask you who ultimately owned and led the
project and how have you worked with your with
the IT team? We very much, as Andy described, we
don't have a lot of the capability. We don't
have this inhouse. So the the project was owned
ultimately we have an AI steering committee.
And we have, it was led out really by my colleague
Jason who's there and myself really. And
it was just a real joint business data kind of
approach. And then we worked with Fintan to get
that expertise around it. And I guess the
biggest thing around it is where the decisions lay,
which the decisions for what we had to do lay with
us as ERO. And we were just really able to work
and partner well to get the specifications
that we needed and what the technology could
do for us. Yeah. And working with your IT team,
have they been involved or? They have but
more around how it interacts and engages with
the rest of our technology and particularly
in the early days when we were anxious about
all the privacy issues what could come in cyber
security. So it was more around what we were doing
wasn't going to break anything. And that it was
going to stay within the rules that we were able
to work within. And even just in the 18 months,
I think somebody mentioned earlier such a fast
change around what's possible and what you can
now actually use and what you can do internally
with the AI. Yeah. But certainly they kept us safe
from that perspective. Yeah. Raewyn, I'll ask the
question and I'll hand you the mic. Interested
at the beginning with the idea and kind of
the decision about whether to go ahead or not.
Was there any big sort of go or no go
decision? What got the project over the line?
Well, and we were very fortunate I think in
that we had some funding available and so we
were looking for some good use cases around the
business. The guidance area seemed like a
really logical place to start. And you know
the team were keen and enthusiastic. This,
it's been very much kind of business led and
we've built our agents kind of off the offsite
premise to DataSync at the moment. We're
looking at sort of bringing them into our
own production environment. So certainly the IT
team are getting involved now. So, but, in many
ways I think with the business leading it
that's enabled us perhaps to go a bit faster
and perhaps to treat it more like you know
these are innovation projects and you need to
kind of use innovation tools and methodology as
opposed to don't apply the kind of methodology
that the IT team might want you to sort of put
you through if it's a $5 million big software
build that's going to last you 5 to 10 years
because otherwise you'll still be here in 18
months time. So just get going I think. And Joe, for someone else starting their agentic AI
journey what would be your top tip on where
to start and is there anything you would have
done differently in retrospect? It's probably a
couple of things from our perspective. So I say
don't boil the ocean. So continuous improvement
over one big change. So look at where it can add
value and start there. In terms of what
we do differently, we spent a lot of time,
a little bit of time, playing with Copilot
and you heard some of the responses that
we got where it couldn't pick up the complexity.
So with some of what we were doing, it was a bit
like an overeager grad. It would come back with
a response if there were two different answers,
it would half the answer. So for an example, if
we said, "How much can I spend when I'm traveling
for work?" and one SOP said $50 and one said
40, it would tell you it was 45. So we
often found it was very keen to give us an answer
but not necessarily the right answer. So for us,
research the technology early and proof
of concept much faster. All right, Andy,
governance and oversight, what's proved most
useful day-to-day rather than perhaps on paper?
Oh, I mean in our situation we attached this
work to existing governance systems. Again,
we're a small organisation and we try and keep the
overheads low. So instead of setting up, you know,
separate boards and separate projects we, you
know, connected into things that were already
operating. So I think that kept things
really easy. But I think the biggest thing
that made a difference certainly in the, I mean we deal with information that that is needs to
be kept you know it's very sensitive,
security is really, really important. So
the biggest thing that made a difference is that
when we first started doing the work we worked
with data and information that was already in
the public domain. So we're not deal dealing with
correspondence from the Attorney-General. We're
not dealing with conversations, you know, with
Ministers or anything like that, or cabinet
papers. We work with the stuff, you know,
the bills, the materials that are
already out there and then we work backwards,
if you like. So we prove the case based on
the material that was already there. And if
we can work there, then it can work with sensitive
materials that aren't already public. Now, if you,
if you're able to take an approach like that,
then it just drastically simplifies the whole risk
profile of the work you're doing. And the risk
profile dictates what kind of governance and what
kind of controls and stuff you need. So,
that's probably been the biggest thing that
made a difference for us. And just while you've
got the mic, what's your and I'm sure that
Brian and Josh won't mind if it's not theirs, have
you got one poc that you're most excited about?
It's like choosing between your children. So
look, we had five poc's. We have
got one that's being taken forward at the
moment into like a production stage. So the
one that we're working on now is explanatory
notes. Many of you will know what I'm talking
about in the room, but basically the notes
that explain to everyone what the legislation is
supposed to do for people who are not actually
going to read all the legislation.
For example, the work that Ackama
did with us on the plain language,
was there was an interesting result.
We actually decided to pivot.
We realised that the plain language guidance was
was helpful, but it turns out that there are other
types of rules that are actually more important
for checking and we can't do everything. So,
we're working on pivoting that to a quality review
tool. So these things take different
different paths. Yeah. Yeah. Brilliant. Thank
you. Joe, I wanted to ask you what's next?
So, we're currently building a brand new end-to-end digital platform for all of our regulatory
services. And so, while we've got some awesome
AI tools, we put in three in the last 8 months,
our next focus is really on how do we incorporate
that into workflow and where are the other things
in workflow that we can use AI and automate
particularly around our what I call our low value
tasks. Through the work that we're doing, we see
quite a big capability shift removing our people
from doing process work into knowledge work. And
so as we are building that end-to-end platform, how
do we find those opportunities to continue to make those shifts? I'll ask Melissa the next question.
So we've heard a lot today about the importance
of people and that this is about culture change
and behavior change. What are your observations
about how your people, people in your organisation
have engaged and responded to this? That's a
really interesting question. And I think
some of the speakers earlier talking about roles
leaving you know people losing roles and things
obviously is really front of mind for a lot of our
people, particularly in certain roles. So we've
worked really hard to be really clear that
it is human-centered. The sort of work we do,
you must, you have to have a human judgment over
it. There's a whole lot of stuff that AI can help
with, but we need a lot of, you know, you need
that human judgment because making a call
around how well a school is doing and what kind
of support it needs is pretty high stakes. So,
there's key elements around that. There's a whole
lot of things around human nature. A some of
the comments I've kind of heard when we've been
talking about it is but it's cheating, isn't it?
You know, if you use that, it's kind of cheating
or right through to great, I can do this for
for everything. And so a lot of the work we're
doing around our people capability and workforce
strategy is picking up the sorts of baseline
skills that our people will need into the future.
We've got some changes happening that allow us
to consider that in slightly different ways and
how we can get efficiencies from that. But I think
at the end of the day, we're an education agency.
Many many of our people come from the education
sector, myself included. So people will always be
first. It's just how you manage your way through
that and provide people with the stepping stones
through that change so they can see what it
means for them and how it will make their
life easier and ultimately it's easy for us. We
can demonstrate over time how it will make life
easier for the young people out there. So that's
the hook for our people. I want to ask you a
similar question Raewyn, but I want to add something
to it. And that's, I don't know if you've got
any colleagues in the room but, I wondered about how people have responded but
also the leadership in the organisation. The other
thing we've talked about a lot is the importance
of leadership and so I wanted to just ask you to
comment on how the leadership in your organisation
has been supporting these efforts and what
you've seen work and maybe what could be better.
I mean the the leadership's been very
supportive and they've, you know, they've
provided the funding and allowed us to get on
with it and to be honest to get on with it
in a reasonably autonomous way because at
the end of the day you know it's just really
about sort of having a go, working out whether we
can do this and and how we use these agents to
to improve our work. We're now sort of
embarking on a journey of probably like many
organisations, you know sort of getting our
building blocks in place around AI in terms of
you know, policies, the appropriate training for
people, the the more sort of ubiquitous how do
we roll, you know we have the tools rolled out
across the organisation but you know where do we
go next and how do we develop from here. And
I think you know WorkSafe as an organisation
put many of its leadership through some
great AI training which just really kind of got
us up the curve as well in terms of you know how
to do some quite sophisticated prompting. Well,
it was for me at the point that I sort of started
and to sort of get on that journey,
but also, you know, what are the building
blocks an organisation needs around AI and
the guardrails that you need in an organisation
to keep yourself safe and how do you need to
be thinking about it from a, from the strategic
perspective in the organisation. So, I think,
you know, that training has really sort of helped
lift people's expectations. We've just
sort of put out some little videos out on
our intranet of what we've been doing and
certainly the ideas on use cases are
starting to flood in. So there's plenty more to
to come. And that's a great segue and I
was I'm going to ask each of you this and I'll
just ask it in order, which is, are you going
fast enough as an organisation? And yeah,
I'll leave it there. I think we probably need
to go faster. I mean, we've got some ideas on
how to do our next developments and to enhance
the tools that we've got. We've got two tools.
We're looking at, you know, can we actually
hitch them together as well and to get a bit
more sophisticated. But I'm kind of conscious
that we're probably well ahead of where the rest
of the organisation is as well. So we do, I
think, the organisation needs to go faster.
Yeah. I I think our speed is appropriate for
our I guess the objectives. Like what we're not
doing, this is not a, we're not
working with these tools to engineer a complete
transformation of our business processes. If
we were doing that, we'd be starting with a
blank sheet of paper and we would be doing things
very, very differently and you'd be wanting to go
at a different speed. What we're doing is we're
looking at it's a change process. You know, when
we one of the outcomes that we started with
in terms of why we're doing this is actually to
help the organisation start to plan out and
and move through this very changeable,
you know, what's about to hit us or what is
hitting us, right? And so from that point of
view we're not trying to, you know, we're not
working at breakneck speed because we're trying
to bring our people along with us. Our people are
in the center of our processes. You know could we
go faster? Absolutely. But we're also running
a business. And so we have to pick our pace
kind of carefully I think. I guess until we're
told otherwise which is always a possibility.
That's a really good question. It's
really two things that we think about
we've been thinking about in this
space, but we could go faster, but we're
also developing our new regulatory services
on a brand new platform at the same time. So,
there's two questions that we're thinking about
as we do that. So, one is where does human
interaction matter the most? And thinking about
that as we design our new services. And the second
one is how do we move faster while maintaining
the trust of both our people and our customers.
So those things are really top of mind. While we
could move a bit faster than what we're doing,
we've got to get the basic capability in place
around our new digital platform first. Yeah,
it's interesting, isn't it? Particularly if
you're a smaller organisation and what we've
learned, we've got a number of use cases and
particularly the one that I was referring to
earlier you need your subject matter experts and
in our case our subject matter experts are the
people out in the field actually, so there's
a real balance between how we manage things like
our actual outputs because that's kind of what
we get paid to do and then how we manage that
across this innovation and making sure that
we can ironically you being able to bring those
people in probably would make their life easier
over time. So we are really carefully managing
the change. So I think like everybody the pace
is probably about right. We've got significant
other change that's happening in the organisation.
We've had significant change. I think it's not
just about the outputs. It's actually you ask me
about people. It's actually about making sure that
people you know can cope with what they've got
and they can absorb the changes that are happening
and lots of things that they need to learn. So
I think for us it's probably about right. And
some of those use cases are going faster and some
of them are going slower. But overall I think
we're pretty comfortable with where we are and
it's been just great to hear where everybody else
is because you just don't know. So yeah, it's
been great opportunity. Thank you everyone for
coming along. This has been a fantastic event
and I hope you've got a lot out of it. And
I think there's definitely appetite for more
of these types of discussions moving forward.
Some of the key points that I got as
a takeaway is that energy and courage is
needed. I don't think there's going to be the
right or the perfect time to deploy AI or to use
AI within your agency. So you need to have
a bit of energy and courage to start. I think
definitely learn by doing otherwise you will get
left behind. So again the the timing is never
going to be quite right. And I think share
and learn from each other. So all of you are on
different journeys and have learned mistakes
and have learned where the opportunities lie
and have learned strategies for getting your
executive on board or getting the funding right.
So I think definitely learn and share from each
other. And I think the the final point is beware
of the slop cannons. I think with AI it's really,
it's really easy to look like you're doing lots
of really good work but actually you know if you
look closer there can be a lot of slop out there.

A little bit of a different story. This is a bit
of telling the story of how we've evolved our use
of AI. So who is this guy? Co-founder
and CEO of Raygun and Autohive. I've built I
started my first business at 23, bringing me up
to 19 years now, but I'm a software engineer at
heart. I love building things. I love creating
things. I'm still a nerd under it all. I
live here in Wellington, New Zealand, and I don't
trust real photos with AI around these days. So,
this is my family. And I've got another son
coming in about 4 weeks. So, I'm excited about
that. So moving on, let's just talk. This this
is my genuine view on AI is that we are totally
underestimating the capability. And I talk about
this from the field of sort of software
or techies. It's really, really easy to get
super excited about this line and be like wow
this is taking off it's going to change the world
everything's going to immediately be different
and then while you get excited by that back
to the point you were making earlier as well
is the human part though is slow all right so if
you're building a business in my case I've have
to think about this line less so than this line
if I want to actually make any money and how
do we actually drive adoption. This however is
also why you want to get started early because it
will never be easier than it is right now to pick
up. I make the reference to computing where prior
to about Windows 95 it was entirely possible
for you to understand every way the computer
worked and these days you'd struggle to understand
a single slice of it. It's just going to build and
compound. And that adoption curve is always
slow with humans. Technology has exceeded us
for a while. I make the reference to Covid when we
started to see businesses coming online and you're
like you've had quarter of a century to do it and
you took a pandemic to finally get a website.
Like that was insane to me. So thinking about
the work in the age of AI. So firstly we talk
about how everybody's job in our business is to
solve a problem. It's not how you do the work,
it's what the problems are you're solving. And
that means you want to be thinking a little bit
more or increasingly about how you orchestrate
the work to be done as opposed to how will you
do the work yourself. And that means that
increasingly the type of work that you're doing
is to manage and fine-tune those orchestrations,
potentially even creating them. I don't anticipate
everybody has to be able to create them, but
you're going to slowly be managing agents and
and that capability. That also means you need
to increase your understanding of externalities
as well as other teams and I guess divisions
because the span of control of an individual is
growing. I saw a picture on the internet the
other day and it showed a product manager,
a software developer and a designer standing in
a Mexican standoff and each of them was saying I
can do your job now. Right? The span of control
is widening. But the difficulty right now is
maintaining quality while increasing velocity.
It's really easy to go fast and create crap.
But it's quite hard to go fast and
create quality outcomes at the moment. So,
we've sort of shifted a little bit and I actually
struggle with this in our business because I don't
actually think that we necessarily hire roles
anymore. I don't I think of it more about the
attitude of the people that we're hiring. And
so we now look for people who have demonstrated
curiosity, strong systems thinking, good
judgment, integrity and speed. And I say
to other business leaders to that point about when
are we getting on to AI? You have just had 3 years
of observing who in your organisation is actually
curious and innovative. They have stood out like a
sore thumb and everybody else is not doing that.
So have you promoted and enabled those people to
spread that innovation in the organisation and
just to be clear and I totally agree with the
the point you made about the authority coming.
You need somebody in in a senior role to really
make it happen. But at the same time those people
should be enabling people across the organisation
that are putting their hands up to be those change
makers. It's not a role, it's not a level, it's
not a salary band, it's the attitude of the people
and you want to elevate those people. They are the
key ones for ensuring that you're not being
obsoleted. That also means credentials are pretty
much dead in my world. I don't care about your
degree. I don't care about anything. I just want
to see what you can do and how you do it. And this is, this is where it's becoming really
important to start thinking about the capability
of the people in the organisation. New Zealand,
you know, and I understand we're a small country,
high employment is important. We can be quite
vulnerable, but in today's world, if you have
people on the team who don't have good judgment,
they just kind of don't do much, right? That's
kind of safe. They don't wreck the place. That's
good. And the people with good judgment,
they're actually getting things done. You know,
we all know those people in our orgs. Let's not
be shy about that. But the problem with AI is that
these people can start to generate a lot of stuff
and so they become a real liability. I heard the, I
heard somebody say it's like running with scissors
and somebody corrected them said no it's like
running with chainsaws now right. But the good news
is, is that for the people that have good judgment
they are absolutely crushing it. So the delta
there, I'll let you take the the photo. Yeah,
is really big and that, you know, I'm obviously
being a little facicious. This is the point about
the human change management. How do we think
about the risks and and all of that. I'll also
say I don't come from the regulatory
world. So I am a bit rough around the edges.
But I was kind of trying to keep count of how many
times we used the word risk, which was a lot, and
how many times did we use the word opportunity,
which is zero. So and Jeff Bezos says that
human beings do have a habit of overestimating
risk and underestimating opportunity. But
there are some risks here because it is a powerful,
powerful tool and it's only getting more powerful
as every day goes by. So I just want to shift
gear now to to telling a bit of the the actual
transition. So quickly here what is Raygun?
So software diagnostics I describe it to people
who aren't in tech that we're like the blackbox
flight recorder for software. So when it blows
up or you have an error or something goes wrong
or it's slow, we actually collect the metadata
that helps you understand why did it blow up, why
was it slow and we help software teams around the
world improve the quality of their software. We are 97% export-based so don't really bother
to market ourselves too much within New Zealand.
We process billions and billions of software
faults per month through our platform.
And some of the international brands that we work
with are folks like, HBO. So, if you watch the
finale of Game of Thrones, all your data went
through our platform. We peaked at tracking data
for 87 million people watching that at one moment,
which was quite a lot of data. Domino's
Pizza, if you ever order a pizza and something
goes wrong, sorry to say, I do know about it.
They're a cool client. And there's thousands
more. And I thought since this is a New Zealand
governmentish related event, I'd acknowledge a
couple of folks that we do work with here.
So the New Zealand police use us to help them
make sure that their software that the police
force have out in the field is actually, you know,
if something goes wrong with that, it can be
a material issue for them. They need to make
sure that that's not a problem. Although I do
joke about the money go around, you know, they
pay Raygun, I pay a speeding ticket, they pay Raygun. Elections. We've worked with them when the
actual elections are on to help make sure that
the websites stay up, they run quickly when people
are sitting there smashing refresh, those sorts of
things. But then 2023 arrives along with AI and
I know it was technically like late 2022, but
this is when I noticed it. And I was a, I was a
teenager in the internet sort of transition time,
but I was a nerd. So I had like, you know, Windows
95 poster on my wall rather than a bikini clad
woman. And so I kind of remembered that time
of going, "Wow, okay, this is a big shift. You
know, what's going on?" And I had that exact same
feeling. I was like, I don't know what's changed
or how it's going to change, but change is a foot.
And so I recognised that we needed to elevate
our understanding in Raygun so that we could have
improved conversations about AI. I didn't want
to just read stuff on the internet. How would we
improve the conversations? So, first one was open
up the wallet. You know, make sure everybody has a
ChatGPT subscription at this point. That was the
AI at the time. Be willing to spend. I said to
the my board in I think it was April, if anybody
leaves the organisation, we're not backfilling
them. Didn't really know if that was a good idea
at the time. It was, we were about 60 people
then and then in May we had an AI week. So we do
have the benefit that Raygun is a SAS business.
That means the money comes in whether we go to
work or not. So we could pause the business
for a week. There was no opportunity cost of build
hours or anything like that, which I do know is
a challenge when I talk to a lot of service
organisations on how do they invest in their
people when there's a very real dollar cost to it.
We just said we're going to do all of the customer
support when we arrive in the morning and then
we're going to do AI stuff for the week. So the
very first thing that we did was we had everybody
in the organisation do our software engineering
hiring test that included the bookkeepers, you
know, customer success people and they could
all suddenly pass our engineering test, even if
they had literally never written a line of code
in their life. And pardon the language, but
I call the moment for when you click to AI as
the oh [ __ ] moment. The bit where you go, this
thing can do stuff. And that was really powerful
for them. The non-engineers suddenly delivering
software by the end of that week. The Friday we
did demos so we put everybody into teams of three
or four, I totally agree with your point that we
need people to do stuff with this to learn, but
we had systems where you know they would talk to
the mic and just say hey and they did call
it Jarvis hey Jarvis I'm thinking about taking
annual leave at Christmas what's the policy to
do this? And it was, we had built a information
store based on our internal documentation and
it spoke back and said well you know you need
to talk to your manager we have a two week shutdown
period you need to do this by this point and
really it was all throw away but it was to get the
team thinking about what was actually necessary
to make these things to make the technology work.
And we made sure that those teams were always
crossfunctional so it wasn't just engineers versus
the design team versus the CS team that sort of
thing they all had to collaborate and it
was a great amount of fun. So yeah, everyone in
the business built AI agents and yeah couple of
months later we actually replaced working with
our advertising agency with a couple of AI agents.
Turns out if you can pull the data out of AdWords,
it's not too difficult to make suggestion that if
you're paying 80 bucks conversion in Iceland but
20 in London, move the budget to London. So it was
quite quite straightforward to do that. Then
2024 and we're primarily a software development
organisation but this is where coding AI really
took off. And just to be clear for for folks here
who maybe don't do any software development which
I imagine is most of you the tools in the software
space are probably the best AI tools that are
around at the moment. You know so you get
very quick feedback on hey did it work? Did it
compile? Does it pass the tests? So it works
really really well in software development.
So we we saw across the industry widespread
adoption of AI coding tools especially in
our organisation. The velocity of our software
delivery really started to pick up. We did
allow all of the AI coding tools. We didn't just
mandate one. I got to say like I feel for the
people who say we've got Copilot. Like oh Jesus
that is, that is trash. Go and get a decent AI.
But my CEO frustration, keep in mind, like I don't
write any code at work, but I do in the weekends
just to keep my skills and understanding
strong, is that the coding tools are great,
but what about everybody else in the business?
So, I would sit there and be writing these
amazing things at home in the weekend for fun, and
I would come to work, and Gemini would offer to
summarise an email, and I'm like, what is this?
This is not moving the needle for anybody else.
So, in my typical fashion, fine. We'll just
build it ourselves because my intention was
to build the most AI enabled organisation in New
Zealand. And so we began building Autohive,
an AI product for our whole business and it
was built initially to scratch our own itch.
So an example of something that you know
we talk about how fast the innovation is
going in the field of AI and especially in 2024
I was like but why are all of these tools single
player mode what we're a business a business
is a collaboration of people yet you're all
sitting there in individual tools just sharing
chats. So we wanted to build something that
was more collaborative I also wanted to think
about what would I see as the CEO for the whole
business's operations how can I ensure a place for
marketing to be able to get all their stuff done,
customer success. Bring it all together. I don't
want everybody just running a separate tool.
This is a a photo that I took that year. This
is an example of a software developer on our
team. And just don't expect you to understand what
you're looking at there, but to set the scene,
he's actually running seven concurrent AI coding
agents at once. And so that is like having a team
of seven people reporting to him that are all
working faster than any human actually can.
And the velocity is just phenomenal. So we come
into 2025 and we ship Autohive in late June.
And that's been quite successful. We got thousands
of users signing up for it. We've had revenue
beginning to grow which is really cool. And demand
for deeper integration with systems. So this kind
of goes again to the agentic use doing real work
not just answering questions. And the analogy
that I often use for this is the sort of ask a
question get an answer is like the email stage of
the internet. So you may remember that was a major
hook for the internet back in the day. It was just
I can now get a message to the other side
of the world in like milliseconds. That's cool.
Nowadays, nobody's getting the internet because
they go, "Yeah, I'm getting an email." Right?
It's all moved further up the value chain.
But right now, we're at the email stage,
but people are starting to ask questions about
how do we actually do work with it. And AI begins
to improve the general office work. So, not just
for our software team. So, our marketing team went
from I think six people down to two and they
like tripled their output. And they are
actually one of our, that after the software team,
they are our most aggressively heavy AI users.
Again sort of a bit a little bit like software you
get a pretty fast feedback loop. This is actually
something that I think is going to be a challenge
for regulators because your feedback loop can be
long. I don't know what the sandboxes were but in
effect you know maybe running simulations against
regulation to understand things as a way to
accelerate that. But way to see this as it rolls
out globally as an AI is the longer the feedback
loop this the longer it's going to take for that
to really take off. So take for example legal
AI the real fitness test on that is probably a
court case. You know that could be a long time to
check whether that contract is actually any good.
So by the end of 2025 we'd seen a 500% increase
in the rate that we were shipping software.
And I know it was 500% because one of the things
that I do in the business is at the end of every
year I present to the team everything that we've
done. So, every major feature, every significant
enhancement or a major bug fix. And typically,
this is a slide deck of about 35 slides, double
column, 15 bullet points. And I do this because
people just forget what they've done in the year.
They remember about the last 90 days. And I like
the team to leave at the end of the year going,
"Holy [ __ ] we got a lot done." Normally, that
deck is about 35 slides. At the end of 2025,
it was 164 slides. It was ridiculous how much
work we got done. This is a quick screenshot of of
AutoHive. So, you can see here we have different
team groups. Ignore some of the testing stuff,
but it can fully automate any type of work
that we're doing. And we're trying to move
as quickly as we can to continue to enhance that
product to be able to accelerate not just for our
customers but also we're using this on our Raygun
business. So I have the benefit that by having two
businesses I can use one as the test and the
other to to produce for it. And so this has
been this has been fantastic for us. We do track
ROI in it as well because this is a major other
question. Is this actually worth the time? And
so this this effectively uses a a mechanism to
estimate the time that would it would take to do
things manually. And so this is across our team
for one month did 4,700 hours worth of work
would have cost based on average salaries about
$171,000. That we probably paid I think it was
$7,000 in AI token costs to achieve that. So a
pretty good ROI. And by the way, the team is now
25 people down from 60. And I do say to people
the way I got there was that I had terrific
headcount churn in 2003 and 2004. And the primary
reason in the exit interviews that I got was
I'm so sick and tired of you talking about AI.
I don't want to know about it. I'm leaving. Which
gave me the benefit that the team that we do have
is all extremely AI pilled now. They love AI and all
of those people are dragging the companies they're
now at down. So, and the an interesting example
talking about Raygun is that because it collects
software faults, one of the messages I said to
the board, I said, "AI is going to destroy the
Raygun business." I said, "I'm selling picks
and shovels to software developers, and I've just
watched a giant German excavator drive past me
on the street that is going to be able to start
fixing this stuff automatically. So the best thing
I can do is build that myself and disrupt myself.
So now we actually have systems that if you were
to use AutoHive, an error would be collected by
Rayun gets sent to a AutoHive agent. It puts the
code in a secure container. It runs a coding agent
inside it. It raises the code into into a pull
request on GitHub. It then has a team of agents
go and review that. It goes in a loop with the
coding agent and once they're all satisfied it
will elevate to being assigned to a human to say
do you want to ship this to production here's the
code change and so this happens now we will have
bug fixes available within about 30 minutes of
a person encountering an issue with our software.
Which is pretty cool. So significant ROI benefits
across the business I wanted to throw in a few key
lessons. So the human in the loop is extremely
important. Extremely important. So, we have for
example, and I've got a screenshot coming up,
but any action that can be dangerous or make
change, it will prompt you to approve it. We
also have native mobile app, so it'll do it via
push notification. And this is going to sound like
a joke, but my dream is that when I wake up in the
morning, I get my phone and there's a pile of push
notifications. I go, "Yes, yes, yes, no, no, yes,
yes" and then when I get out of bed to go brush
my teeth, half my day's work is done. That is the
future we should be striving for. Human behavior,
however, defaults to the easiest path. And this is
where it's actually more like a psychology concern
when thinking about regulation in my opinion
because what we see is people just start going,
"Yep, yep, didn't read it. Yep. Next thing,
whatever." And that's how you end up with Deloitte
in Australia, you know, getting done over for not
reading their their own reports. And making
sure did you really review the outputs. I don't
have a strong answer for this other than maybe
saying do you apply more AI to the problem? But to share one thing that we do in software
development is if you raise a code change,
the typical standard is that the same person
who raises the code change cannot approve the code
change. So you get a two factor type behavior. And
my suspicion is that this might be where this kind
of goes to just say as the person who created it,
you don't really want to self-review somebody else
looking at it going what did you actually do here
might be the mechanism. But we see that a lot.
And then the question is who's liable when an AI
goes wrong. And one of the the things that I find
about this and I've put it in bold there is I'm
watching a lot of people in the tech industry kind
of be like the AI did it wrong. And I'm like,
it's still just software though. Like who was the
person responsible for it? Like how much do you do
you kind of pretend that it's the software's fault
here? So a good example is Amazon's had some major
outages related to their use of AI. Is it really
any different to when a human being is a software
developer goes and runs the wrong SQL statement versus a human being running an AI and letting it
go rogue? My sense is they're actually kind of
the same. But we do tend to consider AI more as
an expert system. So thinking of it as assisting
in decision making and not necessarily making the
decisions for you. Especially at this point
data integrity is still a big problem. Context
management is absolutely critical. This is also
something that's really important to teach people
who are working with AI. And we do need to
upskill everyone to understand this. The better
you understand how to work with an AI, the better
the outputs you'll get from it. And there are
some great organisations around Wellington as
well that are helping like AI supercharger.
And like everything in the world, garbage in,
garbage out still applies. This is just a
quick screenshot of the approval process in in Raygun. So, I get a morning email sent to myself that
analyses my calendar, cross references it with all
my emails, does a deep research with perplexity,
and sends me a report saying, "Here's all of your
meetings today. Here's everything you need to
think about with these people." I thought it would
be kind of gimmicky, but it turned out to actually
be really, really bloody useful. Um, so I went on
a bit long, but thank you very much for having me.
Hi everyone. I'm Breccan. Also used to be
a developer, also running a business for nearly
as long, as probably about the same amount of
time actually. I started mine two years younger,
I think. I've got a couple of little projects
I'm talking about today. The first one is this
piece of work that we did for PCO last year
which has been about improving the drafting
process for legislation. And it's really been
a fascinating piece of work because it's sort of
been done in that trials all the way down.
I know Andy will be on the panel after this
and you can ask him about the broader
context, but I think a big thing there is they
did multiple things, not just our piece of work.
They tried a lot of things, but even inside of our
piece of work, I think one of the key things is
the way that we've been trying lots of stuff and
seeing what works. And you can sort of see that we
sort of did we set up a hypothesis based approach,
you know, that research approach of going
here's all our ideas of what we think will
happen. And then we started looking at how
are we going to apply that to the real world.
And you can see that in the kind of interface
that we built because a lot of this is about
humans and testing it with humans and seeing
if they actually adopt it and what works and
also making it clear enough about what's happening that they, we, can track back their feedback and
everything else and go what's working, what's not
working, what's that. So what you can see here,
so if you see across the top here, it's all these
little tick boxes next to colored things.
And that's basically all of the little agents
that we built to go check for this kind of a
thing and people can turn on which ones they want
on, turn off what they want off and that allows
them to also see looking down the side we've
colorcoded what's feeding back different types of
feedback. So that means we're testing a lot of
ways of applying this content to a particular
piece of drafting. We're able to get feedback
targeted to what were we trying to do and improve
those pieces and people feel like they're getting
good engagement because they, you know,
they can see the feedback improving those
individual areas and also we can target it
to particular things that the workers actually
go these are the hard bits of our job. I think one
of the things that when you just do the kind of
not thinking too hard pieces you go oh yeah doing
the text formatting AI is great at that we can
trust it to do that. There's one thing in there
which is commencement clauses. Which had a
non-trivial amount of dedicated effort because
when we were talking to the drafters we were like
what are the things that you know a young drafter
who's just started takes a lot of years to get
better at it and one of the things that popped
up was commencement clauses. So we put more time
into making the review that the system did of
commencement clauses better because part of
the goal is not just making work easier. It's
in particular making work that is hard easier. Because I think one of the things that people are
learning with AI so far is it's really easy to
get AI to make easy work easier. You know,
we've got less and less need for juniors who can
just go and research a thing on the internet and
tell us what are the five ways I could do this
because the AI is really good at that. But
I think that the interesting thing is really how
can we make the AI help with stuff that is hard.
So this has been a really great little project
of taking that sort of creative approach to having
lots of little bits, seeing which ones will work
and then being able to iterate on that as fast as
we possibly can. And I think that's really key
to having a good experience with rolling out AI in
more complex organisations. And then I just wanted
to talk a little bit about one other project we're
doing recently because I think it plays a really
nice other end of the spectrum for regulators.
This is for a large multinational organisation
that I can't quite talk name but is around
their procurement and they had this problem
where they were they provide funding to lots
of different countries. The countries don't have
the expertise to review against the requirements
that they require that the countries that they're
funding add to all of their procurements. So they
needed a tool that they could actually put out
to the partners that they were effectively
applying policy to, which might sound a little bit
similar to regulators and help them go, hey,
have you actually met this? And if not, start
flagging it up really early so that people
could respond to it and that kind of thing. And my
favorite thing about this project is actually that
we didn't build it. They had a team of their
procurement people that had one young guy who was
decently technically skilled and they built their
entire prototype that proved that it was a good
idea by themselves by just getting started. And then they came to us when they were at we need
this productionised. We need to start, you know,
we've proved that at this level it'll work. We
want to bring it up to the next level. We want it
to be performing more. But they've already proven
the business case by just having some motivated
people do it. And I think that's a really key
part of these projects in organisations is
encouraging people to just build. I think
it's really easy for an organisation to go let's
wait around and like look at the research. The
thing is this market is moving fast. If you go
and wait for someone to put out a think
piece about what AI should I use to do regulation,
you're already months behind. Whereas most of the
stuff is also very accessible. They are trying to
get it into your hands. You can go out and have
a play and you won't do the best outcome, but if
you can't have a play with it and go, "Oh, I can
see how this would work." And start doing that
proof yourself, there's probably lots of other
things that you can improve that way. So maybe do
those ones as your initial investments. And I
think that's a really valuable part of looking at
this because AI just makes it easy to start doing
stuff, start deploying it, and start testing it.
And that would be my big message to everyone is
it's much better to just start doing stuff. We
can help with that but also we can help later
once you've done the first version. Hi, kia ora koutou. My name is Finton Blake Kaihatū, bainistíochta
at DataSing. So I'm going to talk
you through a bit about what we're seeing. We work
a lot with the public sector around
Wellington. We started life as Allana said as a
data engineering firm. And really we're about,
you know, enabling people enabling organisations
to make better decisions with data. And up until
around November 2022 to JD's kind of point
with the explosion of GenAI, it's been restricted
to system kind of data, stuff that's in rows and
columns, but GenAI has opened up this whole world
of unstructured data and it's kind of made
the art of the impossible very possible. So a few
observations I thought we'd start with today
about what what we're seeing around the place
when it comes to the adoption of of AI. No big
surprise Copilot was already mentioned and
pilloried but it's a good place to start. I
think what we see is a lot of kind of
lower value decisions and kind of transactional
stuff which which Copilot lends itself quite
well to but not maybe those higher value
decisions that that have a higher volume
as well of data being thrown at it and
of course the veracity of that is very important.
And what you get out at the other end and really
the main barrier hasn't changed for us in the
last two years I would have said two years ago
maybe there was two things info security slash data
privacy was a big thing and the truth in what's
coming back at you and I think info security
data privacy has got a lot better you know the
controls are there that weren't there before
some of you will be familiar with the term rag
where you can kind of zone your AI in on just
your your estate that was kind of a three-six month
project, two - three years ago. Now it's like a little
bit of a radio button job so that's improved a
hell of a lot but the veracity and the
confidence in what comes back out. Still
a bit of a problem. You know, why we're a
technology firm, you you'll see I'm not the best
with it from from those PowerPoint stock images.
You know, the business process
and the people are very important to us and
I think, you know, my my own personal background
I was the worst mathematician in Ireland in
the year 2000 and I've got the results to prove
that if anyone wants to see them. But one of
the things that I found during that time in
Dublin was the intimidating kind of atmosphere
in a mathematics environment and the
way that you were kind of afraid to ask the stupid
question. And even with data engineering back when
we kicked off in 2021, I felt, you know, terms
like data science were coming into play. Never
heard of it. It's it came about in the 2000s, but
when you dig into it, it's stuff that's 400 years
old. And I think people need to realise that
about AI as well. And be kind of empowered
and encouraged and confident to ask themselves the stupid question.
So we like to start
thinking about you know another Jeff Bezosism
keeping yourself as close to the customer as you
possibly can and with our work with Allana's team
at MartinJenkins we're interested and we like
talking to business owners, service delivery owners,
you know policy owners, and those kind of personas.
Because the other observation I would say around
Wellington in particular is everybody's got a list
of 35 use cases but most of those 35 use cases
are kind of owned and held by it and I think
that needs to shift a little bit and what I
want to talk about today just in the next
few minutes and show you a little bit at the end
aswell is maybe thinking beyond the Copilot
use case and how that can kind of manifest.
So we've got two cars here and I just,
indulge me for a second with the
the transport analogies. You've had a few of them,
but I was listening to the head of Anthropic
there a few weeks back and he said with GenAI,
it's like we've given a teenager a car and
they've come back and they've said, can I have
it in green and can you make it go faster? And
we've just kind of said yes, no problem. But
what we haven't been doing is thinking about the
teenager, the persona of the teenager, the
mindset of a teenager and their appetite for risk
at that age. And it made me kind of extrapolate
that analogy a little bit more. And the first
flight was, if anybody can take a guess in 5
seconds when the Wright brothers hit it off. Not
history buffs. 1903. The first flight
the altitude was just about where that light
is, 2 metres, so a little bit taller than where I am.
And it went for 35 metres. So I don't know, would that
be two tennis courts, there thereabouts? 1903. 1935
was the first checklist for the airline industry.
And if you think about commercial flights, I think
around about then, wars aside, that's when it kind
of took off. You know, if you watch any films,
James Bond was rattling around the Caribbean in
the 1950s, commercial flights were available,
but it took that period of time to kind of move
from that prototype phase into scale usage phase.
And I want to talk about a few of those
little things today. So on the left hand side,
you know, we're in the Wright brothers world
where hopefully they weren't hallucinating,
but they probably were when they thought it up.
There's a lot of ambiguity with where this could
go, I guess. And you know, when we use the
kind of Copilot experience, we don't
really trust what's coming back. We kind of check
it and check it. We've got obvious performance and
scale issues. And in the regulatory world,
we deal with large volumes of data. You can't
throw that up into the kind of chat windows or the
context windows very easily. And as a result,
it's kind of for small scale decision-
making. So, what I'd like to kind of think for you
guys to go away thinking about is what the other
side might look like because it isn't necessarily
a Copilot type experience. And the first thing to
do is, I think, is put the human in the loop and
and put the driver in the car. And that's not to
say there isn't, you know, in the chatbot you are
you're you're continually prompting, but you've
got to think about those humans in the loop and
what they might look in a regulatory world and
the chats that we've been having. I think the
policy analyst is one. You know, maybe doing the do. I think the legal kind of area is another.
You know, what does that look like? And then the
leaders as well. What does that human in the loop
look like? And our little demo that we'll run at
the end, I'd like you to kind of think about where
you pick out those personas. What part they
might play as the human in the loop. Okay. The
other thing is to use AI at scale for those higher volume, higher scrutiny type decisions
is setting the right context.
And in a regulatory process and I'll use a
submissions analysis because we, and we do have
an example coming up of that. There's some
strict rules on that. Like in in a Copilot world,
you can kind of go out to the internet and take
whatever information you can possibly get about
a topic, but when we're in the regulatory world,
I think we've got to be very strict about what
we use because there might be some like legal
reasons that you can only use certain things and
not other things. I know in the world of of
the Commerce Commission, for example, some of the
the consultations they do. So setting the context
is very important. And as the humans in the loop,
you've got to think, I think, really hard about
that, like what is the right context in order
to set those AI agents running. Examples of
that to move away from the jargon, are draft
legislation, submissions themselves that come in,
peripheral acts or maybe some guardrails or other
guidance that you might have. And they all lead to
the map of where we're actually going and the
rubric or the set of instructions for AI to
trivialise that back to the kind of chatbot world.
It's the prompts, but you've got to put a
framework in to say this is how I want the agents
to behave and these are the bounds I want to put
around it. I think you know most of you will be
familiar with the procurement processes here in
for New Zealand government and at the end of all
of these RFPs which we're very familiar with they
give you the evaluation framework and they say 0
to 2 can't do it at all, 9 to 10 exceptional. That's
a rubric, that's a set of instructions
and to use AI at scale you've got to tailor that
to the project that you have and I think it's
really important for the humans to understand
what that is. We heard a bit about evidence and
orchestration there from JD. And
when we're talking about engineered AI across a
project or across a business process, there are
multiple kind of hares running at all points in
time. They run in various sequences. Sometimes
they run in parallel. When you're dealing with
massive troves of documents to engineer
that, you've got to break it up. If you try and
put a pipe across the cook straight in one hit
and it breaks, you're going to have a hard job
fixing the pipe. You're probably going to break
it up into little chunks and put little pipes
in. So, when one gets eaten by a shark, you just
have to replace that little bit somehow. Don't
let make me go any further with that analogy,
please, cause I don't know where it's going. But
it's time immemorial. You break things up
in order to kind of engineer them for
scale. And with that comes orchestration. And
you've got to orchestrate these agents that
are running because they're doing multiple
things for multiple people. But each of them
should go and bring back something to the user,
to the human to say, "This is what I've done." And
then when you get to that court of law, you know,
down the track, you can show this and you can
evidence why it's taken the actions it's taken.
Okay. And I think lastly, and there's been a a bit
of a theme about it today, is just that kind of
that literacy level of AI and that kind of not
being afraid to ask the stupid question.
And I think the more that we keep these in IT, the
less we will raise our AI literacy levels and that
curiosity won't be there. So our our kind
of mission is to make these as intuitive as, and as
kind of interactive as we possibly can. So people
aren't afraid to do that and they don't get kind
of green text on a black screen. And ultimately
it's got to mirror the business process. You
would be, you would be kind of castigated I think
in some tech world for not being disruptive
enough. You know, you have to kind of change things
materially. Think about the the iPod. You're all
of an age you probably remember it but like
before the iPod there was the Walkman and the CD
the thing for CDs. What happened was there was
a load of buttons on these things and then the
iPod came along and it's one button. I'd love to
say that DataSing are about disrupting like one
button, but we're not. And the public sector
in Wellington, I think AI literacy has to be
raised up a little bit before we can disrupt at
that scale. So let's look at existing business
processes where we know there's some opportunity
and have a go at them. Okay, this leads into the
demo. What I would like you to do is think
about those icons on the right hand side of the
context, the rubric, the intuitiveness and see how
you could apply them in in your business process.
So we're taking the use case
here is basically a thousand plus
submissions on a draft piece of legislation
and it's mirroring loosely the process that a
team of analysts would have to work through the
the processing time. Obviously, this runs for 2
minutes. The processing time is 6 hours when we do that from scratch. But I think the real
power of this is the 6 hours is probably weeks
and months in in today's kind of landscape.
And it all has to culminate in something
that goes somewhere else that's used before
a select committee that was brought up
at the at the start of this as well. So,
we'll get to the kind
of the output there shortly.
This is like me showing everyone the baby.
Yes, I've got three kids and it's pretty handsome.
So yeah, we this is a product that
we've developed with with MartinJenkins
and the design of this with real
policy people has been fundamental.
If techies tried to do this themselves, we
would get it wrong and we would spend six
months trying to implement it and kind of shove
a round peg into a square hole. So, you know,
it's got to be driven from that business outcome
perspective. Okay, we're coming to the end. It's
but it may not be a report is the final artefact
either. We do recognise that and I ideally this
would be going back into your system or record
whatever that that might be. But it is it is
it is created pretty quickly. And I think
the other thing with a submissions process as
well just before it finishes you will find themes
over the course of a submissions project that you
then need to update your rubric or your set of
instructions. And that's what this tool enables
us to do. So you're not kind of bound to what
you started with. It discovers new topics and
themes as you use it and it feeds those back in.
That's enough out of me. Thanks very much and I
look forward to meeting you after. So MakerTech, we are a team of entrepreneurial engineers. We
started experimenting with AI about 3 years ago.
We've had a dedicated AI business arm for the
last two. We've done some really good work in
education, in retail, in healthcare. But I'm
going to talk possibly about our most important
work. Shouldn't say that. There's other clients in
the room, but which has been made our work with
DoC and helping them in their important mission to
help protect our unique species and habitats.
So DoC is one of our most complex regulators.
They're responsible for a third of the land
and each year they have to process hundreds
if not thousands of applications, leases,
permits, easements. Have I missed anything,
Joe? Licenses. It's a lot. So just thinking
about one single permit like a commercially
guided tour could involve the DoC team having to
review hundreds of pages of statutory documents.
And to make matters worse often times these
applications are scan PDF documents. So you can't
even use like simple find and replace kind of text
tools. So the problem's evident thousands of hours
spent each year in low value data retrieval
instead of high value conservation science.
So it seems like a clear use case for AI.
It's very dull. It's very laborious. It's
very dear. It's very costly to DoC. But the
problems with the off-the-shelf tools that they
tried is that they got lost in the overlap. So
what DoC is it does is very complex. So if you
take the example of the commercially guided
tour, that tour could span across number
of geospatial locations and across those
locations there may be different bylaws.
There may be different conservation plans
that have different priority in terms of their
hierarchy and the AI tools just got lost in
that overlap. So DoC needed a solution that
could reason and not just retrieve and provide
text. So we built a secure portable AI statsbot
internally. I don't know if you know this
we were channeling the name the AI parallegal but
felt a bit oldfashioned. So AI statsbot won
out. Three key things about this tool. One
works in natural language. So the DoC team can
ask very specific questions and get very specific
answers back. It's contextually smart. So as
I spoke about before, the tool can understand
different plans and different locations and
give very specific answers back. And crucially,
it derisks hallucinations. So direct page
citations, deep links to official documents
so that the team have high trust in the
solution results. Less time in PDF documents,
more time in the field. So three key things.
Efficiency. The research is now in minutes,
not weeks. Maybe that's even hours. Actually, you
said hours before we were speaking, Joe. Maybe
it's hours. So, it's drastically reducing DoC's
admin backlog. Consistency is really important.
The tool doesn't get fatigued by reviewing lots
and lots of documents. It doesn't kind of miss
anything. It can look at everything and then
allow that quality decision- making for the
human to do. And satisfaction. Pretty sure nobody
at DoC signed up to do data mining. They signed
up to kind of protect our environment. This isn't
just a tool in isolation. There's, we've already
started scaling this. So we've worked on
a permission summary AI. So when you've got the
whole application, how do you provide a summit
summary for better iwi engagement? And then beyond this,
there's the obvious potential for them fasttrack
applications. So some work we're doing in
education is relevant here. We're actually
helping a very large education provider in New
Zealand completely fasttrack how they do grading.
So simple binary answers the AI tool can
handle and then the teachers has to focus and the
graders has to focus on more complex, more nuance
questions. All right, three closing thoughts
to take away that have already all been said,
which is fantastic. It doesn't matter if it's
black or white, but humans handle the gray. So,
chap called Herbert Simon. Anyone know that
guy? Not even my friends at the front. Really?
Come on. All right. In the late 70s, Herbert
Simon won a Nobel Prize in economics for his
theory on bounded rationality. I'll give you
the summary. It basically states that humans can
only handle a certain amount of data in their
heads at any one time. So, we always make rubbish
decisions. Right now, AI is a logic beast. So, it
can really help us in terms of capturing all that
information and presenting that information to us
to be better decision makers. I'm still going to
argue for now that humans understand nuance better
than AI. Right. So, while AI is this logic beast,
AI sorry, decision-making and regulation is still
a human craft. Takeway two, well done is
better than well said. Experiment to think and
learn. Paul, you've said this perfectly well.
Don't get too lost in the strategy. You're
probably in one or two camps. You're probably in
the camp of being really excited about AI, really
worried about AI. We tend to think in binary terms
as human beings. The key is just to kind
of start to experiment and think and learn.
The strategy will reveal itself. So don't get too
caught up in journey maps and service blueprints
while everything moves at lightning pace. Get on
the tools and try and build something. This is the
flip side. So the flip side of that is to watch
out for PoC paralysis. Loads and loads of poc's and
prototypes in the market at the moment. Not much
in production. So really think about two things
when you think about how you integrate a tool into
your organisation. First technically obviously
you've got to integrate into your systems
of record but more importantly you've got
to integrate it into your human operating systems
because technology changes fast but people don't.

Tēnā koutou katoa. It is fabulous to have you
all here today. I'm looking around the room and
I see lots of people that I've met over the years,
and you're all wonderful humans. But for those of
you who don't know me, haven't met me before, I'm
Allana Coulon, managing partner at MartinJenkins
and I will be your MC for today. At MartinJenkins,
we have gotten, like I'm sure all of you have,
deeply curious about the way that AI is changing
your world and our world, the new possibilities
that it is creating, but also some of the
risks and challenges that are coming with this
transition and particularly for you as regulators.
Other thing that's really important to say right
up front is that we are co-hosting this event
with the Ministry for Regulation who care deeply
about the capability, the performance and the
practice of regulators as the system, in their
system leadership role. And so it's been great to
work with Paul and Alex and the team from Ministry
for Regulation in planning for this event. This
event has been in the making for a while now and
I wanted to set the context by sharing how the
event came about, who's in the room and why,
what's going to happen this afternoon and
what we hope that you'll take away from today.
But before I get into that, just a show of
hands. Who is actively working with prototyping,
trialing an AI agent in your organisation at
the moment? Yeah. Right. So, not everyone,
but over half of you are already actively
in this game. So, I'm really conscious
we've got a wide range of experience and stages of
progress in this space. And mostly today, this
isn't going to be a really technical event. This
is intended to be real world, practical,
and honest. It's not about hype. It's about
what it looks like in your context as regulators.
This this event came about because in our work
at MartinJenkins we talked to a lot of you,
a lot of agencies and we've been hearing some
of the common challenges and the common desires
and one of those is people saying to us we
just love to talk to other people and find
out what they're doing. We're really busy
doing things in our own organisation but
we don't get enough opportunities to hear what
other organisations like us are up to. So this
kind of was born out of that feedback that we'd
heard from a range of people and then secondly
the Ministry for Regulation are working on
some practical guidance for the use of AI by
regulators and they invited us, they came to have
a chat with us as one stakeholder who might have
some thoughts around that and in the context
of that conversation we realised that we had
a common ambition about the desire to get people
together and share ideas and learn the lessons of
what isn't working and why. So yeah, just again
want to say thank you to Paul and Alex for that
partnership which has been a lot of fun and
it's been good getting to know you guys. So
each of us in this room today is at least
one of three things. We are a leader of a
regulatory system. We are a technology and AI
solutions partner and that might be inside your
organisation or as an external provider. Put a
few of those at the front of the room here and
all of us are organisational leaders navigating
pretty significant change for our organisations.
As regulators you vary greatly in terms of
the size and scope of your role. So we've
got regulators in this room from large Crown
entities and Ministries for whom regulating is
actually only one of the many things that you do.
We've got specialist occupational and professional
regulators. We've got sector specialist regulators
and then we've got some local authorities present
as well who regulate all kinds of things.
And I wanted to make a special shout out to
a couple of representatives that I'm hoping are in
the room from Auckland Council. Anyone here? Yeah,
welcome. I was delighted to see that you
guys made the effort to come down and join us.
And you've only been trumped by Joff
Outlaw, who has actually made it across
the ditch from Sydney to join us for as one of our
guest speakers today. So, thank you for coming.
But while what we might regulate and
who we regulate can be very different,
we actually have a lot in common. And that's
part of what we've been thinking a lot about,
which is that actually there is a lot that is ripe
for improvement in regulatory systems, processes,
and practice. A and a lot of the functions you
perform are very similar. So I wrote some of
them down. You triage and manage complaints.
You issue guidance to inform and educate. You
assess applications. You maintain registers. You
require information disclosure. You audit and
accredit. You investigate non-compliance. You
grant permissions. You communicate enforcement
decisions. And I know as regulators, you'll be
sitting there thinking, "Ah, she missed that,
she missed that." I probably did miss a few
things. But a lot of you are doing some or all of
those things, and the pain points that you share
are also very similar. I wonder if this sounds
familiar. Your people are stretched too thin with
manual processes or disconnected legacy systems
that introduce risk and delay. Laws and policies
aren't keeping pace with changes in the behavior
of regulated parties or technology advancement.
So sometimes you are locked into doing things in
cumbersome outdated ways. This means that there
are missed opportunities to focus on highest risk
and proactively address those emerging systemic
risks and issues because you're too busy managing
down backlogs. That's also because the cases are
getting more complex. And you're also starting to
see increasing use of AI by complainants and
stakeholders when they engage with you,
which means that you're either already seeing or
you anticipate seeing increases in the volume of
engagement and self-representation. And finally,
you're simultaneously experiencing heightened
pressure to increase the use of AI and
AI agents and heightened concern about
the use of AI from your boards, your people,
and from those you regulate. So, paradoxically,
I'm sure it feels like you can't move
fast enough and slow enough to please everybody.
And there's one more commonality that we all
share that I think it's really important to
touch on, and this particularly touches me
personally as a leader of MartinJenkins,
and that is our role as organisational leaders.
I observe, but I'm also sure that I do not fully
grasp the profound how profound these changes are
for MartinJenkins. And honestly, I sometimes wish
that AI would just go away. It's inconvenient
and it's messy and we were all already busy.
And I know there'll be people in the room
and say it's been here for a very long time
and that is true. But there are some things that
have changed. The pace of it, the ubiquity of it,
and the speed with which it is actually evolving
and some of the capability that now is available
and that even if we might not want
to deploy it in our organisations,
our people and our stakeholders are engaging
with it. And so unfortunately, as much as I
think one or two of you might be tempted, I
don't think we can regulate this one away.
So while many of us will probably share concerns
about trust, ethics, privacy, the environment, and
our future, I have a 17-year-old who's thinking
about what he's going to study and do next year,
and that's a tough conversation right now.
Navigating this as leaders in our organisations
is, I think, where it starts and stops. Senior
level role modeling, persistence, energy and
courage are required. And I would emphasize
energy and courage because we have to prioritize
innovation and develop new ways of working
while we're still delivering on some pretty
significant expectations that are in front of us
today in, just to depress us all slightly more,
a constrained fiscal and resourcing environment.
So this afternoon. So first of all what I want to
say is thank you for carving out the time to come
here today. Well done because you are all really
busy but this is really important. So it's great
that we're here in the room together to have this
conversation. So we have a fantastic lineup of
guest speakers. Shortly I am going to hand over to
Paul Delahunty who is the Deputy Chief Executive
of Reviews and System Capability at the Ministry
for Regulation. Paul will then be followed by
Jared Griffiths who will share his fascinating
experience leading the AI journey of Hutt City
Council. We're then going to move to some thought
provoking show and tells from four local New
Zealand owned and founded AI companies that are
developing agentic AI solutions for and with New
Zealand regulators and overseas as well for many
of you. And this will be followed by a panel of
guest speakers from Department of Conservation,
WorkSafe, Education Review Office and the
Parliamentary Counsel Office. And finally,
at the end of the day, I hope that you will
all join us for some light refreshments and to
continue the conversation. Our aspiration for
today is that you leave here with a new idea,
a new connection, someone new that you can
call when you need a bit of help or advice,
and more resolve and confidence to advance AI
agents in your organisation while navigating some
of those risks safely along the way. For me, it's
it's a real privilege to be here and thanks Allana
for reaching out and kind of suggesting the
event because it's it's a real great opportunity
for us to showcase some of what we're doing
and also to connect with with more regulators.
And it's also really good to be in a room full of
individuals whom I suppose you're the ones that
are the early adopters. You're the ones that are
curious. You're the ones that want to innovate.
and so that's great. It's good for us to have
that captive audience. But just like all of you,
MfR is navigating this challenging time just
as MartinJenkins is. And so we've got a lot of
the same questions you do, like how do we use
AI? Could we use AI? And I'll talk to a little
bit about what we're doing because I think our
role, one of our roles is as a system leader.
And so I'll articulate some of what we're doing to
demonstrate that we're taking a few risks because
it does require some risks. In terms of what I
want to cover off is MfR's role as a system role.
A little bit around the regulatory landscape
and then AI and how we can safely use it
together and hopefully give you a few takeaways.
There is one, I have one slide, that's just one
slide. I don't like my name. This is the only
slide I have. So I'm not going to kill you with
death by PowerPoint. And I'll talk to this in
a little bit but I just wanted to cover off the
Ministry's role. So the thing is you as regulators
and we have some solution partners in the room as
well. Is to focus on your individual systems and
how they operate. MfR's role is one to focus on
the system as a whole. And it is broad and it is
deep. We kind of have four key functions. One of
the key functions is to ensure the quality of new
regulation and that is around you know we review
proposals going through cabinet and that's our
second opinion advice. Another key part of our
MfR's role is to improve existing regulation and
respond to issues and that's we have some teams
that do regulatory reviews and also we have a
thing called the Red Tape Tipline that deals with
issues. I've just come from Parliament actually
there's a summit happening at the moment around,
it's a hospitality summit and so we are doing
a review around hospitality so I've got a team
presenting there we just have Minister Seymour
and Minister Upston so I'm in the process of,
I have to constantly switch context from
hospitality now to AI so I'm still making
that transition so apologies if I lose you
at some point. The third part of the role,
key function of the Ministry, is raising the
capability of those who design and operate
regulation and this is the key reason why we've
partnered with MartinJenkins today. And behind me
you'll see we have some guidance coming out. It's
AI guidance. It's a practical guidance. It's not
technical for regulators to help you navigate the
world of AI and innovation. We've tested this with
a number of people and it will be released next
month. And the other key function the Ministry
has is we lead the regulatory management system
and that's kind of around system stewardship.
It's how we bring it all together. So that kind
of ties in a little bit with today as well.
So I think the key thing for me is that AI will
play a significant part in how regulatory systems
operate in the future. And regulation as a system
it must adapt and we must adapt with it. And that
will be uncomfortable and it is going to happen
at pace and that's the the key thing for me is
is the pace at which things is happening.
It's relentless. As part of our building
capability we have a range of tools. One of the
other tools we're releasing some guidance on,
Alex will say this again, this week or next
week, is around regulatory sandboxes. Sandboxes
is a way to safely test regulation. It's kind
of like an experiment and kind of with some
constraints around it. It's a tool that's
been around for the best part of 20 years,
but it hasn't been well utilised in New Zealand.
And we think there's a real opportunity for
sandboxes and AI when partnered together in the
right situation to actually test some things to
keep you all safe. And, but, it's going to require
a bunch of conversations to get there and you need
to have the right regulatory settings in and your
in your legislation. I look at Andrew as I say
that as he's nodding away. So I think there's
a whole bunch of other things that need to be
come together to make that happen. But we've
got some guidance coming out this week around,
regulatory sandboxes. Regulation, AI matters to
regulation , because it will help regulators and
Allana's going to touch on this a little bit.
It will help us kind of target things early.
It will help us understand what matters. Will
help us use our resources more effectively.
And I I want to give some examples of how we're
doing that at MfR ourselves. But the regulatory
landscape is complex and it's one that you all
operate within. We've done some work and we've
identified 259 regulators across New Zealand.
So that number is larger than we had initially
anticipated. And so we're doing some work
around what that looks like. And it's a
complex and crowded space that you operate in.
And it's really complicated for regulators and
for regulated parties. In today's world there are
risks and Allana touched on this a little bit and
those risks are evolving quickly and public
expectations around timeliness and fairness,
transparency, accountability that accountability,
they are growing. AI can help us navigate that
but it can also muddy the waters. And we're
trying to understand how best we can use it
and the questions that we need to ask.
So just a little anecdote at the MfR
select committee hearing just before Christmas
we were asked a series of questions by Chlöe
Swarbrick around how we're using AI around the Red
Tape Tipline. The Red Tape Tipline is essentially
a web form that enables members of the public and
businesses to raise regulatory issues with us.
And are we using AI? Absolutely we are. We'd
be remiss not to and we're a small organisation
dealing with large volumes. But the nature of
her questions were more she really wanted to
understand did we still have people involved in
the process? Were decisions being made by AI or
were people doing it? So fortunately we're able to
give her that reassurance that, that is the case.
We have people involved in these processes but
there's a nervousness and that nervousness is by
leaders, it's by politicians, it's by members
of the public. So the question is how do we
all navigate this and do I have the answers?
No, I don't. So if you're looking for them,
apologies. So in terms of what do we all need
to do? I think I encourage you to learn by
doing. I think and I'll say this as a I can
say I'm a public servant. As a public servant
we can sometimes constrain ourselves by
overthinking things by seeing too much
risk by overanalysing. And I think with AI
because the pace at with which it's changing
we need to actually trial things. We need to
trial things just small keep it small adapt,
learn from it and move on. Yeah, this is
something that we've been doing across our
reviews. So I think about the first regulatory
review that we did which is going about 18
months. We didn't use AI at all. It's 18 months
ago. We're only 2 years old as an organisation.
Our last review which is the telecommunications
review. It's going through cabinet next week.
We use AI on a number of areas. We used it
to analyse legislation. For the first review,
we did that line by line. We had a team of people
reviewing that. I won't say how long it took or
how many people there were, but it wasn't small.
For the last review, we used AI to do that. For
our summary of submissions, so when we go out
and we engage with stakeholders, we use AI to
help us, you know, summarise this, but we still
reviewed things and looked over it. But again,
with our first review, we did that manually. It
took us six weeks. And then we've also used it too
for all of our workshops. So we've run a lot
of workshops and we use it to transcribe those
workshops, pull out the themes and again I
cannot quantify how much time it saved us
but has saved us significant amounts of time and
it's enabled our staff to actually focus on the
parts of the reviews that can add more value.
And the reason I'm sharing this with you is
because you know MfR we have a role as a yeah,
you know we're part of the core public service
what we call again? We're a central agency. I
get this wrong. We're a central agency. Yeah,
apparently this is really important. And
so as you can tell, I registered to that.
And so we have the system role. So I think it's
really important for us in the in the center to
actually share some of our experiences. Have we
got all these things right? Absolutely not. So
with regards to our summary of the submissions
process, I know Carol's sitting there, the first
few times we tried this, we we haven't it wasn't
didn't work that well. But we've got better at it
because we've tried things. We've iterated. We
haven't been afraid to actually get it wrong.
And I think that's where the public service
has tripped itself up in the past. And we're
fortunate we have a Minister and a CE who both
have high risk appetites and I think that gives
us that authorising environment that probably
doesn't exist everywhere. I think that's a key
thing for me in terms of having that authorising
environment. So you as leaders you have a role to
play like Allana asked you the question before
who's using it? So I have my notes. I actually
have a similar question. So I no it's fine. So I'm
going to do it anyway. So it's a little exercise.
So, so if you just want to raise your hand, who
has used AI personally and in work? Just who
uses it in any way, shape or form? Just leave your
hand up. Who has used it today? Oh, most people.
That's even better. And so, and so and then who
has actually deployed or their organisations
have deployed AI into production environment
that's actually changing how their operations
actually work. So, a few more hands going down.
Um so, again, I knew I had a there there. So,
but the reality is a lot of you left your hands
up, which is great, but I don't think that would
be the case across Wellington. And that's where
it's a captive audience. It's the right audience.
And so clearly you have a different risk appetite,
a good level of risk appetite. And so, what I
encourage you all to do is have that conversation
with your leadership teams, with your Minister,
with your boards, if you're Crown entities, if
you haven't already. Because I think finding out
where that risk appetite is will help provide the
authorising environment for which you need to move
forward. So I've got all these notes here. I'm not
following them at all. This is terrible. I knew
this would happen when I come from another event.
That's okay. So let's see what I have missed out
here. Here you go. Start something. So I think
leadership is critical and the thing is we have a
lot of leaders in the room. And the culture that
you create is so important. So I encourage you
to share how you're using it with your staff and
your managers. I've got some managers who embrace
it and others that don't and that's okay. But
understanding that so that we can help them with
the journey that they need to go on. And when I go
back to my example from the the Red Tape Tipline
we have we're in the process of digitising and
improving how we work in that space. And the
team when they've designed things they are very
reflective of that conversation we had at select
committee around needing to ensure that people
are involved in the process and so by having
that conversation out of them listening to it it
has actually really influenced how we've designed
things. I won't keep rabbiting on. In terms of the
takeaways, I I think it's really important that
the, I kind of didn't cover this thing off.
What I want to do here is we are working through
the AI guidance and we will release this in the
next month, but I wanted to offer to you all
an early version of it once I've got the final
approval on before it gets up on our websites.
We've got all your email addresses and
I think it'll be a really valuable tool
for your organisations going forward. The
key thing for me is change just ignore
AI is anchored in a few things and it's good
governance. It's judgment and accountability and
the implementation of AI is no different in terms
of you just need to employ those same principles
to AI. The big difference and Allana touched on
this is the pace at which things are changing. It
is relentless. And I don't think we as individuals
can keep up with the pace of change. So what
I encourage you to do is share with others.
And again Allana touched on this was actually
providing this is providing an event for you to
connect with others but there's no reason for
you to try things on your own. We at MfR all the
trialing we've done to date has been on our own.
But actually as regulators there's some key things
that you do that's very similar. There's no reason
why you can't do some collective trials. I think
that shares the risk. It distributes the risk.
And it also increases the learning that can take
place. And I think this is for me this is part of
the new world that we need to operate in. Because
things are moving so quick I don't think we can
keep up individually. The other key thing for
me is AI adoption is fundamentally a people
and a strategy challenge. It's not a technical
challenge. I think it's a really key takeaway
for each of you and I I'll say this I
look at the vendors on the front row,
I am an ex IT professional and I underline
the word ex, but it's really important that
this is driven from the right place and driving
this from a technology solution perspective is
not the right place sorry I'm sure they'll agree
and then they're the right vendors if they're not
then move on to someone else so yeah. So I guess
to finish off, so our role at MfR is to build
system capability. Your role is to build your
organisation's capability. We're here to help.
We're a very small organisation. So we're going
to try to provide tools like this. We also have
another tool called the RegRoom for those.
Who knows what the RegRoom is? Actually just
see here we go. Okay. A few. So the RegRoom
is another it's an online tool we have for
regulators. That provides a range of training
modules for individuals. Some organisations
have mandated the training, but there will
be a collaboration space on there as well for
regulators going forward. So, we're building
that capability up as well because we want to
connect regulators. So I think yeah, we don't
have all the answers, but we're here to help.
So I guess that's it. I in terms of
for me, thank you for being here. I
thank you for the work you do. I mean, I'm
really looking forward to the conversations
that we have in a little bit from
the other speakers. And I just
want to finish with a short whakatauki
waiho i te toipoto, kaua i te toiroa.
Let us keep close together not far apart.
So thank you very much. Kia ora koutou,
it's a real privilege to be here and just to
acknowledge MartinJenkins and the Ministry of
Regulation organising this afternoons event.
I think what you've heard the power of coming
together and actually sharing what each other
is doing and understanding what's possible. I
think there's a huge benefit in that. And I also
think some of the points that Paul made will be
reinforced and in some of the things I reflect
on as well. Which is also good because you'll
be getting some consistent messages particularly
around leadership and how to approach it. So two
years ago, Hutt City Council made a really
calculated decision that we thought AI could
help us do more with less without necessarily
impacting the services that our community valued.
So today I'm going to share what we learned, how
we approached that, and why I think organisations
really can't wait any longer to transform how
they operate using AI. So our leadership team
kicked off conversations around AI in early 2024.
And at that point, what we were looking to do is
come together as a leadership team and understand
what our collective ambition was and what sort
of risk we were up for as we approached AI in our
organisation. So when we had that conversation, we
were approaching it very much from a perspective
of what are the challenges that we faced and what
tools or approaches could we turn to that would
help us navigate through those. So you'll be
familiar with council's challenges because they're
the challenges of the local government sector. We
have been working through a period of sustained
high inflation and cost of living impacting our
residents. We are fronting up to a decades
long problem of underinvestment in our
infrastructure. And we also are working through
a significant programme of government reform and
having to respond to that. In the face of
that significant challenge to the sector,
we could either look at the scale of it and
shirk from it or what we could do is we can
turn to things like AI and technology as a
way to navigate through. And that's exactly,
that's exactly, what we did with the support of
our Chief Executive at the time, our current Chief
Executive. I led an approach of being an early
adopter in the public service. We didn't have all
the answers at that point, but what we did see was
evidence from overseas that there was definitely
potential in AI use and it could transform the
way that we worked. So in the last 2 years, we've
rolled out AI tools to around 300 of our staff. So
for context, that's about everyone who works in a
desk space role at council. So excluding people
working in libraries or in our swimming pools,
we conservatively estimate that people are saving
around 30 minutes a day using those tools. And for
us, that's about 44,000 hours a year or about
a million dollars in productivity released. In
practice, what that looks like day-to-day is that
we're turning around work faster. It's often to a
higher quality, and we're demonstrating value back
to our rate payers through its use. Our staff have
been particularly outstanding in the way that
they've adopted and embraced AI. So they've
integrated into their work. They understand
the risks and they're using the tools really
robustly. But that's only really part of the
story because we've also had a programme where
we've deployed AI agents and are also using and
looking to adopt more complex solutions. As well,
we're refining our AI and automation tools to
review customer applications and regulatory
services. Things like building consents and
resource consents. We're using it to draft
governance minutes. We're looking to automate our
LIM's process and general administration processes
across the board as well. So something that will
very much have a long list of use cases and we
have an issue with capacity and budget to progress
all of those that we've identified. I'll give an
example of what this transformation looks like
in practice and it's it's quite a straightforward
example but gives you a sense of how we're
achieving some of those benefits. Our democratic
services team at council supports our elected
members run their meetings and generates all
the minutes and actions following those. So for
context at our council there was about 75 formal
meetings in the last calendar year that that team
support them. We developed a simple AI tool that
took transcription, took rough notes and standards
that they were trained on, and is helping that
team turn around all of their documents much
much faster than they were able to do before. It
slashed the time that it takes to generate those
documents. So when a vacancy came up in that team,
the manager had every confidence to say she didn't
need to fill it. And the teams continue to perform
at or exceeding a level than they did before
when they were fully staffed. This is an example,
but what it shows is that shows that AI is more
about than deploying technology. It actually has
a significant people impact as well. So technology
is the enabler, but adopting AI as we have quickly
leads you down a path of reflecting on your
organisation's operating model, what you do,
how you do it, and how your people are working
alongside AI to deliver those services.
So for regulatory organisations like ours and
like yours and as AI progresses and we move
for example to autonomous AI over time we need to
start thinking about things like quality assurance
overseeing the work agents are producing and
ultimately upholding the trust of the services
that we provide as well. Those are exactly the
questions we're working through at Hutt City
Council and it's also I suspect the same questions
that you're all working through as well. So joined
up conversations like this one, national guidance
is also really important to help us navigating
those challenges together. But if you're not
yet thinking about how AI can transform your
operating model, it really ought to be a strategic
conversation you're having as a leadership team.
Because what I can tell you is I've had dozens
of conversations with organisations across New
Zealand and they're at the start of their journey
and looking for practical guidance on where to
start. The appetite's clearly there. They know
AI is something they need to be thinking about,
but they lack the practical tools. And other
frameworks to help them move forward with
confidence and to actually have a crack at this.
Just going to share a couple of key reflections
from my experience leading this work. And a couple
of these points you you've heard already. First,
strongly encourage everyone to approach AI
adoption as a strategic change initiative,
not an IT project led by your IT teams. It's
not something you buy, deploy, and can walk
away from. Not when it transforms processes and
the way your people work. It takes time to build
those relationships, to deploy the technology, and
to work people through what that change means for
them. Secondly, senior leaders need to champion AI
adoption. As a tier 2 leader in my organisation, I
could drive this programme. I could help overcome
and set risk tolerances. I could work through
organisational resistance, and I could ultimately
keep the programme moving. Too often what we see
is organisations are leaving AI adoption to tier
4 managers who toil away on the AI work but don't
have the traction and can't resolve some of those
resistance in in the organisation. So it does need
senior backing. Third, moving too slowly outweighs
the risk of moving forward imperfectly. You won't
have all the answers and your AI journey map
doesn't have to be perfect. We need to learn
as we go. And because organisation change is is so
dramatic what we need to do is is we can't afford
to be left behind through that. These three
reflections are ultimately about leadership
which is critical when it comes to operate,
adopting AI effectively and achieving benefits.
Organisations cannot afford to underinvest
in leadership. It's a capability you must
build because it's what differentiates good AI
adoption from bad. Our work at Hutt City Council
is ultimately a story about the transformative
potential of AI for the public good. We see it
as a way that we can improve our performance and
the services we provide to the community. Yes,
there are lots of considerations to work
through. Yes, there are risks we must manage,
but it's true that New Zealand has been really
late off the mark when it comes to AI adoption
and there's still a lot of opportunity on the
table we need to pursue and we can't ignore
that. In regulation which is obviously our focus
this afternoon. I'm really looking forward to
hearing more about what you do as well because
there's a lot to learn from from the work that
all of us are doing but the potential to deliver
better outcomes and services is genuinely exciting
and AI makes it imminently achievable as well.
So I'm really looking forward to continue this
conversation both in the room and beyond and
looking forward to the rest of the event. Kia ora.