Exploring new product opportunities with Generative AI

A blog by Henry Bacon, Tom Scott, Chloe Langan and Sam Stevens.

Like most people, you’re probably wondering what new opportunities Generative AI offer to improve your customers experience - and it can be a difficult question to answer. In particular, knowing where and how to get started.

At cxpartners, we’ve been curious too. So, we’ve started a series of mini-experiments, prototyping and testing with AI to explore the possibilities, gather insights and gain clarity on what works and what doesn’t.

We’ve found that by following a user centred process of iteratively prototyping and testing with people, you can quickly uncover invaluable insights within a matter of days.

• Using prototyping to help you figure out if Gen AI is suitable for your customers
• How to leverage Gen AI so it works well, and avoid what doesn't
• The possible risks that you'll need to overcome and manage
• Who you'll likely need to involve in the process

To kick off our internal experiments, we created a ChatGPT-style GenAI bot to help people with a topic that has recently interested us - “financial guidance”. We wanted to see whether GenAI could provide UK citizens with effective pension guidance (and therefore help to close the Advice Gap).

Within three days we had evidence to suggest GenAI offered real consumer value in this use case. And determined the steps to take when designing it, so it’s as effective for people as possible.

At cxpartners, we’ve been using rapid prototyping techniques to design better digital services for years, and the approach we used to tackle GenAI was no different.

Using Open AI’s custom GPT builder, we created an AI financial expert, specialising in retirement planning and pensions. We called it PensionBot. 

We gave it instructions defining its expertise, behaviour and unique conversational style so that it would act in an appropriate and specialised way.

We also provided a set of guardrails so that PensionBot wouldn’t act beyond its remit, such as offering formal financial advice or discussing topics other than pensions.

And, lastly, we collaborated with a registered financial adviser. The value of collaborating with an SME can’t be overstated. It meant we could draw on their knowledge, experience and judgement on crucial use cases and scenarios.

Interestingly, after interacting with PensionBot, they were surprised by how well it worked, commenting that its responses to general pension questions were “Pretty much what I would say!”.

Once our prototype was ready, it was time to test it with people.

We recruited participants who wanted help with their pensions and, importantly we ensured each person understood that PensionBot was a prototype and that the information it provided should be researched before assuming it was correct.

We gave each participant 20 minutes to converse with PensionBot, before discussing their thoughts and feedback afterwards.

The entire process took just three days and uncovered a lot of insight. Here is a snapshot of what we learnt.

We were surprised how well PensionBot performed, but there’s significant room for improvement. Whilst extensive testing would be needed for a real-world service, our exploratory testing suggested PensionBot provided useful, accurate and up-to-date pension guidance. However, we identified some important areas for improvement.

Participants often became stuck whilst using PensionBot simply because they didn’t know what to ask. As a result, they found it extremely helpful when PensionBot suggested questions or topics for discussion. As one of our participants remarked; “I’m not sure, tell me what I need to be asking!”

Participants often wanted to know where PensionBot was sourcing its information from. And, when it was able to reveal its sources users weren’t always satisfied.

Whilst we had instructed PensionBot to source content from established sources (such as the FCA), it did occasionally draw information from elsewhere - specifically places that weren’t so familiar to the participants.

This is a significant obstacle to overcome before a service like this could be considered "ready". But, also an opportunity to address by demonstrating its value to users by being explicit about where content is sourced.

We deliberately instructed PensonBot not to give formal financial advice, such as making investment recommendations. Instead, we focused on ensuring the bot's general guidance was accurate.
That said, as we've seen in a growing number of public cases, controlling GenAI's output is a challenge. We explored how to mitigate this, with carefully written instructions and warning messaging for users. However, the risks of 'misbehaviour' and harm are significant. This presents a real challenge for GenAI in use-cases like the Advice Gap. We're planning to research this far more extensively.

As you can see, through rapid prototyping, collaboration with subject matter experts, and user testing, it’s possible to gain important insight in a short space of time.

We’d go so far as to say there is no more effective and efficient way of learning so much about an idea so fast. We do this kind of work day-in day-out and it still surprises us!

If you’re looking to understand the role that technology could play in a service, we’d always recommend a user-centred approach. Doing so helps you get a closer look into the needs and expectations of people, and enables you to de-risk your investment with evidence gathered through research.

A user-centred approach like this couldn’t be more relevant, particularly for a piece of technology like GenAI that is developing and improving at such a rate.

Over the next few months, we’ll continue to experiment with the possibilities of GenAI, its potential application in the real world and how it could be used to help solve real problems.

Perhaps you’d be interested in becoming a contributor and participating in one of our upcoming research sessions. Perhaps you’ve started looking at AI and want to bounce ideas around. Or, maybe you’re grappling with similar questions to us and would like to understand more about howGenAI could be used within your organisation.

If so, we’d love to talk to you. Drop us a note at hello@cxpartners.com if you’d like to collaborate!