I am alarmed at the rush towards use of AI. There have been many technological leaps in my lifetime and many new products for sale. Best practice before committing to a purchase is to draw up a list of advantages and disadvantages. Specifically, what the new goodie does better than the old, and what it does not do well. Or worse, where it might even do harm. I tried this exercise with the free version of ChatGPT.
First principles of communication
My varied career has been in communications, in different settings and for different purposes, most often teaching it. So, starting with first principles, it should be pointed out that communication is shaped from both sides. These are writer purpose and reader purpose. It is then shaped by other variables: the topic, prior knowledge on either side, the setting, channels used, language used, power roles etc. My main argument as set out below is that ChatGPT does not take account of most of these.
A human summary of the government’s rail plans
My experiment used texts about the new railway policy for the UK. Kent and Surrey Bylines had decided to republish an article of about 1,900 words, from Railwatch about this. The article is by Ian Brown, the policy director for the rail campaigning organisation, Railfuture. He is someone who is very knowledgeable about rail, having had a long career in railway management and consultancy.
If you now re-read the article, you could make a quick list of what topics it covers via the sub-headings. You might also note his summary is sparky with brief critical comments. For example, “naïve” in reference to labour relations and his noting of the omission of freight from the policy. His article is not just a summary of the proposals for the railways. It also refers to other policies that will affect railway use, such as new housing. He is writing, of course, for a well-informed readership. Many of the readers will, like himself, have worked in railway management.
An AI analysis of UK rail policy
The first task I gave ChatGPT was: “Summarise in 1,900 words new railway policy in the UK.”
It delivered a text of glowing, but clearly outdated prose. It included the sentence: “the policy recognises the role of the private sector in delivering high-quality rail services.” The summary omitted all the relevant topics that appear in the Railfuture article, such as regional planning, integrating train and bus, trams and so on.
Prompting AI again
So, deciding that my newbie fault was not giving it the right instructions, I tried the instruction: “Write an article of 1,900 words on the policy for transport of the new UK government.” I thought that would force ChatGPT to consider more recent material and to widen the topic across transport planning.
It duly produced a text containing these sub-headings:
- Rail reforms and the creation of the Great British Railways
- Bus service improvement and devolution
- Greener transport initiatives
- High-speed rail and HS2 development
- Road investment strategy and future projects
- Addressing regional inequality through infrastructure
- Funding challenges and economic considerations
- Future-proofing transport systems
Comparing the analyses
It is interesting to compare these sub-headings with those in the Railfuture article, which include:
- Economic stability
- Planning and infrastructure
- Employment rights
My conclusion is that the expert (Brown) sees a rather different configuration of relevant topics to AI. There is no mention at all by AI of employment rights, for instance.
AI’s troubling selection of sources
I wondered whether this is because of the sources used. The ChatGPT article stated clearly that it searched six different sites. These appear in brackets at some paragraphs. They include .gov sites (as one would expect) but also Guernsey Press. Guernsey? It has not even had a railway since 1934! However, in the view of AI it is a valuable source providing, for instance, the gem of information about plans to expand tap-in-tap-out systems to 45 more stations by 2025.
Another source, for a paragraph about regional inequality is ‘Trending Now Infrastructure’. I tried to check what this source actually says, but it was too difficult to navigate to. In any case, the site is advertised as an aggregator of AI sources. Oh dearie me – this seems like serving up a dish already re-heated twice.
Another paragraph sourced from both Guernsey and Trending Now is one on future-proofing transport systems “with a strong focus on technological innovation and long-term sustainability”. The technology listed in it is “smart traffic management systems to reduce congestion” and “digital transformation”. It gives no prominence to new planning legislation, which appears at the start of Brown’s article.
AI’s lack of intelligent analysis
The AI paragraph concludes with the sentence: “The government’s focus on digital transformation extends to its public transport systems, ensuring that new infrastructure projects are designed with the future in mind”. This reeks of sweet-smelling reassurance, typical of a certain kind of public communication, designed to lull our brains and lure us away from the hard points.
This AI text should not be used by those who are seeking a more rounded view of the new policy.
AI misses significant details
I decided to turn to the debate on the new railway bill, which took place in the House of Commons on 29 July. First, I asked ChatGPT: “Summarise in less than 2000 words, the debate in the House of Commons on the railways bill”.
This produced a text of about 200 words, sourced from They Work for You, the House of Commons Library and gov.uk. It was very unsatisfactory in what it left out, so it is rather disturbing to think in future most busy people, some making important decisions, will be relying on summaries of this nature.
Rather better are the research reports lodged in the House of Commons library, which are written by experts with the purpose of informing MPs and those who are about to participate in debates. For example, the report written by Michael Benton on the future of rail is an excellent summary of policy.
But what I wanted was a summary of the recent debate on the second reading of the bill.
A human analysis of the House of Commons debate
I turned to the House of Commons transcript. The session, “Passenger Railway Services (Public Ownership) Bill”, was long because new MPs were making their maiden speeches, all about their constituencies and themselves and not at all about railways. Skipping over these, I concentrated on the speeches about the bill.
It was introduced by Louise Haigh, the new secretary of state for transport, who said: “This is the only response to the failure of privatisation to deliver reliable and affordable services for passengers”, saying that it will “save private sector fees, money can be reinvested in the railways, serving interests of passengers and tax-payers rather than share-holders”.
Nationalisation – ideological or for genuine public benefit?
Helen Whately, a Kent MP (Con, Faversham) was the lead speaker for the opposition, welcoming that “Labour is taking forward our plan for Great British Railways”. Her main question was how exactly the move is going to benefit passengers. Will it just be the same group of people in a different colour shirt? “She could be modernising working practices, instead of bending over backwards to the unions”. She should be saving the Hitachi factory like the Conservative government saved Alstrom. Whateley opposed the bill as “a rushed piece of left-wing ideology”. She hopes it will return after the autumn recess with a bill that jettisons ideology.
Rip-off privatisation, funding requirements and future services
Then it was the turn of various MPs to speak putting forward the arguments for and against, with various amendments on the agenda. Andy McDonald (Lab) spoke of rip-off privatisation. Wera Hobhouse (Lib Dem) pointed out that private operators are already subsidised to run unprofitable services, so even a public system would need financial support – is funding adequate? Also would it not be better to take into public control the failing companies first (like Avanti). Wendy Morton (Con, a former rail minister) asked questions about ticket office closures, accessibility, driver training, and service for seven days a week. She also referred to regional transport planning in the Midlands, the argument of regionalization of transport planning later echoed by the Lib Dems.
Staff strikes, stress and fatigue
The MP for Blackpool referred to the poor service of the Avanti West coast railways: “last week, Avanti’s catering staff went on strike over short-notice changes to their shift patterns, job cuts and enforced overtime, which are causing widespread stress and fatigue”.
He said that a more reliable railway service would be good for Blackpool’s economy so that young people no longer move out to neighbouring cities.
Changes needed to support passengers
In summing up for the opposition, Kieran Mullan argued that nationalisation will not make train tickets cheaper or end strikes. He also specified working practices that are not in the best interest of passengers “the varied approach on mandatory Sunday working, where clearly passengers would be better served by that becoming standard; or annual leave, where again passengers would experience less delays and cancellations if drivers were required to give a reasonable amount of notice”.
Now that is a degree of detail about the sharp points of the labour disputes that have been causing recent strikes, details not much aired in the media so far. “They should be focusing on getting their union backers to stop frustrating reform of our railways”. He also raised points about funding.
The parliamentary under-secretary of state for transport, Simon Lightfoot answered each of these objections one by one, and the debate closed. The amendments put forward were mostly negated because Labour now has a voting majority.
Summaries by AI don’t account for purpose
What has the summary above of a live debate shown about the AI texts? I do not underestimate the challenge of trying to achieve a balanced summary, but the AI versions were excessively bland. I would not have learned at all from them about serious labour problems at the railways. The debate is more revealing because the purposes of the participants is different from that of readers wanting a quick summary. In an adversarial political system, arguments tend to be sharper about differences of opinion.
Similarly, Ian’s article covered the topics differently from AI because his purpose was to identify points that he knows readers of Railwatch would be interested in, as campaigners.
We still need humans – AI is not an expert
Maybe in the future AI may be able to deal better with writer-reader purposes. Maybe it already does and I just have not yet got the knack of telling it clearly enough what to do! But I suspect that increasingly an awful lot of people are dulling their brains by relying on AI systems that use inadequate sources and are just not as good as the experts. So, I think we still need human reporters and writers for some time to come.
Meanwhile, we should all be alert to detect when AI has been used. For Kent and Surrey Bylines, we will tell you if we know that that an article has been written by AI.
