Experiment: What Happens When AI Plans Your Entire Social Strategy?

Are you tired of spending countless hours creating a social media strategy for your business? Have you ever wondered what would happen if you let AI take the wheel and plan everything for you? Would your results improve, or decline? Would your audience love it, or walk away?

To test if AI is really ready to replace social media marketers and strategists, I decided to perform an experiment. I wanted to see what would happen if I asked AI (specifically ChatGPT) to create a strategy, schedule, and copy for my social media marketing efforts.

Here’s what I found out.

Hypothesis: an entirely AI-created social strategy will not perform as well as a human-created one

Before diving into the experiment, let’s clarify our hypothesis. Based on previous experiences with AI-generated content, I anticipated that the entirely AI-created social strategy would not perform as well as one created by a human.

Why? Because AI lacks the ability to truly understand the intricacies of my brand, which is crucial in creating effective social media strategies.

But of course, I was open to being proven wrong. Hey, maybe AI has gotten so advanced that it could create a social strategy that would outperform my own. Would I happily take back the hours I spend agonizing over social media plans? Absolutely.

Methodology

To test this theory, I used ChatGPT, a neural network-based AI platform that generates human-like text, to create a social media strategy for my LinkedIn profile. I only chose one social profile because that’s my most active channel, and I wanted to have clean data to compare with my normal social media efforts.

The tools I used in this experiment are:

• ChatGPT’s GPT-4 model
• Hootsuite for scheduling and advanced analytics
• LinkedIn for simple analytics

The process I used is as follows:

1. Ask ChatGPT to suggest a social media strategy for my business.
2. Ask ChatGPT to suggest a week-long posting schedule based on that strategy.
3. Ask ChatGPT to write posts and suggest images for me to execute the schedule.
4. Compare one week of results to my business-as-usual (BAU) social media efforts, which would be based on a human-created strategy and copy.

Here’s how it all shook out.

Step 1: Prompting

My first step was to input a prompt into ChatGPT, asking it to “act like a social media strategist” and create a LinkedIn strategy for my business.

See my initial prompt below.

ChatGPT came back with a simple but, to my eyes, effective strategy. So far, so good.

From here, I asked it to create a posting schedule.

Okay, not bad.

Next, I asked the AI to write posts for each of the days and suggest images to go with them.

This is where things started to get a little sticky. The posts it suggested were vague and not at all tailored to my brand. Fair enough, maybe I hadn’t given it enough information.

So, I asked it to build on its output and gave it some examples of my recent work.

Here’s what it came up with next.

Not too bad. Not something I’d write myself. But, a step forward at least.

Step 2: Editing

Next step was editing. I realized I definitely needed to revise a few posts as the AI had spit out a lot of vague, empty ideas. I did this by asking ChatGPT to refine its results.

Here’s what the revision of the Tuesday post looked like.

“📊 Data shows that AI is transforming content creation. 70% of businesses now use AI for generating content. Dive into the numbers. #AIandWriting #ContentCreationStats”

Errrrr… what? This was incorrect. A quick Google showed that closer to 34% of all companies use AI for content generation. I asked it to revise again (actually, three times) until I got an accurate post. Ugh.

Once I had the copy down, I noticed the post specified using an infographic image.

I used Jasper.AI to generate this. My prompt was:

“Make an infographic with this statistic related to AI and content creation: Data reveals that AI is reshaping content creation, with 34% of companies now using AI for content generation.”

Here’s the result.

Out of those options, I chose this image.

A few of the other posts took a lot of back and forth between me and ChatGPT. While this was an experiment, I had to protect my brand. Inaccuracies, vague posts, and using the wrong tone were not going to fly here.

Ultimately, I spent three hours going back and forth with the AI to refine these posts. That’s generally as long as it takes me to write them myself.

Step 3: Scheduling

Now, I was ready to schedule these posts. Using Hootsuite, I scheduled the posts to go out over one week.

Luckily, this part was easy, thanks to Hootsuite’s user-friendly platform. This took me all of 15 minutes. Here are the final posts drafted in Hootsuite Planner.

Step 4: Analyze

Here are my content analytics for the week prior to the AI-generated posts.

Okay, nothing crazy, but I did have some interaction.

Here are my content analytics for the week of the AI-generated posts.

Crickets.

Let’s take a look at visitor analytics. This is before AI. Not too bad.

This is after AI.

Again, crickets.

Here’s my follower analytics prior:

And after.

You get the drift.

Results

TL;DR

• The AI-generated social media strategy and posts did not perform as well as my normal efforts.
• It took the same amount of time, even slightly longer, to create an AI-generated plan.
• Inaccuracies and vague posts had to be revised multiple times. Even then, the quality of the copy was not up to my usual standards.
• Visitor engagement and website clicks decreased during the week of the AI-generated posts.
• AI can be useful for generating ideas and inspiration, but it’s not ready to fully replace human-created social media strategies.

Metric	With AI	Without AI
Impressions	1	203
Visitors	0	26
Followers	0	1
Time Spent	3.15 hrs	3hrs

As expected, the AI-generated posts did not perform as well as my usual social media efforts. In fact, they fell flat in terms of engagement and overall reach.

While I was able to schedule them quickly using Hootsuite’s platform, it took a considerable amount of time to refine the copy and ensure it aligned with my brand.

Yet, even then, the posts just weren’t up to snuff with my usual standards. They read like AI copy, and didn’t have the depth, detail, and uniqueness that would make them engaging.

When I saw the posts pop up on my feed, I scrolled right by. And I’m sure my followers did, too. It was just too difficult to give the AI the information needed to write like I would write, and I think that the quality of the posts really affected performance.

What do the results mean?

In conclusion, it’s safe to say that my AI-generated posts flopped. They received zero engagement and had no impact on my visitor or follow traffic.

Here’s a little breakdown of the AI social media strategy results.

1. Vague and unfocused posts

The ChatGPT-generated posts were often too general and not tailored to my business or brand. This resulted in a lack of engagement from my audience, as they likely didn’t resonate with the content.

2. Inaccuracies

As shown in the example above, the AI couldn’t always provide accurate information or statistics. This required multiple revisions and back-and-forth, ultimately wasting time and potentially damaging my brand’s credibility.

3. Lacks human touch

These results suggest that while AI may have made great strides in generating human-like text, it still lacks the ability to truly understand the nuances of your business or brand. This can result in AI-generated content that feels robotic and impersonal, ultimately turning off potential customers.

4. Time-consuming

Even with the heavy editorial oversight I gave these posts, the results were still underwhelming for me, and not something I’d personally create, or want to represent my brand.

AI is still useful for generating ideas, finding inspiration, and even generating specific captions. But, it’s not yet ready to fully replace human-created social media strategies. As shown in this experiment, it required a significant amount of time and effort to refine the AI-generated posts and make them suitable for my brand. This could potentially outweigh any time-saving benefits.

What I’d do differently next time

No surprises here, but giving the robots complete control of my social account wasn’t the most successful strategy. There are plenty of reasons why—and I’ve gone over them above—but none are enough to get me to give up on using AI for social media entirely.

That’s because, for all its limitations, AI is still a total game-changer when it comes to marketing. In a profession as creative and data-driven as ours, AI makes everything from ideation and creation to analysis so much faster. The technology may be in its infancy, but its potential is practically limitless.

And while this experiment was designed to see what happens when AI takes full control, that’s really not what most experts (including the team at Hootsuite) recommend. 

AI is a great tool, but it’s no replacement for your hard-won industry and brand-specific insights.

After all, you’re the expert on your audience, past results, and business goals. If I was to perform this experiment again, I’d use AI as one of many tools in my arsenal. The best strategies include social-specific AI tools (like *ahem* OwlyWriter AI) as a supplemental resource, not the only source of truth.

Save time managing your social media presence with Hootsuite. Publish and schedule posts, find relevant conversations, engage your audience, measure results, and more — all from one simple dashboard. Try it free today.

Get Started