Video Script:
My name is Patrick and I’m from SpyFu.com. Today, I’m going to talk about how we recently did an SEO audit for our site. Things we learned, mistakes we made, and hopefully give you some pointers if you’re ever wanting to do it for your site.
And I think you should. Because it’s working.
*intro*
This is a different type of video because we used a variety of tools and software for this, including but not limited to SpyFu. Good tutorials on how to use some of these 3rd party sites already exist, so I’ll just touch on what they are and how we used them in this video.
Part One - SEO Calamity:
Why did we decide to do this massive SEO overhaul? It’s simple. Our organic ranks started tanking, we started losing a lot of traffic and revenue. Not going to lie, we kind of freaked out.
We wanted to find the culprit, and what we found was a bug on our end dealing with
Robots.txt
Robots.txt basically informs google, and other web crawlers, to index pages on your site, so people can find them. It also allows you to tell google what sections of your site you DON’T want them to index.
We accidentally set a bug in motion that basically told Google not to index any of our main site.
This bug, which was the programming equivalent of turning off the wrong lightswitch, told google to deindex hundreds of thousands of pieces of content pretty much over night.
This lasted for 5 days without us realizing it. We found the problem and reverted it back, but the damage was done. We went from 500,000 pages indexed to 100,000.
We got some of that traffic back. But not all of it, and it was a crushing blow, and an important lesson.
Lesson One - Stay Vigilant
Our SEO has been fairly consistent over the last decade without really applying best practices to all of our content. When you take that approach for a long time, things start to fall through the cracks, including a small bug that tanked our rankings. We thought that what we were doing was good enough, and that was our first mistake - one we didn’t want to make again.
We started by implementing a tool called robotto.org. It'll monitor your robots.txt for changes and alert you. It's free, and it's rad.
But we didn’t stop at fixing the robots.txt, we instead decided that it was time to strike from all angles. Do everything we could to get our SEO in good shape.
The first thing we tackled because is was glaring low hanging fruit was alt text
Alt Text:
We started by adding alt text to every image, both on the main site and our resources.spyfu site.
Alt text is a short, literal, description of an image.
We use Wordpress for our resources site. In your wordpress account you can easily go into Media, Library, Click on an image, and add the alt text right there
The alt text for the main SpyFu required some HTML work. I don’t know HTML, but I was able to do the prep work before handing it over to a capable developer. This meant taking large screenshots of the site, and then inserting callout bubbles for each image on each tab.
And once all of that was done, we decided to analyze every piece of content on resources.spyfu.
We started with a sitemap, that we got from a WordPress plugin called Yoast. With Yoast it’s very easy to save an XML file of your entire sitemap.
We fed this XML file into Screaming Frog. Screaming Frog returns a ton of data including Article Titles, H1s, the specific URL of the content, and even if there’s any existing canonicalization.
We exported the Screaming Frog CSV and fed THAT into URL Profiler.
This tool gave us new information including free backlink stats for each page, and a publish date. It feels strange that the other tools didn’t provide that, but a publish date is HUGE when dealing with aging content.
After we had this snowball of data we put the entire thing into a google doc.
Google Docs.
In general we are big fans of google docs and google sheets. It’s free, and allows multiple people to work on one document or spreadsheet at the same time. Collaboration was key for a project of this size.
For each piece of content there was a row.
For each data point for that content - a column.
To be honest, we didn’t use most of the data. We used a handful of key columns and added some more as the project went on. But we didn’t delete any data either; since we were doing the audit anyway, we wanted to be thorough.
With our spreadsheet all set up, it was time to make sure that our content was optimized for SEO.
Okay, “optimized” might not be the right word.
Lesson Two - Perfection is a Trap
Our goal in this wasn’t to be perfect, but we wanted to be ‘better than yesterday’ which is a work ethic of ours. If your goal is to be perfect, you run the risk of shooting your productivity in the foot, because you’re focusing on something that might be unattainable. But if you are simply ‘better than yesterday’ you can improve your site as a whole, in less time.
This approach was pivotal to this project. We had to sift through almost 700 pages, with 50 data points each. Being perfect wasn’t gonna happen.
A side note here for my fellow video peoples. Around this time I was also gathering and structuring video data for each piece of video content. That project isn’t done yet, but when it is I’ll make another video discussing that process, and I’ll add a link in the description below.
After this massive document was created we knew that we were going to have a problem. In google sheets, anything you change in the document will be changed for everyone. If you add something, delete something, or even sort, that’s how the document will be for everyone.
We found a solution called Filter Views, which allow you to set up views for individual users - like sorting and filtering. If you add or delete data, it works like normal, meaning it’ll change for everyone. But individual filtering and sorting was an absolute game changer for us.
I needed to focus on the content that had more than 0 videos on it. Sidra needed to filter those out so she could focus exclusively on all the articles she’s written.
After we realized this, we added a few more columns to help us organize this list.
I think this is a good time to jump to
Part 2 - Prep Work
Prep work.
Doing prep work is miserable, but it was easily the most important part of this process, and so a big chunk of this video is going to be about… prep work.
If you want to skip past this part, simply fast forward to **TIME STAMP**
Because honestly, THAT’S WHAT WE WANTED TO DO!!
We wanted to jump in, article by article, and just fix it. Do the work, and fix it one at a time. This might make sense for a smaller project, but we were looking at hundreds of posts that needed analyzing and improving. Doing it post by post would’ve been more enjoyable, but taken three times as long.
The best way to make it manageable, as well as to bring other people in to help with the project, was to prep it iteratively.
For us that meant adding a column that
- Identified, in simple terms, what “product” the post was about. Was it about SpyFu Kombat, SpyFu Recon, General Internet Marketing Insights? We specialize in a variety of things, and when we write content we generally like to focus on one “product” so people can find it easier. Having this extremely simple column helped us sort the content into content groups.
- When you sorted by both product and date, you could easily identify what was old and needed to be canonicalized or 301 redirected to something newer and more relevant.
- Since multiple people were working on adding these labels, they weren’t written exactly the same every time, which screwed up the sort (which was the whole point). Not a big deal, just had to go back a couple of times and standardize them as much as we could.
- We also added a few very simple columns including -
- The initials of the person who would fix the content
- A description - which eventually just became notes that we wanted to make about the content
- And a column to X off when the work was done.
- We then identified “what to do” in a column. Which mostly boiled down to if we should 301, Canonicalize, or do an SEO Review of the content.
These labels were extremely important to the fate of the content, and the eventual SEO map of our site.
301 is the harshest. It means if someone clicks on an article, or URL, it will be redirected to another page. This redirect means that it’s impossible for anyone to see the original content ever again. You’re basically telling people, AND google, “this content is worthless and outdated, I don’t want you to see it, I want you to see this instead.”
Canonical is a much softer approach. People can still get to the article if they click it, but you’re letting google know that there is something better. Canonicalizing tells google, “this content is still somewhat relevant, but we have something better.” And google will give the link juice to that better piece of content. We did this with most of our stuff, and so far it seems to have been the right move.
SEO Review was the label we used to identify content that was still relevant and good. We just needed to make sure it was SEO friendly, and if it wasn’t up to snuff (which most of our content wasn’t) we needed to go in there and fix it.
We’ve been asked, “How did you determine what needed to be done with each article?” and I really wish I had a formula for you. But honestly, that part was mostly gut. Sidra and I have been the primary content creators for almost a decade. So if my gut said that a product video was currently irrelevant, I’d label it to be canonicalized or 301’d.
Lesson 3 - Be patient, stay the course
Before I dig into the tools we used - I want to say, this didn’t happen in a day. We went through each piece of content several times, over the course of weeks to identify and label these simple rows one by one. Again, during this time we didn’t actually do the work of reviewing or redirecting, even though we wanted to.
When sticking to this “assembly line” approach, we were eventually able to fill out the necessary information, so we’d be extremely efficient once we got into the actual work. As well as delegate some of the more basic tasks to people who weren’t intimately familiar with the content.
This turned out to be the correct approach. When it came time to actually start the work, we were able to bring Zac on to do things like 301, redirects, and cross linking, on his own time.
Part 3 - The tools we used
I mentioned earlier that we use WordPress for resources.spyfu.com. It’s not perfect, but it’s simple and it works for us.
Using the Yoast plugin for WordPress felt like an SEO cheat sheet. It somewhat gamified SEO for each article. Asking what keywords we wanted to focus on and then giving us pointers on how to have that article rank better for those keywords.
It’s free for most features, and crazy affordable for the premiere version. We probably couldn’t have done this as efficiently without Yoast, and I can’t recommend it enough.
If you’re not on WordPress, but know of a similar tool for other sites, please put it in the comments below. Help other SEO noobs like me
After we started working with the articles themselves we realized that we needed to add a couple more columns to the sheet.
Part 2(again) - Ah crap more Prep Work
- Cross-link - which let us know what content we should interlink to within our own site. Google loves these cross connections, especially when they’re relevant. So we would add a link to the existing text in the content, or add a sentence or two to get those links in there.
- You want to make sure that the words you’re adding a link to are relevant to the content it’s linking to. Don’t just flood your content with random links attached to random words, Google does not like that.
- At this point, external links weren’t our priority, only URLs within SpyFu.
- For every piece of content we double checked that it also linked back to the product page of our site.
- This is a definite regret of mine. When our URL structure changed a few years ago, I never went back to my old content and linked my tutorial videos (which were still being watched) to our current product pages. That simple upkeep could have vastly improved our SEO, which probably would’ve lead to an increase in revenue.
- #regrets
- Focus Keywords - This is the keyword that we told Yoast (and in turn google) that we wanted to focus on.
Fortunately, finding and tracking the best keywords in an industry is our specialty. So let’s finally talk about SpyFu!!!!
Part 4 - Using SpyFu for SpyFu
We used the same techniques we recommend to you!
First we went into SpyFu Kombat and entered in our 3 biggest competitors. We then looked at their “core starting keywords”, which are the keywords that all three of them share. From there we could cherry pick the best and most relevant keywords and add them to a MySpyFu project.
MySpyFu automatically starts tracking these keywords, and how their ranks change for our domain. You can do this with your domain as well!
If you’re unsure what MySpyFu is, I’ll include a link to the tutorial videos in the description below.
**CROSS LINKING FOR THE WINNNNN!!!** (dubstep, doritos, mountain dew)
The joy is that you can go to any research page on SpyFu searching for keywords, and add them to this project. SpyFu will let you know if you’ve already added them, so there won’t be any duplicates.
This MySpyFu project now includes these keywords. We then separated those keywords into different groups, so we could track them easier when the SEO audit was done.
The hope was, that after all of this work, we’d see improvements on these groups of keywords. We knew that it shoullllld, but weren’t sure that it would.
SEO is scary like that. So we started watching the weekly tracking feature like a hawk, praying that eventually our ranks would increase.
I hate to spoil the ending, but they did increase. I probably wouldn’t be making this video if nothing happened, but so far, it’s looking like this crazy audit has been worth it.
Setting up this project before we did the actual work was a smart move too. It gave us a benchmark of what our ranks were before we started. From that point we could really see how much this SEO overhaul moved the needle.
After we had the keywords ready to go, we exported them and put them into another Google sheet for shareability and ease of use.
Finally, we got around to the actual work. Here’s how it went down.
Part 5 - The Actual Work
We set up a variety of filter views, in the google doc. For me, I filtered out everything that didn’t have a video, and that were listed as canonical or 301. Meaning I only got pages with videos, that were ready for an SEO Review. I then sorted this list by product, this was helpful because it made cross-linking easy. Here are the 3 pieces of relevant content about Backlinks, have them crosslink to each other.
I put my initials on all of those videos, so if I ever needed a different view, I could simply filter by my initials.
This is why that prep work was important. Thanks to all the labeling, I finally had a list of content that was tailor made for my eyes.
Doing the work (i feel like this could have “oceans 11” type music in the background):
Let’s do this step by step
- I’d open the URL from the google doc.
- I’d browse the content, maybe watch the video, and then search the Keyword document for a keyword that I think would fit.
- I was already logged into my WordPress account and so I’d just click edit up at the top.
- Add in the cross links, unless Zac had already done it.
- Add the focus keyword into Yoast, and then mostly followed Yoast’s instructions!
- Adding the focus keyword to the first paragraph of content
- Sometimes changing the SEO title if it looked sloppy
- Almost always changing the snippet, occasionally just copying and pasting the first sentence or two from the article itself - because it already contained the focused SEO keyword.
- And then clicking update it until it said the SEO was “Green”,
- or at least yellow.
Lesson 4 - Take the easy road
If the content was too complicated, for example if there were 20 pieces of content for one subject, or the video was kind of old but mayyyyybe not too old. I’d skip it. Sometimes I’d leave a note about why I skipped it.
We weren’t sure if this was the best way of doing things at first, but now we highly recommend this approach.
It’s too easy to get lost in indecision; thinking too long or hard about one article will just halt progress. If you allow yourself to skip anything too complicated you can maintain your momentum of work.
Of course, this meant that I had to come back to it eventually. But by that time I was more familiar with the content, and I sometimes I knew exactly what to do on the second pass. Sometimes I was still stumped and so I’d skip it again.
Unfortunately by the very end, the only content I had left was the really complicated stuff. The stuff I skipped over several times. It did take a lot longer to fix up those articles, but it was easier because I knew I was on the home stretch.
I’ve mentioned a couple of times before, but we also brought our co-worker Zac into the project. While Sidra and I were reconstructing the SEO Review content, we had a boat load of 301s and Canonicals to take care of.
These didn’t need things like focused keywords, or expansive cross-linking. So Zac used his filter view and knocked them out one by one.
How is this done?
You can canonicalize things right in Yoast, under the advanced settings. Simply paste the URL of the article you think is more relevant!
In WordPress the 301 redirects are found under tools, and then Redirection. Scroll to the bottom, enter in the URL you don’t want anyone to see anymore. And then enter the URL of where you want them to go instead.
Doing all of this we were able to work on this thing collaboratively and autonomously. Occasionally we had to discuss what to do with one tricky piece of content or another, but for the most part we just churned away at it.
Do the work, refresh the filter view, repeat until everything has that X marking it as complete. And that was it!
Part 6 - What comes next?
Now it’s about tracking those keywords, and upkeep. Because honestly, this sucked, and I never want to do it again.
Making sure every new piece of content is SEO friendly before we post it is going to be a standard SpyFu practice from now on.
One big regret I have is that a lot of the URLs for these pages weren’t SEO friendly. You don’t really want to change the URL of existing content; but moving forward I’m definitely going to pay much closer attention to it. Either structuring the URL to better explain the content, or include the focus keyword - hopefully both.
But here’s the thing. It’s working.
Our overall SEO numbers are slowly improving. The ranks on our focused keyword groups are increasing across the board.
We weren’t sure if that would be the result and it’s legitimately thrilling and heartwarming.
And the joy of SEO is knowing that work will have a positive impact FOREVER. Months of work for years of payoff.
On a more personal note, we learned a lot. Both on the struggles you guys have to face on a day to day basis, as well as how we can improve SpyFu to make it better for you.
I know some of you out there are screaming at your monitor that there was a better way go about this. If you are that person, please leave a comment below with any tips or tools that make this process better or more bearable.
Thanks for sticking around for this long video! I hope you learned something from our experience, and can apply the lessons we learned to improve the SEO of your site.
As always, thank you for watching.
