[Ep101] - Google Explains Why Not Every URL Is Crawled And Indexed

In 20 minutes, get up to speed on the must-know Digital Marketing updates from the week of 21st March 2022.
1. Shopify Launches Linkpop & Other Features -  Linkpop, a link-in-bio solution from Shopify, enables artists and followers to instantly access and purchase promoted products. Creators on social media networks like Instagram may add the Linkpop link to their profile, and their fans will be led to a Shopify-powered storefront with just one click.

Users who click the link are led to a selected selection of products that they may purchase straight on Linkpop without having to leave the social networking app. Consumers will not have to look for things elsewhere, and marketers will have a better notion of the impact the creator had on the sale.

Separately, Shopify has added a number of features to help advertisers on the site this week.

For example, a new predicted spend tier tool allows marketers to evaluate clients' spending potential into three tiers: high, medium, and low.

Segmentation. The new customer segmentation tool uses the predictive spend tier as a filter. Geographic location, the amount of orders a consumer has placed, and other distinctive behaviors are among the other divisions.

Marketers can now automate their Shopify customer email campaigns. Users can choose from a variety of email themes and design their own procedure for sending and scheduling emails.

Shopify also announced that its Shopify Email service will allow marketers to send up to 10,000 emails for free every month.


2. LinkedIn Allows Company Pages To Publish Newsletters - Company Pages may now establish their own newsletters in the LinkedIn app, which will also include new notifications for Page followers for newsletter changes.

In November, LinkedIn brought newsletters to Creator Mode, giving users a more direct approach to tap both into the growing popularity of newsletters as a connection option and optimize in-app interaction.

Company pages will now be able to participate as well.

As previously stated, the ability to notify Page followers with newsletter updates via an automatic, one-time reminder to your audience for each new issue is a major benefit for brands.

Subscribers will also be able to opt-in to get email updates in the future. It could be a terrific method to stay in touch with your audience, and LinkedIn reports that early testers have had a favorable response.


3. All U.S Instagram Users Can Now Create Product Tags - Instagram will soon allow all users to tag products in their IG posts, starting with users in the United States, as part of its continuous focus on extending its eCommerce offerings. Instagram is now allowing everyone to include direct links to products and companies in their uploads, which was previously only available to certified producers.

This may provide marketers a huge boost in the app for no cost, while simultaneously enhancing Instagram's eCommerce focus and changing customer behavior.

Users who have a company or creative account can tag products in Stories and Reels as well.

When someone tags one of your goods, you'll get a notification, and you'll be able to see all of the marked content on your profile.

At the same time, product tags may be misused, and brands will have control over who can tag their products, as well as the ability to delete tags from their things and brand if they so desire.

According to Instagram, over 1.6 million users tag at least one brand each week, implying that the ability to more precisely tag each product will be an enticing and widely used feature - one that, if it works properly, might help keep the main Instagram feed relevant for a bit longer.


4. Instagram Adds ‘Algorithm-Free’ Feed Sorting For All Users - Instagram has now made its algorithm-free, chronological feed option available to all users, after introducing it in December as part of Instagram CEO Adam Mosseri's testimony before the US Senate amid concerns about how Instagram is damaging to children.

To modify the appearance of your main content, you now have two new Instagram feed display options to pick from:

Favorites — This section shows the most recent postings from the accounts you've added to your 'Favorites' list. You can add up to 50 accounts to your Favorites list, which will help you prioritize updates from these people.

Following — Similar to the conventional Instagram feed, this shows posts from all the accounts you follow in the app.

Both the new 'Favorites' and 'Following' feeds will display your posts in chronological order as they are published. There's no more algorithm tampering – it's the Instagram you used to know and love, with everything in its proper position.

The issue with Instagram's reverse-chronological feed options is that you can't set either as your default – you'll have to manually select your preferred feed presentation every time you check-in.


5. Meta Renames Automated Ad Suite Under Meta Advantage Umbrella - Meta has stated that it is renaming its ad automation products to make the purpose of each more apparent and to improve performance.

Meta's ad automation tools will now be grouped into a single portfolio dubbed 'Meta Advantage.'

Meta's options will be divided into two product lines under the 'Advantage' name. The title change doesn't provide anything new in terms of functionality just yet, though Meta has hinted at some upcoming changes to its automation possibilities.



6. TikTok 'For You' Summit Is Back - TikTok has launched its second 'For You' Summit, this time focusing on the Southeast Asian market, in which it will conduct a series of sessions with platform experts and creators to highlight the app's current marketing potential.

The seminars will focus on measurement, content production, and 'Shoppertainment,' according to TikTok.

This is TikTok's second For You Summit; the first was held in July of last year. TikTok is aiming to expand its eCommerce capabilities and integrations – hence the 'Shoppertainment' reference – and there will be numerous information on the app's newest usage trends and habits to help marketers plan their tie-in campaigns.

You can register for the For You Summit next month here.


7. Google Merchant Center Can Now Be Linked To Google Analytics 4 - You may now link your Google Merchant Center account to your Google Analytics 4 property, according to Google. You may also "watch your conversions from free product listings by connecting your Merchant Center and Google Analytics site," according to Google.

If you already have a Google Analytics (GA4) property, go to the conversions settings page and connect it to your Merchant Center account. This feature will not operate unless you enable auto-tagging, according to Google.

Source: https://support.google.com/merchants/answer/11996548


8. Google Has Launched A New Partner Directory - The new Google Partners directory, which includes new ways to search for partners as well as an overhauled partner program, has been introduced by Google. "Finding the appropriate partner to help you expand your business is now easier than ever," Google stated.

What are the new requirements to be a Google Partner? Google settled on the following requirements to participate:
  • Achieve an optimization score of 70%.
  • Spend $10,000 across all of a partner’s managed accounts within a 90-day period.
  • Have at least 50% of designated account strategists certified in Google Ads, with at least one certification in each product area (Search, Display, Video, etc.) with a campaign spend of $500 (or more) in 90 days.
  • The ability to dismiss recommendations without them counting against your Optimization score

9. Google Explains How They Rank Pages With Abbreviations - A concern about how Google's algorithm handles abbreviations was answered by Google's John Mueller. John went into great detail in response to the question, clarifying that these are simply synonyms and that Google doesn't do anything extra with abbreviations.

“And the short answer is we don’t do anything special with those kinds of things.
We essentially treat them as tokens on a page.
And a token is essentially a kind of like a word or a phrase on a page.
And we would probably recognize that there are known synonyms for some of these and understand that a little bit.
But we wouldn’t really do anything specific there in that we’d like to have a glossary of what this abbreviation means and handle that in a specific way.
So that’s something where, especially when it comes to synonyms, our systems learn these over time.
And for the most part, we handle them when people search and not when we do the indexing.”

If you want to learn more, he recommends that you watch this YouTube video: https://youtu.be/DeW-9fhvkLM


10. Google's Search Algorithm Has Been Modified to Highlight Better Product Reviews - Google has released an overview of how it plans to improve the quality of reviews displayed in Search results.

In order to provide the greatest review insights to searchers, Google is now highlighting product reviews that:
  • Include useful in-depth information, such as a product's benefits or cons, as well as insights on how a product works or how it varies from earlier versions.
  • Come from people who have used the products and can demonstrate what the product looks like or how it is used.
  • Include information that isn't provided by the maker, such as graphics, audio, or connections to other resources that detail the reviewer's experience.
  • Compare and contrast products, or explain how a product differs from its competitors.

In general, these are some pretty excellent criteria for the types of reviews you want anyway, and this is the type of information you'd want to elicit if you're trying to call on your top brand supporters to post about your business/products.

If you need more information, Google has also created a comprehensive guide on 'How to Write Quality Reviews,' which is in line with what it wants to emphasize in its products.

Source: https://blog.google/products/search/more-helpful-product-reviews/


11. Google Explains Why Not Every URL Is Crawled And Indexed - Google's John Mueller published a thorough and open explanation of why Google (and third-party SEO tools) does not crawl and index every URL or link on the internet. He stated that crawling is not objective, that it is costly, that it can be inefficient, that the web changes frequently, that spam and garbage exist, and that all of this must be considered.

Here is what he wrote:
“There's no objective way to crawl the web properly.
It's theoretically impossible to crawl it all since the number of actual URLs is effectively infinite. Since nobody can afford to keep an infinite number of URLs in a database, all web crawlers make assumptions, simplifications, and guesses about what is realistically worth crawling.

And even then, for practical purposes, you can't crawl all of that all the time, the internet doesn't have enough connectivity & bandwidth for that, and it costs a lot of money if you want to access a lot of pages regularly (for the crawler, and for the site's owner).

Past that, some pages change quickly, others haven't changed for 10 years -- so crawlers try to save effort by focusing more on the pages that they expect to change, rather than those that they expect not to change.

And then, we touch on the part where crawlers try to figure out which pages are actually useful. The web is filled with junk that nobody cares about, pages that have been spammed into uselessness. These pages may still regularly change, they may have reasonable URLs, but they're just destined for the landfill, and any search engine that cares about their users will ignore them. Sometimes it's not just obvious junk either. More & more, sites are technically ok, but just don't reach "the bar" from a quality point of view to merit being crawled more.

Therefore, all crawlers (including SEO tools) work on a very simplified set of URLs, they have to work out how often to crawl, which URLs to crawl more often, and which parts of the web to ignore. There are no fixed rules for any of this, so every tool will have to make their own decisions along the way. That's why search engines have different content indexed, why SEO tools list different links, why any metrics built on top of these are so different.”