SEO (Search Engine Optimization) 101

Here is a quick summary regarding how to increase search engine ranking and indexing for website. This is a learning note from https://moz.com/beginners-guide-to-seo 

Indexable Content
Provide alt text for images. Assign images in gif, jpg, or png format "alt attributes" in HTML to give search engines a text description of the visual content.
Supplement search boxes with navigation and crawlable links.
Supplement Flash or Java plug-ins with text on the page.
Provide a transcript for video and audio content if the words and phrases used are meant to be indexed by the engines.

Crawlable Link Structures
Submission-required forms
Links in unparseable JavaScript
Links pointing to pages blocked by the Meta Robots tag or robots.txt
Frames or iframes
Robots don't use search forms
Links in Flash, Java, and other plug-ins
Links on pages with many hundreds or thousands of links
Nofollow instructs search engines to not follow a link

Keyword Usage and Targeting
One of the best ways to optimize a page's rankings is to ensure that the keywords you want to rank for are prominently used in titles, text, and metadata.
Keyword density is divorced from content, quality, semantics, and relevance.

Title tags
The title element of a page is meant to be an accurate, concise description of a page's content. It is critical to both user experience and search engine optimization.
Search engines display only the first 65-75 characters of a title tag in the search results

Meta Tags
Meta Robots
The Meta Robots tag can be used to control search engine crawler activity (for all of the major engines) on a per-page level

Meta Description
Search engines do not use the keywords or phrases in this tag for rankings, but meta descriptions are the primary source for the snippet of text displayed beneath a listing in the results.
Meta descriptions can be any length, but search engines generally will cut snippets longer than 160 characters, so it's generally wise to stay within in these limits.

Meta Keywords:
The meta keywords tag had value at one time, but is no longer valuable or important to search engine optimization.
 
URL Construction Guidelines
Employ empathy
Shorter is better
Keyword use is important (but overuse is dangerous)
Go static
Use hyphens to separate words

Link Building Basics
"Natural" Editorial Links
Manual "Outreach" Link Building
Self-Created, Non-Editorial
 
Five Samples of Link Building Strategies
Get your customers to link to you
Build a company blog; make it a valuable, informative, and entertaining resource
Create content that inspires viral sharing and natural linking
Be newsworthy
 
Canonical and Duplicate Versions of Content
Use the canonical tag within the page that contains duplicate content. The target of the canonical tag points to the master URL that you want to rank for.
<link rel="canonical" href="https://example.com/master-URL"/>
 
Rich Snippets
itemscope, itemtype, itemprop datetime

How scrapers steal your rankings
you'll need to use absolute, rather that relative links in your internal linking structure. This way, when a scraper picks up and copies the content, the link remains pointing to your site.
 
How to Judge the Value of a Keyword
Search for the term/phrase in the major engines
Buy a sample campaign for the keyword at Google AdWords and/or Bing Adcenter
Using the data you’ve collected, determine the exact value of each keyword
 
Understanding the Long Tail of Keyword Demand
popular search terms actually make up less than 30% of the searches performed on the web. The remaining 70% lie in what's called the "long tail" of search.

Tools
Google search console
Bing webmaster
others including moz products

Stop search engine from crawling web site

SEO (Search Engine Optimization) is to promote web site being indexed by search engines (Google, Yahoo, Bing, Baidu etc), however, sometimes webmaster wants to prevent search engines from indexing his web site for various reasons (security, privacy, policy etc), he usually can do 3 things

1. Create robots.txt and put to web site root folder
2. Use password to protect sensitive content
3. Add X-Robots-Tag in Http headers

Page by page control can also use meta tags

1. <meta name="robots" content="nofollow" />
2. <meta name="robots" content="noindex" />

This web page has detailed explanation
http://antezeta.com/news/avoid-search-engine-indexing

Here is one example of robots.txt to disallow search engines to crawl your website (some bad bots may skip this robots.txt)
#Disallow search engines to index your website
#Please put it to the root directory of your website
User-agent: *
Disallow:/