Replication in MongoDB

Introduction

-> Replication provides redundancy and increases data availability.
-> With multiple copies of data on different database servers, replication provides a level of fault tolerance against the loss of a single database server
-> A replica set is a group of mongod instances that maintain the same data set
-> A replica set contains several data bearing nodes and optionally one arbiter node.
-> Of the data bearing nodes, one and only one member is deemed the primary node, while the other nodes are deemed secondary nodes.

Replication

-> Durability
-> Availability
-> Redundant copies across multiple machines.
-> We can say - replicas, copies or backups
-> 'HA' - high availability ( failover )
-> Data Safety ( Durability )
-> extra copies
-> 'DR' - Disaster recovery

'Asynchronous' replication

            

Replica Sets

-> Master - slave replication
-> Can be one primary and multiple secondary’s
-> Or Primary, secondary and an arbiter

                         

Automatic failover

-> During system crash, data will be archived and restored as soon as the system comes back.

                          

Driver - Will reach out to all the members of the set and see who is primary.

Replication Options

--replSet -> replica set name
--logappend -> to overwrite the logs, while start or restart the server
--port -> port no
--dbpath -> path to store datas
--oplogSize -> to specify certain size for oplog in MB, by default - 5% of your available free space on disk
--smallfiles -> just to keep the datafiles little bit smaller and its not advised to use this option in production
--fork -> to run the process background

Starting Replica sets

-> ports
-> dbpath
-> Replica set name

ex

mongod --port 27001 --replSet abc --dbpath /data/db --logpath /data/log --logappend --oplogSize 50 --smallfiles --fork

Initiating the replica set

>rs.initiate() - to initiate a replica set

ex

$mongo --port 27001

>rs.initiate()

>rs.status() - to check the status of the replica set

Adding members to the replica set

>rs.add("hostname:port_no")

ex

>rs.add("localhost:27002")

>rs.add("localhost:27003")

abc:primary > rs.status() - to check the status of the replica set


Some replica set commands

rs.help() -> will show all the commands which is realted to replica set
rs.status() -> checks the replica set status
rs.initiate() -> initiates set with default settings
rs.conf() ->  get the current configuration object from local.system.replset
rs.reconfig( cfg ) -> updates the configuration of the running replica set with cfg
rs.add("hostname:port_no") -> to add a new member in the replica set
rs.remove("hostname:port_no") -> to remove a member from the replica set
rs.slaveOk() -> to read from secondary, should run this command in secondary node
rs.isMaster() -> to check who is primary

Deploy Mongo Clusters in Cloud Manager

Deploy Mongo Clusters in Cloud Manager:
---------------------------------------

Here the steps to deploy the Mongo Clusters in Cloud Manager,

-> Log in to Cloud Manager - https://cloud.mongodb.com/user/login?n=%2Fv2%2F515490147fe227e9f1858066#deployment/topology

-> In the "Deployment Tab" -> Click on "Add" dropdown -> Select "Existing MongoDB Deployment"

-> If its a Sharded Cluster -> Seed a single "mongos" hostname and port no.( Cloud Manager will automatically discover all connected processes )

-> If its a Replica set -> Seed a "Primary" hostname and port no.( Cloud Manager will automatically discover all connected processes )

-> Enable Authentication -> yes

-> Auth Mechanism -> Username/Password(MongoDB-CR/SCRAM-SHA-1)

-> Username -> tommy

-> Password -> *****

-> Click on continue.

-> It will show you the connected servers and processers -> click continue again -> Done.

-> Hold on - It will take a while to get updated in the Dashboard

-> By default, the cluster name will be "Cluster 0" -> We can rename it with the relevant application name/Naming standards

Note:
-----

-> One monitoring agent is sufficient for one entire group

-> Also here we are using SCRAM-SHA-1, In-build MongoDB authentication.

TTL Index in MongoDB

TTL Index: - Time To Live
----------
-> TTL indexes are special single-field indexes that MongoDB can use to automatically remove documents from a collection after a certain amount of time or at a specific clock time.


Create TTL:
------------
-> Ex: To create a Index which deletes the data more than 6 months.

db.eventlog.createIndex( { "lastModifiedDate": 1 }, { expireAfterSeconds: 1.577e+7 } )


collMod - to modify TTL index
------------------------------
db.runCommand( { collMod: "sessions",
                 index: { keyPattern: { lastAccess: 1 },
                          expireAfterSeconds: 3600
                        }
})

o/p:
----

{ "expireAfterSeconds_old" : 1800, "expireAfterSeconds_new" : 3600, "ok" : 1 }


To see all the Indexes in a Coll
---------------------------------
db.people.getIndexes()


To see Indexes in a entire DB
------------------------------
db.getCollectionNames().forEach(function(collection) {
   indexes = db[collection].getIndexes();
   print("Indexes for " + collection + ":");
   printjson(indexes);
});


Deleting a Index
------------------
db.pets.dropIndex( "catIdx" )
db.pets.dropIndex( { "cat" : -1 } )
db.collection.dropIndexes()

Capped Collection in MongoDB

Capped collections
------------------
-> Capped collections are fixed-size circular collections that follow the insertion order to support high performance for create, read, and delete operations.

-> By circular, it means that when the fixed size allocated to the collection is exhausted, it will start deleting the oldest document in the collection without providing any explicit commands.


Creation:
----------
db.createCollection( "log", { capped: true, size: 100000 } )


Specify a maximum number of documents for the collection using the max field:
-----------------------------------------------------------------------------
db.createCollection("log", { capped : true, size : 5242880, max : 5000 } )


-> ordering of results is the same as the insertion order.

To retrieve documents in reverse insertion order - $natural parameter set to -1
-------------------------------------------------
db.cappedCollection.find().sort( { $natural: -1 } )


Check if a Collection is Capped:
---------------------------------
db.collection.isCapped()


Convert a Collection to Capped
-------------------------------
db.runCommand({"convertToCapped": "mycoll", size: 100000});

-> TTL Collections are not compatible with capped collections.

MongoDB Basic Commands

To start the server:
--------------------
Go to -> command promt -> set the path till bin
ex:
C:\mongodb-win32-i386-2.6.4\bin
then -> mongod --dbpath C:\mongodb-win32-i386-2.6.4\bin
the server will start running

create db & select a db
-----------------------
use tommy

create collections
-------------------
use tommy - select the db
db.createCollection("employees")

create collections and insert the data
---------------------------------------
db.employees.insert({"id":111,"name":"tom"})

view the list of dbs
---------------------
show dbs

view the list of collections(tables)
------------------------------------
show collections

view the records in the collections
-----------------------------------
db.employees.find()

Delete a collection
-------------------
db.employees.drop() - to delete the collection
db.employees.remove() - remove all the records from the collection

Delete a speciic doc from the collection
-----------------------------------------
>db.employees.remove({name:"tom"})
>db.employees.remove({name:"tom"},1) - will delete oly one record

to view the current db
-----------------------
db

delete a db
-------------
use tommy - select a db
db.dropDatabase()

note
-----
>in mongodb you dont need to create collections.
>it creates collections automatically, when you insert some documents

insert multiple documents in a single query
------------------------------------------
db.employees.insert([{ },{ }])

to display the results in a formatted way
-----------------------------------------
db.employees.find().pretty()

update
---------
db.employees.update({name:'tomfree'},{$set:{name:'tomm'}})
>db.employee.update({id:111},{$set:{desig:'sr manager'}})
>db.employee.update({"_id" : ObjectId("529589614eff477245718bdc")},{$set:{email:'cristina@gmail.com'}})
- to add an extra field

to update multiple docs
------------------------
db.employees.update({name:'tomfree'},{$set:{name:'tomm'}},{multi:true})

to update multiple docs - with options
--------------------------------------
db.your_collection.update({},{$set : {"new_field":1}},false,true)

db.your_collection.update({},{$set : {"new_field":1}},{upsert:false,multi:true})

Note:
-----

-> In the above example last 2 fields false, true specifies the upsert and multi flags.

Upsert: If set to true, creates a new document when no document matches the query criteria.

Multi: If set to true, updates multiple documents that meet the query criteria. If set to false, updates one document.

-> This is for Mongo versions prior to 2.2. For latest versions the query is changed a bit

to get the sum of the single field:
-----------------------------------
db.Gits_OpenClose.aggregate({$match:{"Year": 2014,"Month": 1}},{$group:{_id: null,total:{$sum:"$Opened"}}})

to update the existing collections with other datas
-----------------------------------------------------
db.employees.save({_id: ObjectId("52945e82e2c023a457cf152e"),name:'tom',id:1111})
>this s by using object_id

to find a specific column
----------------------------
db.employees.find({},{oth:1,_id:0})
>oth - column name
>1 - used to show the specific field
>0 - not to show
>_id:0 - to hide the object id

to limit the records:
----------------------
db.employees.find({},{oth:1,_id:0}).limit(2)

skip method:
------------
db.employees.find({},{oth:1,_id:0}).limit(1).skip(1)
>the default value of skip method is 0.

Sorting:
--------
db.employees.find({},{oth:1,_id:0}).sort({'oth':1})
>1 - ascending
>-1 - descending
db.data.find({},{'Org From Date':''}).sort({'Org From Date':-1}).limit(50)

update:
-------
>db.Employee.update({"_id" : ObjectId("529489e1f2e76698bd1c1616")}, {$set: {"Skill.Sports": "Cricket"}});
>db.Employee.update({_id: ObjectId("529489e1f2e76698bd1c1616")}, {$set: {'Skills.tech':['nosql','java']}});

where conditions
-----------------
>db.employee.find({gender:'f'}) - where gender=f
>db.employee.find({salary:{$gt:50000}}) - where salary>50000
>db.employee.find({salary:{$gt:50000}},{name:1,desig:1,_id:0}) - select name,desig from employee where salary>50000

Indexing
----------
db.employee.ensureIndex({'id':1})
>1 - creating index in ascending order
>2 - creating index in descending order

Aggrigation
------------
db.employee.aggregate([{$group : {_id : "$id", num_tutorial : {$sum : 1}}}])

aggrigation - where
--------------------
db.employee.aggregate([{$match:{salary:{$gt:50000}}},{$group : {_id : "$gender", num_tutorial : {$sum : 1}}}])

Aggre - operations
------------------
count - db.employee.count() - table total count
count - db.employee.count({city:'blore'}) - count of city which is blore
distinct - db.employee.distinct('city')
db.test.count({sports:'football,hockey'},{name:'',sports:''})

in
---
db.case_tbl.find( { empid: { $in: [ 96761, 95939 ] } } )
db.employee.find({id:{$in:[111,222,333]}},{name:'',desig:'',_id:0})

skip() - if wanna skip first 2 docs
-----------------------------------
db.employee.find().skip(2)

total sum:
-----------
db.procount.aggregate({$group:{_id:null,total:{$sum:'$ProCount'}}})

important
----------
db.Employee.find({ "Contact.ZipCode" : "600071"},{"Contact.Address":1})

aggregate:
----------
db.employee.aggregate({$group:{_id:'$city',total_count:{$sum:1},max_salary:{$max:'$salary'},
min_salary:{$min:'$salary'},avg_salary:{$avg:'$salary'}}})

multiple group:
-----------------
db.employee.aggregate([{$group:{_id:{city:'$city',Design:'$desig'},count:{$sum:1}}}])

var a=db.employee.aggregate([{$group:{_id:{city:'$city',design:'$desig'},count:{$sum:1}}}])
printjson(a)

to insert multiple records:
----------------------------
for(var i=1;i<=10;i++)db.test.insert({id:111,name:'tom',desig:'ase',salary:5000})

import csv to mongodb
----------------------
mongoimport --host localhost --db test --collection data --type csv --file Emp_Data.csv --headerline

distinct count
---------------
db.data.distinct('Emp NO').length

OR:
---
db.emp1.aggregate(
  {$match:{$or:[{"DOE.YEAR":null},{"DOE.YEAR":2013}],"DOJ.YEAR":{$lte : 2013}}},
  {$group:{_id:{},
   count:{ $sum:1}}})

some operators:
---------------
$cmp - compare
$eq - equal
$gt - greater than
$gte - greater than or eequal to
$lt - less than
$lte - less than or equal to
$ne - not equal to