Upgrade Config Servers to Replica Set – Outline

SCCC to CSRS

This Outline is based on MongoDB documentation.

Prerequisites 

• All binaries in the sharded clusters must be at least version 3.2.4
• The existing config servers must be in sync 

Procedure

1) Disable the balancer

connect to mongos

# sh.getBalancerState()

# sh.stopBalancer() or

# sh.disableBalancer()

2) Connect a mongo shell to the first config server

Connect a mongo shell to the first config server listed in the configDB setting of the mongos and run rs.initiate() to initiate the single member replica set.

rs.initiate( {

   _id: "csReplSet",

   configsvr: true,

   version: 1,

   members: [ { _id: 0, host: "<host>:<port>" } ]

} )

3) Restart this config server as a single member replica set

sharding:

   clusterRole: configsvr

   configsvrMode: sccc

replication:

   replSetName: csReplSet

net:

   port: <port>

storage:

   dbPath: <path>

   engine: <storageEngine>

4) Start the new mongod instances to add to the replica set. 

• These instances must use the WiredTiger storage engine.
• Starting in 3.2, the default storage engine is WiredTiger for new mongod instances with new data paths.
• Do not add existing config servers to the replica set.
• Use new dbpaths for the new instances. 

sharding:

   clusterRole: configsvr

replication:

   replSetName: csReplSet

net:

   port: <port>

storage:

   dbPath: <path>

  

5) Add the new mongod instances

Using the mongo shell connected to the replica set config server, add the new mongod instances as non-voting, priority 0 members:

rs.add( { host: <host:port>, priority: 0, votes: 0 } )

rs.add( { host: <host:port>, priority: 0, votes: 0 } )

6) Ensure that the new nodes have completed the initial sync 

• Once all the new members have been added as non-voting, priority 0 members, ensure that the new nodes have completed the initial sync and have reached SECONDARY state. 
• To check the state of the replica set members, run rs.status() in the mongo shell: 

# rs.status()

7) Shut down one of the other non-replica set config servers 

• Shut down one of the other non-replica set config servers; i.e. either the second and third config server listed in the configDB setting of the mongos.
• At this point the config servers will go read-only, meaning certain operations - such as creating and dropping databases and sharded collections - will not be available. 

8) Reconfigure the replica set

Reconfigure the replica set to allow all members to vote and have default priority of 1.

var cfg = rs.conf();

cfg.members[0].priority = 1;

cfg.members[1].priority = 1;

cfg.members[2].priority = 1;

cfg.members[3].priority = 1;

cfg.members[0].votes = 1;

cfg.members[1].votes = 1;

cfg.members[2].votes = 1;

cfg.members[3].votes = 1;

rs.reconfig(cfg);

9) Step down the first config server

Step down the first config server, i.e. the server started with --configsvrMode=sccc.

# rs.stepDown(600)

10) Shut down the first config server.

# use admin

# db.shutdownServer()

11) Restart the first config server

Restart the first config server in config server replica set (CSRS) mode; i.e. restart without the --configsvrMode=sccc option:

sharding:

   clusterRole: configsvr

replication:

   replSetName: csReplSet

net:

   port: <port>

storage:

   dbPath: <path>

   engine: <storageEngine>

12) Restart mongos instances 

• Restart mongos instances with updated --configdb or sharding.configDB setting.
• For the updated --configdb or sharding.configDB setting, specify the replica set name for the config servers and the members in the replica set. 

sharding:

  configDB: <rsconfigsver1:port1>,<rsconfigsver2:port2>,<rsconfigsver3:port3>

change to

sharding:

  configDB: csReplSet/ <rsconfigsver1:port1>,<rsconfigsver2:port2>,<rsconfigsver3:port3> 

13) Verify that the restarted mongos instances 

• Verify that the restarted mongos instances are aware of the protocol change. 
• Connect a mongo shell to a mongos instance and check the mongos collection in the config database: 

# use config

# db.mongos.find()

The ping value for the mongos instances should indicate some time after the restart.

14) Connect a mongo shell to the current primary 

• If the first config server uses the MMAPv1 storage engine, remove the member from the replica set. Connect a mongo shell to the current primary and use rs.remove():
• Only if the config server uses the MMAPv1 storage engine. 

# rs.remove("<hostname>:<port>")

15) Shut down the remaining non-replica set config server.

16) Re-enable the balancer

Connect to mongo

# sh.getBalancerState()

# sh.startBalancer()