1. Starting with MongoDB, nodejs and mongoose - part I

Some time ago I started to work on a project that included nodejs and mongodb, I decided to write down as I learn for future reference and maybe others could find it useful as well.

All source code for this post can be found in this git repository

What you will need:

1. Nodejs and npm installed - you can download it from here, if you need to install nodejs on a linux distribution (e.g. Ubuntu, mint), there is a great explanation here

2. MongoDB installed - you can find how to do it here, if you don't want to install mongodb, you can create and use a sandbox db at MongoLab

In this post I will assume that you know how to use node and npm and you have your package.json already ready, we will construct a site for locating airports around the world.

We will start by adding some of our npm dependencies for this project,  we are going to use:

1. express - this will be our web framework to encapsulate all the web server capabilities and make things easier for us, don't worry if you didn't work with express before, you will see it is very straight forward
   

   npm install --save express

2. mongoose - "Mongoose is a MongoDB object modeling tool designed to work in an asynchronous environment."
   

   npm install --save mongoose

Let's open a new file app.js - this will be our main entry point for the application.

our skeleton app will look like this:

//all the dependencies we need
var http = require('http'); //this is a built in package that comes with node and there is no need to install it
var express = require('express');
var mongoose = require('mongoose');

//new app that will work on port 3000
var app = express();
app.set('port', 3000);

//Simple 404 response because we don't have any routing yet
app.use(function(req,res){
    res.status(404);
    res.type('text/plain');
    res.send('404');
});

//start the application and let node listen for requests on port 3000
http.createServer(app).listen(app.get('port'), function(){
    console.log( 'Server started on http://localhost:' + app.get('port') + '; press Ctrl-C to terminate.' );
});

All this app is doing is listening for requests on http://localhost:3000/ but for every request it will answer with 404 - not found as we didn't attach any routing yet.

Now, let's connect to our mongodb, we will do it on the same file for simplicity but its a better practice to keep database logic on a separate, dedicated file.

If you created your account at mongolab, your connection string should look something like this:

mongodb://<dbuser>:<dbpassword>@<mongolab server and port>/<db name>

You can find it on your database page under you account. if your mongo is running locally, your connection string should look something like this:

mongodb://localhost:27017/airports'

27017 is the default port for mongo db unless you changed it, airports is the database we will be working with, once we will enter data into it, mongo will create it and save the data.

Let's add the following just before we create the express app ("var app=express()"):

//connect to our mongo database
var db = mongoose.connection;
mongoose.connect('mongodb://localhost:27017/airports');

//if we have any errors, show them in console
db.on('error', function (err) {
    console.log('connected ' + err.stack);
});

//when we disconnect from mongo, show this in console
db.on('disconnected', function(){
    console.log('disconnected');
});

//when we connect to mongo, show this in console
db.on('connected',function(){
    console.log('connected');
});

In the first line we are creating a mongoose connection object that will be responsible on interacting with mongo driver and mongoDB, on the second line we establish our connection to the airports database.

later on we have 3 listeners for event from this connection with callback, currently all we are going to do is write to the console but we could do other stuff in there, like write to log for example

1. error - in case of an error
2. disconnect - when we get disconnected from the database
3. connect - when we successfully establish a connection to the database.

Now its time to build our mongoDB schema, mongoDB stores data in the database as a collections of documents, object like data. We define the shape of the documents within the collection using schema, mongoDB  has a flexible schema system, which mean it does not enforce a data structure on all the documents in a collection, so we can create our schema now and if there are any changes to be made later on, we can make them without invalidating the rest of the data already stored in the database.

For this example I am going to use a subset of the airports data from ourAirports.com which is an open data aviation web site, you can download a csv file from here, this file is built on a daily basis so it may vary from time to time (you can find the csv file and the subset data file in the git repository of this blog post under the data folder)

Our schema will look like this, using the fields from the csv file:

var airportSchema = new mongoose.Schema({
    "id":String,
    "ident":String,
    "type":String,
    "name":String,
    "latitude_deg":String,
    "longitude_deg":String,
    "elevation_ft":String,
    "continent":String,
    "iso_country":String,
    "iso_region":String,
    "municipality":String,
    "scheduled_service":String,
    "gps_code":String,
    "iata_code":String,
    "local_code":String,
    "home_link":String,
    "wikipedia_link":String,
    "keywords":String
});

As you can see, we define all of our field as String but mongoDB also support the following data types:

• Number
• Date
• Boolean
• Buffer
• ObjectId
• Mixed
• Array

We will talk more about these data types in a later post

Now that we have the definition of the document we are going to store in mongoDB, we can create a model out of it, models are:

"fancy constructors compiled from our Schema definitions"

From the model we can create an instance that will map to a single document in the collection and will handle the work of saving, removing, adding and modifying the document  in the database.

Our model definition will look like this:

mongoose.model( 'Airport', airportSchema );

You can add the schema and model just after the mongoose connection code.

After creating the schema and model it's time to start and insert some data into the database, let's take a look at the first line form large.csv:

45229,AMC,large_airport,Mar de Cortés International Airport,31.35162125,-113.3058643,71,NA,MX,MX-SON,Puerto Peñasco,yes,,,AMC,,http://en.wikipedia.org/wiki/Mar_de_Cort%C3%A9s_International_Airport,

This line matches exactly to our schema, to add this record to the database we will need to create a new Airport model instance from our model:

var record = new Airport({
    "id":"45229",
    "ident":"AMC:,
    "type":"large_airport",
    "name":"Mar de Cortés International Airport",
    "latitude_deg":"31.35162125",
    "longitude_deg":"-113.3058643",
    "elevation_ft":"71",
    "continent":"NA",
    "iso_country":"MX",
    "iso_region":"MX-SON",
    "municipality":"Puerto Peñasco",
    "scheduled_service":yes,
    "gps_code":"",
    "iata_code":"",
    "local_code":"AMC",
    "home_link":"",
    "wikipedia_link":"http://en.wikipedia.org/wiki/Mar_de_Cort%C3%A9s_International_Airport",
    "keywords":""

});

let's go a head and save it to our db:

            record.save( function( err, airport, number affected){
                if(!err){
                    console.log('Saved airport');
                }
                else{
                    console.log(err);
                }
            });

As we said earlier an instance of a model is mapped to a single document in the database and have several methods to handle the CRUD operations (Create, Read, Update, Remove), one of those methods is save, which will insert (or update) the document in the database.

As you can see we are supplying a callback function to the this method which accepts 3 params: err for errors that might occur, airport which is the saved airport and numberAffected which will be 1 when the document was found and updated in the database, otherwise 0. in case there are no errors we will log to the console that we successfully saved the airport and in case there are errors we will log them to the console.

Now that we know how to add one record it is time to add all of our records to the database, we will do it in a separate file as we only going to this once, open a new file loadData.js, in order to read the data from the csv file we are going to use fast-csv by adding it to the project:

npm install --save fast-csv

fast-csv will read lines from the csv file and create a javascript object out of each line, because of the way we defined our airport schema, we can use the object fast-csv is creating for us directly with no need in any transformations.

run this code with node loadData.js

var mongoose = require('mongoose');

var csv = require("fast-csv");



var airportSchema = new mongoose.Schema({

    "id":String,

    "ident":String,

    "type":String,

    "name":String,

    "latitude_deg":String,

    "longitude_deg":String,

    "elevation_ft":String,

    "continent":String,

    "iso_country":String,

    "iso_region":String,

    "municipality":String,

    "scheduled_service":String,

    "gps_code":String,

    "iata_code":String,

    "local_code":String,

    "home_link":String,

    "wikipedia_link":String,

    "keywords":String

});

mongoose.model( 'Airport', airportSchema );

var Airport = mongoose.model('Airport');



//connect to our mongo database

var db = mongoose.connection;

mongoose.connect('mongodb://localhost:27017/airports');



//if we have any errors, show them in console

db.on('error', function (err) {

    console.log('connected ' + err.stack);

});



//when we disconnect from mongo, show this in console

db.on('disconnected', function(){

    console.log('disconnected');

});



//when we connect to mongo, show this in console

db.on('connected',function(){

    console.log('connected');



    //load some data to the database

    csv.fromPath("data/large.csv", {headers : true})

        .on("data", function(data){

            var record = new Airport(data);

            record.save( function( err, user ){

                if(!err){

                    console.log('Saved airport');

                }

                else{

                    console.log(err);

                }

            });

        })

        .on("end", function(){

            console.log("done");

        });

});



//make sure that we are closing the connection to mongo if something happens to node (like Ctrl + C)

process.on('SIGINT', function() {

    mongoose.connection.close(function () {

        process.exit(0);

    });

});

This will add 568 records to our airports collection, in the next post we will handle all the CRUD operations as urls from the webserver