In this tutorial explains about How to Generate and Store Massive data in Node.js. When we build an application we need data to show off how it works. In Node.JS, we can generate massive amount of records by using faker.js.

Prerequisites:

Express – is a popular framework which is a most common framework used in node.js

Faker.js – generate massive amounts of fake data in the browser and node.js.

db-migrate-mysql – used to db-migrate mysql driver.

yargs – It helps you build interactive command line tools, by parsing arguments and generating an elegant user interface.

Install Packages:

First, we need to install node application in your project folder.

node -v

npm init -y

npm i db-migrate-mysql faker fs yargs express

Your package.json file look like in the below

After the installation and create the following files and folders in your project directory.

//root directory
touch database.json
mkdir migrations
mkdir scripts

cd scripts
touch geneate-users.js

In app.js file with following contents

const express = require('express');

const port = 3000;

const app = express();

app.get('/', function (req, res) {
    res.send('hello world')
})

app.listen(port, () => {
    console.log(`Server running on port ${port}`)
})

module.exports = app;

To run your application.

In Browser: http://localhost:3000

Create a database config:

In database.json file with following contents

{
    "dev": {
        "host": "localhost",
        "driver": "mysql",
        "user": "root",
        "password": "",
        "database": "wallet",
        "multipleStatements": true
    }, "sql-file": true
}

Creating a new migration script for Users:

db-migrate create user

This command used to create a users table in mysql. It generate two sql files.

*-users-up.sql file – add your create query here.

*-users-down.sql file – add your drop table command here.

In 20210101074106-users-up.sql file with following contents

/* Replace with your SQL commands */
CREATE TABLE `users` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `name` varchar(40) DEFAULT NULL,
  `email` varchar(50) DEFAULT NULL,
  `balance` int(11) DEFAULT NULL,
  `createdAt` datetime DEFAULT CURRENT_TIMESTAMP,
  `updatedAt` datetime DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  PRIMARY KEY(id)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

In 20210101074106-users-down.sql file with following contents

/* Replace with your SQL commands */
DROP TABLE users;

Create a New Migration Script for Insert & Remove User data:

db-migrate create add-users

Create Massive Fake data using Faker.js:

Goto scripts folder, In generate-users.js file with following contents

Replace your *-add-users-up.sql file <!– REPLACE YOUR SQL FILE NAME –>.

// generator.js
const fs = require('fs')
const faker = require('faker')
const argv = require('yargs').argv

const lines = argv.lines || 1000000
const filename = argv.output || '<!-- REPLACE YOUR SQL FILE NAME -->'
const writeStream = fs.createWriteStream('./migrations/sqls/'+filename)

const createUser = () => {
    const title = faker.name.firstName().replace(/[^\w\s]/gi, '');
    const balance = faker.random.number(10000)
    const email = faker.internet.email();
  
    return `('${title}','${email}',${balance});\n`
  }
  
  const startWriting = (writeStream, encoding, done) => {
  let i = lines
  function writing(){
    let canWrite = true
    do {
      i--
      let post = 'INSERT INTO users (`name`, `email`, `balance`) VALUES ' + createUser()
      //check if i === 0 so we would write and call `done`
      if(i === 0){
        // we are done so fire callback
        writeStream.write(post, encoding, done)
      }else{
        // we are not done so don't fire callback
        writeStream.write(post, encoding)
      }
      //else call write and continue looping
    } while(i > 0 && canWrite)
    if(i > 0 && !canWrite){
      //our buffer for stream filled and need to wait for drain
      // Write some more once it drains.
      writeStream.once('drain', writing);
    }
  }
  writing()
}

//write our `header` line before we invoke the loop
//writeStream.write(`\n`, 'utf-8')
//invoke startWriting and pass callback
startWriting(writeStream, 'utf-8', () => {
    writeStream.end()
})

Run this file in the terminal

node generate-users.js

Then, random records added in *-add-users-up.sql file. You can check that file after executing command.

Finally, Run the migration command in the terminal.

You will see the result in the database.

db-migrate up  // for add and create tables

db-migrate down // for remove and truncate tables.