Friday, January 01 2021
In this tutorial explains about How to Generate and Store Massive data in Node.js. When we build an application we need data to show off how it works. In Node.JS, we can generate massive amount of records by using faker.js.
Express – is a popular framework which is a most common framework used in node.js
Faker.js – generate massive amounts of fake data in the browser and node.js.
db-migrate-mysql – used to db-migrate mysql driver.
yargs – It helps you build interactive command line tools, by parsing arguments and generating an elegant user interface.
First, we need to install node application in your project folder.
node -v npm init -y npm i db-migrate-mysql faker fs yargs express
Your package.json file look like in the below
After the installation and create the following files and folders in your project directory.
//root directory touch database.json mkdir migrations mkdir scripts cd scripts touch geneate-users.js
In app.js file with following contents
const express = require('express'); const port = 3000; const app = express(); app.get('/', function (req, res) { res.send('hello world') }) app.listen(port, () => { console.log(`Server running on port ${port}`) }) module.exports = app;
To run your application.
In Browser: http://localhost:3000
In database.json file with following contents
{ "dev": { "host": "localhost", "driver": "mysql", "user": "root", "password": "", "database": "wallet", "multipleStatements": true }, "sql-file": true }
db-migrate create user
This command used to create a users table in mysql. It generate two sql files.
*-users-up.sql file – add your create query here.
*-users-down.sql file – add your drop table command here.
In 20210101074106-users-up.sql file with following contents
/* Replace with your SQL commands */ CREATE TABLE `users` ( `id` int(11) NOT NULL AUTO_INCREMENT, `name` varchar(40) DEFAULT NULL, `email` varchar(50) DEFAULT NULL, `balance` int(11) DEFAULT NULL, `createdAt` datetime DEFAULT CURRENT_TIMESTAMP, `updatedAt` datetime DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, PRIMARY KEY(id) ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
In 20210101074106-users-down.sql file with following contents
/* Replace with your SQL commands */ DROP TABLE users;
db-migrate create add-users
Goto scripts folder, In generate-users.js file with following contents
Replace your *-add-users-up.sql file <!– REPLACE YOUR SQL FILE NAME –>.
// generator.js const fs = require('fs') const faker = require('faker') const argv = require('yargs').argv const lines = argv.lines || 1000000 const filename = argv.output || '<!-- REPLACE YOUR SQL FILE NAME -->' const writeStream = fs.createWriteStream('./migrations/sqls/'+filename) const createUser = () => { const title = faker.name.firstName().replace(/[^\w\s]/gi, ''); const balance = faker.random.number(10000) const email = faker.internet.email(); return `('${title}','${email}',${balance});\n` } const startWriting = (writeStream, encoding, done) => { let i = lines function writing(){ let canWrite = true do { i-- let post = 'INSERT INTO users (`name`, `email`, `balance`) VALUES ' + createUser() //check if i === 0 so we would write and call `done` if(i === 0){ // we are done so fire callback writeStream.write(post, encoding, done) }else{ // we are not done so don't fire callback writeStream.write(post, encoding) } //else call write and continue looping } while(i > 0 && canWrite) if(i > 0 && !canWrite){ //our buffer for stream filled and need to wait for drain // Write some more once it drains. writeStream.once('drain', writing); } } writing() } //write our `header` line before we invoke the loop //writeStream.write(`\n`, 'utf-8') //invoke startWriting and pass callback startWriting(writeStream, 'utf-8', () => { writeStream.end() })
Run this file in the terminal
node generate-users.js
Then, random records added in *-add-users-up.sql file. You can check that file after executing command.
Finally, Run the migration command in the terminal.
You will see the result in the database.
db-migrate up // for add and create tables db-migrate down // for remove and truncate tables.