Setup
This project was created and tested using Node v9.1.0.
Setup using: npm install
Documentation
Documentation
Firebase Database Self-Serve Backup to BQ/CSV Exporter
This project takes a self-serve backup generated from the Firebase Realtime Database and outputs a flattened CSV as output.
The CSV is made to be uploaded to BigQuery but can be used for any number of purposes.
The term flattened is used to represent the following transformation
{
"a": true,
"b": {
"c": true
}
}
will be converted to
a,true
b,c,true
Getting a Self-Serve Backup
To generate a self-serve backup:
- Go to the Firebase console (
https://console.firebase.google.com/project/<YOUR_PROJECT>/database/backups). - Click on a backup to download it from GCS. (or create a public link to it)
Exporting to a CSV
Run this using: node export.js --inputURL http://link/to/backup.gz --outputPath /tmp/output.gz
Arguments
--inputURL http://path/to/input OR --inputPath /path/to/input
required
This flag indicates where the input resides.
--outputPath /path/to/output
required
This flag indicates where the output csv should reside.
--noCompressedInput
optional
This flag takes no parameters. By default, it is expected that the input is gzipped. If it is not, use this flag to indicate as such.
--noCompressedOutput
optional
This flag takes no parameters. By default, it is expected that the output will be gzipped. If it should not, use this flag to indicate as such.
--fillWithNulls
optional
This flag takes no parameters. By default, the output will contain rows of mixed-width depending on how deep the input data is. By using this flag, the output will be normalized to contain 32 columns.
Upload to BigQuery
- Navigate to the BigQuery console.
- Under the desired Data Set, click "Create new table".
- Use the following params:
- Source Data:
Create from source - Location:
Google Cloud Storage, set the path (e.g.gs://my-bucket/output.gz) - File format:
CSV - Table name:
your-dataset.your-table - Schema: (If you used
--fillWithNulls, you can setAutomatically detect), otherwise clickEdit as Textand input:c1:STRING,c2:STRING,c3:STRING,c4:STRING,c5:STRING,c6:STRING,c7:STRING,c8:STRING,c9:STRING,c10:STRING,c11:STRING,c12:STRING,c13:STRING,c14:STRING,c15:STRING,c16:STRING,c17:STRING,c18:STRING,c19:STRING,c20:STRING,c21:STRING,c22:STRING,c23:STRING,c24:STRING,c25:STRING,c26:STRING,c27:STRING,c28:STRING,c29:STRING,c30:STRING,c31:STRING,c32:STRING - Field delimiter:
Comma - Allow jagged rows:
Check(ONLY if you didn't use--fillWithNulls)
- Source Data:
- Click
Create Table.
This is not an official Google product