Page 1 of 1

Oracle database big data volume import (SqlLoader method)

Posted: Tue Sep 18, 2018 11:18 pm
by youyou
When using oracle database to import large amounts of data, this tool is very fast. According to my test, 100W data is imported into a table in about 7 minutes (about 35 table fields).
The specific implementation method is as follows:

Re: Oracle database big data volume import (SqlLoader method)

Posted: Tue Sep 18, 2018 11:19 pm
by youyou
1. The control file run.ctl needed to create the sqlloader command is as follows
Load data --1. Control file identification
Infile 'F:\sqldata.csv' --2, the data file to be entered can be named.txt or.csv, note the path
Append records into table Table_Name --3. Append records to oracle tables
Fields terminated by ',' --4. Each field is separated
Fields terminatedby X'09' --5. The field terminates in X'09, which is a TAB (delete this line if there is an error)
(Col1, Col2...). 6. Define the corresponding order of columns
Note: the above third command is as follows
A, insert, by default, requires the table to be empty at the start of data loading
B. append, append new records in the table
C, replace, delete the old record and replace the newly loaded record
D. truncate
Control file demo: individual successful cases:
Load data infile 'F:\sqldata.csv' append into table ftotp_tokeninfo Fields terminated by ',' (token,enabled,locked)

Re: Oracle database big data volume import (SqlLoader method)

Posted: Tue Sep 18, 2018 11:21 pm
by youyou
2. Prepare the data file sqldata.csv or sqldata.txt
Note the format in the data file - demo
Data file Demo
00000000000001 0,
00000000000002 0,
00000000000003 0,
Note: 1. The data position in the content file should correspond to the position of the field in the control file above
2. Do not distinguish Numbers and strings in the data file, just separate them with commas
3. After the control file (run.ctl) and data file (sqldata.csv) are ready, enter this directory location under the DOS window
The 2013-3-21 14:53:52 upload
Download the attachment (892 Bytes)
Type the command:
The 2013-3-21 14:53:52 upload
Download attachments (1.69kb)
Carriage return execution;
Note: the userid= Scott/tiger-@orcl orcl in the command is the database (service) name;
The default log file is named: input. Log
The default bad record file is: input. Bad
The result of execution can be viewed in the sqllog.out file under F:\oracle product 10.2.0\db_2\BIN\.

Re: Oracle database big data volume import (SqlLoader method)

Posted: Tue Sep 18, 2018 11:23 pm
by youyou
The following section provides a quick way to create a data file:
public static void main(String[] args) {
try {
// Create the data that will be imported into the database File
String dataFile = "F:/sqldata.csv";
File file = new File(dataFile);
if(file.exists()){
file.delete();
}else{
file.createNewFile();
}
OutputStream output = new FileOutputStream(file);
String dataStr = "{0}, 0, 0\n";
for (int i = 1; i < 100; i++) {
String tokenNum = String.format("%014d", i);
// The first way
output.write(MessageFormat.format(dataStr, tokenNum).getBytes());
// The second way
//IOUtils.write(MessageFormat.format(dataStr, tokenNum), output, "utf-8");
}
if(output!=null){
output.flush();
output.close();
}
} catch (Exception e) {
// TODO: handle exception
}
}

Re: Oracle database big data volume import (SqlLoader method)

Posted: Fri Sep 28, 2018 3:57 am
by summer
Very good,Thanks for sharing

Re: Oracle database big data volume import (SqlLoader method)

Posted: Fri Sep 28, 2018 4:08 am
by youyou
Thank you for your support :P