I try to insert data from a JS function but no success

Hello, with a front end function called from an OnClick Events linked to a Process button, I load a csv file from my disk.
Then the following CodePostalInsertBatch function is called but my table remains desperately empty.
According to my log, the values of the variables cp, mun, reg are valid.
The log five.log(${inserted} lines inserted); displays “22 lines inserted” which should be ok.

Can you tell me what is wrong wih my function? Thank you!

Jean

function CodePostalInsertBatch(five, context, result) {
    if (!context.batchData) {
        return five.createError("Empty Chunk");
    }

    const lines = context.batchData.split(/\r?\n/);

    const sql = `
        INSERT INTO CodePostal (CodePostal, NomMunicipalite, CodeRegionAdm)
        VALUES (?, ?, ?)
    `;

    let inserted = 0;
    const db = five.getDatabaseConnectionByID('ErisiaDB');
    const tx = five.startTransaction(db);

    for (const line of lines) {
        const cols = line.split(";").map(v => v.replace(/"/g, "").trim());
        if (cols.length < 3) continue;

        const cp  = cols[0];
        const mun = cols[1];
        const reg = parseInt(cols[2]);

        if (!cp || !mun || isNaN(reg)) continue;

        //five.log(" " + cp + "-"+mun+"-"+reg);//values are OK

        five.executeQuery(tx, sql, 0, cp, mun, reg);
        
        inserted++;
    }

    five.commit(tx);

    five.log(`${inserted} lines inserted`);
    result.message = `${inserted} inserted`;
    const verify = five.executeQuery("SELECT COUNT(*) AS nb FROM CodePostal");
    five.log("Verification result: " + JSON.stringify(verify));

    return five.success(result);
}

Hi Jean,

Looking at your code, I did not see any errors, unless you want to share your application with me via: elton@five.co.
However, I noticed that you are not testing the result of getDatabaseConnectionByID, startTransaction, and especially the insert ‘five.executeQuery(tx, sql, 0, cp, mun, reg);‘ in which you will know whether the insert has been executed OK or possibly failing silently.

This is one simple example of how to test the DB connection:
const db1 = five.getDatabaseConnectionByID(‘TestAppDB’);

        if (!db1.isOk()) {

            return five.createError(db1, 'Connect to database failed');

        }

        const tx1 = five.startTransaction(db1);

        if (!tx1.isOk()) {

            return five.createError(tx1, 'Create transaction failed');

        }

This is one simple example of how to test the insert, which could be failing silently.

        let queryResults = five.executeQuery(tx, sql, 0, cp, mun, reg);

        if (!queryResults.isOk()) {

            return five.createError(queryResults);

        }

Also, I strongly suggest you log the variable ‘sql‘, copy the SQL statement, and replace the ‘?’ question marks with a valid value and run it in the Queries menu; therefore, you will know if the SQL statement generated is valid.

Please let me know if you have any questions.

Regards,
Elton S

Thank you Elton for your answer. I found my problem: as you suggested, I run my query with the Queries menu and I get this error:
“Error 1364: Field ‘CodePostalKey’ doesn’t have a default value”.

But in my table CodePostal, for the primary key field, the switch “Generated” is toggled on.

If in my query, I add a key in the INSERT command, then insertion is successfull. So the automatic key generation does not seem to be taken into account.

Is that mean that inserting a new record through a javascript function, we need to also insert the primary key value?

Hi Jean,

The switch “Generated” creates a GUI at the form level. For example, if you insert a value into a form, this form does not have a primary key field directly because five will handle it using this switch.

You need to create a valid primary key value (GUI) and add it to the insert statement.

You can generate a valid GUI using the method ‘uuid()’. Example:

const insertResults = five.executeQuery(insertSQL, 0, five.uuid(), Filed1, Filed2, Field3t);

Below are the changes that I believe need to be made (in bold).

1 - Add the primary Key to the insert statement.

const sql = `
INSERT INTO CodePostal (CodePostalKey, CodePostal, NomMunicipalite, CodeRegionAdm)
VALUES (?, ?, ?, ?)
`;

2- Generate a valid GUI.
const codePostalKey = five.uuid();

3 - Add the new GUI to the query execution.

five.executeQuery(tx, sql, 0, codePostalKey, cp, mun, reg);

Please let me know how it goes.
Regards,
Elton S

Hi Elton, it works very well, thanks you!

However, I have a file of 200000 lines of postal codes. This file is only 4600 kb in csv format, but once uploaded in a table, I am very surprised to realize that - according to my sample tests - my table will be over 10 Go !! . The MSAccess version was only 17000 kb. I will have to find a way to connect this data outside of five.co. Maybe you have some suggestions, but I will try to find a solution on my side as well. The goal it to be used on read-only mode by the user (get the city name according to the postal code), and entirely updated by an admin once a year.

Regards,

Jean

Hi Jean,

Could you please provide more details on how the table’s size got over 10GB? And how was this figure calculated?

Additionally, could you please share your CSV file via my email elton@five.co, and the details of your table CodePostal (size of all fields).

Thank you,

Regards,

Elton S

Ah ah, now I understand why my calculation of size is so wrong! I inserted 1000 lines and the fdf files exported is 49 Mb. The CodePostal table is the only table in my sample application, so I assumed that for 200000 lines, the size of the fdf would be 10Gb.

I realized that before that, I tried to insert the 200000 lines csv file and canceled the operation after few minutes. The “DELETE from CodePostal” removed the records inserted but didn’t clean the table so that explains the 49 Mb size.

Is there a way to completely clean a Table after a DELETE operation, something like “Compact database” we have with MS Access? The only solution I found was to delete table and create a new one…

I am sending to you my csv file as well as my sample application to your email address.

Regards,

Jean

HI Jean,

A simple way to clean your data in a table is by recreating the table.
(Five will recreate the ‘Create Table’ script behind the scenes based on your table definition; in this way, you don’t need to delete and create a table any time you want to clear all records from a particular table.)

Only the development data will be clean (Data from the Testing and Production environments will not be changed).

The steps below will delete all records from a table.
Steps:
1 - You can navigate into Tables.

2 - Select the table you want to delete all records from.

3 - Click on the button ‘Create Table’ and wait for the process to finish.
The image below shows the button ‘Create Table’.

1 Like