Server Timeout Sending Large Email Attachment

[RMittelman]

Thanks for reply Elton. I did the same thing but used 10,000,000 for the size. Obviously way too much but it worked. It may need that size when we write the combined PDF to the file.

I was getting same memory issues even during the phase of writing all 75 individual PDFs to the file even before combining. Completely revamped architecture to run steps sequentially, calling server function for each individual PDF. Took longer to run, but now I can write all 75 records without error.

Next step is to work on combining phase.

Thanks…

Ron Mittelman

[eltonsantos]

Thank you for the update, Ron.

It’s good to know you are making progress.

Please keep me posted on how everything goes, so we can also assist as we can.

Regards,

Elton S

[RMittelman]

Hi Elton,

I seem to have made this work. Not only can I render all 75 individual reports, one to each group, but thanks to my good friend ChatGPT, I’ve been able to combine all reports into a single pdf. Or if too large, it will combine enough to reach an arbitrary limit (currently 5 MB), then name it Group Roster-Part 1, then Group Roster-Part 2, etc. then it will attach multiple combined parts to the email. This combining is sensitive to the 5 MB limit, so larger reports will have parts containing fewer group reports. This is fine.

I have yet to test all 75 reports. it takes a long time to render 75 reports on the server. But I’ve tested up to 6-8 reports, temporarily changing the limit to 2 MB for testing, and it worked. I will now change the limit back to 5 MB and test all 75 reports.

It is unclear if the internal issue you mentioned before is still pertinent to me, because my solution now works (pending the full test).

I’ve asked this before, but is there any way to change the title of the attachment icon to the file name that we supplied to the report in code, rather than the arbitrary string of digits.pdf? That would be very user-friendly.

Would it be better to ask this question in the “Feature Request” topic?

Thanks again for all of your help in this matter!!!

[eltonsantos]

Hi Ron,

In the next release, which is still under testing, we are making the GUID displayed for the report name optional.

Regards,

Elton S

[RMittelman]

Great News… All 75 rosters were successfully emailed to me. They were divided into about 6 files which were attached to the email. That will be nice to see the names in the attachments when that is released. Any lead time on that?

Thanks…

Ron Mittelman

[eltonsantos]

Hi Ron,

It is good to hear that you managed to get it working.

We don’t have a lease date yet, but as soon as we are ready, I’ll let you know when your account can be upgraded.

Regards,

Elton S

[RMittelman]

Hi Elton,

I have a followup question regarding the code you gave me:

function SetPDFQuality(five, context, result)  {

result.update('ErrErrorOk', '', '');
result.setOptions({
    ImageQuality: 0.60,
    ImageType: 'jpeg',
});

return result;

}

I tried this code, and it definitely reduces the file size. For example, roster for 2 groups at 0.6 have sizes of 280,716 and 265,824.

At 0.1, the sizes are 127,156 and 121,036

So the questions:

1: It obviously reduces the file size, but I can’t see any differences between the 2 reports attached to my emails. Why?

2: Will this code attached to the proper event also affect the quality of the report when I run it from client? I also don’t see any difference between the reports at the 2 numbers mentioned.

Thanks…

[eltonsantos]

Hi Ron,

When optimising a PDF, the engine usually removes unusual data and compresses objects and metadata, and decreases resolution… Those changes can affect data that isn’t visually noticeable at normal viewing sizes. Therefore, you don’t see any difference.
Example: The settings I previously provided reduce the PDF size whilst maintaining the same quality, which is why you don’t see much difference.

Also, it depends on the report and what you are plotting: charts, images, or texts.

If you change the settings, you may notice a difference in the file’s size and quality.

ImageQuality: (quality percentage )

ImageType:(‘jpeg’ or ‘png’)

Regarding question 2. Yes, this code runs on the report (event On Run), and the report is always generated on the back end, even if you expose the action in a menu.

Therefore, regardless of whether you call it via a callback function, a process or via the menu, if this function is on the On Run event, it will always be called.

Reagrds
Elton S

[RMittelman]

I think something is going wrong on the server. I’m trying to send about 9 different emails, each with 1 or 2 reports attached. My logging is showing only calling the command to execute the report 9 times, but I end up receiving 27 emails. Finally available server memory is exhausted and I get that blank message. when I click OK, five restarts.

Tried it with the executeAction commented out, and the logs show that area of code only being hit the 9 times as expected.

Is it possible that some process is restarting on the server?

Please let me know if you need the FDF file and instructions for recreating the issue.

Thanks…

[eltonsantos]

Hi Ron,

Could you please share the application via email or via driver, and the detailed steps used, so I can further investigate it?

Thank you.
Elton S

[eltonsantos]

Hi Ron,

Thank you for sending your application and details.

After investigating, I can confirm that we have experienced this scenario before.

We already have a solution for this scenario, and your version should include this fix.

You need to call the function executeFunction with the background option; it basically will shift the entire execution to a different session and no longer stress the front end.

When using this new option, you will notice that the execution of teh callback function will finish straight away. It means that the callback function no longer waits for the execution to finish to return; therefore, in the future, I suggest you only use this functionality as the last step in your code, as you won’t have the return from the backend to apply further validations, as it will be executed in parallel.

The function signature is available at: executeFunction() | Five | Low-Code For Real Developers for future references.

This is one example of how to use it:

const executeOptions = { background: false };

five.executeFunction(‘TestFunction’, null, null, JSON.stringify(executeOptions), ‘’, function (result) {

five.log('result: ' + JSON.stringify(result));

});

This is how I used it in your code, in the function ‘GenerateEmails’, when calling the server-side function GenerateEmailsServer under the function SendGeneratedEmails.

five.executeFunction(‘GenerateEmailsServer’, parms, null, JSON.stringify({ background: true }), ‘’, function(serverResult) {

Can you please test yourself if that works for you, and if you have any issues, please let me know.

Regards,
Elton S

[RMittelman]

Thanks so much for working on this issue. I’m slightly confused, so I hope you will indulge me for a moment.

If I understand your answer, nothing in the callback function from executeFunction command will be pertinent any more, as the last server call will be done in parallel instead of waiting for it to finish and run the callback, correct? If so, consider the following code:

    const opts = { background: true };
    const serverFunction = 'GenerateEmailsServer';
    five.executeFunction('GenerateEmailsServer', parms, null, JSON.stringify(opts), '', function(serverResult) {
        // old version: five.executeFunction('GenerateEmailsServer', parms, null, '', '', function(serverResult)

        if (!serverResult) {
            _five.showMessage(`${serverFunction} returned no result.`);
            return;
        }

        if (serverResult.serverResponse && serverResult.serverResponse.errorCode !== 'ErrErrorOk') {
            msg = serverResult.serverResponse.message ||
                  serverResult.serverResponse.Message ||
                  'Error generating emails.';
            _five.showMessage(msg);
            return;
        }

        let results = {};
        try {
            results = JSON.parse(serverResult?.serverResponse?.results || '{}');
        } catch (e) {
            results = {};
        }

        five.log(`GenerateEmails: results=${JSON.stringify(results)}`);

        // advise emails are complete
        if (results.EmailType === 'LEADERS') {
            msg = `Emails sent to ${results.SentCount || 0} group(s)`;
        } 
        else if (results.EmailType === 'BOARD') {
            msg = `Emails sent to ${results.SentCount || 0} board position(s)`;
        } 
        else {
            msg = `Email sent to ${results.ToCount || 0} recipient(s)`;
        }
        ShowToastMessage(five, {Message: msg, Timeout: 5000, XPos: 'center', YPos: 'bottom'});
    });

You will notice I already customized the executeFunction call by sending the options. But everything in the callback is now somewhat useless, right?

So testing that the server function even returned a response, plus interpreting that response and passing information from it back to the user will no longer work? We can’t advise the user how many emails were sent?

If so, how do we know the function did its job?
And should I just remove everything in the callback?

Please clarify if I am misunderstanding your reply.

you said it was stressing the front end. This I don’t understand. If the problem was in the server end, how is the front end being stressed? Please forgive my relative ignorance about how web apps work, as I’m somewhat new to them.

Thanks…

OH, forgot to ask: if this runs in a parallel process, does that process have access to all of my Five variables? Very Important!

[eltonsantos]

Hi Ron,

When using this functionality, five still have access to the properties, and when you execute, you still get the correct emails and data. The only difference is that the process no longer waits for the callback function to finish.

Instead of adding the validation in the callback function, you can use the five.showMessage in the server function (GenerateEmailsServer) to display messages to the user.

For example, look at the images. I simulated an issue on line 452, and this is the result displayed on the front end; therefore, you can still display information to users.

Additionally, you do have the code ‘throw new Error’, which will create an issue and display in the front as well.

When I meant stress the front, I did not mean the client or server function; I meant the entire process running. (Sorry for the misunderstanding.)

Please let me know if the solution meets your requirements.

Regards,
Elton S

[RMittelman]

So I tried the new code, and everything was working pretty well. Then I tried to send 2 reports to every group leader. It seemed to work for a while, even though it took a long time create the emails. I never actually received any emails, and eventually got the dreaded memory error:

I can’t even see the logs at this point because the moment I dismiss the empty message, Five restarts.

Here is the newest FDF with all of the most recent changes, including running the emails in parallel process.

OOPS, for some reason the file is now too large to paste here. Please find it on the OneDrive FDFs folder.

Please advise ASAP via email or here if you no longer have the OneDrive shortcut.

Any idea why it got almost twice as large as the last time I exported?

Thanks…

UPDATE:
FDF file was so big because in test mode I’m not deleting the staged PDFs. I’ve deleted them and now it is somewhat smaller
BrandeisConejo-20260606-182158236990862.fdf (7.2 MB)

[RMittelman]

some of the email types work. GENERIC is fine. if I’ve specified a split report (right now Group Roster is only one), it generates reports for every selected group and saves them to a table, along with any other non-split reports. Then it goes through a Generate Attachments phase, which combines the split reports back into a single report and attaches it to the email, as well as any non-split reports selected.

It used to need 5 or so attachments because my code keeps combining individual PDFs until the combined size matches the maximum configured in settings. This was 5 MB. I increased the setting to 10 MB and it still works, but only needs 2 parts for that report instead of 5 or more.

It would be very helpful if the attachments could be named the desired name instead of the guid-like name. I think you said that was coming. Any ETA?

For LEADERS type emails, it seemed to work, even though before I got the memory error. I don’t know if it’s now successful becaused I raised the memory limit from 5 - 10 MB?

However, the BOARD emails still don’t work. It renders all 75 group rosters plus other non-split reports selected and writes them to the table. After that, I don’t know what is happening, because when the memory issue happens, if I click OK to the blank message, five restarts itself. So I can’t see any logs.

For this report, it combines the rosters by portfolio, into 4 files, one each for portfolios A-D. May be more than 1 file each portfolio if they get larger than the 10 MB memory limit.

BrandeisConejo-20260607-234759976739039.fdf (7.2 MB)

I hope you can help me figure out why this is failing in the final server function, even though I’m running it in parallel as you suggested.

I added some messages to see how far the process has gotten. After initial staging, and rendering, and building the attachments, when it goes to actually execute the code to build the emails, I put five.showMessage(‘Starting Email Generation’) or something like that, but the message never shows up, even though you said it would from a server function.

[eltonsantos]

Hi Ron,

Thank you for sharing the behaviour in detail.

I will investigate and update you when I make any progress.

Observation: I am assuming the steps are the same as when you sent it via email; otherwise, can you please share the steps you are using, especially if you are selecting more records?

Regards,

Elton S

[RMittelman]

Thanks for the quick answer.

BrandeisConejo-20260608-030244239691187.fdf (7.2 MB)

This version is slightly newer. I was having issues with my settings form, which I’ve fixed.

The first mail type, GENERIC, is the simplest. just put the email address you want to send to, then pick reports to attach, and which groups are involved. You don’t need to do anything else except click Generate Emails button.

The other 2 types, LEADERS and BOARD, don’t require (or allow) you to put an email address. Just pick the reports and groups to process. The Send-to addresses are picked automatically from the database. in the case of LEADERS, it finds the Group Leader and Coordinator of each group selected, and uses them for the Send-To. Similarly, the BOARD emails look at the tables for Board Positions and Board Reports to get the Send-Tos.

Very Important: Please don’t send these as-is, or you will be sending live emails out to my members. Change as follows:

Go into Setup>Settings. There are 2 pertinent settings, TestEmails and TestPdfAttachments. Make sure TestEmails is set to true. If so, the emails will go to one particular email address instead of all the group leaders or board members.

There are 2 other settings, EmailFromAddress and EmailFromName. These control who the emails will actually go to if TestEmails is true.

You can leave the email info as me, but if you are testing it may be better to temporarily set them to yours. After any settings are changed, the application saves them as variables. You can always click the ShowFiveVariables action button at the very top. this will show all five variables, including all of the settings.

When the emails are done being sent, the staged PDF files are deleted from the table unless you want to inspect the table later. in that case, turn on the TestPdfAttachments setting and they won’t be deleted. You can probably not do that, as I’ve already tested that feature.

The Reports and Groups selected will control which reports get printed and which emails get sent. For the LEADERS, I’ve successfully been able to select all groups, and have only tried 2 reports, GroupRoster and GroupSchedule. The first one is a split report, and the second one isn’t. Those reports/emails seemed to work.

What doesn’t work is the BOARD emails, where I selected EVERY report and EVERY group. The GroupRoster will be summarized by portfolio when it comes time to send out those emails, as per the Board Positions and Board Reports tables, which you can see by looking at Setup>Board>Board Positions. so the Group Roster, if selected will be sent to the group registrars and vp’s that are associated with each portfolio.

BUT, remember to set that TestPdfAttachments setting to true so they don’t really get the emails, but I (or you) will.

Sorry, very verbose. Please reply with any questions.

[eltonsantos]

Hi Ron,

It seems that the Report type ‘MEMBER LIST’ is responsible for the error.

I only got the same error you have reported when I selected this Report.

It may be related to an internal loop that does not handle this record/report type properly.

You can test it by simply selecting the Report type ‘MEMBER LIST’ and one Group; it seems that nothing happens, but if you wait for a while, it will eventually fail.

Or for a quicker response, you can select this report and another one that works, and only one group.

Observation: In your test, can you please test each report type individually, because even though I have not looked at the code, it seems to be an error of logic behind the scenes, as only one type is falling for me.

Hope it can help you. Please let me know that goes.

Regards,

Elton S

[RMittelman]

Thanks for your help on this, Elton.

There is definitely something going on with this report (Member List).
Here is my query ShowStageReportPDFs

SELECT
  `StageReportPDFs`.`StageReportPDFsKey` AS `StageReportPDFsKey`,
  `StageReportPDFs`.`UserKey` AS `UserKey`,
  `StageReportPDFs`.`BatchKey` AS `BatchKey`,
  `StageReportPDFs`.`IsSeparatedByGroup` AS `IsSeparatedByGroup`,
  `StageReportPDFs`.`Seq` AS `Seq`,
  `StageReportPDFs`.`CreatedAt` AS `CreatedAt`,
  `StageReportPDFs`.`ReportID` AS `ReportID`,
  `StageReportPDFs`.`RunKey` AS `RunKey`,
  `StageReportPDFs`.`GroupKey` AS `GroupKey`,
  `StageReportPDFs`.`GroupName` AS `GroupName`,
  `StageReportPDFs`.`PortfolioKey` AS `PortfolioKey`,
  `StageReportPDFs`.`Portfolio` AS `Portfolio`,
  `StageReportPDFs`.`PdfSize` AS `PdfSize`,
  `StageReportPDFs`.`PdfData` AS `PdfData`
FROM
  `StageReportPDFs`
ORDER BY
  `GroupName` ASC

Here are partial results. I had to scroll horizontally to see the important columns.

The first thing I notice is the first 3 records, which are for MemberList, are much larger in the PdfSize column, over 10 MB. And the report itself is missing. As opposed to the last report, GroupSchedule. There you can see a much smaller size and you can also see the beginning of the report PDF.

So maybe it’s not a logic issue, maybe it’s something wrong with the report itself.
I was able to bring the report up in the client then download it, and it was definitely 10 MB or larger.

Any ideas why this would happen? I notice the PdfData field in the table is binary, and size is 10,000,000. That would explain it.

I could simply increase the size to 20,000,000 or some other reasonable number, or figure out why this report is that large. It’s a simple list of members, but it is about 30 pages long.

Any ideas?

Thanks…

UPDATE: Actually it is not over 10 MB if I run the report in the client and download it. In that case, it is 7.6 MB. Still huge. The over 10 MB size is when I stage the PDF, as shown in the image above.

UPDATE 2: I increased the size of the binary column “PdfData” to 13,000,000. The StageReportPDFs table still shows 10,775,804 and doesn’t show the actual data in the query results, but that may be a limitation of the viewer, because this time the report did finish and I received the email. So I guess it was the table column size.

However, is there any way to make this report smaller? I’m already using 0.3 for the image quality setting.

[eltonsantos]

Hi Ron,

Regarding the column size, you can definitely, you can define the size like:

On my end, I have defined it as MEDIUMTEXT (16777215), 16 MB.

TINYTEXT | 255 (2 8−1) bytes
TEXT | 65,535 (216−1) bytes = 64 KiB
MEDIUMTEXT | 16,777,215 (224−1) bytes = 16 MiB
LONGTEXT | 4,294,967,295 (232−1) bytes = 4 GiB

I could not find exactly what the issue is, but it could be the same issue that caused the memory spike. I noticed the PDF was sent to a few users and, as you mentioned, it has a certain number of pages. When testing my local environment, it does work, but it takes 8 minutes to finish (which is too long).

My suggestion: check whether there is any part of the code that can be optimised, such as loops and code that can be called only once.

Also, you could try to call the function ‘RenderStagedReportServer’, in the background, it could solve the issue, but if you do so, you may need to re-engineer your code, as you will no longer have the return to continue, for example, when testing it on my side, I did not get any email because all the code was executed at once without waiting for the callback.

Regards,

Elton S