My GenerateEmails form is working. I can successfully send an email with a report attached.
When I accidentally tried to send a somewhat large report, it thought for a while then came back with a server timeout message.
Is this something that can be fixed, and how long of a process will cause this message?
Thanks…
Hi Ron,
Thank you for bringing this scenario to our attention.
Could you please upload an updated version of your application to the shared folder and send me an email at elton@five.co with the steps so I can try to reproduce your scenario?
Thank you
Elton S
Hi Ron,
After investigating this scenario, I have identified two possible issues in your report design:
1 - Timeout when selecting multiple study groups:
The timeout occurs when multiple study groups are selected. This happens because Five has an internal limit of 3 minutes to generate a report. If the report is not completed within this timeframe, the process fails due to a timeout.
2 - Email not sent due to attachment size limitations:
Depending on the number of study groups selected, the email may not be sent because email providers (such as Gmail) have limits on attachment size.
For example:
When generating the report with the first 10 study groups selected, the PDF size is approximately 8 MB.
When selecting the first 20 study groups, the file size increases to around 15 MB.
An improvement can be implemented by adding a function to the “Study Group Roster” report to the event “On Run”. With this function, you can pass a configuration object to the report to control settings such as image quality and format.
For example, when selecting the first 10 study groups, the PDF size can be reduced to approximately 2 MB, compared to the previous 8 MB.
Below is an example of this function using an options object:
function SetPDFQuality(five, context, result) {
result.update('ErrErrorOk', '', '');
result.setOptions({
ImageQuality: 0.60,
ImageType: 'jpeg',
});
return result;
}
Regards,
Elton S
Thanks for this answer, Elton.
This is a major disappointment, that there is a hard timeout after 3 minutes. I will try your code in the OnRun event. While that may help, I don’t know if it will totally solve the issue.
Can you please point me to documentation regarding this function you are suggesting? It’s not clear what the 0.60 means. I guess the lower the quality number the smaller the file?
Also, I’m confused by the ImageType: ‘jpeg’ item. Why is a PDF file having anything to do with a jpeg image?
Finally, is there ANY way that 3-minute limit can be overridden? This seems like a big show-stopper for anyone who has large reports to generate in the background.
Thanks…
Hi Ron,
I have notified the team regarding increasing the timeout. I will keep you informed of any updates.
Meanwhile, and I believe you may need it, you can generate a PDF for each studyGroup, then merge them into a single PDF when you send the email.
This is an example of how to achieve it, which I am regenerating 4 reports, combining and merging them into 3 PDF files:
function DoAddAndMergeAttachments(five, context, result) {
const reportResult0 = five.executeAction('StaffExtension', {});
if (reportResult0.isOk() === false) {
return five.createError(reportResult0);
}
const reportResult1 = five.executeAction('WebProductsReport', {});
if (reportResult1.isOk() === false) {
return five.createError(reportResult1);
}
const reportResult2 = five.executeAction('PositionReport', {});
if (reportResult2.isOk() === false) {
return five.createError(reportResult2);
}
const reportResult3 = five.executeAction('StaffReport', {});
if (reportResult3.isOk() === false) {
return five.createError(reportResult3);
}
const pdfs1 = \[\];
const pdfs2 = \[\];
// Array pdfs1 with reports (StaffExtension and WebProductsReport)
if (reportResult0.report) {
pdfs1.push(reportResult0.report);
}
if (reportResult1.report) {
pdfs1.push(reportResult1.report);
}
// Array pdfs2 with reports (PositionReport, and StaffReport)
if (reportResult2.report) {
pdfs2.push(reportResult2.report);
}
if (reportResult3.report) {
pdfs2.push(reportResult3.report);
}
// Array pdfs3 with reports (StaffExtension, WebProductsReport, PositionReport, and StaffReport)
const pdfs3 = \[\];
pdfs3.push(...pdfs1, ...pdfs2);
//Store the PDF utility method.
const pdfUtils = five.pdfUtils();
//Merge reports (StaffExtension and WebProductsReport) into PDF pdf1
const mergedPDF1 = pdfUtils.merge(pdfs1);
if (mergedPDF1.isOk && mergedPDF1.isOk() === false) {
return five.createError(mergedPDF1, 'Failed to merge pdf1 files for attaching');
}
let attachResult1 = five.addAttachment(mergedPDF1, "pdf1.pdf");
if (attachResult1.isOk() == false) {
return five.createError(attachResult1);
}
//Merge reports (PositionReport, and StaffReport) into PDF pdf2
const mergedPDF2 = pdfUtils.merge(pdfs2);
if (mergedPDF2.isOk && mergedPDF2.isOk() === false) {
return five.createError(mergedPDF2, 'Failed to merge pdf2 files for attaching');
}
const attachResult2 = five.addAttachment(mergedPDF2, "pdf2.pdf");
if (attachResult2.isOk() == false) {
return five.createError(attachResult2);
}
//Merge all reports (StaffExtension, WebProductsReport, PositionReport, and StaffReport) into PDF pdf3
const mergedPDF3 = pdfUtils.merge(pdfs3);
if (mergedPDF3.isOk && mergedPDF3.isOk() === false) {
return five.createError(mergedPDF3, 'Failed to merge pdf3 files for attaching');
}
const attachResult3 = five.addAttachment(mergedPDF3, "pdf3.pdf");
if (attachResult3.isOk() == false) {
return five.createError(attachResult3);
}
return five.success(result);
}
Below is the definition for the SetPDFQuality function (we are in the process of updating our documentation). The properties definition is:
ImageQuality: Set the quality of the pdf content. Example: default is 1, meaning 100%, 0.6 means 60%.
ImageType: The report generates an image for each page during PDF creation, which is then compiled and converted into a final PDF.
The supported types are ‘png’ and ‘jpeg’
Regards,
Elton S
Thanks for the suggestions, Elton.
You may be right about needing to generate in smaller batches (one roster at a time), then put them together. That would also have an advantage of each roster having “page 1 of 2”, “page 2 of 2” instead of “page 1 of 40” which happens when I generate a single report for many groups.
It’s unclear if this is a huge advantage or not. The main thing is avoiding the timeout. where does the timeout occur? is it when the executeAction(‘reportName’, {}) is happening? If so, that would explain the timeout.
It looks like this is a server function, only because the name starts with Do. But it’s not the server function itself that is timing out, but the executeAction, right?
Hi Ron,
Indeed, my example was executed on the Do event (Server-side).
The timeout is set on the report internally (by default), and not in the executeAction function, as an action can also be Process, Mailmerge, etc.
Regards,
Elton S
I’m curious about your example of generating reports individually then combining them. your example function begins with the word “Do”, so I’m inferring that this is a server function. Please correct me if I’m wrong.
With on-demand reports, the server function is called to do the staging if needed, then either way, the report is actually generated inside the callback function. On demand report does not necessarily time out if I’m generating the full (largest) report. But this doesn’t guarantee that it won’t time out in the future as data grows.
Definitely this is an issue with generating emails with reports attached. In that case, the actual generating is controlled in a server function, and the executeAction is also called in the server function instead of from the client.
So now I’m wondering if I should create a new server function dedicated to generating a single report, then I can either call it from inside the callback function (if doing on-demand report). The problem is that we can’t call a server function from another server function. So I could make the final function to generate the single report into a library, that could be called from either client or server, called GenerateSingleReport.
This library could decide whether to iterate groups and generate a single report at a time then combine, or generate a full report. This decision can be based on a variable set from my ReportSettings table.
To do this, I would either need to create 2 different queries (one joining to SelectedGroups table, which is done currently, and one exactly the same but without using the join to SelectedGroups), then decide which one to use based on some metadata. But I believe that would require me to have 2 different report definitions, since the parameters are hard-coded in the report definition.
Alternatively, do you think I can somehow make the join in the query optional, based on a variable or something? That way I still only need a single report/query. But how, in five, would I do that?
Thanks for taking the time to read this post. I feel like we are getting closer…
UPDATE:
I figured out that I don’t need to change the query at all. When calling the report from client-side, just include all records based on the join with the SelectedGroups table.
When running the reports to attach to the emails, this is done server-side. There, I can control which of the originally selected records in the SelectedGroups table are selected. I’ve already written code that will snapshot the groups where IsSelected = true, then reset so only the desired group is selected, then run the report with the original query, then reset the SelectedGroups to where it was from the snapshot when I start the process. I’ve already tested that code, and it should work.
This lets me call the report from either client-side or server-side.
I need to verify the following:
Thanks…
Hi Ron,
Thank you for the update and for teh extra questions, here are the answers:
1 - The timeout is always on the server-side, which is where the reports are executed.
2 - I cannot guarantee it, as it depends on how long it takes to generate the report. That’s why I mentioned that splitting the report would help you, so that each report execution has a timeout to finish (in the context of a group of many studies).
3 - This is still under investigation. The team works across different demands, including improvements, new features, and issues. I will keep you posted about the issue.
4 - In my example, I used the function that attached the reports to an email. It means that when I call the email, I do not pass the attachment property to it (as everything is generated inside).
Yes, this function is attached to the event “Do Merge Record“ server-side,
Observation: this was only one example; you could also generate all reports in the main server-side function and pass the array with many attachments, then the function that attaches the email can merge the reports.
Hope this information can help you.
Regards,
Elton S
Thanks Elton. This does help somewhat, but I need some clarification.
I’m not sure I stated my questions properly.
1:
The documentation doesn’t address this issue. I’ve been able in the past to generate the report from the client-side, inside of a callback function after doing some prep work by calling a server function. I’ve also been able to generate reports directly from a server function (before that stopped working). I wanted to know if the timeout issue you talked about may be there regardless of whether the executeAction to generate the report comes from client-side or server-side.
2:
The report I’m referring to is the GroupRoster. Since no groups have more than 20-25 members or so, this is NOT a large report. However, doing one for all groups does get large. Is the report generation via executeAction subject to the same possible timeouts even if called from client-side?
4:
I am working toward a similar technique as your observation, where I generate reports on the server-side, and if they are too large (like the GroupRoster), I can generate them in smaller chunks and combine. If not too large, just generate them normally. This question is mainly around whether I should change my technique and do this also when doing on-demand reports (which don’t involve emailing or attaching).
I think I should always do the reports server-side, because of the timeout issue, but then I don’t know how to render them client-side if they are on-demand reports. I only know how to generate them client-side (in the callback function). In fact, I suspect that on-demand reports to be displayed in the browser must be called client-side. True?
I have completely re-engineered my process for generating reports, especially those that are large, and need to be split by group (currently only GroupRoster).
Before, I had a couple of staging tables for their respective reports. now I have changed the architecture so all reports have a staging table, which use UserKey and RunKey to keep the records unique. This helps solve the issue where the staging wasn’t completely done before the report was to be generated, causing only fragments of the reports to be generated.
Any report that is indicated it needs to be split by group will now generate for all SelectedGroups, but only 1 group at a time. When this is complete, I combine the appropriate attachments per your earlier examples.
Still, if I select a lot of groups to be included in the report, I still get the timeout error (at least I think it’s a timeout error, as the error message is blank). At this point, the Five session gets automatically logged off. But if I select a small number of groups for the report, it works properly.
Please advise any progress at your end.
Hi Ron,
I had made further analysis and update the team about this issue; this is one of our priorities, and hopefully the changes will come soon.
Could you please share the new version of your application, so I further analyse it using your latest version?
Call you also share the steps you are using?
Regards,
Elton S
Thanks for the reply Elton.
Here is the newest fdf.
BrandeisConejo-20260507-170458618907847.fdf (7.0 MB)
To recreate the issue:
Go to Processing>GenerateEmails.
Select GENERIC. Enter your email address in the To field. Select Group Roster in the Reports tab. Select any number of groups in the Groups tab, then click the Generate Reports button. If you chose a small number of groups, it works fine. In the background, each GroupRoster is generated individually, then combined into a single PDF, which is attached to the email. It will take some time, but you should receive an email. If you chose a large number of groups, there is a long pause then you get a blank error message and the session logs itself off and tries to connect again.
Thanks…
Hi Ron,
Thank you for the details and the new File. I received an error related to the size of the PDF when running 40 study groups. This scenario is not an issue from Five. This is a limitation of Gmail, where the size of the email (message, body, and Attachments) cannot be exceeded; you may be able to increase it through your Google Workspace (if required later).
To help reduce the PDF file size, you can decrease its quality while the PDF is generating.
Please, add this function to the On Run event of the Report.
function SetPDFQuality(five, context, result) {
result.update('ErrErrorOk', '', '');
result.setOptions({
ImageQuality: 0.60,
ImageType: 'jpeg',
});
return result;
}
With this change, I managed to select 40 groups, and the final size was 9MB.
where previously, I ran 20 groups, and the file size was 15MB. which is a huge difference.
On top of that, I ran with all groups (75), and it works!
Observation: This will help but not solve your issue if there are too many groups, because as you increase the size of the file, it will eventually reach Gmail’s limitation. Hopefully, you won’t face this issue.
Regards,
Elton S
Hi Elton,
This fix doesn’t work for me. I selected all groups to be included in the roster and I still got the error after some time.
It is hard to understand why you are convinced it is an error with Gmail. the error message from five shows up, and is blank. Not only that, when I click OK to the blank error message, five then closes. I don’t get why Five closes after an error. It would be helpful if the error was specific about what went wrong.
Hi Ron,
When I previously tested your application, I used a local version, which had misled my analyses.
I have made a further analysis of your current application in a similar environment to your account.
I have sent an email with all findings.
Reagrds
Elton S
Thanks so much for all your efforts, Elton. It was very complete!
I’m trying to get a sense of when the memory error happens. I certainly understand how keeping all group roster reports in memory will use up too much memory. Even writing the data to a table instead of keeping it in memory uses a lot, especially as we get to the end, and the combined pdf has grown huge.
I wonder if you can enlighten me as to whether the combinePdfs methods use a bunch of extra memory, besides the combined pdf and the one or ones to be added to the combined pdf. Does the pdf library create duplicates in memory of the data it is combining?
If I have 5 PDFs to add to the combined PDF, I think the technique would be to read the combined PDF from the temp table and push it to the array in memory of those to be added, then send that array to the CombinePdfs function, which will combine all and then update the temp table with that new combined PDF.
If I’m using your technique of writing the individual PDFs to the temp table as they are generated, then reading X number of them, pushing them to an array, then combining, how much memory might that use in addition to the PDF data itself?
I’m trying to analyze whether I need to write the individual PDFs to the temp table, or keep them in memory, but have a maximum number of them in memory at a time. for example, generate a pdf, keep in memory, repeat until I have 3-5 of them, then push them into an array and combine. The exact number to keep in memory before combining into the full PDF depends on how much memory the PDF utilities use for that operation.
It may be better to do it one PDF at a time, but that seems like it would create its own overhead and slowdowns having to read combined PDF, combine the single new PDF into it, then write it back to temp table. It seems that would be inefficient.
So I’m looking for some perspective to decide how many group PDFs to accumulate before trying to combine, then clearing that memory before repeating.
I hope I haven’t confused you with my rambling.
Hi Ron,
1 - Combining PDFs usually uses more memory than just the final PDF size.
During a merge, memory may include:
Current combined PDF + Next PDF or group of PDFs + New combined PDF being created + Internal working memory used by the PDF utility
It means that a 10 MB PDF may take more than 10 MB in memory to process.
Observation, of course, we need to take into account how complex it is to get all the data to generate the report.
The example I sent to you, the memory improvement comes from changing how the process is structured.
Instead of generating all PDFs and keeping them active in memory, the process now does this:
Generate one PDF → Save it to temporary storage → Stop holding that PDF in the active process → Move to the next PDF.
So yes, technically, this can include setting the PDF variable to null, but the bigger improvement is that each report is handled in a short backend step. When that step finishes, the local variables used for that report are no longer needed and can be garbage-collected by the runtime.
This does not force memory to be released instantly, but it removes the references that were keeping the PDFs active.
2 - A possible middle-ground later would be to merge small batches, such as 3–5 PDFs at a time, but I would start with one-at-a-time merging for stability. Once we know the average PDF size and performance, we can tune the batch size if needed.
(Regarding the memory, it will be difficult to know as it depends on how many reports you want to process at once, and how long it takes to process each one).
Just in case it helps, here’s a thought after looking at your scenario:
Rather than creating a staging table during the process itself, could you create it while the user is in the front end, selecting the information? As they make their selections, you could create one record for each group and add it to a primary key that is generated on the fly. Then, when the process runs, there would be no need to clean and rebuild the table each time—you would simply read from it. After the report and email are generated, you could update the status of that record.
You could also have a scheduled job run daily based on the record status and date.
Regards,
Elton S
I’m trying your technique. I’ve rewritten the code to write each pdf to a new table. since I cannot create temporary tables in Five, I created a permanent table that I clear, then fill up. it was working for a bit, but now it overflows the PdfData field. here are my fields:
SELECT
StageReportPDFs.StageReportPDFsKey AS StageReportPDFsKey,
StageReportPDFs.UserKey AS UserKey,
StageReportPDFs.BatchKey AS BatchKey,
StageReportPDFs.Seq AS Seq,
StageReportPDFs.ReportID AS ReportID,
StageReportPDFs.RunKey AS RunKey,
StageReportPDFs.GroupKey AS GroupKey,
StageReportPDFs.GroupName AS GroupName,
StageReportPDFs.PortfolioKey AS PortfolioKey,
StageReportPDFs.Portfolio AS Portfolio,
StageReportPDFs.PdfSize AS PdfSize,
StageReportPDFs.PdfData AS PdfData,
StageReportPDFs.CreatedAt AS CreatedAt
FROM
StageReportPDFs
First I defined PdfData as Memo type field. This seemed to work, but now it is giving me an error that it is overflowing the field. the size was defined as 10,000,000 so that should have been enough for any reasonable PDF for a single group. I even changed it to 20,000,000 but that didn’t work either.
Your example from email showed the field as longblob, but since I can’t get to MySQL, I need to change to one of the Five-accepted types. I tried Binary, but it defaults to 65535 max length. can a binary field be increased to 5 or 10 million?
thanks…
Hi Ron,
This is the definition of the field that I was using to support a 4 GB MB longblob.
Before, I had defined it with a size of 16,777,215 bytes → 16 MB, but I was getting an issue, so I needed to increase it.
After you change the table, please, click on the button ‘Create Table’ to recreate your DB structure based on the new definition.
Regards,
EltonS
