Problem Processing Large Reports on Server

How do I?
1

In my Brandeis Conejo application report generation on the server seems to have issues with large reports.

The report generation consists of several phases:

  1. Stage report data. runs the report queries and writes the resulting data into a staging table such as StageMemberList This is called from client function GenerateEmails, which calls StageReportsServer, and uses a callback function.
  2. The callback then runs a new function, RenderStagedReports, which first calls CleanupReportPDFBatchesServer so old batches are removed. Then it calls RenderStagedReportServer to generate the PDF file and write the returned base64 text to the StageReportPDFs table. This also has a callback function in the client.
  3. The BuildEmailAttachments client function calls BuildEmailAttachmentsServer to actually sort through the data and build the attachments. This also has a callback function.
  4. The client function SendGeneratedEmails is called, which in turn calls GenerateEmailsServer function which runs in the background, so no callback processing is done.

Mostly this works fine, but if I have a LARGE report, it will not process. I end up with a blank message in Five, and when I click on the OK button, Five restarts.

To reproduce the issue:

  1. Go to Processing > Generate Emails.
  2. Select Board Members for the email type.
  3. Under Reports, select all reports.
  4. Under Groups, select all groups.
  5. Click Generate Report button.

The first time I ran this, the error seems to occur during the rendering phase, because nothing happens for a long time, then the error occurs. In this scenario, I saw a bunch of reports were staged, but the error seemed to occur when this report was trying to be rendered or staged.

When I restart Five and try this with only the Member List report, then it works.

The main difference between the original code and current code is that instead of attaching the reports to the emails, I am uploading them to FilePost, a web file sharing and storage site. Then I put the returned shortcut into the outgoing emails, and they can be clicked on by the recipient and see/download the report.

If you want to try it, I recommend going into Setup>Settings and change the EmailFromAddress to your own for testing.

Log on as Admin with the credentials I gave you before. This enables you to take advantage of the pdf caching, so most reports don’t need to be rendered again.

The FilePost API key is stored in the Settings table, so you don’t need to do anything other than run the application.

So it seems like the Member List is still the culprit. I redid the run but this time did not include the Member List, and it ran completely. I have no idea why that report is so large. I have the PdfImageQuality setting set to 0.5.

One more point: This time the failure seems to be in the Render Reports phase, not the last phase of sending the emails. We may have been premature in assuming the issue was in the last phase, and making the last server function run in the background.

I hope you can help me figure this out.

Thanks so much…

2

Hi Ron,

The memory issue is probably happening because the process still keeps large base64 PDFs and attachment arrays in memory before sending.

Your flow is much better than generating everything at once, but steps 3 and 4 may still be the problem:
1 - BuildEmailAttachmentsServer
2 - builds attachment objects
**3 - GenerateEmailsServer reads them
4 - sends emails
**
For large reports, can you try to avoid building all attachments first? Use a streaming/batch pattern instead:

  1. Stage report data
  2. Render one report/batch PDF
  3. Upload that PDF to FilePost immediately
  4. Save only the FilePost URL in the table
  5. Clear/delete the base64 PDF from staging
  6. Repeat for next report/batch
  7. Send emails using links instead of PDF attachments

Therefore, you do not pass the PDF back to the client, nor do you keep all PDFs in an attachment array.

So, as soon as each PDF is generated:

PDF generated as base64:
1 - upload that single PDF to FilePost
2 - get FilePost link
3 - Save only the link in your table
4 - remove/clear the base64 PDF
5 - move to the next PDF

You can also use ChatGPT to help you with this.

Please let me know how that goes.
Regards,

Elton S

3

Hi Elton,

Thanks for the reply.

I’ve started to completely re-write the process. the sequence was:

  1. Stage report data
  2. Render reports individually, save to table
  3. Build attachments. This will collect split reports and combine them appropriately, depending on the flow i’m in. It was saving the rendered and combined reports in memory.
  4. Generate the emails. During this phase, the pre-built attachments are uploaded, then upload response is used to add text to the email.

The sequence now is:

  1. Stage report data
  2. Render reports individually, save to table
  3. Build attachments. same as above, but now also upload each attachment, and add the upload response to the in-memory attachments list instead of the base64 pdf.
  4. Generate the emails. this has been modified to not try to upload the files

This change is only implemented for the GENERIC flow so far, and only for non-split reports. Other flows still work the original way.

I’ve ascertained that the memory error occurs in the rendering phase, and only if the MemberList is selected. So this very large report is causing an issue during rendering. It renders fine when I use the Processing > Generate Reports menu and generate the report on the client. The issue happens when using the Generate Emails flow, causing the report to be generated on the server.

Is there any way to figure out if this report can be made smaller, or what is happening on the server when the report is being rendered? I’m using 0.5 as the PdfQuality.

Thanks…

4

Hi Ron,

Could you please share the latest version of your application in the shared folder?

Thank you.

Regards,
Elton S

5

Newest FDF file is on the OneDrive folder.

Remember:

Processing > Generate Reports > MemberList seems to work.
Processing > Generate Emails > Generic is the problem.

You need to enter any email address in the To field. It will end up going to me anyway, because TestEmails setting is set true.
You can either change that setting to false, or change the EmailFromAddress to your own, and leave TestEmails setting true.

Either way, you may need to select one or more Interest Groups in the appropriate tab, and select the report you want to test in the Reports tab.

The only issue seems to be when rendering the MemberList, which is a very large report. About 900 members, 30 pages. I hope we can make that smaller, or at least figure out how to get more memory on server.

I haven’t actually worked on optimizing the memory usage in the Rendering phase, and we may have large report objects in memory. Not sure.

Thanks for looking at this, Elton.

6

This may be fixed.

I found that the timeout was happening during report rendering, not during sending. So I re-engineered the report rendering and don’t get that error anymore. I also removed the code that caused the actual emailing server functions to run in the background, so now it runs like any server function, including having a callback that I can code.

The revised FDF file is up on OneDrive.

Note, I’ve only fixed one workflow, which is GENERIC NON-SPLIT reports to use the new architecture. Feel free to test this, but I will give a more extensive testing effort once I’ve updated the code for all flows (split reports, LEADERS, BOARD, etc).

Thanks…

7

Hi Ron,

I am pleased to know that you got it working.

I will have a look at the final version.

Thank you for sharing your solution.

Regards,
Elton S