FAQ re: Practical Techniques for Improving Application WAN Performance
First of all, let me just say that I’m grateful for the awesome response to my 2011 FileMaker DevCon session on Practical Techniques for Improving Application WAN Performance. As you might expect, there were a lot of questions from attendees – both during the session and afterwards – and hardly enough time to get to all of them.
The letter below came from one such attendee, who was gracious enough to let me answer him publicly via this post. I’ve broken up his email so my answers could be placed inline and in the correct sequence.
Hopefully his questions match some of your own and this response will prove helpful. If not, please feel free to email me and I’ll do my best to respond promptly, perhaps publicly 🙂
A huge thank you to you for your session on practical techniques for improving WAN performance. This is a thing that I am constantly striving to get better with and any little tidbit or technique to help improve performance is always appreciated, and I found a number of them in your talk so cheers! I have a couple of questions I noted from the talk (too shy to ask them during the talk or at the end!)
Thanks for writing and for your questions. A few brave souls hung out after the session to see the WireShark demo (more on that later) and to ask some follow-up questions. Thanks too for letting me reply to your questions publicly, for the benefit of all.
The first one is concerning wireshark, I've downloaded it today and been playing around with it, can you let me know how you configure it for FM testing? So far I've just been capturing port 5003 traffic but wondering if there are any more specific filtering you use in your testing?
The updated slides and sample file (sent to FileMaker Inc. this week; I took a week off after DevCon) include some of this, and I promised during the session to blog with step-by-step instructions or a video for how to run a WireShark capture, but it basically boils down to four (4) steps:
- Decide which interface you need to capture. e.g., if you run the tests locally like I do, you would capture loopback0 aka lo0; if you are capturing network traffic on your ethernet port, it’s likely ethernet0 or en0.
- Set the capture filter to ‘tcp port 5003’. That’s the WireShark syntax for packet filtering. I suspect many packet filtering apps use a similar syntax. YMMV.
- Display and export only the columns you need. The how-to post on capturing will break this down more, but the gist of the issue here is that WireShark capture a LOT of information about each packet, whereas all I needed for my analysis was, in effect, timestamp, source and destination port (so I could tell which traffic was ‘up’ to the server vs. ‘down’ to the client) and bytes.
- Export as comma-delimited to a consistent file for easy import into a template. Again, stay tuned for the related post.
I guess there is one more thing worth mentioning, and that is the idea of layering on a display filter, e.g. just displaying/exporting/plotting traffic in one direction. In short:
- If you plan to gauge the traffic ‘by sight’, i.e. in the filtering tool, then consider using a display filter, e.g. ‘tcp.srcport == 5003’.
- If you plan on exporting to analyze the traffic elsewhere, capture it all and plot ‘up’ and ‘down’ traffic separately, for additional insight.
The second thing was a point you mentioned in your talk that had me thinking. You mentioned that when in form view, Filemaker downloads the record you are on (providing there is a field from that record on the layout) but it also downloads I believe 25 other records in that found set as a pre-fetch action. Can you confirm that this is indeed the case (I cannot find it on the slides).
This has implications for us where we use an interface/data separation and all interaction with the system is via the interface record. This lets us control record viewing to one at a time without any pre-fetch, BUT on the interface table itself we may have many interface records at any given time. If the found set on interface is always kept to 1 record (the logged in users) will this download additional records in the table providing they are not in the found set?
If the found set is one (1) record and at least one field from that record is visible on the layout, then the only data downloaded to the client should be:
- Whatever graphical elements are required for the layout to be drawn.
- Data from all the text, date, number, timestamp and stored calculation fields.
- Data from any container fields in that record, so long as they are visible on the layout.
- Data from any unstored calculation fields in that record, so long as they are visible on the layout.
- Data from any summary fields in that record, so long as they are visible on the layout.
A couple of caveats:
- It’s not entirely clear to me in your example if the interface record is displaying data via relationships, or just data from externally referenced Table Occurrences (TOs). If it’s the former, than the answer might be that only one record’s worth of data comes down at a time, as all the ‘local’ record data is really related, in a one-to-one relationship.
- Those last two items above (unstored calcs and summary fields) can expose you to additional data from related records, e.g. such fields often access related records, either directly or indirectly.
- Similarly, conditional formatting is often based on unstored data, and thus might call related record data into play (e.g., a status color that changes based on related line item fulfillment).
If I ever get into a situation where all interface records are in the found set, would they all be downloaded providing there were less than 25? It had always been my assumption only the current record in form view would be downloaded, but your comments got me thinking!
Yes – if there are less than 25 records in form view, you’ll download data all of them. If you had more than 25, when you flip to record 26, you’ll get the next 25, etc. Also, keep in mind that any sorting or summarizing of the found set (or a related set of records, at the TO or portal level) can also trigger ‘other-record’ data to start moving too.
I have also taken my files offline and set the last opened layout to a blank layout with no records and a blank table, haven't noticed a significant drop in packets transmitted but probably to be expected as my startup/shutdown layouts are generally small anyway. Also in the process now of zeroing out all of my development/data layouts initially, hopefully this gives a little boost.
It might, but as someone in the audience pointed out, YMMV. For example:
- ‘Zeroing out’ records on a layout (i.e., scripting a change from the default state of ‘Show All Records’ to the desired state of ‘Show No Records’) is not really per-table or per-layout as much as it is per-context, i.e. per-table-occurrence.
- As each new window opens, only the foreground context (i.e., the TO behind the layout you’re on) inherits the found set, all the rest revert to ‘Show All Records’, so you’ll need to manage new windows in your planning.
- If you never take a user (or allow them to freely navigate) to a context, then zeroing it out may be an unnecessary effort that simply generates extra traffic (the layout load and find process).
One thing I did notice is that on startup of one of my larger solutions, the total packets transmitted during startup from woe to go, was about 6,100 packets. Of those, about half were transmitted before the startup script began running which seems quite high given the layouts it goes to are blank. Can you shed any light on what is downloaded prior to the startup script running, other than the layouts required? Does FM download the entire script/layout catalog, or all the field definitions for the table in question (ie Interface)?
First of all, I’d probably focus less on packet count than bytes transmitted.
As pointed out by FileMaker engineers, FileMaker 11 isn’t fully optimized (yet) in terms of how many remote calls a client makes to the server during a ‘transaction’ (for lack of a better term) or in how the packets are sized (a legacy networking issue that hopefully will be improved in future versions). In addition, there’s little you can do to micro-manage packet count.
Furthermore, a lot of those packets may be the ones going back to the server from the client, simply acknowledging receipt of some other packet, i.e. display-filtering the packet capture for ‘srcport or ‘destport’ will let you see which ‘direction’ packets are moving.
That said, items (schema, structure and data) that come to mind and which could be downloaded during the startup process:
- Items on the layout specified in File Options.
- Data triggered to move on layouts that are displayed (per the list a few questions above) during the startup process.
- Data triggered to move by scripts that run (and which can perform operations on sets of local and/or related records, which will call those records’ data into play) during the startup process.
- Data triggered by access privileges that are invoked (and which can use calculations to derive table- or record-level access) during the startup process.
- Schema for the tables and fields needed for the layouts, scripts and conditional-access-privileges that are invoked during the startup process.
- Some amount of relationship schema data in general, for a dependency check, depending on what files you already have open and where any open files are hosted. I have not done tests of this yet, but I suspect this is fairly small and consists primarily of a cross-reference of External Data Sources and associated TOs.
What about calculations on the interface table that reference other tables, does this start a cascade of downloading of all tables in question (their field definitions?) Any insight would be hugely appreciated.
Yes, unstored calculations (which are often unstored either due to the presence of a globally stored field or a related field being present in the calculation) can trigger data from related records to start being accessed as well. Depending on how you structure your calculations, this can easily cascade from parent to child to grandchild, ad infinitum.
One little thing I noticed in your talk also which isn't on the slides but got my attention when you mentioned it. You said at one point that summary functions are to be avoided in some situations as they download all record data, which is fine, but you also mentioned that the Get ( Foundcount ) function has the same effect of downloading the records. I believe this to be untrue, and that Get ( FoundCount ) is actually super fast as it is a single # returned from server requiring no record downloads. In our solutions we have replaced Count summary with a call to Get(FoundCount) and it has hugely improved speeds when we simply need a count, in fact it is almost instant for millions upon millions of records counted. Just wanted to let you know.
You’re right, that was an error on my slide. Someone pointed this out after the session as well and it has been corrected in the slide. A better example of a calc to avoid would have been an aggregate function, e.g. Count ( Record::ID ).
Thanks again for the great session - it was definitely the best one I went to and by far the most useful, look forward to more sessions on this topic in the future!
Considering the phenomenal cast of speakers, I am humbled. Thanks!
It’s especially gratifying to know my efforts are helping people like you build better applications.
Oh – and one more thing – please fill out the speaker evaluation form and let FileMaker Inc. know you want more of this topic in the future! 😉
Mark Richman is an Owner, the President, a FileMaker Certified Developer and a FileMaker Authorized Trainer at Skeleton Key.
About Skeleton Key
Skeleton Key is an accomplished team of technology consultants who solve business problems. We specialize in the rapid development of custom applications, integrating Macs and PCs in the professional workplace, and providing personalized training.
Despite our end-to-end technical skills, we are consultant first and technologist second. We know that you don’t just need technology. You need to know that the technology you choose to deploy will provide the results you desire.
Skeleton Key is a Platinum Level FileMaker Business Alliance company, an Authorized FileMaker Trainer, a member of the Apple Consultants Network and a Microsoft Registered Partner.