I measured the BUILD times of the Flutter Gallery app to make sure we aren't introducing a lot of overhead by turning on track-widget-creation on in profile builds. The Flutter Gallery has 3 BUILD's before the first LAYOUT. Here are the results:

This was tested on an iOS device. At first blush there doesn't appear to be a significant change in BUILD times as a result of tracking widget creation.

I ran a performance test with debugProfileBuildsEnabledUserWidgets turn off an on as it is currently written:

(Tested with the flutter gallery on an iOS device)

I switched to a HashMap since that should technically be faster and take up less memory. With the same gallery test I got 6.5ms 2.1ms 2.3ms.

@Hixie I chose you to review this since you've done work on the Timeline events and we talked about this feature in the past.

I implemented the approximation and set the sampling rate to 2,000 hz. Then I ran the stocks microbenchmark with the flag on comparing it with baseline dart:

Before the change we were at 831.9 µs, so we just added 61 µs, would this be acceptable to you @dnfield? @bkonyi are you open to approximation if I can give you some data about the error?

Here is the patch to dart:

diff --git a/runtime/vm/os_macos.cc b/runtime/vm/os_macos.cc
index eed635df0d2..5a4c52dbe79 100644
--- a/runtime/vm/os_macos.cc
+++ b/runtime/vm/os_macos.cc
@@ -105,7 +105,14 @@ int64_t OS::GetCurrentMonotonicMicros() {
   return GetCurrentMonotonicTicks() / kNanosecondsPerMicrosecond;
 }
 
-int64_t OS::GetCurrentThreadCPUMicros() {
+namespace {
+struct ThreadTimeInfo {
+  int64_t threshold = 0;
+  int64_t last_thread_time = 0;
+  int64_t last_clock_time = 0;
+};
+
+int64_t ExpensiveGetCurrentThreadCPUMicros() {
   mach_msg_type_number_t count = THREAD_BASIC_INFO_COUNT;
   thread_basic_info_data_t info_data;
   thread_basic_info_t info = &info_data;
@@ -121,6 +128,28 @@ int64_t OS::GetCurrentThreadCPUMicros() {
   return thread_cpu_micros;
 }
 
+int64_t approximate(const ThreadTimeInfo* info, int64_t clock_time) {
+  return (clock_time - info->last_clock_time) + info->last_thread_time;
+}
+
+thread_local ThreadTimeInfo* s_info = nullptr;
+} // namespace
+
+int64_t OS::GetCurrentThreadCPUMicros() {
+  int64_t clock_time = GetCurrentTimeMicros();
+  if (s_info == nullptr) {
+    s_info = new ThreadTimeInfo();
+  }
+  if (clock_time > s_info->threshold) {
+    s_info->last_clock_time = clock_time;
+    s_info->last_thread_time = ExpensiveGetCurrentThreadCPUMicros();
+    s_info->threshold = clock_time + 500;
+    return s_info->last_thread_time;
+  } else {
+    return approximate(s_info, clock_time);
+  }
+}
+
 int64_t OS::GetCurrentThreadCPUMicrosForTimeline() {
   return OS::GetCurrentThreadCPUMicros();
 }

@dnfield nevermind the approximation question, let's land this turned off and I can follow up later about turning it on with improvements to Timeline in a different PR.

I implemented the approximation and set the sampling rate to 2,000 hz. Then I ran the stocks microbenchmark with the flag on comparing it with baseline dart:

test time
exact thread time 969.4 µs
approximate thread time at 2000 hz 893.4 µs
Before the change we were at 831.9 µs, so we just added 61 µs, would this be acceptable to you @dnfield? @bkonyi are you open to approximation if I can give you some data about the error?

Here is the patch to dart:
...

@rmacnak-google might have a better way of doing this that doesn't involve approximating thread time and is quite a bit faster (quick results show ~3x improvement on some simple C programs when using the current GetCurrentThreadCPUMicros() implementation and one that uses clock_gettime_nsec_np(CLOCK_THREAD_CPUTIME_ID) / 1000 instead). I'll follow up later whether or not this approach pans out.

I just tested build_bench.dart with this flag on with the 2 new patches we've done to make Timeline events faster.
flag off: 789.5
flag on: 930.6

It didn't lower the percentage of overhead. It just dropped the overall numbers. I'm going to walk away from this now that the low hanging fruit has been explored.