Instrumenting involves accurately naming every small action within your product's app, such as button clicks and new screen loading. If you've worked as a data analyst, you have probably encountered this challenging task.
Why is it such a beast?
Imagine trying to keep track of every single file and folder on a massive shared computer network. Each one needs a specific name and location. You need to track who created each file, when it was last modified, and what it contains. Without a clear system, you're left with a disorganized mess, trying to find the file you need among countless others.
Once you've tackled that, you test each event to ensure you got it right. Plus, there's the responsibility of maintaining them. Just thinking about it is enough to make your head spin.
The Early Days of Instrumentation
When I first started learning about instrumentation, it went something like this.
The product manager placed a Figma design on my desk and said, "Just get the data." As I looked at the screen, I wondered, "Great! But what's the deal with these events? Where's the connection to the grand business scheme of things?"
There were no KPIs, no guidelines, just a big pile of events staring at me like an unsolved Rubik's cube.
The Rise of User Behavior Analysis (and the Unrealistic Expectations)
The cycle continued until early 2022, when "user behaviour analysis" became the latest trend.
Questions like "Why can't you create a user behaviour tool in Looker or display the user funnel in Looker?" started coming from all directions.
All I could do was roll my eyes to the heavens, thanking the stars that my poker face was hidden behind a webcam. The thought alone was like asking ChatGPT to pilot a spaceship. Could it happen with some wildly advanced AI and a ton of training data? Maybe. But was it the best, most efficient use of our resources? Absolutely not.
The Segment & Amplitude Dream (Turned Nightmare)
Then comes the grand plan: we decide to implement Segment and Amplitude in our analytics toolkit. Now, you might be thinking we hit the jackpot, right? Nah. In fact, here are the "best" parts:
- The "Track All the Things" Trap: Our motto was "Track All the Things!" Why did we do this? Honestly, I think we were overcome by FOMO (Fear Of Missing Out). We were afraid that it could be crucial if we missed even one small event, and we'd have no way to go back. But in reality, we ended up with over 200 events, no clear focus, and our product teams were utterly overwhelmed. Testing was a nightmare - like finding a needle in a haystack with a blindfold on.
- The Generic Naming Disaster: We used the same naming system for all events. For example, when a user ordered a product, it was labelled as "button clicked" in our system, with the specific action buried in a property. It seemed like a good idea, but it led to a flood of 'button clicked' events, making it hard for users to find important data. This created a bad user experience in Amplitude. As a result, our stakeholders bombarded us with questions, and we spent most of our time training them.
- The "All-In" Approach Backfires: We took the "all or nothing" approach. We were neck-deep in more work than we could handle, as we had to refactor all our events later. We could've spotted the issues sooner if we'd tried this out with one product team and maybe tested the waters before diving in.
- The Missing Product Managers: Our product managers decided to skip this round, leaving data analysts like me to create event names. The outcome? A bunch of poorly named events that didn't contribute to anything. It felt like we were swimming in data with no direction.
- The Google Sheets Bottleneck: We chose Google Sheets as our tool. It's great, but managing over 100 events, coordinating with more than ten project managers, and communicating with over twenty developers and data analysts feels like trying to host a big banquet in a small kitchen. Google Sheets is excellent but not suitable for this task.
Learning from Experience: A Better Approach to Instrumentation
What would it be if I had a magic wand and could change things?
- Prioritize Essential Actions: Focus on instrumenting the actions that truly matter for your product, ensuring quality over quantity.
- Test and Iterate: Start with one product area to gather feedback and assess the effectiveness of your instrumentation within Amplitude.
- Embrace Best Practices: Adhere to Amplitude's naming conventions and guidelines to ensure consistency and clarity.
Conclusion
Our instrumentation journey taught us the value of focusing on the essentials, testing early, and following best practices. It's a constant learning process, but with thoughtful planning and patience, you can turn data into a powerful tool for your product. Remember, it's not about tracking everything; it's about making the data work for you
