MARK5826
Product Analytics
Term 2, 2024
Assessment 1: Quizzes
Description of assessment task
ONLINE QUIZ 1 (5%)
Description: This is an online multiple-choice test, consisting of 10 questions covering content in week 1-3. Timing: The Online Quiz will be available on 11th June Tuesday Week 3, and you must finalize your attempts to complete it by 11.59pm on 16th June Sunday. The quiz will be disabled after this time so make sure that you have completed it before then. My advice is to complete it a few days early to avoid any glitches that may occur.
ONLINE QUIZ 2 (10%)
Description: This is an online multiple-choice test, consisting of 20 questions covering content in week 1-5. Timing: The Online Quiz in Week 5 will be available on 25th June Tuesday, and you must finalize your attempts to complete it by 11.59pm on 30th June Sunday. The quiz will be disabled after this time so make sure that you have completed it before then. My advice is to complete it a few days early to avoid any glitches that may occur.
Approach to the assessment task
Read the questions carefully and select the correct answer. You may attempt this quiz as many times as you choose (on each attempt questions will be sampled from a bank in a randomized order). The highest mark will be recorded. You will receive your mark and right answers right after the quiz closes.
Structure
This assessment task constitutes 15% of your overall mark for this course. Each question is worth 0.5 mark. You will receive 0.5 mark for answering a question correctly. Feedback and results will be provided through the Moodle quiz activity once the quiz window closes and after you have attempted the quiz.
Submission instructions
Remember you can take the quizzes as many times as you like until the quiz closing date. The highest mark will be recorded as the final mark.
Assessment 2: Customer Problem Identification using Mixpanel
Description of assessment task
Each student will write a report to investigate a business problem about new product development in the digital space. The case will enhance students' analytical, qualitative, and strategic decision-making skills as well as their knowledge of multidisciplinary marketing approaches.
Approach to the assessment task
You have been tasked to develop a dashboard report for the team managing the media streaming platform. They are looking to set up metrics that will help them to review their products’ performance. Use what you have learned about defining useful and effective metrics and your knowledge of Mixpanel to help them out!
The team has listed the following three focus areas, along with some questions for each one.
1. Core product
1.1. How much time do our users spend on watching videos each week?
1.2.Do users come back to view more videos every week?
2. Improving activation
2.1. How good are we at helping new sign-ups to quickly find and watch an interesting video?
3. Improving subscription revenues
3.1.What percentage of new sign-ups are upgrading to a subscription? (Hint: Use the Purchase event with purchase type = “Subscription”. You may need to create “custom event property”)
Use the Mixpanel demo media project to build the reports and add them to a board. Include any context (such as your definition of the metric) as text next to your report. The purpose of this assessment task is to help you harness, manage, and communicate business information effectively using multiple forms of communication across different channels.
Submission instructions
Create a product analytics dashboard report for the on-demand streaming platform. Make sure that you:
• Revisit the Mixpanel Board function. A Board in Mixpanel is a collection of reports, text, and images or videos, which allow you to view all your most important metrics at a glance.
• Include the link to the assignment cover sheet.doc, so that your marker can access to your board.
• Always provide rational to the insights you have gathered.
• If the period has not been provided in the question, set your own period to answer the questions. Make sure there is enough data to base conclusions on. Communicate the selected time range and include most recent data.
• Name your cover sheet as Tutorial_PA_Mixpanel_zID.doc (or .docx). For example: W14A_PA_Mixpanel_z1234567.doc.
• Submit the cover sheet via an upload link on Moodle by 7th July Sunday 11:59 pm. Late submission results in a 5% penalty for every day late, with a maximum of 5 days, impacting the grade negatively.
Assessment 3: Sentiment analysis, topic modelling, A/B test
Description of assessment task
After customers purchase and use products or services, they often share their experiences on online review platforms. Many companies attempt to identify customer problems and unmet needs from large-scale product review data by using natural language processing techniques such as sentiment analysis and topic modelling and. Recent machine learning methods enable the automatic categorization of text data, although these methods are not yet perfect. Your task is to generate new product or service ideas based on your analysis.
Specifically, choose 2 out of 3 questions. Maximum 500 words per question plus submission of the Orange workflow.ows file(s). For each specific task conducted, interpret the results and transform. the results into words (findings) i.e. based on your results, what further information and recommendations (e.g. managerial actions) you can provide to Colgate or LDC (depending on the questions you've chosen).
Approach to the assessment task
1. Sentiment analysis
Description of the dataset
With total 1000 rows, the dataset consists of 500 positive and 500 negative rows Colgate product reviews. All the rows have been generated through prompts given to ChatGPT-4. There is no redundancy/duplication in the rows since GPT generated only max 50 rows in one-go. The rows were carefully inspected to remove any duplicate found at run-time. This dataset can be used for basic sentimental analysis as this dataset mirrors real-world customer product reviews.
Here, sentiments on toothpaste reviews are collected and stored as .csv file, consisting of 2 attributes:
• Review: review of Colgate toothpaste.
• Sentiment: sentiment related to the review, i.e., positive or negative.
Instructions of this analysis
Perform. the following tasks:
a) Train a natural language model from scratch (you can choose the complexity of your model) that predicts the sentiment score (positive or negative) of a given review. To perform. this task, the dataset needs to be split a priori in a 80-20 train-validation dataset, respectively, at random. The inputs of your natural language models can be either “Document embeddings” or “Bag of words”, depending on your personal preferences. Remark: To perform. this task, it can be that you need to perform. some pre-processing steps.
b) Evaluate the performance of your trained neural network model by calculating the precision, recall and F1-score measurements on the validation dataset.
c) Predict sentiment from text. Visualise the Sentiment Analysis using either Scatter Plots or Heat Map (your choice of method and variables).
2. Topic modelling
Description of the dataset
Product reviews are important for brands to not only develop new product features, but also continuous improvement to achieve product superiority over their competitors, command higher market shares and a premium price relative to their competitors. Here, a corpus of toothpaste product reviews was published in a British supermarket in the United Kingdom. It is stored as .csv file, consisting of 6 attributes:
• Rating: between 1 to 5
• ReviewText: Text of the toothpaste review in English
• SubmissionTime: Date of publishing for the product review in YYYY-MM-DDTHH:MM:SS.000+00:00 format
• Title: Text title of each review in English
• UserLocation: Location of the reviewer in city, region, country format
• UserNickName: Reviewer's nickname
Instructions of this analysis
Perform. the following tasks:
a) Apply the Latent Semantic Analysis and Latent Dirichlet Allocation technique to study the topic focus of toothpaste’s reviews. Characterize them by exploring the most frequent words in each topic.
b) How do these topics evolve through time in the product review?
Remark: To perform. this task, it can be that you need to perform. some pre-processing steps.
3. A/B Test
Description of the dataset
Reviewing the LDC case study again. LDC tested its product at each stage of the customer journey. LDC has implemented experimental design for InstaMoney. The meaning of each variable is explained the sheet called “Variable Legends”.
Instructions of this analysis
Perform. InstaMoney Prototype A/B testing results (both conversion and lift) for the following features:
a) Generating Prospects
b) Onboarding
c) Credit Assessment
d) Payment Reminder
e) Query Resolution .
Based on the A/B testing results, how can LDC incorporate the best feature into InstaMoney?
Submission instructions
• You are expected to choose 2 out of 3 questions.
• Questions 1 and 2 are designed for Orange, consisting of a clear workflow so that the steps and conclusions you make can be easily followed and verified.
• Question 3 is designed to be answered using excel although Orange also allows you to select the plotting method in Scoring method to conduct statistical analysis, such as t-test and ANOVA. Both tools are acceptable. The key is to draw conclusion to ensure the best product features.
• Create a report using Microsoft Word, including cover page, index, headings, and visuals.
• Screenshot of your completed workflow from Orange.
• Include in your report answers to the TWO questions outlined above. Limit your response per question to a maximum of 500 words (single space, 12 fonts, 2.5cm margins, Calibri). Hence, a total number of 1000 words can be submitted.
• Always provide rational to the insights you have gathered.
• Feel free to customize the report based on specific findings and nuances from your analysis.
• Name your report as Tutorial_PA_Orange_zID.doc (or .docx). For example: W14A_PA_Orange_z1234567.doc.
• Name your Orange workflow as Tutorial_PA_Orange_QuestionID_zID.ows. For example: W14A_PA_Orange_Q1_z1234567.ows. OR W14A_PA_Orange_Q2_z1234567.ows.
• If Q3 and excel is chosen, name your file as W14A_PA_Excel_Q3_z1234567.xls (or .xlsx).
• This assignment contains three submission parts: report, workflow (1st question of your choice), workflow (2nd question of your choice). Submit the files via the upload link on Moodle by 21st July Sunday 11:59 pm.
• Late submission results in a 5% penalty for every day late, with a maximum of 5 days, impacting the grade negatively.
Assessment 4: Group Project
Description of assessment task
You will develop a new data product with your group members. This project provides you with an opportunity to take your knowledge and skills of the product analytics learned in the course and apply them to a real-world problem. The purpose of group presentation is to communicate what you are building, who it is for, and how it will deliver value to end users and to the business.
Approach to the assessment task
Milestones, scaffolding, informal feedback opportunities during tutorials.
• Week 3: Group formation with each group of 4-5 students. Elect the roles of CEO (group leader), developer, product manager, product marketing manager, UX designers. Confirm the app product or app service. (Keep a diary re: teamwork challenges, solutions and skills developed)
• Week 4: Part 1 Creating “Customer Journey Map”
• Week 7: Part 2 Creating “Measurement Framework”
• Week 9: Part 3 Creating “Tracking Plan”
Week 10: Presentation
Submission instructions
As you are aware now, there are a number of delivery dates for this project. Although they are formative coursework which does NOT count towards the final mark, completing the task in the respective week helps to progress your group project. You’re encouraged to engage your tutor during tutorials to receive oral feedback and use this feedback to make real-time shifts in your learning trajectory. Make sure you work with your group and individually to format and complete the delivery following these requirements:
• Always provide rational to the insights you have gathered and use academic support to strengthen your arguments.
• For presentations, make sure you work on a clear and professional presentation. Remember that you do not need to include all the materials you have gathered. You will need to make a group decision to organise the most important materials.
• As part of your slide presentation, please add speaker notes to each slide. These notes are important because they provide context, explanations, and additional details that may not be evident from the slide content alone. The notes help you convey your message effectively to the audience and ensure a comprehensive understanding of your presentation.
• All group members must be involved in the presentation.
• Name your presentation as Tutorial_PA_zID-groupleader.ppt (or .pptx). For example: W14A_PA_z1234567.ppt. Note that only ONE member of the group submits the deck of slides via Turnitin.
• Submit the file via an upload link on Moodle by 29th July Monday 11:59 pm. Late submission results in a 5% penalty for every day late, with a maximum of 5 days, impacting the grade negatively