Hello and welcome to Microsoft OpenHack, a challenge oriented hack event. This experience will be presented as a series of challenges, each one adding a new level of depth and complexity to a machine learning solution for computer vision.
Teams should already be assigned and seated together. During your days of hacking, teams will attempt to solve as many challenges as possible.
Team coaches are the first point of contact, and are available for technical support during the hack, but they will not solve challenges or provide answers for teams.
Adventure Works, a major outdoor and climbing gear retailer, wants to understand customer behavior by learning more about the gear that consumers wear; and plans to accomplish this with powerful Microsoft and open-source technologies for computer vision.
Your team has been invited to work with Adventure Works and build a series of computer vision solutions based on the image data they have collected.
Each challenge covers a different aspect of machine learning for computer vision, and all of them will require you to write Python code. You can reference the Python 3.6 documentation at https://docs.python.org/3.6/.
Challenges do not include step-by-step instructions. Your team must work together to determine a solution for each challenge using the hints and documentation references provided, with guidance from your coach when needed. There may be mulitple ways to complete each challenge correctly. Your coach will validate your solution for each challenge before enabling access to the next one.
One Final Tip
Read everything carefully!
The OpenHack team has worked hard to ensure that each problem is solvable and that all required details are included in the challenge briefs
Briefs are very carefully written and worded to give clues toward the solution
Reading challenges thoughtfully is the best way to figure out successful approaches, as nuances can be easily missed
Coaches will not give out the answers, but rather will be a resource to fill in any missing pieces in knowledge and skill, as well as help guide in the right direction
Before working on a computer vision solution, the data science team at Adventure Works wants to standarize on a collaborative development environment that will work well with common Python libraries.
After provisioning the development environment, you can use it to explore the images that Adventure Works has collected, and prepare them for use in training a machine learning model.
Adventure Works wants to understand customer behavior by learning more about the gear that consumers wear, and you have collected and pre-processed the product catalog images, ready to start building a machine learning solution.
Machine Learning solutions can be complex and require time, expertise, data preparation, ongoing maintenance and deployment. As an initial approach, the data science team at Adventure Works wants to try a using the Microsoft Custom Vision cognitive service as a quick and easy way to build a solution that can categorize different kinds of jacket that a customer might want to buy.
Adventure Works wants their data science team, including your team members, to begin learning how and when to perform custom machine learning with powerful, more programmatic APIs. Becoming proficient in machine learning takes some time, however, beginning with a high-level API, that is stable and has good documentation, is a great start. Some of the reasons to move on to custom machine learning include:
To explore different algorithms to optimize what works best for the data
To create a workflow with more control over your process
To deploy a model to a specific architecture or configuration
To overcome limits in bandwidth or data size for a service
The Adventure Works data science team wants to try deep learning algorithms on the data to see if the accuracy of gear classification improves. The team anticipates an influx of catalog and outdoor pictures soon, and deep learning shines when the data size gets bigger. Also, the Data Science team is eager to learn more about deep learning and convolutional neural networks (CNNs) for image analysis because CNNs naturally lend themselves to performing well on complex data such as images, and can require less pre-processing and feature engineering.
A CNN consists of a series of layers in which weights are applied to the pixel values in the images based on filters that are convolved across the image (hence the name). The data is passed through the layers in a series of iterations (or epochs) in which you train the model using a subset of the image data, and validate it with another subset (usually, within each epoch the training and validation data is passed through the network in multiple mini-batches). For each batch of data passed through the network, a loss function is used to calculate the error (or loss) in the predictions based on the known labels in the source data, and then a process of backpropagation is used to adjust the weights and decrease the amount of loss.
It’s not just enough to be able to build a world class machine learning model, you need to know how to deploy it so that it can be consumed by team members, developers, and 3rd parties to provide useful functionality to applications and services for Adventure Works customers.
In this challenge you will use Azure Machine Learning Services and the Azure Machine Learning SDK to operationalize your model as a scoring endpoint in the cloud (specifically, as a containerized web service), so that developers at Adventure Works and elsewhere can write consumer applications and use this scoring endpoint for prediction.
Adventure Works has partnered with a local guide service to help them collect images of mountaineers and climbers in an effort to encourage the use of helmets for safety.
The Adventure Works data science team wants to experiment with creating a custom object detection model using a convolutional neural network (CNN). This custom model should perform the same object detection task as the model previously built with the Computer Vision service, detecting whether or not each person in an image is protected by a helmet.
Adventure Works has a new project for the team - building on top of the helmet detection system for safety, Adventure Works would like to identify trails that need improvements for safer travel, as another service to the community and way to advertise their brand. This will be solved with semantic segmentation.