The Story

Pilly is an AI-powered medication tracker which won first place 🏆 at the Codebrew Health Hackathon 2020.

In September 2020, along with four other Monash University students (and close friends), I entered the Codebrew Health Hackathon which aims to innovate and create digital solutions at the intersection of health and technology.

We had a weekend to design, develop and pitch an application which fit into one (or more) of four categories: Health Delivery, Pandemic Control & Assistance, Medications, and General Wellbeing. After consulting with some friends in Biomed and Medicine about common problems faced by patients, we decided to tackle the topic of medications. Around 250,000 Australian hospital admissions every year are medication related, and the annual cost of medication-related problems in Australia surpasses $1.5 billion.

Our solution: Pilly. Pilly is a webapp which uses AI to helps consolidate complex combinations of prescription medications for Australians. Pilly attempts to understand your prescriptions, and can notify you when it’s time to take their meds. Being a digital platform, it also helps translate critical usage instructions to dozens of languages to help make medication more accessible to non-fluent English speakers. 

Our team consisted of students from many disciplines, from medicine and commerce to computer science and software engineering. Bringing such diverse viewpoints to the table allowed us to create a product that was people-focused, not bogged down by technology for the sake of appearing complex, and tackles an issue which was real and relevant to many Australian individuals and families.

Our Github repository with the full code is available here. View our video pitch for Pilly below.

Simplifying Medication Management

The prototype Pilly application is a React PWA which can be installed onto both iOS and Android devices through Chrome. 

The application uses the device’s camera to allow users to scan their medication boxes. An image of the box is uploaded to cloud storage and is analysed though our ML pipeline powered by Cloud Vision and AutoML Entity Classification to determine the medication, dosage, schedule and other useful attributes.

All this information is displayed on a clean, easy to use UI designed to be accessible for users with varied technical abilities.

Extracting Details From Medication Labels

We used Cloud Vision API to extract the text from the medication labels which we then passed on to a custom AutoML Data Labelling model to extract the medication title, dosage, frequency and active ingredients from the medications.

We had a team creating the datasets for the model to train on based on a small number of scans and online medication data from the Pharmaceutical Benefits Scheme and NPS Medicinewise. To generate enough data to train our model for our prototype, we generated various permutations of medication labels from our small initial dataset with added noise. Although this was not the best approach for preparing data, due to our time restrictions we had to work quickly to start the training process which can take up to six hours.

We also passed the predictions through a heuristic function to determine the medication type through its position in the text and its similarity to other medication names in a large medication list. Ultimately we were able to create a pipeline which was able to pick up the medication names and important usage details to a reasonable degree of accuracy given our short turnaround time.



Timely Notifications

Once all a user’s medications are on Pilly is when the application really shines. Pilly has a cloud-based notification system powered by Firebase Cloud Messaging, which can remind users on multiple devices when it’s time to take their medications.

Loved ones can even decide to set up their own notifications so that a carer can receive updates if a user forgets to take their critical medications.

Our model can also pick up on warnings and highlights in medication packaging. Notifications can then show users important warnings about their medications and relevant usage instructions.

Making Medications Accessible

A key benefit of Pilly is that it is a digitised medication management systems. Once we have matched a medication to information in our database, we can provide translated usage and warning information to patients in their native language using the Google Translate API (or other more curated sources).

Tens of thousands of Australians, especially elderly immigrants, struggle with English comprehension, so by providing vital medication information in their mother tongue, Pilly can help bridge the accessibility gap in current medication fulfillment.



Within the roughly 36 hours we had to decide on our idea, build the prototype, create a pitch deck and record our submission video, our small student team at BrightSeeds were able to bring together a functional application which won over the hearts of the judges.

I’m extremely grateful to have had this opportunity to work with my friends to bring this project to fruition, and for it to be recognised for its merits by the judging panel was simply an added bonus.

I would like to thank Marc Porciuncula, Mabel Li, Arosha Dissanayaka, Mathen Jose and Sho Yamamoto for working tirelessly and burning the midnight oil to bring Pilly to life and to have it look and work as well as it did during our live demo. I hope we get to work together again in the future. It was also extremely rewarding to play around with AutoML and using the existing ML solutions available through GCP, and I look forward to potentially integrating these features in future projects.

Until then, remember to take your meds👋!